Recently I had an argument with a search engine expert. We are both members of the search engine club and thanks to Google this club has gained some prestige. Developing a search engine is very much like skinning a cat. Although I have never had an opportunity to skin a real cat and I actually like cats I still think the analogy holds. The first caveman would face many choices – not only which tools and animals to use but also making strategic decisions of performance, product definition and quality standards. In the end, there would be just enough wasted dead animals outside the cave to reach a compromise which would satisfy a minimum set of requirements. That was, if you like, our good old search engines. The altavista and the yahoo. After enough generations of search engines and cat skinners a standard way would be formalized. I am not familiar with the skinning standards but I’m sure there must be one somewhere. Search has the IR – information retrieval theory. It has it’s own vocabulary of terms. BTW, one of the terms is “term”. We have documents and frequency and similarity and ranking and … etc’. In the end every search engine would have those concepts or equivalent.
So what’s there to argue about when all search engines implement the same well known theory?
A lot apparently. Every detail actually has a big impact on performance, product and quality. The best choice is also context dependent. What is an excellent choice for one context is a total disaster in another. Searching documents on the web is not like searching addresses which have short, rigid and repeating sequences of words.
Arguments like this one happens all the time. Companies are shifting their attention to new areas and paying the tuition for learning and tailoring a unique solutions. Later, the same developers or external consultants come to new insights. That’s the nature of the business where the requirements are changing all the time. Obviously the ideal approach is to learn as much as possible as soon as possible but usually the research resources are limited, especially if the product is not in the center of attention.
And here is the conclusion – there is more than one way to skin a cat.
Each approach has its cons and pros and the decision takes are always based of what we know at now and our predictions for the future. When any of the first assumptions changes the direction of development must be carefully reassessed. Ideally, the differences, taken the evolution is done often enough, should be relatively small and the appropriate adjustments can be made. Sometime, however, the present or the future predictions is changed so much that a whole different way should be taken. In those moments, you realize that you are no more skinning the same cat, or worst, not skinning at all while should perform another task – maybe not even violent (like using a third party search engine).
Those decisions are the important one. As long as the same cat is skinned the technique doesn’t matter so much. Those arguments can a nice for a coffee break but in the end it is a waste of time. The other type of arguments is critical. In my opinion the competency of the management staff is visible when they show the ability to tell the first argument type from the other one.
