Is reading the tech help emails my company subscribes to I found the following, which i think is part of the problem and why I continually hate search engines. Not that meta data is a cure all since spammers will fake it, but it could help.
http://www.library.uq.edu.au/iad/ctmeta4.html
"
3. Why isn't an Internet search engine good enough?
The problem relates to the underlying nature of the World Wide Web. In the early 1990s, "surfing" the World Wide Web was popularised in the mass media. These days, the concept of browsing the Web is little used.
The Web has become a two-edged sword. It is now very easy to publish information, but it is becoming more difficult to find relevant information [EC, p.4]. For outsiders and casual users, much of the useful material is difficult to locate and therefore is effectively unavailable [DC1, p.2].
At the global level, Internet search engines were developed to search across multiple Web sites. Unfortunately, these search engines have not been the panacea that some people had hoped for. Every search engine will give you good results some of the time and bad results some of the time. This is what information scientists term "high recall" and "low precision". The high recall refers to the well known (and frustrating) experience of using an Internet search engine and receiving thousands of "hits". It is popularly known as information overload. The low precision refers to not being able to locate the most useful documents. The search engine companies do not view the high hit rates as a problem. Indeed, they market their products on the basis of their coverage of the Web, not in the precision of the search results.
The Working Group on Government Information Navigation outlined the problems with Internet search engines:
relevant information can be missed because sites contain types of resource in addition to HTML text (e.g. images, databases, PDF documents);
the search engines frequently do not harvest every page on a site, but often only the top two or three hierarchical levels, thus missing significant documents which, on larger and more complex sites, may be located in lower levels of the hierarchy;
search engines, especially the more comprehensive ones, may index sites on an infrequent basis and may therefore not contain the most current data; and
irrelevant information can be retrieved because the search engine has no means (or very few means) of distinguishing between important and incidental words in the document text. [WGGIN, p.2]
The introduction of the <META> element as part of HTML coding, was in part, an attempt to encourage search engines to extract and index more structured data, such as description and keywords. However, search engines are rather proprietorial in recognising <META> tags. It ranges from no support at all, to reasonable. Details are available from Search Engine Watch [SEW]. As far as I am aware, none currently supports metadata schemas. It is the proverbial "chicken and the egg" situation. Web page authors and publishers do not invest in providing metadata if the indexing services do not utilise it and harvesters do not collect metadata if there is not enough data available. The other problem is the malicious "spoofing" of search engines, making them return pages that are irrelevant to the search at hand or pages that rank higher than their content warrants.
Support for <META> tags by search engines designed for local Web servers varies from non-existent to good. Some of the specialist packages include support for Dublin Core or other metadata schemas."
