begin van de zoekmachine
is klein begonnen. Voelspriet kijkt voor de fijnproevers terug naar het
begin. Lees alles over de oorspronkelijke techniek achter de zoekmachine
en de reden van oprichting.
Hoe het begon
De grootvader van alle zoekmachines, Archie, is
opgericht in 1990. Lees alles over de eerste zoekmachine Archie en diens
A History Of Search Engines
: over Archie, Aliweb, Excite en
Search Engine Players, A Brief History:
over AltaVista, Excite, Hotbot en Webcrawler
Sergey Brin van de Computer Science Department,
schreef in 2000 een document over
"The Anatomy of a
Large-Scale Hypertextual Web Search Engine". Dit is de
uitgebreide versie van zijn studie. Het beschrijft de doelen van Google en
gaat uitgebreid in op populariteitsmeting van websites, in vaktaal "Page
The Anatomy of a Large-Scale Hypertextual Web Search Engine
"I have a web robot which is a Java app."
"BackRub is a 'web crawler' which is designed to traverse the web."
"If your question is not answered here [...] call  and ask for Larry."
" This is a demo of the Google Search Engine. [...] Number of Web Pages
Fetched: 24 million"
"Current Repository Size: ~25 million pages"
"Index contains ~25 million pages (soon to be much bigger)"
"Google can make you feel lucky!"
"Here is our main logo full size, created using GIMP. If you want to hack
on it, here is the XCF file."
" Google Receives $25 Million in Equity Funding"
AltaVista Site White Paper
Het noodlijdende AltaVista, lange tijd de lieveling van
professionele researchers, schrijft in AltaVista Site
White Paper meer
over Scooter, de automatische robot die websites besnuffelt. (Scooter
bestaat nog steeds). Dit document was lange tijd
niet meer verkrijgbaar op internet!
Unlocking the true value and promise of the
Traversing the Internet has always been a bit
like exploring outer space. One could wander indiscriminately and make
many useful discoveries. Just as easily, though, hours (thankfully, not
light-years) could pass with nothing of value to show for the effort.
Like its figurative cousin, the cyberspace of
the Internet is vast; but unlike the world of Voyager and Magellan, the
Internet is finite. And while the Internet remains essentially
unstructured, it is possible-with enough sophistication and power-to
catalogue the entire realm. To index every word on every page of every
Web site. To bring order and meaning to an otherwise unwieldy behemoth.
Digital has proved it can be done-and has done
The AltaVista site-launched on December 15,
1995-radically altered the way we view and use the Internet. On the
surface, one sees a simple searching interface, not unlike other tools
available through a standard Web browser. Behind the scenes, however,
the world's most sophisticated "web crawler" software and most powerful
computers have compiled (and continue to update) a complete index of the
entire Internet. For the first time, it is possible to find and retrieve
useful information from across the vast expanse of the Internet in
AltaVista has changed everything. It is no
longer necessary to know the address of a particular home page, only to
begin following the trail of hyperlinks to your eventual goal. AltaVista
takes you to precisely where you want to be from the start-pointing you
to relevant Web pages regardless of where they reside on a particular
site. You can then follow the links from there as desired. The
painstaking task of classifying every Web page into logical groups is a
thing of the past, before it even became a full reality. Today,
AltaVista puts the entire contents of the Internet at your fingertips,
transforming it into a bona fide business, education, and
A higher view of the Internet
What exactly is AltaVista? It is a whole new
class of Internet technology, developed in the research laboratories of
Digital Equipment Corporation. To better understand AltaVista, let's
first look at it from the user's perspective.
Finding useful information on the Internet in
AltaVista, from a user's point of view, is a
query system for finding useful information on the Internet. Accessible
through the World Wide Web from any standard Web browser, AltaVista
provides a simple interface for entering a few words pertaining to your
topic of interest. AltaVista then produces a prioritized list of all the
Web pages that contain at least one mention of the word or words in your
inquiry-the more mentions, the higher the priority in the listing.
What's more, each reference in the list is hyperlinked to the actual Web
page, so you simply click and you're there. Bear in mind, the entire
process takes only seconds.
Want to know the annual rainfall in Nepal? How
about the latest earnings from Fifth Third Bancorp? Care to hear your
favorite musician talk about her latest release? Or watch a clip from
her video? Looking for the latest breakthrough on osteomalacia? Need
data on the effects of carbon monoxide on evergreens? Want to find an
old friend? Or catch up on the news from down under?
If it's on the Internet, you can find it in
seconds using AltaVista. Moreover, if something new appears on the
Internet tomorrow-or the next day, or any day-AltaVista will know about
it. That's because AltaVista maintains a comprehensive database of every
word on every Web page on the Internet-and this database is growing
At last count, the AltaVista database contained
11 billion words indexed from over 22 million Web pages. This is a
database of practically everything on the Internet (some pages are
excluded, as discussed below). And it's accessible in an instant.
How do they do that?
The technology behind AltaVista is truly
revolutionary. It's a combination of super-sophisticated software and
Collecting data and making it useful
The software consists of the query tool
described above, a data collector (a.k.a. web crawler, spider, robot),
and an indexer. The AltaVista data collector, dubbed Scooter, is the
fastest known web crawler in existence. Scooter looks at 2.5 million Web
pages per day, every day, and brings back the contents of those pages to
its host computer for indexing.
Scooter is known as a "polite" web crawler; that
is, it obeys the rules of the Standard for Robot Exclusion. This means
that Scooter checks a special file at each Web site before visiting any
of its pages. This file may contain a listing of certain pages that the
site's Webmaster does not want traversed by a web crawler. Scooter will
not fetch any pages on the list.
Scooter is polite in another respect, too. It is
simultaneously accessing and fetching thousands of Web pages at a time.
Yet, it imposes minimal load on a Web server, to avoid inconveniencing
the site in way. To accomplish this, Scooter waits after performing a
fetch before it retrieves another page from the same site. By invoking a
delay that is a factor of 100 times the duration of the fetch, Scooter
accesses slower systems much less frequently that fast ones. In fact,
Scooter never uses more than 1% of the resources of a given system while
it is retrieving pages.
Now, what do we do with all this content? Enter
the Ni2 indexing software. Ni2 indexes an astounding 1 gigabyte of text
per hour, producing links to every word on every Web page brought back
by Scooter. The Ni2 index is the key software that allows you to enter a
few words in the query interface and instantly retrieve a listing of
relevant Web pages.
One of the most important features of Ni2 is its
ranking system-the means of prioritizing the results of a query. Ni2
looks through the documents in the index and considers each document
that includes at least one of the words in your query. Then, using a
method known as "collection frequency weighting," it calculates a score
for each matching document, placing the most relevant and useful
documents at the top of the list.
Providing results in an instant
The AltaVista software is optimized for
Digital's 64-bit Alpha technology, which enables the query interface,
Scooter, and Ni2 indexer to perform at unbelievable speeds.
To process inquiries from the AltaVista site, we
use a pair of AlphaStation 250 4/266 systems, each with 256 MB of RAM
and 4 GB of hard disk. Running on the AlphaStation systems is a custom
multi-threaded Web server, which sends queries to the Web indexer and
News indexer. With just these two relatively small systems we easily
handle up to six million hits per day to the AltaVista site. About 90%
of the queries are to search the Web and approximately 10% for
Scooter runs on a DEC3000/900 AlphaStation with
1GB RAM and 30 GB of hard disk with RAID 5 to ensure data integrity. The
sole job of this computer is crawling the Web, fetching content and
sending it to the Web indexer.
The Ni2 Web indexer runs on an AlphaServer 8400
5/300 (a.k.a. TurboLaser), which includes 10 processors, 6 GB of RAM,
and 210 GB of hard disk with RAID 5. The TurboLaser is the most powerful
computer built by Digital, holding a Web index that is larger than 30
GB, while providing responses to most requests in less than a second.
Our News Server runs on an AlphaStation 400
4/233 system, with 160 MB RAM and 24 GB of hard disk with RAID 5. This
server maintains a current news spool for the News Indexer and serves
the articles via http to those users who simply want to read news using
the ease of their standard Web browser.
The New Indexer runs on an AlphaStation 250
4/266 system, with 196 MB of RAM and 13 GB or hard disk. This machine
keeps an up-to-date index of the news spool, handling the constant
turnover of thousands of news articles to ensure the most current
information is presented when you make a query.
AltaVista-A Brief History
Where did AltaVista come from? AltaVista emerged
from Digital's Palo Alto research laboratories in the spring of 1995.
Begun as a way to demonstrate the sheer power of the AlphaServer 8400
TurboLaser computer running an Oracle database, the demo quickly bloomed
into a research challenge to achieve the impossible.
Defying common wisdom, a small band of
researchers in Palo Alto set out to index the entire Web-something long
considered unobtainable. Yet, with benchmark results showing the
TurboLaser and Oracle performing 100 times faster than any of the
nearest competitors, nothing seemed impossible.
The right place at the right time
It was a research environment that fostered the
growth of an innovative idea. And it was the personality of one research
facility in particular that transformed that idea into a practical
reality. Digital's Palo Alto lab is filled with forward-looking
scientists and engineers-people who feel an urgency to move their
projects into the real world as products or technology demonstrations.
The original idea for AltaVista came from Paul
Flaherty, who saw it as a way to showcase the Very Large Memory Database
capability of the TurboLaser/Oracle duo by indexing the Internet. With
Digital's experience in 64-bit computing and collective expertise with
Internet technologies, the foundation was already laid. The state of the
Internet was in high growth, but still without much structure. So the
timing was ideal. What was needed was a way to fetch Web pages and index
Louis Monier developed the super-spider,
Scooter, from scratch with the sole intention of making AltaVista a
reality. This software is simply the result of Louis' skill and the
brain trust of Digital's research labs. All the resources needed to
create the world's fastest and most complete web crawler were there in
one place. In fact, there is probably nowhere else in the world that
could have supported this effort.
The Ni2 Web indexing software started out as a
way for Mike Burrows to organize his personal e-mail. Then, in response
to a dare back in 1991, he refined the code to index all of the articles
in all of the newsgroups on the Internet. Mike's original approach
scaled very well and proved capable of handling an enormous amount of
text-a million documents could be indexed. With a bit more "tinkering,"
he revised the code further, making the indexer useful and fast in
dealing with an even larger set of documents. It was just good timing
that Mike finished the second incarnation of the indexer-Ni2-when the
AltaVista project was taking off. It proved to be the perfect test of
his new software with the huge set of data that Scooter was returning
from the Web.
Through a combination of ideal research
conditions and excellent timing, AltaVista had arrived.
AltaVista-A Long Future
AltaVista grew out of pure research, but it was
no accident. It was borne of an environment that values creativity and
AltaVista, today, is without doubt a showcase
for the technological leadership of Digital computers and software. And
it will remain so. But it also holds fundamental value as a tool for
using the Internet in our every-day business and personal lives. Its
sudden and remarkable popularity-attracting over six million hits per
day without any promotion-is a testament to that fact.
AltaVista marks a turning point in the way we
view and use the Internet. It makes it possible for anyone to gain value
from resources on the Internet, without wasting hour after hour in the
process. It also has implications for how Web sites are structured. For
instance, the home page, which has been the traditional first point of
entry to a site-defining it, setting the tone, and providing links to
further detail-may never be seen by the typical visitor. Through
AltaVista, Web travelers could land on a page anywhere in a particular
site, based on their specific interest. With AltaVista, in fact, the
entire Web is treated as one huge site-the only home page left is
AltaVista itself. It has proved to be the most natural starting point of
More of a good thing-mirror sites
Where do we go from here? With the tremendous
growth in popularity of the single AltaVista site in Palo Alto-the
number of daily hits grew by more than 2,000 percent in just three
months-the natural course of action is to deliver more of a good thing.
We will start by establishing mirror sites around the world.
An AltaVista mirror site exactly replicates the
index and search capabilities of the Palo Alto site. Mirror sites will
be licensed using a franchise model, which guarantees exclusive rights
for the owner to provide AltaVista services within a geographical
territory. While mirror sites will harness AltaVista technology, the
format and layout of their Web pages will be managed by the mirror site
owner. But for anyone using a mirror site, you will still recognize the
AltaVista "look and feel"-with the same quality, performance, and
availability that makes the original site so useful.
With the development of mirror sites, AltaVista
will be available to more Internet users than ever before, all around
the world-truly establishing it as the leading Internet search engine.
Particularly for users outside of the U.S., response times will increase
significantly. And with geographic distribution of mirror sites, we can
support local languages and reflect the culture of the region.
Getting even more useful information
The research environment that produced AltaVista
is alive and brimming with new ideas, new technological breakthroughs.
This is sure to produce expanded capabilities for the existing site and
future mirror sites.
Inevitably, the Web index will continue to grow,
as more sites appear and as Scooter penetrates the Web even deeper than
it is currently. In fact, rather than performing periodic crawls,
Scooter will soon crawl the Web continuously, updating the index even
more frequently than is possible today. In addition, Digital researchers
are tackling the difficult challenge of indexing Asian Web pages, where
multiple character encodings present unique difficulties. As we have
seen, though, nothing is impossible. And the result of this extra effort
simply means a greater wealth of information and resources available to
more and more Internet users.
Another enhancement to the site will be new
ease-of-use features to assist the user in refining queries for more
precise results. For example, a query on "bonds" might prompt the user
Through such prompting and refinement, AltaVista
will make it simpler for a new user and more effective for advanced
users to produce valuable results. Another refinement technique may take
advantage of "query builds"-using the query mechanism for multiple
searches built on each other. By successively refining each query, the
user can narrow down the number of matches and, again, focus in on the
most pertinent results.
Adding value to commercial content
There is also life for AltaVista beyond the
search site. The powerful search and indexing technology in AltaVista is
written in portable code, which has huge implications for where and how
this technology can be utilized.
One of the first likely applications is as a
value-added capability to selected content sites on the Internet. In
this scenario, a content site would incorporate AltaVista with some
level of added value beyond simply returning a list of Web links. The "raw"
data produced by AltaVista would undergo post processing and
reformatting by the host site. For example, a publishing company could
provide search capabilities across various categories of book titles.
The results of a search might produce a listing of matching titles,
additional titles by the same author, other selections with similar
subject matter, etc. The possibilities here are nearly limitless.
AltaVista at work
If we can index the entire World Wide Web, and
help the average Internet surfer find practically anything in a matter
of seconds, just think what we could do for today's corporations,
universities, and government agencies.
With all that Internet technologies have brought
to the public arena, many enterprises are recognizing the value of
adopting the same capabilities inside their operations-on private
intranets. Many have set up TCP/IP networks (the same as the Internet)
and are populating desktops with standard Web browsers, such as
The number of intranet Web pages is growing ever
more rapidly. For anyone on the enterprise network-as on the
Internet-spending time just in pursuit of information draws directly
away from productive endeavors. And unless information can be located
and retrieved, its value diminishes significantly. AltaVista technology
will be adapted for enterprises, to help them unlock the full value of
all the information on their intranets.
AltaVista Team Edition
A "Team" edition of AltaVista, for instance,
will provide an AltaVista query interface allowing browser users to
instantly find useful information anywhere on their private intranet. A
Scooter-like crawler will crawl the intranet, fetching internal Web
pages. In turn, an Ni2-like indexer will compile a complete, full-text
index of the entire body of material. The Team edition will only crawl
behind the corporate firewall and, initially, will only fetch HTML (Web)
information. As the technology evolves, however, all corporate data,
regardless of format, will be crawled and indexed.
AltaVista Enterprise Edition
An "Enterprise" edition will extend the reach of
the intranet crawl out onto the Internet. This will be a tailored
solution, enabling customers to specify a particular subset of the
Internet to be crawled. In addition, the Enterprise edition will enable
distributed enterprises, with numerous remote, branch, or home workers
connected across an Internet backbone, to be included in the crawl. The
result will produce a thorough index of all information in an
enterprise-unlocking hidden assets and maximizing the value contribution
of every individual.
AltaVista Personal Edition
Also planned is a "Personal" edition of
AltaVista, designed for the individual desktop. The Personal edition
will offer the same robust data gathering, indexing, and query
capabilities as the original AltaVista, specially tuned and packaged for
personal use. The key here, as with all AltaVista search tools, is speed
and depth. No other desktop tool on the market could match the
performance of AltaVista. You will have a single interface to query all
the information on your desktop-regardless of format-and locate relevant
files in seconds. Imagine being able to instantly find a choice piece of
data, buried in an e-mail message from three years ago. That's the power
of AltaVista brought to your desktop.
AltaVista technology has spawned a whole new
vision of the Internet, providing a higher view from which to locate and
access valuable resources. And it has inspired broader use of Internet
technologies across the enterprise and around the world.
Digital is building on the success of AltaVista,
with an entire family of software products. Using Internet technology as
a common, ubiquitous environment, AltaVista software provides users with
dynamic, global capabilities for exchanging information and ideas. It
combines the vast resources of the Internet, the navigational ease of
standard Web browsers, and the value of existing intranet assets.
AltaVista is truly revolutionary in scope,
breaking through traditional barriers that limit communication and
information access. It is both a vision and a reality, offering
immediate rewards as it inspires new, innovative technology. It is a key
that unlocks all the world of the Internet has to offer. So wherever
your business takes you, start your journey with AltaVista.