Algorithms For Dummies

  Getting the Right Web page 

     209

All these search engines worked by having specialized software autonomously 

visit the web, using domain lists and testing hyperlinks found on the visited pages. 

These spiders explored each new link in a process called crawling. Spiders are pieces 

of software that read the pages as plain text (they can’t understand images or 

other nontextual content).

Early search engines worked by crawling the web, collecting the information from 

spiders, and processing it in order to create inverted indexes. The indexes allowed 

retracing pages based on the words they contained. When you made a query, such 

inverted indexes reported all the pages containing the terms and helped score the 

pages, thus creating a ranking that turned into a search result (a list of ordered 

pages, ranging from the anticipated most useful page to the least useful page).

The scoring was quite naive because it often counted how frequently the keywords 

appeared on pages or whether they appeared in the titles or in the header of the 

page itself. Sometimes keywords were even scored more if they mixed or clustered 

together. Clearly, such simple indexing and scoring techniques allowed some web 

users to take advantage by using various tricks: