Ask the Chatbot Games & Quizzes History & Society Science & Tech Biographies Animals & Nature Geography & Travel Arts & Culture ProCon Money Videos
Web crawler
References
Britannica AI Icon

Web crawler

software
Also known as: crawler
Written by
The Information Architects of Encyclopaedia Britannica

The Information Architects maintain a master list of the topics included in the corpus of  Encyclopædia Britannica, and create and manage the relationships between them.
The Information Architects of Encyclopaedia Britannica
Last updated

Quick Summary

A web crawler, also known as a bot or spider, is a software program designed to systematically search and collect data from web pages. These programs navigate the internet by following hyperlinks from one page to another. The information gathered by crawlers is used to build an index for search engines. Search engines then use this index to present users with a ranked list of relevant web pages in response to their queries. Web crawlers are essential for search engines to discover and organize the vast amount of information available on the World Wide Web.12

Summary created from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles.
3 Britannica Sources
  1. web crawler
  2. search engine
  3. crawler

Ask Britannica AI about this topic

Britannica doesn't have a full article on this topic, so we've curated a list of our relevant coverage.