spider

computer program

Written by

The Information Architects of Encyclopaedia Britannica

Last updated

July 19, 2026

Quick Summary

A web crawler, also known as a spider, is a computer program designed to systematically search the World Wide Web for specific keywords. These programs explore the internet by following hyperlinks from page to page, collecting data to build an index.12

While spiders are known for spinning silk webs to catch prey, a web crawler is a digital tool. It's used by search engines to gather information from websites, which then helps determine how pages rank in search results. The process involves bots or crawlers that rapidly search for data, far faster than a human could.34512

Summary created from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles.

5 Britannica Sources

Ask Britannica AI about this topic

Britannica doesn't have a full article on this topic, so we've curated a list of our relevant coverage.

CAPTCHA
…programs, known as bots and spiders, from gaining access to websites. A CAPTCHA, which may consist of letters, numbers, or images, is distorted in some manner to prevent recognition by computers but not so distorted that a human with normal vision cannot identify the code and retype it.
Read More
…programs, known as bots and spiders, from gaining access to websites. A CAPTCHA, which may consist of letters, numbers, or images, is distorted in some manner to prevent recognition by computers but not so distorted that a human with normal vision cannot identify the code and retype it.
Read More