deep web
-
What is the deep web?
The deep web is the portion of the internet that search engines can't index, including fee-for-service content, unindexed sites, and private databases. Coined in 2001 by computer scientist Michael K. Bergman, the deep web is also referred to as the invisible or hidden web.
The deep web is not the same as the dark web, which makes up only a small fraction of the overall web. The dark web uses encryption software for anonymity, and requires specific software such as Tor to access it.
The deep web is estimated to be about 500 times larger than the surface web. It contains as much as 96% of online content, including legal, medical, and financial databases, password-protected email accounts, and fee-based media services like Netflix.
3 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
How is the deep web different from the surface web?
The surface web is the portion of the internet that is readily accessible to the general public and can be easily indexed by search engines. Software robots known as "spiders" or "crawlers" capture and index content on the surface web, allowing search engines to rank it. These crawlers scan websites with common domains (such as .com, .org, and .net) and follow embedded links to discover more content.
The deep web comprises the rest of the internet content, about 96%, which is not indexed and is therefore hidden from standard search engines. This includes content behind paywalls, password-protected information, and private databases. Websites on the deep web use methods to block spiders and prevent indexing, like CAPTCHAs, multiple IP addresses, and password protection.
4 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
Why can't regular search engines find deep web content?
Regular search engines can't access deep web content because they use software robots called "spiders" or "crawlers" to capture, index, and rank content on the surface web. These spiders follow embedded links to uncover additional content. However, the deep web contains a vast amount of unindexed content that is not easily searchable by the general public.
Websites on the deep web employ various methods to block spiders and prevent indexing, including CAPTCHAs, multiple IP addresses, non-HTML content, password protection, and unlinked content. Some content is also only accessible through a fee or specific software. While the deep web is often conflated with the dark web, the deep web contains mostly benign sites, such as password-protected email accounts and paid subscription services, and is estimated to be 400â550 times larger than the surface web.
3 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
What kind of information can be found on the deep web?
The deep web contains web pages not indexed by standard search engines like Google. It is much larger than the surface web. The deep web includes a wide array of content, such as:
- Password-protected email accounts
- Fee-based video and media on-demand services
- Financial records
- Legal and medical databases
- Corporate intranets
While the deep web is often conflated with the dark web, the dark web is a small portion of the deep web where users and website owners use encryption software to remain anonymous. The dark web is associated with illegal activity, but is also used by political activists and journalists who risk retaliation if discovered.
4 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
Is the deep web the same as the dark web?
The deep web and the dark web are not the same. The dark web is a small portion of the deep web that requires specific software for access and ensures anonymity through encryption. This makes it a common space for illegal activities, though it's also used by activists and whistleblowers.
The deep web includes any content not indexed by standard search engines like Google, such as password-protected emails, subscription services, and sites behind online forms. The deep web is much larger than the surface web and the dark web, containing mostly benign content.
5 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
Why do some websites choose to stay on the deep web?
The query assumes websites choose to stay on the deep web. However, the deep web consists of Internet content not searchable by standard search engines. The dark web, a small fraction of the deep web, uses encryption software for anonymity, often associated with illegal activities.
The deep web contains mostly benign sites, such as password-protected email accounts, parts of paid subscription services like Netflix, and sites accessible only through an online form. The dark web is used by political whistle-blowers, activists, and journalists who may be censored or risk political retaliation if discovered by their government.
3 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles.
deep web, a part of the Internet that extends beyond the reach of search engines such as Google, Yahoo!, and Bing because it includes unindexed sites, fee-for-service (FFS) content, and private databases such as password-protected e-mail accounts. As such, it is a widely effective source of mass media. The term âdeep webâ was coined in 2001 by computer scientist Michael K. Bergman, who differentiated it from the âsurface web,â where openly viewable and retrievable content resides. The deep web is also known as the âinvisible webâ or âhidden web,â but it should not be confused with the âdark web,â where encrypted content with hidden IP addresses resides. Called âdarkâ because it is accessible with anonymity and only through certain networks and software such as Tor, this part of the Internet represents a small fraction of the overall Web.
The deep web represents a vast array of data and content. In fact, it is likely some 500 times larger than the surface web and may contain as much as 96 percent of online content. Much of this content is in the form of databases that are accessible only by password or subscription or for a fee.
How the deep web works
Content that resides on the surface web is accessible because software robots called âspidersâ or âcrawlersâ capture and index it, and search engines assign it rankings. These systems typically scan websites that contain .com, .org., .net, or a similar domain as well as some data and posts at social media sites. As these spiders capture Web pages, they follow embedded links to uncover additional content. Later, when people search for specific content, the results appear in a search engine such as Google, where the content is then openly viewable.
" height="20" alt="HEW Logo">
With only about 4 percent of all online content freely accessible (making up the surface web), the remainder is tucked away in the deep web. This means there is no easy, direct way for the general public to search this vast amount of unindexed content. In some cases, websites use various methods to block spiders and prevent indexing. These methods include using CAPTCHAs, multiple IP addresses for the same content, non-HTML content or data that spiders cannot pick up, password protection, and unlinked content.
Types, viewability, and risks of deep web content
There are numerous types of deep web content. These include websites with registration requirements; fee-based video and media on-demand services such as Netflix, HBO Max, Apple TV+, Amazon Prime, and Spotify; and various password-protected entities. Among these are e-mail systems, legal and medical databases, corporate intranets, document libraries, image archives, financial records, scientific databases, and police and government resources as well as gaming sites, cloud storage services such as Dropbox or iCloud, and private sites and services that are not registered and thus do not appear in search engines. However, university researchers and commercial search services such as Google and Microsoft have explored ways to index deep web content, and law enforcement agencies have attempted to develop deep web crawlers that can spot illicit activities, including drug dealing, sex trafficking, and terrorist activity. The success of these efforts has been limited.
There is no inherent risk associated with searching the surface web or entering the deep web. In fact, both acts are common and part of daily life and business today. However, some websites contain digital dangers such as malware, viruses, spyware, and keyloggers (a kind of surveillance software that can monitor and record every keystroke on a computer). It is also possible to find sensitive data on the deep web that can be stolen or abused, and, of course, the possibility of encountering individuals who engage in cybercrime and unethical, even harmful, activities is always a risk in the online world.