content moderation

Every day, billions of people worldwide sign onto the world’s social media platforms, consuming and producing massive amounts of digital media material for dissemination to friends, their communities, and the world. These media and the resulting activity related to their consumption—users’ status updates, likes and favorites, comments, posts, photo and video uploads, and so on—constitute what is known in the social media industry as user-generated content (UGC). The system of review and sometimes removal of UGC is known as content moderation. This entry further describes UGC and content moderation, then discusses the impetus for and outcomes of commercial content moderation, content moderation as a labor practice, and the relationship between content moderation and the law.

In terms of scale, recent statistics about how much content is generated on mainstream social media platforms are astounding: in the early 2020s, there are around 500 hours of content uploaded to YouTube every minute, and more than 90 million photos are uploaded to Instagram every day.

UGC is central to sustaining or even driving increased engagement from users, creating a cycle of ongoing behavior of both production and consumption that characterizes user relationships with platforms. It is this cycle of participation and consumption and the reliance upon UGC that together serve as the basis for the business model of mainstream social media platforms, whose user base size and engagement, demonstrated by such participation, are key metrics used to articulate value when soliciting advertisers, arguably the platforms’ primary customers. Yet a business model predicated on opening these powerful, instantaneous, and global dissemination social media channels to anyone who wishes to upload to them also serves as a worrisome potential site of risk. Platforms have mitigated this risk (to their own brand and to their users) through gatekeeping, review, and removal of unwanted, deleterious, dangerous, or illegal content known as content moderation.

Discover how ordinary objects work, and expand your understanding of the familiar.

SIGN UP

" height="20" alt="HEW Logo">

Thank you for signing up!"

Back to homepage

By clicking "Sign Up", you consent to the Britannica Privacy Policy and Terms of Use.

Content moderation is an intervention practice that may consist of automated/computational or human-driven processes, or a hybrid of both, that involve the identification and evaluation of UGC against platform- or site-specific rules or guidelines, legal restrictions prohibiting certain kinds of content, or other matters of taste or social norms. The evaluation is followed by a judgment or verdict rendered upon the UGC which, in turn, is followed by a range of actions, including removal, age restrictions, labeling, or account suspension. Content moderation is therefore a primary mechanism of rules enforcement, adherence to legal standards, behavior control, and management of UGC on mainstream social media platforms.

Content moderation practices on mainstream social media platforms and sites have their roots in long-standing volunteer moderation and self-organized community management practices from earlier iterations of the Internet’s online social spaces and are related to the community management of contemporary platforms organized around volunteer community management. The intervention activity on mainstream social media sites, however, has evolved into commercial content moderation. The latter is distinct from other forms of community management and is characterized by highly organized activity coordinated by the platforms themselves, a large-scale and global workforce with particular linguistic and cultural expertise, and made up of a for-pay workforce of professionals employed as commercial content moderators part- or full-time.

The Impetus for and the Outcomes of Commercial Content Moderation

The phenomenon of content moderation activity and its results, particularly when industrialized, organized, and done at scope and scale, directly impacts the landscape of social media platforms by being a bidirectional gatekeeping mechanism, determining both what is allowed to stay up and what is deleted. The latter can rarely ever be perceived or apprehended in a meaningful way due to the platforms’ long-standing orientation toward opacity and secrecy regarding their commercial content moderation practices, policies, and techniques. This opacity can lead to suspicion, mistrust, and other negative consequences.

In recent years, the concept and phenomenon of content moderation has gone from obscure industry parlance among those who either require the practice or undertake it to a concept covered more and more frequently by technology journalists, studied by academic researchers, debated by civil society advocates, and interrogated by policy makers. This kind of engagement expanded greatly after 2016–17, when platforms and their parent firms were just barely acknowledging the fact that they had legions of workers, often contractors and subcontractors, deployed worldwide to work around the clock on contending with the UGC firehose and subsequent user reports, or flags, on abusive material.

Explore Britannica Premium!

The trusted destination for professionals, college students, and lifelong learners.

SUBSCRIBE

Mainstream social media platforms have been criticized for UGC depicting shocking incidents of abuse, sexual assault, suicide, and murder that have circulated across their networks and to which users have then been exposed. Social media firms have been criticized, too, for other more insidious types of deliberately misleading, inaccurate, and polarizing material disseminated on their platforms to a variety of potential ends—to confuse issues, sway political beliefs, sow social or political discord, foment hostility toward particular groups of people, and so on, much of which has been shown to be the work of organized campaigns acting on behalf of state interests. For platforms, risks to their brand and to their users lie in both. Increasingly, their response has been to gesture overtly and tacitly to their commercial content moderation practices as the primary line of defense, alternately invoking them as being at fault for misses and as the solution for ongoing problems, present and future.

Google and Facebook each have committed to a global workforce of tens of thousands of workers to contend with the commercial content moderation needs of their platforms and products. These are only two of the many firms, niche to massive, that rely on content moderation. In addition, firms have committed to building up new, specialized teams for particular aspects of UGC evaluation. These include the so-called ad raters at Google, news feed raters at Facebook, search quality raters who inform Google’s algorithms, fact-checkers, and other types of specialists who constitute new and developing areas of commercial content moderation beyond generalist work. Although the firms are typically cagey about how they will build up such a large workforce in such a short period of time, the likelihood is that a large portion of these workers will be globally dispersed and will often work in third-party business process outsourcing firms, commonly known as call centers. The world’s call center capital is the Philippines, where many firms specialize in content moderation services targeted to a global clientele.

Content Moderation as Labor Practice and Worker’s Well-Being

One sustained area of concern, and an early lens through which commercial content moderation has been studied and written about by journalists, is from the perspective of workers’ well-being. The work and working conditions of moderators who labor in a professional capacity for social media platforms have been the subject of research and reporting since at least 2010, when The New York Times published a story describing the conditions of some commercial content moderators working as third-party contractors in rural Iowa.

What the 2010 Times article and subsequent academic and journalistic investigations revealed was that the labor of reviewing thousands of posts, images, and videos per day in order to seek out policy and other types of violations can be, on the one hand, rote or repetitive, while on the other hand, can expose workers to content that is disturbing, abusive, shocking, graphic, violent, or frightening. Complicating this situation is the fact that many frontline commercial content moderation jobs are low status and low wage, are located in a variety of countries and cultural contexts around the globe, may not provide mental health–related coverage to their workers, or may be located in places where the seeking of such help is not socially acceptable. Therefore, the issue of resiliency support for workers both during their terms of employment and after has been of increasing concern to the workers themselves and to those who advocate on their behalf.

In 2016, a landmark lawsuit was filed in the state of Washington, United States, on behalf of two workers who had served as commercial content moderators employed by Microsoft for a decade and who alleged they had both been rendered permanently and wholly disabled by post-traumatic stress disorder due to the material they were asked to view and adjudicate as a main function of their jobs. In September 2018, a former Facebook content moderator, who had worked for Facebook doing moderation work via a third-party contracting firm, filed a lawsuit in the state of California alleging similar harms and seeking class-action status for the lawsuit. Both lawsuits were settled.

Google and Facebook are now on record as having many thousands of commercial content moderators working for them worldwide; accounting for the social media industry as a whole around the world, the actual number of people doing commercial content moderation is likely in the hundreds of thousands. Given the relative newness of this labor form and the fact that the long-term effects of working as a commercial content moderator are largely unknown, it is likely that the allegations of harms stemming from professional moderators’ working conditions will increase in the years to come. Individual firms have done a better job addressing the working conditions and the support mechanisms for their commercial content moderators at their own worksites, but the industry is working on safety guidelines and best practices.

Content Moderation and the Law

An additional source of pressure on social media firms’ content moderation practices is coming from new policy and laws that set particular moderation standards in order for platforms to operate within a given jurisdiction. One example of this is Germany’s Network Enforcement Act, known as NetzDG, which mandated that for German users, social media moderation practices must be aligned with Germany’s anti-hate speech laws. The law applies to any social media platform of more than 2 million registered users, which encompasses virtually all of the mainstream social media platforms and their parent firms. The result was the firms’ mass hiring of commercial content moderators with German language ability, cultural sensitivity, and local knowledge, and the development of specialized tools and services for the German user base, in order to comply with the requirements to report UGC that violates German law.

The costs of noncompliance are high: 50 million euros for cases of obviously illegal content not removed within 24 hours of a complaint. Other legislative bodies have begun asking pointed questions about firms’ content moderation practices, long kept secret, and it is likely that other jurisdictions and sovereign nations will follow Germany’s lead by demanding acknowledgment of their local laws in moderation practices.

The policies and reach of social media firms have evolved quickly but often without a big picture understanding of context, implications, and consequences for individual users and larger corporations alike. Further, moderating content often takes place without detailed transparent, user-friendly policies that allow moderators’ decision-making to be clearly understood by the public. Pressure from activists, civil society advocates, legislators, and others will likely yield greater transparency and insight into the content moderation ecosystem—its people, programs, policies, and practices—at mainstream social media firms in years to come.

The original version of this article was published by SAGE Publications in The SAGE International Encyclopedia of Mass Media and Society.