- Related Topics:
- surveillance capitalism
- data
big data
-
What is big data?
Big data refers to extremely large and complex datasets that can't be processed using traditional database management tools. The term, which originated in the mid-1990s, is traditionally characterized by the three V's: volume, velocity, and variety. Volume refers to the large size of datasets; velocity is the speed at which data are produced and analyzed; and variety refers to the many different types of data.
Nearly every industry uses big data to detect patterns, recognize fraud, and more. New tools, such as Hadoop and Spark, have been developed to analyze big data, but working with these datasets can be challenging because storing large amounts of data requires a significant investment in equipment.
2 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
Why is big data important in computer science?
Big data is important to computer science because it has spurred the rapid evolution of computational science, especially with the increase in the amount of data transmitted from scientific instruments. Big data is characterized by volume, velocity, and variety. The challenges of working with big data include storing large amounts of data, ensuring data accuracy and completeness, and maintaining data security, especially when it contains sensitive information. New tools, such as Hadoop and Spark, have been developed to help analyze big data.
3 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
What are some common sources of big data?
Big data is characterized by volume, velocity, and variety and is used in nearly every industry in the 2020s. Entertainment companies use consumer-generated data to determine which song or video a consumer may want to see or what kind of movie or television series to produce. Banks rely on big data to find patterns that could indicate fraud or persons who may be a credit risk, and manufacturers use big data to detect faults in the production process and to avoid costly shutdowns by finding the best time for equipment maintenance.
The Internet of Things (IoT) also contributes to the sources of big data. IoT consists of the Internet Protocol (IP) and Transmission Control Protocol (TCP) which provide the standards and rules for devices to connect to the internet and to each other.
4 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
What challenges do people face when working with big data?
Working with big data presents several challenges. Storing massive datasets demands significant investment in equipment, often requiring specialized data centers that consume vast resources like water for cooling. Ensuring data accuracy and completeness becomes increasingly complex as the volume of data grows. Moreover, maintaining data security is crucial, especially when handling sensitive personal information. New tools like Hadoop and Spark, supported by the Apache Software Foundation, have been developed to analyze big data, but the scale and complexity of these datasets require careful management and infrastructure.
1 Britannica SourceCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
How do computers store and analyze big data?
Computers store big data in specialized data centers, which require significant investment and energy to maintain. NoSQL databases are often used for storing these datasets, as they efficiently handle unstructured data like text, video, or audio.
To analyze big data, computers use tools like Hadoop and Spark, often maintained by open-source foundations. Data analysis involves descriptive statistics, exploratory analysis, and predictive modeling. Techniques such as machine learning, regression analysis, cluster analysis, and anomaly detection are employed to identify trends, relationships, and unusual patterns within the data. These analytical insights facilitate data-driven decision-making in various fields.
3 Britannica SourcesCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles. -
How is big data used in real-world applications like healthcare or business?
Big data is used across many industries. Streaming services use consumer data to suggest what users may want to watch or listen to, and to decide what movies or shows to produce. Banks analyze big data to detect fraud and assess credit risks. Manufacturers use it to find defects in production and to predict when equipment maintenance should occur.
Tools such as Hadoop and Spark help to analyze these datasets, which may be stored in NoSQL databases. Managing and securing big data involves challenges like ensuring data accuracy and protecting sensitive personal information, and also requires significant investments in equipment and data centers.
1 Britannica SourceCreated from Britannica articles using AI. AI can make mistakes, so verify using Britannica articles.
big data, in technology, a term for large datasets. The term originated in the mid-1990s and was likely coined by Doug Mashey, who was chief scientist at the American workstation manufacturer SGI (Silicon Graphics, Inc.). Big data is traditionally characterized by the “three V’s”: volume, velocity, and variety. Volume naturally refers to the large size of such datasets; velocity refers to the speed with which such data are produced and analyzed; and variety refers to the many different types of data, which can be in text, audio, video, or other forms. (Two further V’s are sometimes added: value, referring to the usefulness of the data; and veracity, referring to the data’s truthfulness.)
Since the term big data was coined, the amount of data has grown exponentially. In 1999 an estimated 1.5 exabytes (1 exabyte = 1 billion gigabytes) of data were produced worldwide; in 2020 that number grew to an estimated 64 zettabytes (1 zettabyte = 1,000 exabytes). About the turn of the 21st century, big data referred to datasets of a few hundred gigabytes each; in 2021 EsNet, the U.S. Department of Energy’s data-sharing network, carried more than 1 exabyte of data.
In the 2020s nearly every industry uses big data. Entertainment companies, particularly streaming companies, use the data generated by consumers to determine which song or video a given consumer may want to see next or even to determine what kind of movie or television series the companies should produce. Banks rely on big data to find patterns that could indicate fraud or persons who may be a credit risk. Manufacturers use big data to detect faults in the production process and to avoid costly shutdowns by finding the best time for equipment maintenance.
New tools have been developed to help analyze big data. Such datasets are often stored in NoSQL databases. Traditional databases are in tabular format, with rows and columns, and the computer language SQL was designed with such relational databases in mind. However, extremely large datasets whose data are unstructured (that is, tending to be qualitative, such as text, video, or audio) are called NoSQL, because SQL may not be the best tool for working with such data. Some of the most popular tools for working with big data, such as Hadoop and Spark, have been maintained and developed by the Apache Software Foundation, a nonprofit organization that supports many open-source software projects.
Working with big data presents certain challenges. Storing large amounts of data requires a significant investment in equipment. Specialized buildings called data centres are used by companies such as Google, Amazon, and Microsoft for storing data, and the largest data centres require billions of litres of water per year to keep the buildings cool. Basic data analysis problems such as ensuring that the data are accurate and complete become much more difficult as the amount of data grows. Data security is very important, particularly when the data contain sensitive information about individuals and their habits.