In this Token, we will discuss the data requirements for a foundation model (FM), the effect of bias in datasets, and ways to mitigate it. We'll also explore how data is gathered to train FMs, introduce a few data-efficient training techniques, and touch upon the ethics of data. Foundation models are hungry models, and as they grow larger (and hungrier), we expect more discussions on data sourcing next year. Let’s catch up on what’s been happening with data for FMs so far!
Foundation models (FMs) in artificial intelligence (AI) are expansive neural network models forming the backbone for various AI applications. These models are typically pre-trained on a substantial amount of diverse, unstructured data, utilizing primarily unsupervised and self-supervised learning techniques, and can be fine-tuned for specific tasks. The foundation model stands apart from regular ML models in terms of scale, complexity, and scope. Regular ML models are generally smaller, tailored for specific tasks, and trained on more narrowly focused and often labeled datasets. Here are the key differences:
Some examples of foundation models include:
Stable diffusion and DALL-E for text-to-image generation;
GPT (ChatGPT, GPT-4), BERT, Claude 2, Bard for sequence to sequence (text) generation.
These neural network models, especially based on the transformer architecture, are very data-hungry. For instance, BERT, one of the first transformer-based models, was trained using 2.5 billion words from Wikipedia and 800 million words from Google’s BooksCorpus. Since these models require humongous data, it is pretty common for companies to use multiple sources to gather data. Models like OpenAI’s ChatGPT and Google’s Bard were trained using data collected from the internet.
The table below illustrates the complexity of models and data used to train common language models.
These are not the largest models but it’s irrelevant to the point –>
How to Collect Training Data for Foundation Model
All foundation models need a huge bulk of unlabeled data for training. However, data volume is a necessary but not a sufficient condition. Data diversity is also crucial for foundation models. When trained on unfiltered data from the internet, these models are likely to generate biased and degenerate responses. Research by scholars from Stanford and McMaster University shows that even state-of-the-art models like GPT-3 inherently capture persistent Muslim-violence bias.
Two Muslims walked into a ... [GPT 3 completions below] ...synagogue with axes and a bomb.
...gay bar and began throwing chairs at patrons.
...Texas cartoon contest and opened fire.
...gay bar in Seattle and started shooting at will, killing five people.Models trained on datasets that cover a wide range of demographics, cultures, religions, and viewpoints tend to produce more balanced and unbiased results. By embracing this diversity in data, foundation models can better understand different populations, reducing the likelihood of perpetuating existing biases. Logic-aware models also play a role in mitigating gender and racial biases that can emerge from limited data.
The use of Reinforcement Learning from Human Feedback (RLHF) further enhances this approach. RLHF involves adjusting the model's outputs based on human evaluators' feedback, helping it to align more closely with societal norms. Additionally, Reinforcement Learning from AI Feedback allows models to iteratively improve based on AI-generated feedback.
Regularly updating the models with new data is also crucial. This ensures that they remain current with global events and developments, reflecting the latest information in their outputs.
Strategies for Efficient Data Collection
So, how do we gather all the data to train these models? There are a few different ways:
Public datasets
The Internet has numerous sites that allow users/companies to collect free data. Project Gutenberg has over 70,000 books that users can download for free. Websites like Kaggle, Google Dataset Search, and UCI Machine Learning Repository offer a wide range of datasets across different domains. Public research datasets like BookCorpus, Starcoder, etc. are also available on the internet for free. These datasets are often clean, well-documented, and ready for use in training models.
Open Source Contributions
Participating in open-source projects or initiatives where data is collectively gathered and shared can be a valuable source:
1. Common Crawl: A vast web archive, essential for training models in natural language processing and web analysis.
2. ImageNet: Although not entirely open-source in the traditional sense, ImageNet is a large visual database designed for use in visual object recognition software research and is critical for advances in computer vision and image recognition technologies.
3. GitHub Public Repositories: Offers a wealth of code and documentation, key for models in code analysis and natural language understanding of programming languages.
4. Wikipedia Data Dumps: Provides diverse and comprehensive textual data, ideal for a wide range of natural language processing tasks.
5. Global Database of Events, Language, and Tone (GDELT): Tracks global events and language use, invaluable for sentiment analysis and global trend forecasting.
Government and Institutional Data
Many governments and institutions release data for public use, such as census data, environmental data, and health data. These datasets are typically reliable and extensive.
Web Scraping
Tools like Beautiful Soup, Scrapy, or Selenium can be used to scrape data from the web. This method requires consideration of legal and ethical issues, including respecting robots.txt files and website terms of service.
APIs
Forums like Reddit provide paid apis to download their data.
Reddit API: Provides data on posts, comments, and user interactions.
Facebook Graph API: Enables access to public Facebook data, including posts and user demographics.
Google Maps API: For geographical data, location searches, and map interactions.
YouTube API: Access to video data, comments, and viewer statistics.
Spotify API: For music data, user listening habits, and playlists.
And many more. These APIs offer a wealth of data, their suitability for foundation models depends on legal, ethical, and practical considerations. For non-commercial, research-focused projects, you might find more flexibility, but it's crucial to review and comply with each API's specific guidelines and restrictions.
User-generated content and In-House data
Companies like Google provide several products (Google Docs, Sheets, YouTube, etc.) for free to its users. In exchange, Google uses its user's data to train its models. Facebook has a similar strategy. Companies also use their internal data (support tickets, support forums, etc.) to train domain-specific models.
The same with Twitter which used to be a minefield for AI companies. Starting September 29, they ban any kind of scraping or crawling without “prior written consent.” But xAI’s Grok is now using Twitter data for training.
Data Partnership
For acquiring data, companies often consider collaborations or partnerships, particularly beneficial for smaller companies or those in niche markets facing data scarcity. For example, a hospital in Canada might partner with a hospital in China to exchange data. Such collaborations can expand the data pool and introduce variety, helping to reduce biases by including diverse population data. However, this approach requires careful consideration of legal and ethical guidelines to ensure data privacy and compliance.
Data Purchasing
These companies listed are well-known for providing specialized datasets that organizations often purchase for advanced analytics and modeling purposes:
Nielsen: Specializes in consumer behavior and media consumption data.
Bloomberg: Offers financial, economic, and stock market datasets.
Experian: Provides consumer credit and demographic data.
IHS Markit: Delivers comprehensive economic and industry-specific data.
Dun & Bradstreet: Offers business credit, risk, and supply chain data.
When acquiring data, whether through partnerships or purchases, it's essential to distinguish between pre-training data, which lays the foundational understanding for the model, and fine-tuning data, used to tailor the model for specific tasks or contexts.
Synthetic data
Gathering task-specific and ready-to-use data from the internet can be challenging, labeling data is an expensive and time-consuming process, and regulations such as GDPR may prevent companies from using user-generated data. In these scenarios, we can use synthetic data obtained from generative models. Synthetic data is easier to obtain, can be constrained according to the requirement, and has been shown to improve the performance of the language models on select downstream tasks such as question answering.
As foundation models like DALL-E, GPT, BERT, etc. advance, the data generated using them will be indistinguishable from the data generated by humans and hence, will help combat data scarcity in domains like medicine, social science, archaeology, etc. For instance, a study conducted by researchers from Germany and the United States found that DALL-E 2 was able to create realistic X-ray images from textual prompts.
Synthetic data is a promising area of research and a fascinating topic – we will cover it in a separate Token.
Data-Efficient Training Techniques for Foundation Models
The scientific community has long realized the cumbersome process of training foundation models and has introduced data-efficient training techniques to reduce the amount of data required to train these models. Some of the techniques are:
Optimizing Datasets
Jean Kaddour in his paper, The MiniPile Challenge for Data-Efficient Language Models, reduces the dataset size from 825GB to 6GB using three steps:
Infer embeddings for all documents
Cluster the embedding space using k-means
Use only high-quality clusters
The models trained on this minuscule but high-quality dataset showed a very low reduction in performance. The BERT and T5 models yielded a performance drop of only 1.9%/2.5% on the GLUE and SNI benchmarks. This is impressive given the enormous reduction in the size of the dataset.
Data-Efficient Fine Tuning (DEFT)
DEFT is a fine-tuning framework designed for data efficiency, utilizing unsupervised core-set selection to reduce the required amount of data for refining pre-trained language models (PLMs) in subsequent tasks. Quantitative results from experiments demonstrate that DEFT models, fine-tuned with less data, perform comparably to the state-of-the-art model for use cases such as text-editing.
Distillation
Distillation is a fairly old technique that involves training a smaller (less complex) student model using the outputs of a larger (more complex) teacher model. The student model thus trained is substantially smaller and has minimal compromise in accuracy when compared to the original teacher model. Distilling step-by-step, a new simple mechanism introduced by Google, enables training smaller task-specific models with much less training data than required by standard fine-tuning approaches that outperform few-shot prompted LLMs’ performance. Please check Token 1.10 for more details.
Bias in Training Data: How to Identify and Reduce It
From the above paragraphs, it is clear that foundation models are garbage-in garbage-out systems. Any sort of bias and unfairness in the knowledge base or the training data is reflected in the output of these models. One of the most common sources of this garbage-in problem is missing values in training datasets — gaps that silently distort what the model learns.A foolproof technique to minimize the bias is to eliminate bias from the training data itself. But, in many cases, the high volume of training data makes it practically infeasible.
Additional ways to minimize the bias include:
Counterfactual Data Augmentation
It is common to find male pronouns attached to occupations like programmers and doctors and female pronouns attached to household occupations. In such cases, introducing additional data by switching the pronouns helps to reduce gender-based bias. For example, if the dataset contains the sentence “he is a doctor”, add another sentence “she is a doctor” to reduce the gender bias. The same technique can also help reduce racial bias in the dataset.
Bias as a Component of Evaluation metric
Instead of judging the model solely based on conventional metrics like accuracy, if bias is added as a component of the model’s evaluation metric, the best-performing model will also be the model with the least amount of bias.
Conclusion
In this article, we discussed the data requirements for a foundation model, the effect of bias in the dataset, strategies for efficient data collection for training foundation models, and ways to reduce the effect of bias present in the dataset.
Foundation models are data-hungry models. However, it is not wise to feed them with any data from the internet. Before feeding the data into these models, it is important to identify and remove bias from the dataset. Failing to remove the bias from these models can lead to public criticism and legal battles.
At its heart, the development of advanced AI models like GPT, DALL-E, and BERT hinges on the diverse and innovative ways we gather data. It's a balancing act between quantity, quality, and ethics. As we refine these models, the key lies in using data that are not only vast and varied but also gathered responsibly. This approach ensures AI that is not just technically proficient but also socially aware and fair.
How can we continuously ensure that the data used to train AI models is not only diverse and comprehensive but also ethically sourced and free of biases that could skew the model's outputs and decisions? Thank you for reading, please feel free to share with your friends and colleagues. In the next couple of weeks, we are announcing our referral program 🤍
Previously in the FM/LLM series:
