Data Collection & Sourcing

🗂 Data Sources

Every machine learning project begins with a fundamental question: where does the data come from? The source shapes everything downstream — quality, bias, licensing constraints, and ultimately model performance. Understanding the trade-offs between different data sourcing strategies is a core practitioner skill.

📊 Data Volume Planning

One of the most common questions in applied ML is "how much data do I need?" The honest answer depends on model complexity, task difficulty, and the quality of your labels. Here are evidence-based heuristics to guide collection efforts before you commit significant resources.

🕸 Web Scraping & APIs

Web scraping enables collection of data at a scale and recency that no public dataset can match, but it comes with technical, legal, and ethical complexity. Understanding how to scrape responsibly is as important as knowing how to scrape effectively.

Minimal Python Scraper Outline

import requests
from bs4 import BeautifulSoup
import time
import hashlib
from urllib.robotparser import RobotFileParser

# ---- Check robots.txt ----
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

BASE_URL = "https://example.com/articles"
DELAY    = 2        # seconds between requests
seen     = set()    # URL deduplication

def scrape_page(url):
    if not rp.can_fetch("*", url):
        print(f"Blocked by robots.txt: {url}")
        return None

    resp = requests.get(url, headers={"User-Agent": "MyResearchBot/1.0"}, timeout=10)
    resp.raise_for_status()

    content_hash = hashlib.md5(resp.content).hexdigest()
    if content_hash in seen:
        return None          # exact duplicate
    seen.add(content_hash)

    soup = BeautifulSoup(resp.text, "html.parser")
    return {
        "url":   url,
        "title": soup.find("h1").get_text(strip=True) if soup.find("h1") else "",
        "body":  soup.find("article").get_text(separator=" ", strip=True) if soup.find("article") else "",
    }

# ---- Paginated crawl ----
results = []
for page_num in range(1, 20):
    url  = f"{BASE_URL}?page={page_num}"
    data = scrape_page(url)
    if data:
        results.append(data)
    time.sleep(DELAY)

# For large-scale scraping, use Scrapy + Redis for distributed crawling
# pip install scrapy scrapy-redis

🧪 Synthetic Data Generation

When real data is scarce, sensitive (medical records, financial fraud), expensive to label, or contains privacy constraints, synthetic data bridges the gap. The key question is always whether the synthetic distribution is close enough to the real one to be useful for training.

⚖ Data Licensing & Ethics

The legal and ethical landscape around training data has shifted dramatically. High-profile lawsuits (Getty Images v. Stability AI, The New York Times v. OpenAI) and incoming EU AI Act requirements mean practitioners must understand licensing before they collect, not after.

Bias in Data Sourcing

Bias enters ML systems long before modelling begins. Common sourcing biases include:

• Selection bias — datasets reflect who had access to the data-generating system (e.g., medical datasets skew toward patients who sought care)
• Historical bias — datasets encoding past discrimination (hiring data, criminal justice records) perpetuate it
• Measurement bias — different quality sensors or annotation standards across demographic groups
• Label bias — annotators bring their own cultural assumptions; use multiple annotators and measure inter-annotator agreement (Cohen's kappa)
• Representation bias — CommonCrawl-derived datasets over-represent English and Western perspectives; under-represent Global South populations
• Provenance documentation — use Datasheets for Datasets (Gebru et al.) or Data Cards to record collection methodology, known limitations, and intended uses