> For the complete documentation index, see [llms.txt](https://docs.burla.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.burla.dev/all-examples/data-processing-examples.md).

# Large-Scale Data Processing

Large-scale data processing examples for files, corpora, table scans, and ordinary Python data code.

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td><strong>Query 2.4TB Parquet in 76s</strong></td><td>Run a DuckDB query over 1,000 Parquet files on 10,000 CPUs and combine the results.</td><td><a href="/pages/98L1G4g9GrVTTy641yZv">/pages/98L1G4g9GrVTTy641yZv</a></td><td><a href="/files/oFIAsVD6hNnlxdJ44ywi">/files/oFIAsVD6hNnlxdJ44ywi</a></td></tr><tr><td><strong>Distill 571M Amazon reviews</strong></td><td>Read 275 GB of JSONL with HTTP Range requests, deterministic scoring, heap reducers, and a second Unhinged Mode pass.</td><td><a href="/pages/BWdAJuSYdRcWGvdRIoRu">/pages/BWdAJuSYdRcWGvdRIoRu</a></td><td><a href="/files/qRrthFNz7eVhoXp92NW9">/files/qRrthFNz7eVhoXp92NW9</a></td></tr><tr><td><strong>Scan 2.76B NYC taxi trips</strong></td><td>Scan taxi and FHV month files, keep pickup counts small, and classify zones after the full scan.</td><td><a href="/pages/pDdLW6LmaxrXMOGhY2Eh">/pages/pDdLW6LmaxrXMOGhY2Eh</a></td><td><a href="/files/ogQ4STtDE5DOCK92IkHH">/files/ogQ4STtDE5DOCK92IkHH</a></td></tr><tr><td><strong>Map geotagged Flickr photos</strong></td><td>Reverse-geocode public photos and build country signatures from user-written tags.</td><td><a href="/pages/eZdCoQ5oWGW1QiFPYBzl">/pages/eZdCoQ5oWGW1QiFPYBzl</a></td><td><a href="/files/4FhDbemo1UXKRMRN2E5j">/files/4FhDbemo1UXKRMRN2E5j</a></td></tr><tr><td><strong>Summarize 1M GitHub READMEs</strong></td><td>Shard README Parquet by deterministic ids, score with inspectable rules, and reduce category stats.</td><td><a href="/pages/KVTCMNqnZhZJzSFtu5mu">/pages/KVTCMNqnZhZJzSFtu5mu</a></td><td><a href="/files/xdBdtqWQyGYdEiQuHIk5">/files/xdBdtqWQyGYdEiQuHIk5</a></td></tr><tr><td><strong>Audit 5,000 Parquet files</strong></td><td>Compute one QA row per object so bad shards are easy to triage.</td><td><a href="/pages/FCMAhAAfS1i3ncq5HuHQ">/pages/FCMAhAAfS1i3ncq5HuHQ</a></td><td><a href="/files/T5K2f8GlxT1eGkBNDF8o">/files/T5K2f8GlxT1eGkBNDF8o</a></td></tr><tr><td><strong>Parallelize pandas apply</strong></td><td>Partition by user id, keep the row function intact, and write enriched Parquet chunks.</td><td><a href="/pages/fFrT39PFVRNoXkzIniJD">/pages/fFrT39PFVRNoXkzIniJD</a></td><td><a href="/files/S6jlzgy6Ye0nFakABUby">/files/S6jlzgy6Ye0nFakABUby</a></td></tr></tbody></table>