# Large-Scale Data Processing

Large-scale data processing examples for files, corpora, table scans, and ordinary Python data code.

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td><strong>Query 2.4TB Parquet in 76s</strong></td><td>Run a DuckDB query over 1,000 Parquet files on 10,000 CPUs and combine the results.</td><td><a href="/pages/98L1G4g9GrVTTy641yZv">/pages/98L1G4g9GrVTTy641yZv</a></td><td><a href="/files/oFIAsVD6hNnlxdJ44ywi">/files/oFIAsVD6hNnlxdJ44ywi</a></td></tr><tr><td><strong>Distill 571M Amazon reviews</strong></td><td>Read 275 GB of JSONL with HTTP Range requests, deterministic scoring, heap reducers, and a second Unhinged Mode pass.</td><td><a href="/pages/BWdAJuSYdRcWGvdRIoRu">/pages/BWdAJuSYdRcWGvdRIoRu</a></td><td><a href="/files/qRrthFNz7eVhoXp92NW9">/files/qRrthFNz7eVhoXp92NW9</a></td></tr><tr><td><strong>Scan 2.76B NYC taxi trips</strong></td><td>Scan taxi and FHV month files, keep pickup counts small, and classify zones after the full scan.</td><td><a href="/pages/pDdLW6LmaxrXMOGhY2Eh">/pages/pDdLW6LmaxrXMOGhY2Eh</a></td><td><a href="/files/ogQ4STtDE5DOCK92IkHH">/files/ogQ4STtDE5DOCK92IkHH</a></td></tr><tr><td><strong>Map geotagged Flickr photos</strong></td><td>Reverse-geocode public photos and build country signatures from user-written tags.</td><td><a href="/pages/eZdCoQ5oWGW1QiFPYBzl">/pages/eZdCoQ5oWGW1QiFPYBzl</a></td><td><a href="/files/4FhDbemo1UXKRMRN2E5j">/files/4FhDbemo1UXKRMRN2E5j</a></td></tr><tr><td><strong>Summarize 1M GitHub READMEs</strong></td><td>Shard README Parquet by deterministic ids, score with inspectable rules, and reduce category stats.</td><td><a href="/pages/KVTCMNqnZhZJzSFtu5mu">/pages/KVTCMNqnZhZJzSFtu5mu</a></td><td><a href="/files/xdBdtqWQyGYdEiQuHIk5">/files/xdBdtqWQyGYdEiQuHIk5</a></td></tr><tr><td><strong>Audit 5,000 Parquet files</strong></td><td>Compute one QA row per object so bad shards are easy to triage.</td><td><a href="/pages/FCMAhAAfS1i3ncq5HuHQ">/pages/FCMAhAAfS1i3ncq5HuHQ</a></td><td><a href="/files/T5K2f8GlxT1eGkBNDF8o">/files/T5K2f8GlxT1eGkBNDF8o</a></td></tr><tr><td><strong>Parallelize pandas apply</strong></td><td>Partition by user id, keep the row function intact, and write enriched Parquet chunks.</td><td><a href="/pages/fFrT39PFVRNoXkzIniJD">/pages/fFrT39PFVRNoXkzIniJD</a></td><td><a href="/files/S6jlzgy6Ye0nFakABUby">/files/S6jlzgy6Ye0nFakABUby</a></td></tr></tbody></table>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.burla.dev/all-examples/data-processing-examples.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.