Large-Scale Data Processing

Query 2.4TB Parquet in 76s

Run a DuckDB query over 1,000 Parquet files on 10,000 CPUs and combine the results.

Distill 571M Amazon reviews

Read 275 GB of JSONL with HTTP Range requests, deterministic scoring, heap reducers, and a second Unhinged Mode pass.

Scan 2.76B NYC taxi trips

Scan taxi and FHV month files, keep pickup counts small, and classify zones after the full scan.

Map geotagged Flickr photos

Reverse-geocode public photos and build country signatures from user-written tags.

Summarize 1M GitHub READMEs

Shard README Parquet by deterministic ids, score with inspectable rules, and reduce category stats.

Audit 5,000 Parquet files

Compute one QA row per object so bad shards are easy to triage.

Parallelize pandas apply

Partition by user id, keep the row function intact, and write enriched Parquet chunks.