Production Data Workflows

ETL 10K S3 files to Postgres

Transform gzipped JSON files with idempotent inserts while protecting Postgres with max_parallelism.

Resize an image corpus

Chunk image keys, resize with Pillow, write outputs to S3, and stream a manifest.

Run a 2M-user API backfill

Backfill user profiles while keeping provider limits explicit in chunk size, sleeps, retries, and max_parallelism.

Scrape 1M web pages

Scrape static HTML with polite pacing, retries, error rows, streaming output, and a global concurrency cap.

Run 1B option simulations

Split the run into independent chunks, then reduce sums and squared sums locally.