Examples for file-parallel ETL, image processing, external service limits, and operational backfills.
ETL 10K S3 files to Postgres
Transform gzipped JSON files with idempotent inserts while protecting Postgres with max_parallelism.
max_parallelism
Resize an image corpus
Chunk image keys, resize with Pillow, write outputs to S3, and stream a manifest.
Run a 2M-user API backfill
Backfill user profiles while keeping provider limits explicit in chunk size, sleeps, retries, and max_parallelism.
Scrape 1M web pages
Scrape static HTML with polite pacing, retries, error rows, streaming output, and a global concurrency cap.
Run 1B option simulations
Split the run into independent chunks, then reduce sums and squared sums locally.
Last updated 2 months ago
