General Use Cases Make your Python script 10x faster. A guide for turning slow Python loops into fast parallel cloud jobs.
If your script does the same work one item at a time, it is often much faster to run many items at once in the cloud.
This guide shows a simple way to convert a normal Python loop into a parallel Burla job.
Before you start
Make sure you have already:
installed Burla: pip install burla
connected your machine: burla login
started your cluster in the Burla dashboard
Step 1: Find the part of your script that repeats
A good first target is work that looks like this:
Copy from time import sleep
items = [ 1 , 2 , 3 ]
results = []
for item in items :
sleep ( 1 )
results . append ( item * item )
print ( results ) This loop is slow because it waits for each item to finish before starting the next one.
Step 2: Move repeated work into one function
Put the per-item work into a function that takes one input and returns one output.
Step 3: Run the function in parallel with Burla
Pass your function and a list of inputs to remote_parallel_map.
Burla runs one function call per input on separate cloud machines.
Step 4: Check that your function is safe to parallelize
Your function should:
use only its input to do work
avoid shared local state between calls
This is safe:
This is not safe:
Step 5: Measure your speedup
Use a small timer so you can compare local and Burla runs.
For independent work items, Burla is often much faster than a local loop.
What to do next
Once this pattern works, apply the same function-plus-input-list shape to your real script.
If your script works with many files, continue with Process thousands of files quickly.