Last updated
Last updated
burla.remote_parallel_map
Run any python function on many remote computers at the same time.
See our for a more user friendly description of how to use remote_parallel_map.
Run provided function_
on each item in inputs
at the same time, each on a separate CPU. If more inputs are provided than there are available CPU's, they are queued and processed sequentially on each worker. remote_parallel_map
can reliably queue millions of inputs.
While running:
If the provided function_
raises an exception, the exception, including stack trace, is re-raised on the client machine in a way that looks like it was running locally.
Your print statements (anything written to stdout/stderr) are streamed back to your local machine, appearing like they would have if running the same code locally.
When finished remote_parallel_map
returns a list of values returned by each function_
call.
Parameters
Name
Description
function_
Callable
Python function. Must have single input argument, eg: function_(inputs[0])
does not raise an exception.
inputs
List[Any]
Iterable of elements passable to function_
.
func_cpu
int
(Optional) Number of CPU's made available to each running instance of function_
. Max possible value is determined by your cluster machine type.
func_ram
int
(Optional) Amount of RAM (GB) made available to each running instance of function_
. Max possible value is determined by your cluster machine type.
background
bool
(Optional) remote_parallel_map
returns as soon as your inputs and function have been uploaded. The job then continues to run independently in the background until done.
spinner
bool
(Optional) Set to False
to hide the status indicator/spinner.
generator
bool
(Optional) Set to True
to return a Generator
instead of a List
. The generator will yield outputs as they are produced, instead of all at once.
max_parallelism
int
(Optional) Maximum number of function_
instances allowed to be running at the same time.
Returns
Type
Description
List
or Generator
List of objects returned by function_
in no particular order. If Generator=True
, returns generator yielding objects returned by function_
in the order they are produced.
Questions? , or email jake@burla.dev. We're always happy to talk.