burla.remote_parallel_map

Run an arbitrary python function on many computers at the same time.

remote_parallel_map(
  function_,
  inputs,
  parallelism=-1,
  func_cpu=1,
  func_ram=1,
  func_gpu=None,
  verbose=False,
  dockerfile=None,
  packages=None,
  api_key=None,
)

Run provided function_ on each item in inputs at the same time, in the cloud, each on a separate CPU, up to 4000 CPUs. If more than 4000 inputs are provided, inputs are queued and processed sequentially on each worker.

By default, the local python environment is cloned on all remote machines. To prevent this pass packages=[].

Parameters
Name	Description
function_	Callable Python function. Must have single input argument, eg: function_(inputs[0]) does not raise an exception.
inputs	Iterable[Any] Iterable of elements passable to function_.
parallelism	int (Optional) Target number of function_ instances running in parallel. Set to -1 for maximum: min(4000, len(inputs)).
func_cpu	int (Optional) Number of CPU's available to every instance of function_. The maximum possible value is 96.
func_ram	int (Optional) Amount of RAM (GB) available to every instance of function_. The maximum possible value is 360.
func_gpu	Literal["A100"] (Optional) GPU available to every instance of function_. If func_gpu is not None maximum parallelism is reduced to 100.
verbose	bool (Optional) Set to False to prevent status indicator from being displayed.
dockerfile	Union[str, pathlib.Path] (Optional) Path to dockerfile. If present, and never seen before, a docker image is built, and all instances of function_ are run in this image. If present, and seen before, the previously built image is used to maintain low latency. If None, Burla reverts to default behaivior, which is to clone the local python environment on all remote machines.
packages	Iterable[str] (Optional) Iterable containing names of pipy packages to install in environments where function_ is run. This argument will override the default behaivior, which is to clone the local python environment on all remote machines.
api_key	str (Optional) API key, for use in deployment environments where burla login cannot be run.

Returns
Type	Description
List	List of objects returned by function_. This list is not ordered.

---

Questions?
Schedule a call with us, or email us. We're always happy to talk.