Burla
GitHubDiscordBook a Demo
  • Welcome
  • Overview
  • API-Reference
  • CLI-Reference
  • Installation: Self-Hosted
  • Installation: Fully-Managed
  • Examples
  • About
  • Brick
Powered by GitBook
On this page
  • API-Reference
  • burla.remote_parallel_map
PreviousOverviewNextCLI-Reference

Last updated 4 days ago

API-Reference

burla.remote_parallel_map

Run any python function on many remote computers at the same time. See our for a more user friendly description of how to use remote_parallel_map.

remote_parallel_map(
  function_,
  inputs,
  func_cpu=1,
  func_ram=4,
  background=False,
  spinner=True,
  generator=False,
  max_parallelism=None,
)

Run provided function_ on each item in inputs at the same time, each on a separate CPU. If more inputs are provided than there are available CPU's, they are queued and processed sequentially on each worker. remote_parallel_map can reliably queue millions of inputs.

While running:

  • If the provided function_ raises an exception, the exception, including stack trace, is re-raised on the client machine in a way that looks like it was running locally.

  • Your print statements (anything written to stdout/stderr) are streamed back to your local machine, appearing like they would have if running the same code locally.

When finished remote_parallel_map returns a list of values returned by each function_ call.

Parameters

Name

Description

function_

Callable

Python function. Must have single input argument, eg: function_(inputs[0]) does not raise an exception.

inputs

List[Any]

Iterable of elements passable to function_.

func_cpu

int

(Optional) Number of CPU's made available to each running instance of function_. Max possible value is determined by your cluster machine type.

func_ram

int

(Optional) Amount of RAM (GB) made available to each running instance of function_. Max possible value is determined by your cluster machine type.

background

bool (Optional) remote_parallel_map returns as soon as your inputs and function have been uploaded. The job then continues to run independently in the background until done.

spinner

bool

(Optional) Set to False to hide the status indicator/spinner.

generator

bool (Optional) Set to True to return a Generator instead of a List. The generator will yield outputs as they are produced, instead of all at once.

max_parallelism

int

(Optional) Maximum number of function_ instances allowed to be running at the same time.

Returns

Type

Description

List or Generator

List of objects returned by function_ in no particular order. If Generator=True, returns generator yielding objects returned by function_ in the order they are produced.


Questions? , or email jake@burla.dev. We're always happy to talk.

overview
Schedule a call with us