parsley
Abbreviation for “Parallel Script Library” parsley It lets you take computing jobs and cut up them throughout a number of methods utilizing roughly the identical syntax as present Python. Pool objects. It additionally lets you be part of completely different computing duties into multi-step workflows, which could be executed in parallel, in sequence, or utilizing map/scale back operations.
Parsl lets you run native Python purposes, but in addition run some other exterior software utilizing shell instructions. Your Python code is written like regular Python code, aside from a particular perform decorator that marks the entry level to your work. The job submission system additionally provides you fine-grained management over how issues are executed on targets; for instance, the variety of cores per employee, how a lot reminiscence per employee, CPU affinity checks, how usually to ballot for timeouts, and so forth. .
An important function that Parsl provides is a set of pre-built templates for submitting work to a wide range of high-end computing sources. This not solely consists of primary merchandise like AWS or Kubernetes clusters, however reasonably supercomputing sources (assuming you’ve entry) like Blue Waters, ASPIRE 1, Frontera, and so forth. (Parsl was co-developed with the assistance of lots of the establishments that constructed such {hardware}.)
