Module Nproc.Full

module Full: sig .. end

type ('a, 'b, 'c) t 

Type of a process pool. The type parameters correspond to the following:

• 'serv_request: type of the requests from worker to master,
• 'serv_response: type of the responses to the requests,
• 'env: type of the environment data passed just once to each worker process.

val create : ?init:(Nproc.worker_info -> unit) ->
       int -> ('a -> 'b Lwt.t) -> 'c -> ('a, 'b, 'c) t * unit Lwt.t

Create a process pool. create nproc service env returns (ppool, lwt) where ppool is pool of nproc processes and lwt is a lightweight thread that finishes when the pool is closed.

service is a service which is run asynchronously by the master process and can be called synchronously by the workers.

env is arbitrary environment data, typically large, that is passed to the workers just once during their initialization.

val close : ('a, 'b, 'c) t -> unit Lwt.t

Close a process pool. It waits for all submitted tasks to finish.

val terminate : ('a, 'b, 'c) t -> unit

Terminate the processes of a pool without waiting for the pending tasks to complete.

val submit : ('a, 'b, 'c) t ->
       f:(('a -> 'b) -> 'c -> 'd -> 'e) -> 'd -> 'e option Lwt.t

Submit a task. submit ppool ~f x passes f and x to one of the worker processes, which computes f service env x and passes the result back to the master process, i.e. to the calling process running the Lwt event loop.

The current implementation uses the Marshal module to serialize and deserialize f, its input and its output.

val iter_stream : ?granularity:int ->
       ?init:(Nproc.worker_info -> unit) ->
       nproc:int ->
       serv:('a -> 'b Lwt.t) ->
       env:'c ->
       f:(('a -> 'b) -> 'c -> 'd -> 'e) ->
       g:('e option -> unit) -> 'd Stream.t -> unit

Iterate over a stream using a pool of nproc worker processes running in parallel.

iter_stream runs the Lwt event loop internally. It is intended for programs that do not use Lwt otherwise.

Function f runs in the worker processes. It is applied to elements of the stream that it receives from the master process. Function g is applied to the result of f in the master process.

The current implementation uses the Marshal module to serialize and deserialize f, its inputs (stream elements) and its outputs. f is serialized as many times as there are elements in the stream. If f relies on a large immutable data structure, it should be putting into env in order to avoid costly and repetitive serialization of that data.