scheduler Module

class pipelet.scheduler.NullHandler(level=0)

Bases: logging.Handler

Extension of the logging handler class.

emit(record)

Avoid warnings.

class pipelet.scheduler.Scheduler(pipe)

Feed tasks to an execution queue.

A scheduler is made of a execution queue which is feeded with tasks. The scheduler coordinates the execution of the pipeline, building tasks for each segment and product.

abort_scheduling(t)

Abort the processing.

check_in()

Register a worker.

A worker call this method to check in

check_out()

Unregister a worker.

A worker call this method to check out

get_pipe()

Return the pipe object bound to the scheduler

get_task()

Return the next task of the queue.

push_next_seg(seg)

Push the segment task list to the queue.

Segment repository is created at this stage, where a copy of the code, visu and argument files are saved. The parent segment outputs are combined to form the task list.

put_task(t)

Add task to the queue.

There is one task for each input which are the products of the parent segment.

requeue(t)

Requeue a running task.

run()

Start the scheduler.

Start a tracker and fill the task queue with the first segment task.

store_meta_seg(t)

Store meta information for segment.

Store meta info at the segment level on disk (to rebuild db).

task_done(t)

Tell the tracker when a task has been succesfully completed.

The product list is updated with the task outputs. If the completed task is the last of the segment task list, next segment is pushed to the queue.

task_failed(t)

Tell the tracker when a task has failed.

If the failed task is the last of the segment task list, next segment is pushed to the queue.