tracker Module

class pipelet.tracker.SqliteTracker(pipe)

Bases: pipelet.tracker.Tracker, threading.Thread

Fast sqlite-based tracker.

Tasks are saved in a sqlite3 data base. Format is: TODO

add_queued(t)

Add a list of task to the queue.

Add a task to the data base, and return it with its new id and status.

clean(seg)

Remove queued and running tasks from data base.

This is called only at the init of the scheduler.

close()

Close database connexion.

flush()

Empty the sql requests queue in one go.

As most of the asynchronous requests are updates, and the change are commited to the sqlfile after completion, calling flush too often with few requests may seriously hinder the performances.

get_list_status(seg, status)

get_list_status_old(seg, status)

Return the task list for a given segment and status.

get_status(t)

Return task status.

gettest(seg, status)

gettest2(seg, status)

run()

Start the tracker.

Use the shared connection to the sqlite data base, and regularly call flush to empty the requests queue, and commit the result to the db.

segments_registration()

Register all segments in the db.

The registration includes connectivity between segments.

stop()

Stop the tracker.

Empty the request queue by adding None.

update_param(t)

Update the segment with new tag.

update_status(t, status)

Update the task with new status.

Update the task status in data base. If not done task_output is set to None.

class pipelet.tracker.Tracker

Track the submission of objects through files in the segment directory.

The numerous file access may result in a serious performance bottleneck. Nevertheless, it is the only work tracker that may be compatible with the ancestral version of pipelet.

add_queued(t)

Add a task to the queue.

Add a task to the data base, and return it with its new id and status.

get_done(seg)

Return the list of done tasks.

get_failed(seg)

Return the list of failed tasks.

get_list_status(seg, status)

Return the task list for a given segment and status.

get_queued(seg)

Return the list of queued tasks.

get_running(seg)

Return the list of running tasks.

get_status(t)

Return task status.

update_status(t, status)

Update the task with new status.

Update the task status. If not done task_output is set to None.