utils Module

Define various tools used by the pipe engine.

pipelet.utils.close_logger(logger)

Close a logger

class pipelet.utils.crc32(s='')

Hash object using the crc32 algorithm.

base32_digest()

Return a base32 digest of the checksum

Checksum is computed using the crc32 algorithm.

>>> print crc32('test').base32_digest()
3B7X4DA

update(s)

Update this hash object’s state with the provided string.

pipelet.utils.create_pipe(pipename, prefix=[])

Create an empty pipe environment including main, defaulf segment, prefix and log directories.

This function is called from the script pipeutils.

pipelet.utils.flatten(seq)

this generator flattens nested containers such as

>>> l=( ('John', 'Hunter'), (1,23), [[[[42,(5,23)]]]])
>>> for i in flatten(l): print i,
John Hunter 1 23 42 5 23

By: Composite of Holger Krekel and Luther Blissett From: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/121294 and Recipe 1.12 in cookbook

pipelet.utils.get_hashkey(s, starthash)

Return a unique key which identify code string using cheksum algorithm.

Comment lines and blanks are not taken into account in code string.

>>> print(get_hashkey("a = 5", 0).base32_digest())
ZNBR63A

pipelet.utils.get_log_file(pipe, name)

Return log file name

Current date time is prepend to the log file name.

pipelet.utils.init_logger(name, filename, level=10)

Initialize a logger.

pipelet.utils.is_code_file(s)

Check wether the file is a python code file based an the file extension.

pipelet.utils.make_dict(l)

Convert a list into a dict keeping duplicated entries.

>>> d = make_dict(zip(['a']*10+['b'], range(10)+[1]))
>>> print d
{'a': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 'b': 1}

pipelet.utils.parse_disk(pipedir)

Return list of pipe instance starting from pipedir.

pipelet.utils.rebuild_db_from_disk(pipedir, sqlfile=None)

Rebuild db from disk.

pipelet.utils.reduced_code_formatting(s, unsafe=False)

Return a code string with some part of the non syntactic information removed.

if unsafe is True remove some syntactic information like spaces in string literals.

>>> print reduced_code_formatting('''def pouet():
... #lets try it
... a,  b = foo ( bar )
... ''')
def pouet():
a,  b = foo ( bar )

pipelet.utils.spl(s)

pipelet.utils.str_date()

Return a string representation of the date time now %y/%m/%d %H:%M:%s

pipelet.utils.str_file(obj)

Object string representation with special caracters replaced.

>>> str_file((1,))
'1_'

pipelet.utils.str_web(a)

String representation without the quotes and spaces that disturbs the web interface.

>>> print(str_web(['a', ' b']))
[a,b]

pipelet.utils.upgrade_webconfig()

Upgrade pipeweb config with new convention.

Use .pipelet directory for all config/log files

pipelet.db_utils.add_tag(db_file, segid, tag)

Add new tag to the database

Check that the tag do not exist yet for this id

pipelet.db_utils.change_root(sql_file, old_prefix, new_prefix)

Update the data root path in the database.

pipelet.db_utils.del_tag(db_file, tag)

Delete tag from the database

pipelet.db_utils.deltask(db_file, lst_task, report_only=False)

Delete a tasks instances.

Delete all products directories of a tasks instance.

pipelet.db_utils.get_last(db_file)

Return the last computed segment in a given database.

Return a seg-id

pipelet.db_utils.get_lst_date(db_file)

Return the list of existing dates

Date strings are picked from queued_on field corresponding to the first task of each segment.

pipelet.db_utils.get_lst_seg(db_file)

Return the list of segment name

pipelet.db_utils.get_lst_tag(db_file)

Return the list of existing tags

Tags are ; separated in the db.

pipelet.db_utils.old_to_new(old_file, new_file)

Convert an old database to the new database format.