adios_db.util package

Assorted utilities

A few right here:

class adios_db.util.BufferedIterator(iterable)

Bases: object

An iterator that can have stuff put back

Give it an iterable, and it will create an iterator that iterates, but you can push things back on to be returned in future next calls.

In the common case, this would be used to put back an item, but you could use it more generally.

See the test code in test_utilities

push(item)
adios_db.util.sigfigs(x, sig=6)
adios_db.util.strip(value)

Submodules

adios_db.util.db_connection module

adios_db.util.folder_collection module

class adios_db.util.folder_collection.FolderCollection(folder)

Bases: object

Ducktyped class that acts like a MongoDB oil collection, but acts upon a file folder instead.

The file folder is assumed to be a base folder, and we will assume the filesystem structure is that of the noaa-oil-data project. As such, the oil records are saved in a path like:

f’{folder}/oil/{oil_id_prefix}/{oil_id}.json’

find_one_and_replace(filter, replacement, upsert=True)

Insert our json record into the filesystem

For the Exxon records, there is no source field that we can use to generate an ID. Each record does have a unique name however.

So we generate the IDs in this class using an automated method. In the initial case, the IDs start at 0 and increment with each subsequent record.

This poses a problem when re-running the importer with new records in the set, because unless the records are processed in exactly the same order, it is likely a bunch of records will be saved with different IDs than they previously had.

The solution is to query the filesystem (typically a git repo), and build an index associating a record’s name with an ID. This will tell us if a particular named oil has had a previous ID.

  • if the named oil has a previous ID, then use it

  • otherwise, generate the next ID and use it.

replace_one(filter, replacement, upsert=True)

adios_db.util.many_many module

ManyMany

class for managing a many to many relationship in a pair of dicts

The two mappings are “left” and “right”

The left one is originally populated by the initializer.

After that they are equivalent – when you add something to one the other is updated.

Currently there is no way to remove anything

YOu can access copies of the mappings with:

ManyMany.left and ManyMany.right

They are copies, so that mutating them won’t break the internal data.

class adios_db.util.many_many.ManyMany(initial_data=None)

Bases: object

add_to_left(key, value)

add a new value to the left dict

Parameters:

  • key – the key the value is to be added to

  • value – the value to be added

If the key is already there, the value will be added to the corresponding set

A new key and set will be created if it is not already there.

add_to_right(key, value)

add a new value to the right dict

Parameters:

  • key – the key the value is to be added to

  • value – the value to be added

If the key is already there, the value will be added to the corresponding set

A new key and set will be created if it is not already there.

property left

A copy of the left dict

It’s a copy, so it won’t change the internal ones if mutated

property right

A copy of the right dict

It’s a copy, so it won’t change the internal ones if mutated

adios_db.util.settings module

some common methods for getting some application settings for the oil database

adios_db.util.settings.convert_str_to_type_value(str_in)

I really don’t see why the config parser can’t just do this, at least as an option. It would obviate the need for a utility module like this.

adios_db.util.settings.default_settings()
adios_db.util.settings.file_settings(config_file, section='app:adios_db')

adios_db.util.term module

Here is where we will keep a few things that will make printing stuff out on a terminal a bit nicer.

class adios_db.util.term.TermColor

Bases: object

classmethod change(str_in, color)

Surround our string with terminal codes that will change its color and then change it back

color_codes = {'blue': '\x1b[94m', 'bold': '\x1b[1m', 'cyan': '\x1b[96m', 'darkcyan': '\x1b[36m', 'end': '\x1b[0m', 'green': '\x1b[92m', 'purple': '\x1b[95m', 'red': '\x1b[91m', 'underline': '\x1b[4m', 'yellow': '\x1b[93m'}