populse_db package

class populse_db.Database(database_url, timeout=None, create=False, echo_sql=None)

Bases: object

Entrypoint of populse_db for creating DatabaseSession object given an URL that identify the underlying database engine.

Creating a Database doesn’t connect to the database engine. It just parses the URL (ensuring that it corresponds to a valid engine) and stores it. The Database is a Python context manager that must be using a with statement that connects to the database and creates a database specific object that implements the API of DatabaseSession.

Example:

from populse_db import Database

db = Database('sqlite:///tmp/populse_db.sqlite')
with db as dbs:
    dbs.add_collection('my_collection', primary_key='id')
    dbs['my_collection']['my_document'] = {
        'a key': 'a value',
        'another key': 'another value'
    }

Database modification within a with statement is done in a transaction. If an exception occurs before the end of the with, a rollback is done and the database is not modified. By default, this transaction allows to read the database in parallel. The first database modification will acquire an exclusive access to the database. It is possible to request an exclusive session (preventing any other access to the database by other processes or threads) by using the exclusive property:

from populse_db import Database

db = Database('sqlite:///tmp/populse_db.sqlite')
with db.exclusive as dbs:
    # All other access to the database are blocked
    # until the end of this "with" statement.

:any:Database is a reusable context manager. It means that it is allowed to use it in several consecutive with:

from populse_db import Database

db = Database('sqlite:///tmp/populse_db.sqlite')
with db as dbs:
    dbs.add_collection('my collection')

with db as dbs:
    dbs['my collection']['my document'] = {}

Database is a reentrant context manager. It is possible to use a with statement within another with that already use the same Database. In that case, the same context is returned and the transaction is not ended in any of the inner context, it is terminated when the outer context exits:

from populse_db import Database

db = Database('sqlite:///tmp/populse_db.sqlite')
with db as dbs1:
    # Connection to the database and transaction starts here
    dbs.add_collection('my collection')
    with db as dbs2:
        # here: dbs1 is dbs2 == True
        # No new connection to the database is done
        dbs['my collection']['my document'] = {}
    # After the inner with, connection to the database is not
    # closed and transaction is ongoing (no commit nor rollback
    # is done)
# After the end of the outer with, transaction is terminated and
# connection to the database is closed.

On a multithreaded application, each thread using Database context manager gets its own DatabaseSession instance with its own database connection.

methods:

Creates a Database instance.

Parameters:

database_url –

URL defining database engine and its parameters. The engine URL must have the following pattern: dialect://user:password@host/dbname[?key=value..]. To date dialect can only be sqlite but postgresql is planned.

Examples:

  • /somewhere/a_file.popdb

  • sqlite:///foo.db

  • postgresql://scott:tiger@localhost/test

begin_session(exclusive, create=None)
end_session(rollback)
property exclusive
session(exclusive=False, create=None)

Subpackages

Submodules

populse_db.database module

class populse_db.database.DatabaseCollection(session, name)

Bases: object

add(document, replace=False)
add_field(name, field_type, description=None, index=False, bad_json=False)

Adds a field to the database

Parameters:

  • collection – Field collection (str, must be existing)

  • name – Field name (str, must not be existing)

  • field_type – Field type, in (‘string’, ‘int’, ‘float’, ‘boolean’, ‘date’, ‘datetime’, ‘time’, ‘json’, ‘list_string’, ‘list_int’, ‘list_float’, ‘list_boolean’, ‘list_date’, ‘list_datetime’, ‘list_time’, ‘list_json’)

  • description – Field description (str or None) => None by default

  • index – Bool to know if indexing must be done => False by default

Raises:

ValueError –

  • If the collection does not exist

  • If the field already exists

  • If the field name is invalid

  • If the field type is invalid

  • If the field description is invalid

delete(filter)

Delete documents corresponding to the given filter

Parameters:

filter_query – Filter query (str)

document(document_id, fields=None, as_list=False)
document_id(document_id)
documents(fields=None, as_list=False)
documents_ids()
filter(filter, fields=None, as_list=False)

Iterates over the collection documents selected by filter_query

Each item yield is a row of the collection table returned

filter_query can be the result of self.filter_query() or a string containing a filter (in this case self.fliter_query() is called to get the actual query)

Parameters:

  • filter_query –

    Filter query (str)

    • A filter row must be written this way: {<field>} <operator> “<value>”

    • The operator must be in (‘==’, ‘!=’, ‘<=’, ‘>=’, ‘<’, ‘>’, ‘IN’, ‘ILIKE’, ‘LIKE’)

    • The filter rows can be linked with ‘ AND ‘ or ‘ OR ‘

    • Example: “((({BandWidth} == “50000”)) AND (({FileName} LIKE “%G1%”)))”

  • fields – List of fields to retrieve in the document

  • as_list – If True, document values are returned in a list using fields order

has_document(document_id)
parse_filter(filter)
remove_field(name)

Removes a field in the collection

Parameters:

  • collection – Field collection (str, must be existing)

  • field – Field name (str, must be existing))

Raises:

ValueError –

  • If the collection does not exist

  • If the field does not exist

set_settings(settings)
settings()
update_document(document_id, partial_document)
update_settings(**kwargs)
class populse_db.database.DatabaseSession

Bases: object

Base class

methods:

__getitem__(collection_name)

Return a collection object given its name.

add_collection(name, primary_key='primary_key')

Adds a collection

Parameters:

  • name – New collection name (str, must not be existing)

  • primary_key – New collection primary_key column (str) => “index” by default

Raises:

ValueError –

  • If the collection is already existing

  • If the collection name is invalid

  • If the primary_key is invalid

add_document(collection, document)

Deprecated since version 3.0: Use del db_session[collection].add(document) instead. See DatabaseCollection.add().

add_field(collection, name, field_type, description=None, index=False)

Deprecated since version 3.0: Use db_session[collection].add_field(...) instead. See DatabaseCollection.add_field().

collections()

Iterates over collections

Returns:

generator

commit()
default_primary_key = 'primary_key'
execute(*args, **kwargs)
filter_documents(collection, filter_query, fields=None, as_list=False)

Deprecated since version 3.0: Use del db_session[collection].filter(...) instead. See DatabaseCollection.filter().

get_collection(name)

Deprecated since version 3.0: Use db_session[name] instead

get_collections()

Deprecated since version 3.0: Use collections() instead

get_collections_names()

Deprecated since version 3.0: Use (i.name for i in db_session) instead

get_document(collection, document_id, fields=None, as_list=False)

Deprecated since version 3.0: Use db_session[collection].document(...) instead. See DatabaseCollection.document().

get_documents(collection, fields=None, as_list=False, document_ids=None)

Deprecated since version 3.0: Use db_session[collection].documents_ids(...) instead. See DatabaseCollection.documents_ids().

get_documents_ids(collection)

Deprecated since version 3.0: Use db_session[collection].documents_ids(...) instead. See DatabaseCollection.documents_ids().

get_field(collection, name)

Deprecated since version 3.0: Use db_session[collection].fields.get(name) instead. See DatabaseCollection.fields.

get_fields(collection)

Deprecated since version 3.0: Use db_session[collection].fields.values() instead. See DatabaseCollection.fields.

get_fields_names(collection)

Deprecated since version 3.0: Use db_session[collection].fields.keys() instead. See DatabaseCollection.fields.

has_collection(name)

Check if a collection with the given name exists.

has_document(collection, document_id)

Deprecated since version 3.0: Use db_session[collection].has_document(...) instead. See DatabaseCollection.has_document().

remove_collection(name)

Removes a collection

Parameters:

name – Collection to remove (str, must be existing)

Raises:

ValueError – If the collection does not exist

remove_document(collection, document_id)

Deprecated since version 3.0: Use del db_session[collection][document_id] instead. See DatabaseCollection.__delitem__().

remove_field(collection, field)

Deprecated since version 3.0: Use db_session[collection].remove_field(...) instead. See DatabaseCollection.remove_field().

rollback()
set_settings(category, key, value)
set_values(collection, document_id, values)

Deprecated since version 3.0: Use db_session[collection].update_document(...) instead. See DatabaseCollection.update_document().

settings(category, key, default=None)
populse_db.database.check_value_type(value, field_type)

Checks the type of a value

Parameters:

  • value – value to check

  • field_type – type that the value is supposed to have

Returns:

true if the value is None or if the value is of that type, False otherwise

populse_db.database.json_decode(value)
populse_db.database.json_dumps(value)
populse_db.database.json_encode(value)
populse_db.database.python_value_type(value)

Returns the Python type corresponding to a Python value. This type can be used in add_field(s) method. For list values, only the first item is considered to get the subtype of the list.

Examples:

  • python_value_type('a value') == str

  • python_value_type([]) == list

  • python_value_type([1, 2, 3]) == list[int]

  • python_value_type(['string', 2, {}]) == list[str]

  • python_value_type({'one': 1, 'two': 2}) == dict

populse_db.database.str_to_type(str)

Convert a string to a Python type.

Examples:

  • str_to_type('str') == str

  • str_to_type('list[str]') == list[str]

populse_db.database.type_to_sqlite(type)

Like type_to_str(type) but for internal use in SQLite column type definitions in order to avoid conversion problems due to SQlite type affinity. See https://www.sqlite.org/datatype3.html

populse_db.database.type_to_str(type)

Convert a Python type to a string.

Examples:

  • type_to_str(str) == 'str'

  • type_to_str(list[str]) == 'list[str]'

populse_db.filter module

class populse_db.filter.Field

Bases: str

class populse_db.filter.FilterToSQL(dbcollection)

Bases: Transformer

Instance of this class are passed to Lark parser when parsing a document selection filter string in order to create an object that can be used to select items from the database. FilterToSQL implements methods that are common to all engines and does not produce anything because the query objectis specific to each engine. Therefore, engine class must use a subclass of FilterToSQL that implements the following methods:

build_condition_all build_condition_literal_in_list_field build_condition_field_in_list_field build_condition_field_in_list build_condition_field_op_field build_condition_value_op_field build_condition_negation build_condition_combine_conditions

all(items)
build_condition_all()

Return a selection query that select all documents. This query is directly given to the engine and never combined with other queries.

build_condition_combine_conditions(left_condition, operator_str, right_condition)

Builds a condition that combines two conditions with an operator.

Parameters:

  • left_condition – condition object returned by one of the build_condition_*() method (except build_condition_all)

  • operator_str – string containing one of the BOOLEAN_OPERATOR defined in the grammar (in lowercase)

  • right_condition – condition object returned by one of the build_condition_*() method (except build_condition_all)

build_condition_field_in_list(field, list_value)

Builds a condition checking if a field value is a constant list value

Parameters:

  • field – field object as returned by Database.get_field

  • list_value – Python list containing literals

build_condition_field_in_list_field(field, list_field)

Builds a condition checking if a field value is in another list field value

Parameters:

  • field – field object as returned by Database.get_field

  • list_field – field object as returned by Database.get_field

build_condition_field_op_field(left_field, operator_str, right_field)

Builds a condition comparing the content of two fields with an operator.

Parameters:

  • left_field – field object as returned by Database.get_field

  • operator – string containing one of the CONDITION_OPERATOR defined in the grammar (in lowercase)

  • right_field – field object as returned by Database.get_field

build_condition_field_op_value(field, operator_str, value)

Builds a condition comparing the content of a field with a constant value using an operator.

Parameters:

  • field – field object as returned by Database.get_field

  • operator_str – string containing one of the CONDITION_OPERATOR defined in the grammar (in lowercase)

  • value – Python value (None, string number, boolean or date/time)

build_condition_literal_in_list_field(value, list_field)

Builds a condition checking if a constant value is in a list field

Parameters:

  • value – Python literal

  • list_field – field object as returned by Database.get_field

build_condition_negation(condition)

Builds a condition inverting another condition.

Parameters:

condition – condition object returned by one of the build_condition_*() method (except build_condition_all)

build_condition_value_op_field(value, operator_str, field)

Builds a condition comparing a constant value with the content of a field withusing an operator.

Parameters:

  • value – Python value (None, string number, boolean or date/time)

  • operator_str – string containing one of the CONDITION_OPERATOR defined in the grammar (in lowercase)

  • field – field object as returned by Database.get_field

condition(items)
conditions(items)
date(items)
datetime(items)
empty_list(items)
field_name(items)
keyword_literal(items)
keyword_literals = {'false': False, 'null': None, 'true': True}
list(items)
negation(items)
no_list_operators = {'<', '<=', '>', '>=', 'ilike', 'like'}
number(items)
quoted_field_name(items)
sql_operators = {'!=': 'IS NOT', '==': 'IS', 'ilike': 'LIKE'}
string(items)
time(items)
populse_db.filter.filter_parser()
Returns:

A singleton instance of Lark grammar parser for filter expression

populse_db.filter.literal_parser()

This is used to test literals parsing

Returns:

An instance of Lark grammar parser for parsing only a literal value (int, string, list, date, etc.) from a filter expression. This is used for testing the parsing of these literals.

populse_db.filter.to_sql(value)

populse_db.info module

populse_db.test module