dsl

Domain specific langauge (DSL) for use in AnnotationStore queries and indexes.

This module facilitates conversion from a restricted subset of python to another domain specific language, for example SQL. This is done using eval and a set of provided globals and locals. Mainly used for construction of predicate statements for AnnotationStore queries but also used in statements for the creation of indexes to accelerate queries.

This conversion should be assumed to be on a best-effort basis. Not every expression valid in python can be evaluated to form a valid matching SQL expression. However, for many common cases this will be possible. For example, the simple python expression props[“class”] == 42 can be converted to a valid SQL (SQLite flavour) predicate which will access the properties JSON column and check that the value under the key of “class” equals 42.

This predicate statement can be used as part of an SQL query and should be faster than post-query filtering in python or filtering during the query via a registered custom function callback.

An additional benefit is that the same input string can be used across different backends. For example, the previous simple example predicate string can be evaluated as both a valid python expression and can be converted to an equivalent valid SQL expression simply by running eval with a different set of globals from this module.

It is important to note that untrusted user input should not be accepted, as arbitrary code can be run during the parsing of an input string.

Supported operators and functions:

  • Property access: props[“key”]

  • Math operations (+, -, *, /, //, **, %): props[“key”] + 1

  • Boolean operations (and, or, not): props[“key”] and props[“key”] == 1

  • Key checking: “key” in props

  • List indexing: props[“key”][0]

  • List sum: sum(props[“key”])

  • List contains: “value” in props[“key”]

  • None check (with a provided function): is_none(props[“key”]) is_not_none(props[“key”])

  • Regex (with a provided function): regexp(pattern, props[“key”])

Unsupported operations:

  • The is operator: props[“key”] is None

  • Imports: import re

  • List length: len(props[“key”]) (support planned)

Compile options:

Some mathematical functions will not function if the compile option ENABLE_MATH_FUNCTIONS is not set. These are:

  • // (floor division)

Functions

json_contains

Return True if a JSON string contains x.

json_list_sum

Return the sum of a list of numbers in a JSON string.

py_is_none

Check if x is None.

py_is_not_none

Check if x is not None.

py_regexp

Check if string matches pattern.

sql_has_key

Check if a dictionary has a key.

sql_is_none

Check if x is None.

sql_is_not_none

Check if x is not None.

sql_list_sum

Return a representation of the sum of a list.

Classes

SQLExpression

SQL expression base class.

SQLJSONDictionary

Representation of an SQL expression to access JSON properties.

SQLNone

Sentinel object for SQL NULL within expressions.

SQLRegex

Representation of an SQL expression to match a string against a regex.

SQLTriplet

Representation of an SQL triplet expression (LHS, operator, RHS).