Tables

Your automation can keep data on the platform: you declare the tables in code, datallog push carries the declaration along, and the platform creates what is missing. There is no database console, no connection string and no migration to write.

Requires the datallog package 0.4.21 or newer.

Declaring

from datallog import table, Text, Number, Date, Bool, Json

leads = table("leads", fields={
    "company": Text(),
    "stage": Text(index=True),
    "value": Number(),
    "next_action": Date(),
    "active": Bool(),
    "payload": Json(),
})

table() runs on your machine, validates right there and returns the declaration as plain data. Declare it at the top of the automation file, or in a module they import — what matters is that the line runs when the code is loaded.

The types

Type	Holds	Becomes, in the database
Text()	text of any length	text
Number()	exact numbers, money included	numeric
Date()	a date, always as YYYY-MM-DD	date
Bool()	true or false	boolean
Json()	a whole object or list	jsonb

index=True creates an index on that column. Use it on the ones you filter or sort by; every index takes space and costs on every write, so it is not for all of them.

Every table gets an automatic id column, which you neither declare nor write.

The names

Table and column names accept lowercase letters, digits and _, starting with a letter, up to 63 characters. Reserved as column names: id, order_by, limit, offset and columns; reserved as table names: tables and schema.

The keys are in English on purpose: they travel over the network and show up in the platform's screens.

Publishing

datallog push

The output says how many tables it found:

Found 2 declared tables

The declaration is additive: a new table is created, a new column is added, and a column you removed from the declaration is not dropped — its data stays. Removing for real is a request to us, on purpose: a push from an old checkout must not drop a column.

Publishing the same declaration twice does nothing the second time: what exists stays as it is.

Two caps apply per push: 1599 columns per table (Postgres refuses more than 1600, and one is the id) and 200 tables per declaration.

What the platform refuses, and what you can catch

The same validation runs on both sides — on your machine, so the error points at your file, and on the server, because what arrives over the network is what someone chose to send. The errors are importable:

from datallog import InvalidIdentifier, InvalidManifest, UnknownType, TableRedeclared

try:
    table("Leads!", fields={"company": Text()})
except InvalidIdentifier as error:
    print(error)
    # invalid table name: 'Leads!'. Use only lowercase letters, digits and _,
    # starting with a letter, up to 63 characters.

Error	When
InvalidIdentifier	a table or column name off the rule, or reserved
UnknownType	a type that does not exist ("Texto", "string")
InvalidManifest	wrong shape: a table with no fields, a repeated field, a cap exceeded
TableRedeclared	the same table declared twice with different fields
DataEngineError	the base of them all, to catch in one go

Declare a table in one place

Two automations declaring leads with the same fields are fine — that is copy and paste, and the platform treats them as one. With different fields you get TableRedeclared: extract the declaration into a module and import it in both.

Changing a table that already exists

Adding a column is just declaring it and pushing. Changing the type of a column that already holds data is refused — turning Text into Number with rows written loses data, and the platform would rather say so than convert in the dark:

the manifest changes the type of a column that already exists, and changing
type with data inside loses data: leads.value is text in the database and
numeric in the manifest

The way through is to declare another column with the right type, migrate the content with your automation, and stop writing to the old one.

Writing and reading

The table that table() returns is what you use inside the automation. There is no URL to assemble and no credential to pass: it already knows how to talk to the platform.

from datallog import automation, table, Text, Number, Date

leads = table("leads", fields={
    "company": Text(),
    "stage": Text(index=True),
    "value": Number(),
    "next_action": Date(),
})

@automation()
def run(seed=None):
    lead = leads.insert({"company": "Acme", "stage": "open", "value": 4500})
    leads.update(lead["id"], stage="in conversation")
    return leads.find(stage="in conversation", limit=20)

Operation	What it does
insert(row)	writes one row and returns it whole, with the id
insert_many(rows)	writes a batch and returns only the ids
find(**filters)	the matching rows, as a list of dicts
find_with_total(**filters)	(rows, total), to paginate showing "31 of 1,200"
get(id)	the row, or None if it does not exist
count(**filters)	how many rows match, without fetching any
update(id, **fields)	changes the given fields and returns the updated row
delete(id)	deletes; True if it did, False if it was already gone
aggregate(...)	count, sum, average, min and max, with group by

Filtering

A filter is column=value, and None looks for the empty ones:

leads.find(stage="open")                        # equality
leads.find(stage="open", company="Acme")        # both conditions
leads.find(next_action=None)                    # with no date set
leads.count(stage="lost")

limit, offset, order_by and columns are parameters, not filters — and that is never ambiguous, because the engine reserves those four as column names:

leads.find(stage="open", order_by="-value", limit=50, offset=100)
leads.find(columns=["company", "value"])        # only what the screen uses

columns is the biggest saving on this route

Measured while porting our CRM: a kanban needed 11,859 bytes and was receiving 643,514 — 54 times more — because the table had a large Json column nobody was going to draw. Ask for the columns you use.

Math in the database, not in the automation

leads.aggregate(
    [{"function": "sum", "column": "value", "as": "total"},
     {"function": "count", "as": "how_many"}],
    group_by="stage",
)
# [{"stage": "open", "total": 91500, "how_many": 20}, ...]

Summing a thousand rows by fetching all thousand is paying bandwidth and memory for one number.

When something goes wrong

from datallog import InvalidValue, TableNotFound, RowNotFound

try:
    leads.insert({"value": "R$ 4,500.00"})
except InvalidValue as error:
    print(error)  # Postgres refused a value in this operation: … send numbers
                  # without currency symbols or thousand separators …

get and delete already handle a row that is gone — they return None and False. An update on a row someone else deleted raises RowNotFound.

Outside the platform it does not work, and it says so

The credential comes from the execution's environment. Running on your own machine, any operation raises "no execution credential: reading and writing tables only works inside a Datallog automation".

In a project that runs against something other than production, set datallog_api_url in the project's environment variables — without it the automation talks to the production account.

Where to see them

In the dashboard, under Usage → Data, you get the account's tables, the size of each one, the plan's cap and today's cost.