Tables & data flow

Data moves between nodes as a Table — an immutable, zero-copy facade over a single pyarrow.Table. A step reads Tables on its input ports and returns a Table; that value is what the wire carries to the next node.

Usage#

from lsdtools import Table

t = Table({"city": ["Lima", "Cusco", "Ica"], "pop": [9_700_000, 420_000, 240_000]})
big = t.filter(t["pop"] > 300_000).sort("pop", descending=True)
big.print()
# → city:["Lima","Cusco"] pop:[9700000,420000]

Tables flow between steps#

Each step returns a Table, and each input port receives one. A @tool.shape step is just a function Table → Table; wiring two nodes connects one's return value to the other's port:

from lsdtools import Tool, Table

tool = Tool("geo")

@tool.shape
def big_cities(t: Table, floor: int = 300_000) -> Table:
    return t.filter(t["pop"] > floor)

The engine caches each step's output Table by its inputs, so an unchanged upstream node is never recomputed.

Immutable — verbs return new tables#

Every verb returns a new Table; nothing mutates in place. That is what makes the cache trustworthy — a step can't accidentally alter its input:

clean = (t
    .rename({"pop": "population"})
    .drop("city")
    .with_column("k_people", t["pop"] / 1000)
    .sort("population", descending=True)
    .head(10))
# t is unchanged; clean is a new table

The core verbs: filter, select, drop, rename, with_column, sort, head, join, group_by(*keys).agg(col="mean", …), plus null_report() and to_dicts().

Columns and expressions#

Index a column with t["name"] to get a Column. Columns support arithmetic and comparison operators that build new columns or boolean masks — feed a mask to filter:

mask = (t["pop"] > 300_000) & t["city"].is_valid()
t.filter(mask) # keep rows where the mask is true
t.with_column("mega", t["pop"] > 1_000_000) # a derived boolean column

Operators: + - * / > >= < <= == != & | ~, plus is_valid() / is_null().

Building and reading tables#

The Table(...) constructor accepts columns, row dicts, or a pyarrow table, and reads files directly:

Table({"x": [1, 2, 3]}) # from columns
Table([{"x": 1}, {"x": 2}]) # from row dicts
Table.read_csv("data.csv", delimiter=";")
Table.read_parquet("data.parquet")

To write results out, use a deliver step with t.write_csv(path) or t.write_parquet(path).

The .arrow escape hatch#

When you need a pyarrow (or pandas-via-arrow) operation the Table doesn't wrap, drop to the underlying pyarrow.Table through .arrow, then wrap the result back with Table(...):

import pyarrow.compute as pc
from lsdtools import Table

col = pc.utf8_upper(t.arrow["city"]) # any pyarrow.compute kernel
result = Table(t.arrow.set_column(0, "city", col))

.arrow is zero-copy — it hands back the very buffers the Table wraps, so reaching in and back out is cheap. It is the sanctioned way past the verb set; you never need lsd.* internals for data.

Related#

Learn — Parameters vs ports · Shape steps

API — Table · Source · Tool

Examples — Clean a CSV · Your first tool