# Tables & data flow
URL: https://lsd.tools/docs/tools-tables
Updated: 2026-07-17

> The Table is the currency between steps — an immutable, zero-copy facade over one pyarrow.Table. Steps receive Tables on their input ports and return a Table; verbs like filter, sort and join always return a new Table rather than mutating in place. Build one with Table(...), read files with Table.read_csv/read_parquet, and reach the underlying pyarrow via .arrow.

Data moves between nodes as a [`Table`](/docs/api-table) — an **immutable, zero-copy** facade over a
single `pyarrow.Table`. A step reads Tables on its input ports and returns a Table; that value is
what the wire carries to the next node.

## Usage

```python
from lsdtools import Table

t = Table({"city": ["Lima", "Cusco", "Ica"], "pop": [9_700_000, 420_000, 240_000]})
big = t.filter(t["pop"] > 300_000).sort("pop", descending=True)
big.print()
# → city:["Lima","Cusco"] pop:[9700000,420000]
```

## Tables flow between steps

Each step returns a `Table`, and each input **port** receives one. A `@tool.shape` step is just a
function `Table → Table`; wiring two nodes connects one's return value to the other's port:

```python
from lsdtools import Tool, Table

tool = Tool("geo")

@tool.shape
def big_cities(t: Table, floor: int = 300_000) -> Table:
    return t.filter(t["pop"] > floor)
```

The engine caches each step's output Table by its inputs, so an unchanged upstream node is never
recomputed.

## Immutable — verbs return new tables

Every verb returns a **new** `Table`; nothing mutates in place. That is what makes the cache
trustworthy — a step can't accidentally alter its input:

```python
clean = (t
    .rename({"pop": "population"})
    .drop("city")
    .with_column("k_people", t["pop"] / 1000)
    .sort("population", descending=True)
    .head(10))
# t is unchanged; clean is a new table
```

The core verbs: `filter`, `select`, `drop`, `rename`, `with_column`, `sort`, `head`, `join`,
`group_by(*keys).agg(col="mean", …)`, plus `null_report()` and `to_dicts()`.

## Columns and expressions

Index a column with `t["name"]` to get a `Column`. Columns support arithmetic and comparison
operators that build **new columns** or boolean masks — feed a mask to `filter`:

```python
mask = (t["pop"] > 300_000) & t["city"].is_valid()
t.filter(mask) # keep rows where the mask is true
t.with_column("mega", t["pop"] > 1_000_000) # a derived boolean column
```

Operators: `+ - * / > >= < <= == != & | ~`, plus `is_valid()` / `is_null()`.

## Building and reading tables

The `Table(...)` constructor accepts columns, row dicts, or a pyarrow table, and reads files directly:

```python
Table({"x": [1, 2, 3]}) # from columns
Table([{"x": 1}, {"x": 2}]) # from row dicts
Table.read_csv("data.csv", delimiter=";")
Table.read_parquet("data.parquet")
```

To write results out, use a [deliver step](/docs/tools-deliver) with `t.write_csv(path)` or
`t.write_parquet(path)`.

## The `.arrow` escape hatch

When you need a pyarrow (or pandas-via-arrow) operation the `Table` doesn't wrap, drop to the
underlying `pyarrow.Table` through `.arrow`, then wrap the result back with `Table(...)`:

```python
import pyarrow.compute as pc
from lsdtools import Table

col = pc.utf8_upper(t.arrow["city"]) # any pyarrow.compute kernel
result = Table(t.arrow.set_column(0, "city", col))
```

`.arrow` is zero-copy — it hands back the very buffers the `Table` wraps, so reaching in and back
out is cheap. It is the sanctioned way past the verb set; you never need `lsd.*` internals for data.

## Related

**Learn** — [Parameters vs ports](/docs/tools-params-ports) · [Shape steps](/docs/tools-shape)

**API** — [Table](/docs/api-table) · [Source](/docs/api-source) · [Tool](/docs/api-tool)

**Examples** — [Clean a CSV](/docs/ex-clean-csv) · [Your first tool](/docs/ex-hello-tool)

## FAQ
### Do Table verbs modify the table in place?

No. A Table is immutable — every verb (filter, sort, with_column, drop, rename, join) returns a new Table and leaves the original untouched. This is what makes caching safe and pipelines reproducible.

### How do I use a pyarrow or pandas function LSD doesn't wrap?

Read the underlying pyarrow.Table from t.arrow, do your work, and wrap the result back with Table(pa_result). The .arrow property is the sanctioned escape hatch; it is zero-copy.