Shape steps

A shape step is a transform: it takes one or more Table input ports and returns a Table. It is the workhorse node — filter, join, reshape, compute.

Usage#

from lsdtools import Tool, Table

tool = Tool("scores")

@tool.shape
def top(t: Table, limit: int = 10) -> Table:
    """Keep the highest-scoring rows."""
    return t.sort("score", descending=True).head(limit)
# → t: input port · limit: a number field

t: Table is an input port you wire from an upstream node; limit: int is a parameter rendered as a number field.

Combining multiple input ports#

Add more than one Table parameter and each becomes its own input port on the node — wire a different upstream node into each. Combine them however you like and return one table:

@tool.shape
def enrich(facts: Table, dims: Table, on: str = "id") -> Table:
    """Join a facts table to a dimensions table."""
    return facts.join(dims, on=on, how="left")

enrich(load_facts(), load_dims(), on="id") # wire both ports in Python

On the canvas the node shows two inlets, facts and dims; in Python you pass them positionally or by keyword. The parameter name is the port label — so name your ports for what they carry.

Parameters alongside ports#

Everything that isn't a Table (or StepContext) is a form field. Mix ports and parameters freely; a parameter with no default is required, one with a default is optional:

@tool.shape
def bin_col(t: Table, column: str, bins: int = 10) -> Table:
    """Bucket a numeric column into `bins` equal ranges."""
    ...

Progress and cancel#

For long transforms, add a StepContext parameter to report progress and honour cancellation. It is injected by the engine and excluded from the cache key:

from lsdtools import Tool, Table, StepContext

@tool.shape
def heavy(t: Table, ctx: StepContext = None) -> Table:
    out = []
    for i, part in enumerate(split(t)):
        if ctx.cancelled:
            break
        ctx.progress(i / len(t), "processing")
        out.append(transform(part))
    return concat(out)

Emitting several tables#

A shape step can produce named output ports with outputs=[...] and return a dict — useful for a split or a decomposition. See Multiple outputs:

@tool.shape(outputs=["keep", "reject"])
def partition(t: Table, floor: float = 0.0) -> dict:
    mask = t["score"] >= floor
    return {"keep": t.filter(mask), "reject": t.filter(~mask)}

Related#

Learn — Load steps · Deliver steps · Tables & data flow

API — Tool · Table · Context

Examples — Clean a CSV · Multi-output mesh