How an LSD package fits together

One idea: run and show#

An Engine is the whole project. It holds two kinds of thing, and that split is the entire model:

• Entities you run — a pipeline of steps.
• Views you show — a piece of UI.

                         ┌──────────────────────────────────────────┐
                         │  Engine   (workspace dir · event bus)     │
                         └───────────────┬───────────────┬──────────┘
              engine.add(entity)  ───────┘               └───────  engine.add(view)
                  → run() / run_all()                        → show()

     RUN  (pipelines) ─────────────────────      SHOW  (UI) ─────────────────────────
       Source → LOAD ─Table→ SHAPE ─Table→          View  — ONE class:
                       DELIVER → file, or              • no kind → a form (params + @action)
                       return a ViewerPayload          • a kind  → table · chart ·
       Entity owns the steps + @tool.action                        flowchart · custom
       Template stamps ready-made entities            a view watch(entity)es to get its output

              entity runs ─► output + ViewerPayload ─► ( framework wires the bus ) ─► watching views

The symmetry is exact: engine.add(entity) then engine.run() mirrors engine.add(view) then engine.show(). Everything the Engine holds is either something you run or something you show.

The run side — a pipeline#

A pipeline is a chain of typed steps. A Table (a friendly, zero-copy wrapper over Arrow) flows through the ports:

• LOAD — brings data in (source → Table). A Source parameter injects the reader.
• SHAPE — transforms it (Table → Table).
• DELIVER — ends the pipeline. It either writes a result out (Table → file / API) or returns a ViewerPayload (LayerPayload / ChartPayload / LegendPayload) to show it; the framework persists the payload and routes it to any view that watches the entity.

Steps are cached on their parameters: change one knob and only what depends on it recomputes. An Entity owns an ordered set of steps and any right-click actions; a Template turns a dropped file into a ready-wired entity.

from lsdtools import Tool, Table, Engine

tool = Tool("scores")

@tool.load
def numbers(count: int = 5) -> Table:
    return Table({"n": range(count)})

@tool.shape
def doubled(t: Table) -> Table:            # `t: Table` is an input port
    return t.with_column("n", t["n"] * 2)

Engine().run(doubled(t=numbers(count=5))).print()

The show side — a view#

A View is one declarative class, presented as a window when the desktop is present or a console rendering otherwise (so the same code runs headless). It is param() fields plus @action buttons, with optional data:

• with no kind it renders as a form — an interaction surface of params + actions;
• with a kind (table, chart, flowchart, or a custom one) it renders that representation of data.

from lsdtools import views, Engine

engine = Engine()
engine.add(views.chart(kind="histogram", title="Grades"))
engine.show()          # a window, or console text when headless

A View never imports GTK — the desktop registers the renderer and LSD stays front-end-agnostic. The same View a package contributes with @tool.view also works standalone through show().

The coupling — watch(entity)#

The run side and the show side never reference each other. A view gets an entity's output with one verb — watch:

from lsdtools import Engine, views

engine = Engine()
engine.add(drillholes(collar="collars.csv", survey="survey.csv"))   # an entity to run
grid = views.table(title="Assays").watch("drillholes")              # a view that follows it
engine.add(grid)
engine.run_all()      # the entity runs → the view holds its output; a re-run refreshes it
engine.show()

view.watch(entity) holds that entity's output and refreshes on every re-run — the framework does the bus wiring (you never call bus.on). Any order works: watch before or after either is added, and watching an entity that has already run pulls its output immediately (which is what makes drag-and-drop feel instant). A deliver step that returns a ViewerPayload is routed the same way — to whatever watches the entity. The desktop's drag-drop, the 3-D checkbox, and right-click "Show in ▸" are all sugar over this one verb.

For live, mid-run signals (progress, a training epoch) steps still ctx.emit custom events — payload return is for the terminal deliver artifact, emit is for transient events while a step runs.

The front door — @tool.*#

You never wire any of this by hand. One Tool object collects the whole package through decorators:

Two rules the model enforces: a step only ever gets a StepContext (the full Context is rejected — it keeps run() pure and cacheable), and the same View has two homes (mounted in the desktop, or shown standalone).

Related#

Start — What is LSD? · Write your first tool

Learn — The Tool object · Naming: which is which

API — Engine · Table · UI overview