Example: file loader

Real data never arrives with the column names you want. This load step reads a file or a SQL query and lets the user map their columns onto canonical x, y, z — with a columnmap widget in the node form and fuzzy auto-detection when they leave it blank.

The code#

# tool.py
from typing import Annotated

from lsdtools import Tool, Source, Table
from lsdtools.advanced import columnmap_param, parse_column_map, resolve_column_map, apply_column_map

tool = Tool("points", label="Points")


@tool.load(source_kinds=("file", "sql"))
def load_points(
    src: Source,
    columns: Annotated[str, columnmap_param(["x", "y", "z"])] = "",
) -> Table:
    """Load a file/SQL query and map its columns onto x/y/z (blank = auto-detect by name)."""
    t = src.read()
    wanted = parse_column_map(columns) # "x=easting,y=northing" → {"x": "easting", …}
    resolved, missing = resolve_column_map(t.columns, wanted, required=["x", "y", "z"])
    if missing:
        raise ValueError(f"could not map required column(s): {missing}")
    return Table(apply_column_map(t.arrow, resolved))

What each line does#

• @tool.load(source_kinds=("file", "sql")) — the picker offers both a file source and a SQL query. src: Source injects a reader; src.read() returns a Table.
• columns: Annotated[str, columnmap_param(["x", "y", "z"])] = "" — a normal str parameter, but the columnmap_param metadata tells the inspector to render a column-map editor seeded with the target rows x, y, z. columnmap_param lives in lsdtools.advanced — the escape hatch for domain widgets in a signature.
• parse_column_map(columns) — parses the "target=source,…" grammar into a {target: source} dict (empty when the field is blank).
• resolve_column_map(t.columns, wanted, required=[...]) — resolves the (possibly partial) map against the file's real columns: explicit entries win, and any required target left unmapped is auto-detected (exact → case-insensitive → fuzzy). Returns (resolved, missing).
• apply_column_map(t.arrow, resolved) — selects the mapped source columns and renames them to the targets, returning a raw pyarrow.Table; Table(...) re-wraps it. t.arrow is the zero-copy escape hatch to pyarrow.

Run it in Python#

from lsdtools import Engine, Source, Table

# a survey file with domain-specific headers → map them explicitly
Table({"easting": [10.0, 11.0], "northing": [20.0, 21.0], "elev": [5.0, 6.0]}).write_csv("survey.csv")

pts = load_points(Source.file("survey.csv"), "x=easting, y=northing, z=elev")
Engine().run(pts).output.print()
# → x y z
# 10 20 5
# 11 21 6

# a file already using X/Y/Z → leave the map blank; it auto-detects (case-insensitive)
Table({"X": [1.0], "Y": [2.0], "Z": [3.0]}).write_csv("xyz.csv")
Engine().run(load_points(Source.file("xyz.csv"))).output.print()
# → x y z
# 1 2 3

The second positional argument to load_points(...) is the columns parameter — a plain string in the column-map grammar. In the desktop it is the column-map editor instead.

Try changing it#

• Add aliases={"z": ["rl", "elevation"]} to resolve_column_map so common survey names auto-map.
• Pass keep_unmapped=True to apply_column_map to carry the file's extra columns through.
• Read a database instead: load_points(Source.sql("warehouse", "select * from collars")).

Related#

Learn — Load steps · Parameters vs ports · UI overview

API — Source · Table · lsdtools.advanced

Examples — Clean a CSV · Multi-output mesh