Reducing a season of spectrometer runs, reproducibly
Reduce 900 spectrometer CSVs against a calibration NetCDF into publication-ready spectra and a reproducible dataset export — no scripts.
Drop months of CSV logs from all 200 sensors on the canvas, clean and align them onto one clock, then compute windowed means and spreads with real uncertainty. Out comes a set of figures and a per-sensor summary table — and because it is a pipeline, not a notebook, your co-authors re-run it by opening one file instead of guessing which cell to run first.
Two hundred sensors logging for months is exactly the workload that breaks a notebook. Cells run out of order, a cleaning step gets tweaked and never re-applied upstream, and the figure in the draft no longer matches the code that supposedly made it. The data is fine — the process around it is what quietly rots. A pipeline fixes the order by construction, so reopening the canvas always reproduces the same numbers.
A mean with no spread is a claim with no error bar, and reviewers notice. Carrying the standard-error band through every window means the figures ship with their uncertainty attached, not bolted on at the end. Because the window length is a live parameter, you can show a co-author how the trend firms up as the window widens instead of arguing about it.
Add sensor 201 next month and its logs drop into the same folder; the canvas picks them up on the next run and the summary table grows a row — no new script, no edits. When a co-author opens the pipeline file, they get the cleaning rules, the alignment clock, and the window settings exactly as you left them, and they re-run the whole analysis from one file.
Drag the archive of long-run CSVs — 200 sensors, months of samples each — onto the canvas. The @load node groups them into one array entity keyed by sensor ID, infers the timestamp and value columns, and previews row counts per sensor. Files parse lazily, so a multi-gigabyte archive opens at once.
Add a @shape cleaning node with a rolling MAD outlier check and a stuck-value detector. Bad samples flag in red on the preview and drop from the working series, while the raw stays untouched underneath. Sensor 118's rail-stuck afternoon lights up before you have even scrolled past it.
Chain a sync node and pick a one-minute common grid. Set interpolation for the analog channels and hold-last for the slow ones, and watch 200 ragged timelines snap into one aligned table. That aligned table is a cached intermediate, so every statistic downstream recomputes instantly.
Add a windowed-stats node and compute mean, standard deviation, and a standard-error band per sensor over a rolling window. Expose the window length as a live parameter and drag it; every curve and its uncertainty band rebreathe from the cached aligned table. No rerun, no waiting.
Finish with a @deliver node. Render a small-multiples figure of the per-sensor trends with their uncertainty bands, and export a per-sensor summary table — count, mean, spread, and window settings — as CSV. The pipeline file ships with it, so a co-author reproduces every number by reopening the canvas.
Read long-run CSV logs from all 200 sensors as one lazily-parsed, per-sensor entity.
Reject bad samples, resample onto one clock, and roll up windowed stats with uncertainty.
Render per-sensor trend figures and export a reproducible summary table.
Reduce 900 spectrometer CSVs against a calibration NetCDF into publication-ready spectra and a reproducible dataset export — no scripts.
Model post-closure pit-lake rebound in LSD — krige borehole heads against a NetCDF recharge grid and deliver a live 3D drawdown model plus report.
Validate 63 LAS drillhole logs against collar surveys, krige a lithology block model, and export one glTF the whole team can open.
Build churn features from Postgres events and Parquet snapshots, catch a leaky window visually, and score the holdout with ONNX — on one canvas.
Score a million-row Parquet batch with an ONNX model — training-identical normalization, live previews, scored Parquet and metrics out.
Turn raw LAS/LAZ sweeps from a mobile robot into a nav-ready occupancy grid and 3D map — registration to export with zero scripts.
Sync CSV sensor logs with live REST telemetry, flag outliers and dropouts, and score every robot pass/fail before the field trial.
Turn folders of line-scan images and a label CSV into an ONNX defect-segmentation pipeline with overlay QA and a per-coil defect report.
Turn GeoTIFF orthomosaics into a stitched land-cover raster with per-class area stats, using tiled ONNX segmentation you can preview at every step.
Align CMM measurement points to a nominal CAD mesh, compute point-to-surface deviation, and export a color-mapped glTF model with a pass/fail report.
Resample a week of machine sensor logs, baseline each tag, and flag drift and step changes on a live dashboard with a ranked maintenance shortlist.
Join a live meter feed to 85 campus building meshes, aggregate it hourly, and paint a 3D heat-map that repaints as energy use shifts, plus a weekly report.
Drag the archive of long-run CSVs — 200 sensors, months of samples each — onto the canvas. The @load node groups them into one array entity keyed by sensor ID, infers the timestamp and value columns, and previews row counts per sensor. Files parse lazily, so a multi-gigabyte archive opens at once.
Chain a sync node and pick a one-minute common grid. Set interpolation for the analog channels and hold-last for the slow ones, and watch 200 ragged timelines snap into one aligned table. That aligned table is a cached intermediate, so every statistic downstream recomputes instantly.
Add a windowed-stats node and compute mean, standard deviation, and a standard-error band per sensor over a rolling window. Expose the window length as a live parameter and drag it; every curve and its uncertainty band rebreathe from the cached aligned table. No rerun, no waiting.
Finish with a @deliver node. Render a small-multiples figure of the per-sensor trends with their uncertainty bands, and export a per-sensor summary table — count, mean, spread, and window settings — as CSV. The pipeline file ships with it, so a co-author reproduces every number by reopening the canvas.