Batch-scoring a million rows with an ONNX model
Score a million-row Parquet batch with an ONNX model — training-identical normalization, live previews, scored Parquet and metrics out.
Wire Postgres events and Parquet snapshots into windowed churn features where every intermediate table is cached and previewable. A leaky recency feature shows up as a too-clean histogram before training, not as a production incident. The same canvas exports the time-based split and scores the holdout with ONNX.
Every churn post-mortem has the same shape. The model looked great in the notebook — AUC deep in the 0.90s — and useless in production. Along the way the features were rebuilt three times (pandas for exploration, SQL for the warehouse, a serving job in Go), and one of the copies quietly measured the future.
Ours did exactly that. The training set was assembled months after the snapshots it described, and one recency feature was computed relative to export day. Every churned account looked long-dormant. Perfect signal, completely fake.
Two sources, both realistic and both messy. An events table in Postgres — 38M rows of logins, page views, support tickets, and payment failures — and twelve monthly account snapshots in Parquet, one row per account per month, each carrying a churned_within_90d label.
The pipeline in this walkthrough is one artifact. The join, the windows, the split — each is a live node with a cached intermediate you can open and sort. That's what makes the leak findable in seconds. You don't audit code, you look at the data between the steps. And because the ONNX scoring node runs in the same pipeline, the features the model sees at inference are, by construction, the ones it trained on.
Drop a PostgreSQL node on the canvas and point it at the `events` table. The @load step previews the first rows instantly — 38M rows of user_id, event_type, and timestamp, already typed. No SELECT written yet.
Drag twelve monthly `snapshot_*.parquet` files onto the canvas together. LSD unions them into one entity, derives a `snapshot_month` column from the filenames, and caches the result — 2.1M account rows with a `churned_within_90d` label.
Add a @shape join on `user_id`, then window steps for 7-, 30-, and 90-day activity — login counts, support tickets, failed payments. Each window is its own live node, and clicking any of them opens the cached intermediate table it produced.
Sort the feature preview by correlation with the label and `days_since_last_login` glows at 0.96, its histogram split into two clean islands. Click the node and the cause is right there in the expression — the window was anchored to the export date, not the snapshot date. Re-anchor it and the histogram relaxes into honest overlap.
Add a split step with the boundary on `snapshot_month` — train on the first nine months, test on the last three. Both splits are cached intermediates, so you inspect class balance in each before anything leaves the canvas.
A @deliver step writes both splits to Parquet for your trainer. When training finishes, drop the exported `churn.onnx` back on the canvas, wire it to the test split, and the confusion matrix fills in right next to the pipeline that built the data.
Pull the raw events table from Postgres and union twelve monthly account snapshots from Parquet.
Join, window, and normalize into 42 features with every intermediate cached and inspectable.
Export a time-based train/test split to Parquet and score the holdout with the trained ONNX model.
Score a million-row Parquet batch with an ONNX model — training-identical normalization, live previews, scored Parquet and metrics out.
Model post-closure pit-lake rebound in LSD — krige borehole heads against a NetCDF recharge grid and deliver a live 3D drawdown model plus report.
Validate 63 LAS drillhole logs against collar surveys, krige a lithology block model, and export one glTF the whole team can open.
Turn raw LAS/LAZ sweeps from a mobile robot into a nav-ready occupancy grid and 3D map — registration to export with zero scripts.
Sync CSV sensor logs with live REST telemetry, flag outliers and dropouts, and score every robot pass/fail before the field trial.
Turn folders of line-scan images and a label CSV into an ONNX defect-segmentation pipeline with overlay QA and a per-coil defect report.
Turn GeoTIFF orthomosaics into a stitched land-cover raster with per-class area stats, using tiled ONNX segmentation you can preview at every step.
Align CMM measurement points to a nominal CAD mesh, compute point-to-surface deviation, and export a color-mapped glTF model with a pass/fail report.
Resample a week of machine sensor logs, baseline each tag, and flag drift and step changes on a live dashboard with a ranked maintenance shortlist.
Join a live meter feed to 85 campus building meshes, aggregate it hourly, and paint a 3D heat-map that repaints as energy use shifts, plus a weekly report.
Calibrate live strain telemetry, place gauges on the bridge mesh from a CSV, and color the structure by stress live — plus an alert log on every exceedance.
Turn a CT DICOM series into an ONNX organ segmentation you QA in 3D, then export an annotated volume and a per-organ volumetrics report.
Drag twelve monthly `snapshot_*.parquet` files onto the canvas together. LSD unions them into one entity, derives a `snapshot_month` column from the filenames, and caches the result — 2.1M account rows with a `churned_within_90d` label.
Sort the feature preview by correlation with the label and `days_since_last_login` glows at 0.96, its histogram split into two clean islands. Click the node and the cause is right there in the expression — the window was anchored to the export date, not the snapshot date. Re-anchor it and the histogram relaxes into honest overlap.
A @deliver step writes both splits to Parquet for your trainer. When training finishes, drop the exported `churn.onnx` back on the canvas, wire it to the test split, and the confusion matrix fills in right next to the pipeline that built the data.