How robot-learning teams build training datasets from raw fleet logs

The problem: the best training data is buried in logs you can't search

A single robot streams cameras, lidar, and telemetry all day. A fleet of them makes 0.5–2 TB per robot per day. About 99.9% of that is never useful again. The 0.1% that is — a failure, an edge case, a clean demonstration — is what your model needs to learn from. But it is scattered across petabytes of logs.

To use it you have to find it. And finding one moment across the whole corpus is the hard part. Hot storage plus a search database is far too expensive at petabyte scale. Cold archives like Glacier are cheap but cannot be searched. So most of the data — and the training signal inside it — just sits there.

What a training dataset actually is

A robot training dataset is a set of episodes: short, time-synced clips of the robot acting, each with the camera, sensor, and action data for that moment. Imitation-learning and robot-foundation-model teams train on these episodes, packaged in a format like LeRobot or RLDS. The quality of the episodes decides how well the model learns — a smaller, sharper set beats a huge, noisy one.

How teams do it today — and why it breaks

Today most teams curate by hand. An engineer scrubs through recordings, or writes a one-off filter for a signal they already know about, then copies the clips out. This only finds what you already know to look for, and no one can review petabytes by hand.

The teams that go further build their own search pipeline on Spark or Iceberg. It works, but running it is a full engineering project on its own — the kind only the largest labs can afford. Everyone else deletes data they will later wish they had kept.

The method: search a fleet three ways at once

Finding the right moments needs three kinds of search over the logs, combined in one query:

• Metadata — which robot, what time range, which topics or message types.
• Signal — a time-series pattern, like "torque above 5 for 2 seconds and velocity near zero."
• Semantic — frames that match a description in plain words, like "gripper slipping on a transparent object," using image embeddings.

Used together they find moments no single filter would — a torque spike that also looks like a slip ranks highest. That turns "search my whole fleet" from a manual chore into one call.

Keep the raw data in your own bucket

The raw logs never have to move. The approach is to keep them in your own object storage and build only a compact index — around 1–3% of the raw size — that points back into your bucket. Search runs over the index; matched clips are read from your bucket with short-lived signed URLs. Nothing raw is copied out of your account, which removes the biggest objection to putting fleet data anywhere.

How Coldstack does this

Coldstack, built by Vantar, is a retrieval layer for robot logs that implements exactly this. You point it at your bucket of MCAP logs. It builds and hosts a compact index on object storage. Then you search the whole fleet by metadata, signal, and meaning in a single call, and export the matches straight to a LeRobot-compatible dataset. Raw data stays in your bucket the whole time.

It is early, and we are picking a few design partners for the first version. If your team is curating training data from fleet logs by hand, we would like to talk.

Questions