Prepare a machine-learning dataset

You need one training dataset assembled from spectra spread across several .mtrl archives: aligned onto one shared grid, split so related runs never leak across the train/test boundary, and stamped with a provenance digest you can cite later.

Record claims through sample.claim (or inspect them through sample.claims) and then assemble features and claim-derived targets with the core builder this guide shows.

This guide assumes: you have finished the tutorial path, you can save and load archives after Save, load, and verify, and you have met assertions in the scientific assertions workflow.

The mtrl.ml package builds that dataset across archives, and it never writes to them: it mutates no archive and commits no revision. You describe the dataset you want in one flat build_dataset call, and it assembles a read-only result with a full provenance manifest. In this example two specimen archives hold FTIR spectra measured on different grids — one fine, one coarse — and each carries an assertion naming its polymer. The example builds both archives from scratch so it runs standalone.

from pathlib import Path
from tempfile import TemporaryDirectory

import numpy as np

from mtrl import InstrumentRun, MaterialData, Mtrl
from mtrl.assertions import AssertionDraft, SubjectRef
from mtrl.ml import build_dataset


def specimen_archive(directory, sample_label, target_label, axis):
    application = Mtrl(sample_id=sample_label)
    sample_uid = application.assign_sample_uid()
    content = np.empty(axis.size, dtype=[("wavenumber", "<f8"), ("absorbance", "<f8")])
    content["wavenumber"] = axis
    content["absorbance"] = np.linspace(0.1, 0.9, axis.size)
    run = InstrumentRun(run_name="ftir-scan-01")
    checkpoint = MaterialData(content=content, type="raw", checkpoint_name="spectrum")
    run.add_checkpoint(checkpoint)
    application.add(run)
    draft = AssertionDraft(
        subject=SubjectRef.sample(sample_uid),
        property_id="polymer.identity",
        value=target_label,
    )
    application.assert_properties([draft])
    return application.save(Path(directory) / f"{sample_label}.mtrl")


with TemporaryDirectory() as directory:
    fine_axis = np.linspace(400.0, 4000.0, 200)
    coarse_axis = np.linspace(400.0, 4000.0, 120)
    first_path = specimen_archive(directory, "polymer-film-01", "polypropylene", fine_axis)
    second_path = specimen_archive(directory, "polymer-film-02", "polyethylene", coarse_axis)

    dataset = build_dataset(
        [first_path, second_path],
        channel="raw",
        x_field="wavenumber",
        y_field="absorbance",
        target_grid=np.linspace(500.0, 3900.0, 100).tolist(),
        target="polymer.identity",
    )
    print(f"features shape: {dataset.features.shape}; channels: {dataset.channel_names}")
    print(f"class names: {dataset.class_names}; targets: {dataset.targets.tolist()}")

    bundle = dataset.split_grouped(test_size=0.5, seed=42)
    print(f"train rows: {len(bundle.train)}; test rows: {len(bundle.test)}")

    print(dataset.summary())

    sklearn_bundle = bundle.train.to_sklearn()
    print(f"dataset digest: {sklearn_bundle.dataset_digest}")

features shape: (2, 100); channels: ('raw',)
class names: ('polyethylene', 'polypropylene'); targets: [1, 0]
train rows: 1; test rows: 1
MLDataset — 2 rows; 2 groups; 1 channel
  target: 'polymer.identity' (classification); classes: 'polyethylene', 'polypropylene'
  channels:
    [0] 'raw' — 100 points on 'wavenumber' (span 500.0–3900.0); resample-linear, resampled
  auxiliary: none
  metadata detail: standard
  group by: sample-uid
  excluded: 0 candidate runs
  digest: sha256:89327935f52952ee86d514bf0d8787d75f7295ff89b5cece59fc74bf7ed9f1b2
dataset digest: sha256:89327935f52952ee86d514bf0d8787d75f7295ff89b5cece59fc74bf7ed9f1b2

The last line labels the canonical dataset digest dataset digest; that digest derives from the exact source records, whose identities mint per run, so your digest differs from this one. The bundle you fit on is bundle.train.to_sklearn(): it serves the training rows alone and carries the canonical digest of the parent dataset, because a split subset keeps its parent’s provenance manifest. Everything above it is stable, except the digest: line inside the summary, which carries that same canonical digest. dataset.summary() recaps the prepared dataset: the row, group, and channel totals, the target property with its task type and sorted class vocabulary, the one signal channel with its point count, coordinate field, span, alignment mode, and resample flag, the auxiliary basis, the metadata detail level, the group kind, and the excluded candidate count. The first line prints features shape: (2, 100); channels: ('raw',): two rows, one per archive, aligned onto the 100-point target grid even though the archives hold a 200-point and a 120-point spectrum. The second line prints the sorted class vocabulary ('polyethylene', 'polypropylene') and the encoded targets [1, 0]. The split puts one film in each subset, which the third line reports as train rows: 1; test rows: 1. The last line prints the canonical dataset digest: sha256: followed by 64 hexadecimal digits.

What just happened?

build_dataset([first_path, second_path], ...) is a reader, not a writer. It walks the sources in order, collects one candidate row per run, applies its filters, resolves the target from the active assertions, aligns every channel onto one coordinate grid, and projects the per-row metadata. The result is one read-only MLDataset. The sources may be paths, path strings, already loaded Mtrl containers, or ArchiveSourceInput items; an ArchiveSourceInput pins the revision that one source reads with as_of_revision.

The flat call is the routine route, and the whole recipe is declared up front, so the sources and the arguments alone reproduce the dataset. channel names the one channel, and stage selects one committed checkpoint state per run — here the raw stage. x_field, y_field, and target_grid resample the heterogeneous spectra onto your shared target grid, which is why the feature matrix has 100 columns. The advanced route spells the same recipe as one DatasetSpec and calls build_ml_dataset(); there its GridAlignmentSpec names the other alignment modes, where STRICT refuses mixed grids outright and RESAMPLE_PCHIP and RESAMPLE_CUBIC need the vector-mtrl[interp] extra. A channel can also run governed operations in memory through on_the_fly_steps, so preprocessing joins the recorded recipe without touching the archives.

Caution — on_the_fly_steps resolves each operation_id through the registry that builds the dataset. The historical identifiers mtrl.arithmetic.normalise-peak and mtrl.arithmetic.zero-at-point are not in the default registry. Record the equivalent step first as an ordinary callable transform, then select the committed state that step produced. A grid report still classifies a historical fixed identifier, because it does not execute the step. Control the operation registry gives the retirement details and the replay limits.

target names the property that supplies each row’s label, read from the active assertions on the sample — here the polymer.identity claim that each archive carries. The default group_by='sample-uid' is your leakage control: every run of one physical sample forms one group, and split_grouped shuffles groups, not rows. Replicates of one film therefore always land in the same subset. One seeded generator drives the shuffle, so the same seed always produces the same subsets and the same split_manifest_digest on the returned bundle. Pass val_size= to carve out a validation subset, and stratify=True to balance the class pools.

The dataset carries its own audit trail. dataset.provenance holds the canonical dataset_digest; the same sources and the same recipe always produce the same digest, which is the handle you cite in a tracker or a report, and the flat call and a hand-built DatasetSpec of one logical dataset agree on it. Every excluded run leaves one machine-readable reason in dataset.provenance.excluded_records, so nothing is dropped silently. When you hand the dataset to an experiment tracker, tracking_tags() flattens the dataset facts into a tag mapping, and log_to_mlflow() logs the same mapping to MLflow as tags and parameters. The framework bridges load lazily: to_sklearn() needs no scikit-learn installation, to_pandas() needs the vector-mtrl[tabular] extra, and to_torch_dataset() and to_tf_dataset() raise MissingOptionalDependencyError with installation guidance when their framework is absent.

Read the dataset as a frame

dataset.to_pandas() needs the vector-mtrl[tabular] extra. The default frame is not a training table. It holds one row per dataset row and the metadata columns that the selected detail level keeps, plus the encoded target in target_encoded when the dataset has one. It holds no signal columns at all.

dataset.to_pandas(include_features=True) adds the signals as feature_0000, feature_0001, and so on. Here one 100-point channel gives 100 such columns. The names encode a position only: they hold no field name, no unit, and no coordinate. Auxiliary features stay out of both frames, even with include_features=True, so a frame can hold fewer model inputs than to_sklearn().feature_matrix.

Use dataset.channel_names and dataset.coordinate_grid to map a column back to a physical position. The channel names are in stacking order, and the grid holds the shared axis. For one channel, feature_0000 matches coordinate_grid[0]. For several channels, the flat index is channel_index * len(coordinate_grid) + point_index. The dataset exposes those two attributes; it exposes no per-column accessor.

The frame is a detached view. Editing it changes no dataset array, and it holds no dataset digest column. Cite dataset.provenance.dataset_digest with a result.

The labelled views

Three labelled views answer the per-point question that feature_0000 cannot. Each one is detached in the same way, needs the same vector-mtrl[tabular] extra, and returns one row per dataset row with the positional index 0..n-1, so the views and the dataset rows line up:

dataset.metadata_frame() holds the same columns as to_pandas(): the selected metadata plus target_encoded, and no signal or auxiliary column. Cells keep their exact Python values.
dataset.signal_frame() holds the signal points alone, without any auxiliary column, as one (channel, coordinate) column per point under a two-level MultiIndex. The channel level repeats dataset.channel_names in stacking order and the coordinate level repeats dataset.coordinate_grid in the order that grid holds. Nothing is sorted or renamed, no unit is attached, and no scientific conversion is guessed: the labels carry exactly what the dataset declares.
dataset.auxiliary_frame() holds the auxiliary columns alone, as one (source, feature, category) column per model input under a three-level MultiIndex. category is empty for a numeric feature and holds the typed canonical category key for a one-hot column. A dataset without auxiliary features gives a (n_rows, 0) frame.

The auxiliary labels come from the dataset, not from the frame’s own rows. A split subset or a row subset therefore keeps every one-hot column of its parent, in the parent’s order, even where a column is all zeros because that subset lost one category. A dataset that was built before the labels existed, or one assembled by hand without them, falls back to the placeholders ('unknown', 'auxiliary_0000', '') and reports frame.attrs['mtrl.auxiliary_labels_resolved'] is False; a dataset that resolved its labels reports True. This is a presentation view, not an archive export and not a reversible training table.

The scikit-learn bundle answers a different question. dataset.to_sklearn() gives feature_matrix, which flattens every signal channel and joins the auxiliary columns. Beside it sit targets, groups, class_names, sample_ids, metadata, and dataset_digest. Prefer that bundle when you need every model input in one matrix and the provenance handle beside it. A split subset serves that bundle through bundle.train.to_sklearn(), over its own rows, so the matrix you fit on stays bound to the rows it holds. to_pandas cannot replace it for a complete training table.

Neither view decides the safety questions. Group leakage stays in the group split, target conflicts stay in the conflict policy, units stay in the metadata selection, and alignment stays in the channel grid specification. A frame only presents rows that those decisions already fixed.

Caution — A grouped split works at group level, so the subsets are only as granular as your groups. This toy study has two groups, and test_size=0.5 puts one film in each subset: it demonstrates the mechanics but cannot train anything. Real studies need enough samples per class to fill every subset.

Tip — When several active claims compete for the target property of one sample, the conflict policy on TargetSpec decides what happens: FAIL raises ConflictingAssertionError, EXCLUDE_SAMPLE drops the contested sample, and PREFER_METHOD picks one claim by method. Choose the policy deliberately.

Required install: vector-mtrl[tabular]

from pathlib import Path
from tempfile import TemporaryDirectory

import numpy as np

from mtrl import InstrumentRun, MaterialData, Mtrl
from mtrl.assertions import AssertionDraft, SubjectRef
from mtrl.ml import (
    AuxiliaryFeatureSpec,
    DatasetSpec,
    FeatureChannelSpec,
    GridAlignmentMode,
    GridAlignmentSpec,
    MetadataSelectionSpec,
    TargetSpec,
    build_ml_dataset,
)


def specimen_archive(directory, sample_label, target_label, mass_fraction, axis):
    application = Mtrl(sample_id=sample_label)
    sample_uid = application.assign_sample_uid()
    content = np.empty(axis.size, dtype=[("wavenumber", "<f8"), ("absorbance", "<f8")])
    content["wavenumber"] = axis
    content["absorbance"] = np.linspace(0.1, 0.9, axis.size)
    run = InstrumentRun(run_name="ftir-scan-01")
    run.add_checkpoint(MaterialData(content=content, type="raw", checkpoint_name="spectrum"))
    application.add(run)
    application.assert_properties(
        [
            AssertionDraft(
                subject=SubjectRef.sample(sample_uid),
                property_id="polymer.identity",
                value=target_label,
            ),
            AssertionDraft(
                subject=SubjectRef.sample(sample_uid),
                property_id="polymer.pp_mass_fraction",
                value=mass_fraction,
            ),
        ]
    )
    return application.save(Path(directory) / f"{sample_label}.mtrl")


with TemporaryDirectory() as directory:
    axis = np.linspace(400.0, 4000.0, 60)
    first_path = specimen_archive(directory, "polymer-film-01", "polypropylene", 40.0, axis)
    second_path = specimen_archive(directory, "polymer-film-02", "polyethylene", 4.0, axis)

    spec = DatasetSpec(
        channels=(
            FeatureChannelSpec(
                channel_name="raw",
                stage="raw",
                alignment=GridAlignmentSpec(
                    x_field="wavenumber",
                    y_field="absorbance",
                    mode=GridAlignmentMode.STRICT,
                ),
            ),
        ),
        target=TargetSpec(property_id="polymer.identity"),
        auxiliary_features=(
            AuxiliaryFeatureSpec(name="polymer.pp_mass_fraction"),
            AuxiliaryFeatureSpec(name="polymer.identity", encoding="one-hot"),
        ),
        metadata=MetadataSelectionSpec(),
    )
    dataset = build_ml_dataset([first_path, second_path], spec)

    metadata = dataset.metadata_frame()
    signals = dataset.signal_frame()
    auxiliary = dataset.auxiliary_frame()
    print(f"metadata columns: {list(metadata.columns)[:2]}; index: {metadata.index.tolist()}")
    print(f"signal shape: {signals.shape}; column levels: {list(signals.columns.names)}")
    print(f"auxiliary shape: {auxiliary.shape}; column levels: {list(auxiliary.columns.names)}")
    print(f"first auxiliary columns: {auxiliary.columns[:3].tolist()}")

metadata columns: ['sample_uid', 'sample_id']; index: [0, 1]
signal shape: (2, 60); column levels: ['channel', 'coordinate']
auxiliary shape: (2, 3); column levels: ['source', 'feature', 'category']
first auxiliary columns: [('assertion', 'polymer.pp_mass_fraction', ''), ('assertion', 'polymer.identity', '{"type":"string","value":"polyethylene"}'), ('assertion', 'polymer.identity', '{"type":"string","value":"polypropylene"}')]

The metadata view holds only metadata columns, so the first line prints the two leading metadata names beside [0, 1]. The signal view holds one (channel, coordinate) column per point, which for two films and one 60-point channel is (2, 60). The auxiliary view holds one column for the numeric mass fraction and one column per one-hot category, so it is (2, 3) and prints the three (source, feature, category) labels in that order.

Where next

Align mixed-resolution members — the same mixed-grid problem when you aggregate runs inside one archive.
Handle missing extras — how MTRL reports a missing optional dependency, and how to install the right extra.
Reference — every DatasetSpec field, alignment mode, and conflict policy, with exact signatures.