Model Tracking: How to Record Experiments, Metrics and Artefacts

Your notebook shows promising scores, but can you identify the data, code and parameters behind each one? A model tracking record lets a teammate reproduce a run, compare it fairly and recover the selected artefact. The fictional churn dataset supplies three comparable runs whose lineage, metrics, versioned artefacts and promotion decision can be checked end to end.

Model tracking: the questions every run must answer

An experiment is a named investigation, such as churn-baseline. A run is one execution inside it, such as r001, r002 or r003. An artefact is a file produced by a run. The tracker is the index connecting inputs, code, configuration, results and output files. Tracking is the evidence layer: it records what a run used and produced. Converting a notebook into an executable training pipeline and proving a clean rerun are separate responsibilities.

A teammate should be able to identify who ran an experiment, when, with which dataset snapshot, commit, environment, parameters, random seed, evaluation split and threshold. Notebook cell order and a filename such as best-model-final-v2.joblib fail this reproducibility test.

The ML & Data Science hub provides the broader learning map. Approaches to AI covers choosing a modelling approach; tracking preserves the evidence it produces.

Model tracking fields: inputs, configuration, results and outputs

A compact run record needs six parts:

1. Identity: experiment_name, run_id, owner and UTC start time.

2. Data: dataset URI, version and checksum.

3. Code and environment: Git commit and locked package versions.

4. Parameters: every explicit value and every default that affects training.

5. Metrics: split name, decision threshold, metric definition and whether higher or lower is better.

6. Artefacts: URI, file size, checksum and format.

For the constructed runs, the common dataset is churn-v3.csv: 12,000 rows, 20 input features and checksum prefix 4f9c2a7e. A 70/15/15 split gives 8,400 training rows, 1,800 validation rows and 1,800 untouched test rows. Every run uses commit a17c9d4, python==3.11.9, scikit-learn==1.5.2, seed 42 and classification threshold 0.50.

For a fair comparison, change the model and declared parameters while holding the dataset snapshot, split, seed, commit and metric definitions fixed. A changed shared input creates a new comparison group.

Model tracking worked example: compare three churn classifiers

The validation set contains 1,800 rows: 450 positive churn cases and 1,350 negative cases.

Recompute r003 from its confusion matrix:

• Accuracy = (TP + TN) / total = (355 + 1,230) / 1,800 = 1,585 / 1,800 = 0.8806, which rounds to 0.881.

• Precision = TP / (TP + FP) = 355 / (355 + 120) = 355 / 475 = 0.7474, which rounds to 0.747.

• Recall = TP / (TP + FN) = 355 / (355 + 95) = 355 / 450 = 0.7889, which rounds to 0.789.

• F1 = 2 x TP / (2 x TP + FP + FN) = 2 x 355 / (2 x 355 + 120 + 95) = 710 / 925 = 0.7676, which rounds to 0.768.

ROC-AUC 0.916 comes from scores across thresholds. Confusion-matrix metrics use threshold 0.50. “F1 = 0.768” is not comparable without the validation split, class counts and threshold. Keep the 1,800-row test split untouched during selection.

Model artefacts: save the evidence needed to reproduce r003

The immutable r003 folder contains:

• runs/r003/model.joblib, 8.4 MB, checksum prefix 7b1e4a2c

• preprocessor.joblib, 23 KB

• feature-schema.json, with 20 ordered feature names and types

• metrics.json, with the confusion-matrix result, accuracy, precision, recall, F1, ROC-AUC and threshold 0.50

• confusion-matrix.png, labelled TP 355, FP 120, TN 1,230 and FN 95

• requirements-lock.txt

• run-manifest.json, linking every file to dataset checksum 4f9c2a7e and commit a17c9d4

Keep parameters and individual metric values where they can be searched. Charts, models, preprocessing objects and schemas are artefacts; the manifest joins both sides. Saving only model.joblib loses the preprocessing order and makes inference uncertain.

Use immutable paths and checksums. A production-candidate pointer may move, but never overwrite runs/r003/.

Model selection: turn “best” into a recorded promotion rule

Set the promotion gate before selecting a run: validation F1 at least 0.750, recall at least 0.770, ROC-AUC at least 0.900, and serialised model size at most 10 MB.

• r001 fails F1, recall and ROC-AUC.

• r002 fails F1, recall and model size.

• r003 passes with F1 0.768, recall 0.789, ROC-AUC 0.916 and size 8.4 MB.

Record that r003 was selected under the rule of F1 at least 0.750, recall at least 0.770, ROC-AUC at least 0.900 and model size at most 10 MB, along with the approver, timestamp and a pointer to the untouched test record. Do not tune after seeing its result.

Different error costs may need another gate. Recording the actual rule keeps “best” interpretable.

Model tracking traps: high scores can still be invalid

Suppose r004 uses churn-v4.csv and adds days_since_cancel, a value known only after prediction. This leaks the outcome. Its matrix is TP 419, FP 30, TN 1,320 and FN 31. Precision is 419 / 449 = 0.933, recall is 419 / 450 = 0.931, F1 is 838 / 899 = 0.932, accuracy is 1,739 / 1,800 = 0.966, and ROC-AUC is 0.973.

Reject r004. Its dataset version breaks comparability, while the unavailable feature breaks real-world validity.

Other common failures also have direct fixes:

• An unrecorded seed makes splits irreproducible. Track the seed and split ID.

• Accuracy alone hides class-specific errors. Store the full metric set and confusion matrix.

• An unlocked environment can change behaviour after a library upgrade. Save a lock file or container digest.

• Overwriting a shared model path destroys lineage. Use immutable, versioned artefact paths.

Keep secrets, raw personal data and huge binaries out of metadata. Track approved references and checksums; keep artefacts in controlled object storage.

Model tracking questions in interviews and practical assessments

Interviewers and practical assessments can test the same reasoning through three prompts:

1. Calculate r003 recall and F1 from its confusion matrix.

2. Explain why r004 must not outrank r003.

3. If only model parameters and F1 were logged, name the missing records that would prevent reproduction.

Good reasoning selects r003 under the stated gate, not the highest raw score. A metric needs its dataset, split, threshold and definition; a deployable artefact needs preprocessing and schema. Assessments may vary the values, but the decision rule and reproduction requirements stay the same.

Model tracking checklist and the next learning step

Before closing a run, check identity, data version and checksum, code and environment, parameters and seed, split and threshold, metric direction, artefact manifest and selection decision. Another person should reproduce r003 and explain its selection without the original notebook.

For broader machine-learning study, continue with the Artificial Intelligence (AI) course. If programming fluency is the immediate gap, start with the Python course. The Skills courses page is the wider catalogue route.

A trustworthy model is a reproducible run, a complete evidence trail and a decision that still makes sense later.