Model Registry & Artifact Storage

🗄️ What Is a Model Registry

A model registry is the system of record for trained models. It sits between your experiment tracking system and your deployment infrastructure, answering the question: "Of all the model versions that exist, which one should be running in production right now — and who decided that?" A registry is not a file server. It enforces a lifecycle, tracks lineage, and gates promotion through defined stages.

Without Registry vs With Registry

🔧 MLflow Model Registry

MLflow's Model Registry is the most widely deployed open-source registry. It is tightly integrated with the MLflow Tracking system — you register a model directly from a run's artifact, and the registry maintains the full link back to the run's parameters, metrics, and metadata. As of MLflow 2.x, aliases (Champion/Challenger) replace the older stage terminology.

Register and Transition a Model in Python

import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient(tracking_uri="http://mlflow.internal:5000")
MODEL_NAME = "fraud-detector"

# ── Option A: Register during training (from log_model) ────────────
with mlflow.start_run() as run:
    # ... training code ...
    mlflow.sklearn.log_model(
        model,
        artifact_path="model",
        registered_model_name=MODEL_NAME,   # auto-creates or adds version
    )
    run_id = run.info.run_id

# ── Option B: Register an existing run's artifact ─────────────────
result = mlflow.register_model(
    model_uri=f"runs:/{run_id}/model",
    name=MODEL_NAME,
)
version = result.version
print(f"Registered as version {version}")

# ── Wait for registration to complete ─────────────────────────────
import time
for _ in range(10):
    mv = client.get_model_version(MODEL_NAME, version)
    if mv.status == "READY":
        break
    time.sleep(2)

# ── Add descriptive metadata ──────────────────────────────────────
client.update_model_version(
    name=MODEL_NAME,
    version=version,
    description="XGBoost v3; trained on 2024-11 data; AUC=0.972, F1=0.891",
)
client.set_model_version_tag(MODEL_NAME, version, "dataset_version", "v2024-11")
client.set_model_version_tag(MODEL_NAME, version, "approved_by", "akumar")

# ── Assign Champion alias (MLflow 2.x preferred approach) ─────────
client.set_registered_model_alias(MODEL_NAME, "challenger", version)

# After shadow / A/B testing passes — promote challenger to champion
client.set_registered_model_alias(MODEL_NAME, "champion", version)

# ── Load by alias in serving code ─────────────────────────────────
champion_model = mlflow.sklearn.load_model(f"models:/{MODEL_NAME}@champion")

# ── Searching registered models ───────────────────────────────────
for mv in client.search_model_versions(f"name='{MODEL_NAME}'"):
    print(f"  v{mv.version}: {mv.current_stage} | {mv.description[:50]}")

# ── Legacy stage transitions (MLflow 1.x / 2.x compat) ───────────
client.transition_model_version_stage(
    name=MODEL_NAME,
    version=version,
    stage="Production",
    archive_existing_versions=True,   # archives previous Production version
)

# ── Get current production version URI for serving ────────────────
prod_versions = client.get_latest_versions(MODEL_NAME, stages=["Production"])
prod_uri = f"models:/{MODEL_NAME}/{prod_versions[0].version}"
print(f"Production model URI: {prod_uri}")

☁️ Cloud Registries

All major cloud providers offer managed model registries deeply integrated with their ML platforms. For teams already committed to a cloud provider, these offer better security, audit logging, and integration with managed training and serving infrastructure than self-hosted MLflow.

📁 Artifact Management

An artifact is any file output of a training run that needs to be preserved. Artifact management is the discipline of storing these files reliably, making them addressable by content (not just path), and ensuring that the right set of artifacts is always co-located when a model is deployed.

Artifact Lifecycle

📋 Model Cards & Documentation

A Model Card (Mitchell et al., 2019) is a short structured document attached to a model that describes its intended uses, performance characteristics, and limitations. Originally a research proposal, model cards are now a regulatory expectation in financial services (EU AI Act, SR 11-7), healthcare, and hiring — and a best practice everywhere else. A model without a card is an undocumented system in production.

Model Card YAML Template

# model_card.yaml
# Store this as an artifact in the model registry alongside the weights

model_details:
  name: "fraud-transaction-detector"
  version: "3.2.1"
  type: "XGBoost binary classifier"
  description: >
    Detects fraudulent credit card transactions in real time.
    Trained on anonymised transaction logs from 2022-2024.
  license: "Proprietary — internal use only"
  contact: "[email protected]"
  training_date: "2024-11-15"
  framework: "XGBoost 2.0.3 / scikit-learn 1.4.0"
  registry_uri: "models:/fraud-detector@champion"

intended_use:
  primary_use: "Real-time fraud scoring at payment authorisation time"
  intended_users:
    - "Fraud operations analysts"
    - "Payment processing pipeline (automated)"
  out_of_scope:
    - "Account takeover detection (different model)"
    - "Transaction amounts > $50,000 (insufficient training data)"
    - "Cryptocurrency transactions"

training_data:
  description: "Anonymised Visa/Mastercard transactions, US market, 2022-01 to 2024-10"
  size: "142M transactions (0.18% positive class)"
  preprocessing: "SMOTE oversampling; StandardScaler on amount; frequency encoding on merchant_id"
  note: "Full schema cannot be disclosed for privacy reasons"

evaluation_data:
  description: "Held-out 20% split, stratified by fraud rate, 2024-09 to 2024-10"
  size: "28.4M transactions"

metrics:
  overall:
    roc_auc: 0.9724
    precision_at_0.5: 0.8312
    recall_at_0.5: 0.7644
    f1_at_0.5: 0.7964
    false_positive_rate: 0.0021

  per_group:
    card_type:
      visa:   { roc_auc: 0.9731, f1: 0.7998 }
      master:  { roc_auc: 0.9708, f1: 0.7901 }
    transaction_type:
      card_present:     { roc_auc: 0.9688, f1: 0.7812 }
      card_not_present: { roc_auc: 0.9751, f1: 0.8043 }

limitations:
  - "Performance degrades for merchants with < 100 historical transactions"
  - "Not calibrated for transaction amounts > $10,000"
  - "Assumes feature pipeline version >= 2.4.0; older pipelines produce different feature values"
  - "May underperform during major retail events (Black Friday) — consider threshold adjustment"

ethical_considerations:
  - concern: "Geographic disparity"
    mitigation: "Per-region performance monitoring; alert if regional FPR diverges by > 0.5%"
  - concern: "Fraud pattern evolution"
    mitigation: "Monthly PSI checks on input features; retrain trigger if PSI > 0.2"

caveats:
  - "Retrain when monthly fraud rate changes by > 15% relative to training baseline"
  - "Model card must be updated before any Production transition"
  - "Threshold may need recalibration after any retraining event"

model_card_version: "1.0"
last_updated: "2024-11-15"
approved_by: "akumar"