Design a system for managing a distributed data versioning system.

Question

Anonymous · Accepted Answer

I would design it as a Git-like control plane for large datasets: immutable,
content-addressed data stored in distributed object storage, with lightweight
version metadata, branches, commits, merges, and transactional reference
updates. The central goals are reproducibility, efficient sharing, safe
concurrent writes, strong lineage, and the ability to roll back or reconstruct
any dataset version exactly. Immutable commits and content-addressed objects are
proven patterns for guaranteeing a point-in-time view without copying all data
for every version.[docs.lakefs
[https://docs.lakefs.io/concepts/internals/]][epicgames.github
[https://epicgames.github.io/lore/explanation/system-design/]]

1. CLARIFY REQUIREMENTS FIRST

In an interview, I would begin by asking a few questions:

> “Before I design it, I would clarify whether this is intended for structured
> tables, arbitrary files, streaming data, ML datasets, or all of these;
> expected data volume and number of concurrent users; whether branches and
> merges are required; consistency requirements; and whether this is a
> multi-region system.”

Then I would make reasonable assumptions:

AreaAssumptionData typesFiles plus structured datasets such as
Parquet/CSV/JSONScalePetabytes of data, billions of objects, many
teamsStorageCloud object storage or distributed blob storeVersioningDataset-,
partition-, and file-level versionsCollaborationBranching, commits, tags,
comparisons, merges, rollbackConsistencyStrong consistency for commit
visibility; eventual replication acceptable for immutable payloadsSecurityTenant
isolation, RBAC, encryption, auditing, retention/legal holdConsumersData
engineers, analysts, data scientists, pipelines, ML training and reporting
systems

A useful framing is:

> “I would separate the large immutable data plane from the small transactional
> metadata control plane.”

2. HIGH-LEVEL ARCHITECTURE

┌─────────────────────────────────────┐
                 │ CLI / SDK / REST API / UI            │
                 │ commit | branch | diff | merge       │
                 │ checkout | tag | rollback | lineage  │
                 └─────────────────┬───────────────────┘
                                   │
                 ┌─────────────────▼───────────────────┐
                 │ Versioning Control Plane             │
                 │                                     │
                 │ • Repository & branch service       │
                 │ • Commit/snapshot service           │
                 │ • Metadata catalog                  │
                 │ • Merge/conflict-resolution engine  │
                 │ • AuthN/AuthZ and audit service     │
                 │ • Garbage-collection coordinator    │
                 └───────┬───────────────────┬─────────┘
                         │                   │
             conditional │                   │ metadata/query
              ref update │                   │
                         ▼                   ▼
          ┌────────────────────┐   ┌─────────────────────┐
          │ Transactional      │   │ Search/Lineage       │
          │ Metadata Store     │   │ Index                │
          │ branches, refs,    │   │ commits, schemas,    │
          │ leases, tags, ACLs │   │ provenance, audit    │
          └────────────────────┘   └─────────────────────┘
                         │
                         ▼
       ┌────────────────────────────────────────────────────┐
       │ Immutable Distributed Data Plane                     │
       │                                                     │
       │ Content-addressed object/chunk store                │
       │ Raw files, columnar partitions, manifests, indexes  │
       │ Hash-keyed objects; replicated across regions       │
       └────────────────────────────────────────────────────┘

The key architectural decision is that the system never overwrites a committed
data object. A version is a metadata snapshot that points to immutable objects.
This allows safe sharing, deduplication, auditability, rollback, and fast
branching. Systems such as lakeFS use immutable committed metadata and
object-storage-backed versioning, while content-addressed storage designs use a
small mutable reference layer only for branch heads and similar
pointers.[docs.lakefs
[https://docs.lakefs.io/concepts/internals/]][epicgames.github
[https://epicgames.github.io/lore/explanation/system-design/]]

3. CORE DATA MODEL

REPOSITORY

A repository is the top-level namespace for a logical data domain.

Repository
- repository_id
- tenant_id
- name
- default_branch
- storage_namespace
- retention_policy
- encryption_policy
- created_at

Example:

repo: sports-media-analytics
storage namespace: s3://analytics-prod/sports-media/
default branch: main

BRANCH

A branch is a mutable named pointer to the latest immutable commit.

Branch
- repository_id
- branch_name
- head_commit_id
- version_number
- protection_policy
- created_by
- created_at

Examples:

main       → commit C1042
playoffs   → commit C1077
experiment → commit C1084

COMMIT / DATASET SNAPSHOT

A commit is an immutable point-in-time view of an entire dataset or namespace.
It forms a directed acyclic graph, similar to a source-control commit graph.

Commit
- commit_id
- repository_id
- parent_commit_ids[]
- root_manifest_hash
- schema_version
- author
- message
- created_at
- pipeline_run_id
- source_lineage[]
- quality_validation_results
- statistics

A commit has one parent for a normal update and two or more parents for a merge.

C1000 → C1001 → C1002 → C1003
                    \
                     → C1010 → C1011

A dataset snapshot graph is a useful abstraction because commits are immutable
nodes, branches are named pointers to the latest reachable snapshot, and merges
generate new commits with multiple parents.[docs.lakefs
[https://docs.lakefs.io/concepts/internals/]][arxiv
[https://arxiv.org/html/2605.00676v1]]

MANIFEST

A manifest maps logical dataset paths or partitions to immutable physical
objects.

{
  "dataset": "game-events",
  "schema_version": "v14",
  "partitioning": ["event_date", "league"],
  "entries": [
    {
      "logical_path": "event_date=2026-09-28/league=MLB/part-0001.parquet",
      "content_hash": "sha256:4e2b...",
      "size_bytes": 268435456,
      "row_count": 2100000,
      "schema_hash": "sha256:79ac...",
      "stats": {
        "min_event_time": "2026-09-28T17:00:00Z",
        "max_event_time": "2026-09-28T17:04:59Z"
      }
    }
  ]
}

The root manifest can be represented as a Merkle tree:

Root Manifest Hash
   ├── Partition Manifest: league=NBA
   │      ├── 2026-09-28/part-001 → hash A
   │      └── 2026-09-28/part-002 → hash B
   │
   └── Partition Manifest: league=MLB
          ├── 2026-09-28/part-001 → hash C
          └── 2026-09-28/part-002 → hash D

Merkle trees make diff, merge, integrity verification, and selective reads
efficient because unchanged subtrees have the same hash and do not need to be
traversed. lakeFS uses a two-layer Merkle-tree structure for committed object
metadata, while other distributed version-control designs similarly use
content-addressed immutable fragments and tree traversal for diffs.[docs.lakefs
[https://docs.lakefs.io/concepts/internals/]][epicgames.github
[https://epicgames.github.io/lore/explanation/system-design/]]

CONTENT-ADDRESSED DATA OBJECT

Every physical chunk is named by a cryptographic hash of its canonical content.

Object key:
objects/sha256/4e/2b/4e2b8f...

Object metadata:
- content_hash
- encryption_key_id
- compressed_size
- uncompressed_size
- content_type
- checksum
- created_at
- replica_locations

If two dataset versions use the same unchanged partition, both manifests point
to the same object hash. The system stores the data only once.

4. WRITE AND COMMIT FLOW

STEP 1: CREATE A WORKSPACE

A user or pipeline begins with a base snapshot:

workspace = checkout(
  repository = "sports-media-analytics",
  branch = "main",
  base_commit = "C1042"
)

The workspace contains:

* Base commit ID.

* Read set.

* Staged additions, deletions, and modifications.

* Schema changes.

* Quality-validation results.

* Pipeline/run metadata.

STEP 2: UPLOAD NEW OR CHANGED DATA

The client splits large files into content-defined or fixed-size chunks,
computes hashes, and uploads only objects not already present.

Input partition: 1 TB
Changed data: 25 GB
Already-existing chunks: 20 GB
New upload: 5 GB

A content-addressed store supports deduplication and immutable object identity:
the same content resolves to the same object hash, while a data change produces
a different hash.[atlan
[https://atlan.com/know/llm-training-data-versioning-strategies/]][thedataops
[https://www.thedataops.org/data-version-control-dvc/]][scalablebook.apartsin
[https://scalablebook.apartsin.com/part-2-distributed-data/module-08-distributed-storage-data-loading/section-8.9.html]]

STEP 3: BUILD A NEW MANIFEST

The client or commit service creates new manifests only for changed partitions
and ancestor nodes. Unchanged manifest entries are reused.

Base Commit: C1042
Changed Partition: event_date=2026-09-28/league=NBA
New Commit: C1043

Only the affected leaf manifest, parent manifest, and root manifest must be
rewritten.

STEP 4: VALIDATE BEFORE COMMIT

The commit service runs configurable gates:

* Schema compatibility.

* Required-field validation.

* Duplicate-record thresholds.

* Null-rate and completeness checks.

* Referential-integrity checks.

* Partition and event-time consistency.

* PII/classification policies.

* Data-quality expectations.

* ML-data validation, if relevant.

* Policy checks for protected branches.

For live sports/media analytics, this is particularly important. A data version
might include not only raw event feeds, but also source identity, event time,
ingestion time, feed quality, model version, enrichment version, and
reconciliation state.

STEP 5: ATOMICALLY ADVANCE BRANCH HEAD

The commit is created as immutable metadata. The branch head is advanced only if
it still points to the expected base commit.

UPDATE branches
SET head_commit_id = 'C1043',
    version_number = version_number + 1
WHERE repository_id = 'sports-media-analytics'
  AND branch_name = 'main'
  AND head_commit_id = 'C1042';

This is a compare-and-swap operation.

* If one row is updated, commit succeeds.

* If zero rows are updated, another writer advanced the branch first.

* The losing writer rebases, merges, or retries.

A common distributed version-control pattern is to write immutable content first
and then atomically conditionally update the small mutable “latest
pointer.”[epicgames.github
[https://epicgames.github.io/lore/explanation/system-design/]]

5. READ AND CHECKOUT FLOW

A read request resolves a repository and version reference:

GET dataset://sports-media-analytics/main/game-events
GET dataset://sports-media-analytics/commit/C1042/game-events
GET dataset://sports-media-analytics/tag/championship-final/game-events

The system performs:

1. Resolve branch/tag/commit to a stable commit ID.

2. Fetch the root manifest.

3. Prune partitions based on query predicates.

4. Resolve required object hashes.

5. Read immutable objects from the nearest replica/cache.

6. Return data plus version metadata.

For example:

SELECT *
FROM versioned_dataset(
  repo => 'sports-media-analytics',
  ref  => 'main',
  path => 'game-events'
)
WHERE league = 'NBA'
  AND event_date = '2026-09-28';

The query engine pins the resolved commit ID at the beginning of the query. Even
if main advances while the query is running, the query sees a consistent
snapshot.

6. BRANCHING, MERGING, AND CONFLICT RESOLUTION

BRANCHING

Creating a branch is inexpensive because it creates only a pointer:

CREATE BRANCH playoffs
FROM main@C1042

No data is copied. Both branches share immutable objects until one branch
changes a partition.

This makes it useful for:

* Data-science experiments.

* Backfills.

* New enrichment pipelines.

* Partner-specific transformations.

* Schema migrations.

* Historical correction work.

* Testing new quality rules.

MERGE

A merge identifies a common ancestor and compares logical paths or partition
keys.

main:      C1042 → C1043 → C1044
playoffs:  C1042 → C1050 → C1051

merge(playoffs → main)

Three-way merge algorithm:

base = C1042
target = C1044
source = C1051

For each logical key:
    If target == base and source != base:
        take source
    If source == base and target != base:
        take target
    If source == target:
        take either
    Otherwise:
        conflict

CONFLICT TYPES

Conflict typeExampleResolutionFile/partition conflictBoth branches change the
same Parquet partitionSelect one version, run semantic merge, or manually
resolveRow-level conflictBoth branches update the same player-event recordUse
primary key + record version + domain rulesSchema conflictOne branch renames a
field while another adds a dependent transformationRequire schema migration
reviewDelete/update conflictOne branch deletes a partition, another modifies
itExplicit human or policy-based decisionQuality-policy conflictSource branch
does not meet main-branch quality thresholdsBlock merge until checks pass

For append-only event data, merges can often be automatic using deterministic
partitioning and event IDs. For mutable records—such as corrections to player
metadata, royalty calculations, or ad-attribution records—the system needs
record-level conflict semantics.

7. HANDLING STREAMING DATA

For a near-real-time system, I would not create a commit for every event. That
would create excessive metadata churn.

Instead, I would use a two-stage model:

Live stream → mutable ingestion buffer → micro-batch seal → immutable commit

STREAMING FLOW

1. Events arrive through Kafka, Event Hubs, or equivalent.

2. The ingestion service writes append-only raw events to a durable landing
    zone.

3. A stream processor deduplicates, validates, enriches, and groups events by
    event time.

4. Every configurable interval—such as one minute or five minutes—the system
    seals a micro-batch.

5. The sealed micro-batch becomes an immutable partition/object set.

6. A commit captures the newly available data and associated quality metadata.

7. Late-arriving events create correction commits rather than mutating
    historical data.

Example:

Commit C1200: Game 7 events through 19:30:00 UTC
Commit C1201: Events from 19:30:00–19:31:00 UTC
Commit C1202: Late corrections for 19:28:00–19:30:00 UTC

Consumers can choose:

* Live view: Latest branch head, accepts low-latency provisional data.

* Validated view: Latest commit satisfying quality/completeness criteria.

* Final view: Reconciled version after the event closes.

This supports both immediate production use and accurate downstream reporting,
including revenue- or royalty-sensitive workflows.

8. CONSISTENCY AND MULTI-REGION DESIGN

CONTROL PLANE

For branch-head updates, tags, permissions, and commit publication, I would use
strongly consistent transactional metadata storage:

* Globally consistent SQL database.

* Consensus-backed key-value store.

* Single-writer-per-repository or per-branch leadership, where appropriate.

* Conditional write/CAS semantics.

The critical invariant is:

> A commit is never partially visible. A reader sees either the old valid
> snapshot or the new valid snapshot.

DATA PLANE

Immutable data objects can replicate asynchronously across regions because their
hash verifies integrity.

Region A: Primary object write
Region B: Asynchronous replica
Region C: Asynchronous replica

For a commit to become visible globally, I would choose based on durability
requirements:

* Standard mode: Publish after primary-region durable write; remote reads may
   temporarily fetch from primary.

* Critical mode: Publish only after replication quorum or required-region
   replication completes.

* Live-event mode: Publish metadata quickly but route readers to the source
   region until replicas are ready.

CONFLICT AVOIDANCE

For high-write workloads, avoid many writers directly committing to the same
branch. Use one of these patterns:

* Partition ownership: one writer per league/game/date partition.

* Per-producer branches merged by an orchestrator.

* Append-only event log plus periodic compaction commits.

* Optimistic concurrency with compare-and-swap on branch heads.

9. METADATA, LINEAGE, AND GOVERNANCE

Every commit should include more than a pointer to files. It should be a
complete reproducibility record.

{
  "commit_id": "C1043",
  "parents": ["C1042"],
  "author": "live-event-ingestion-pipeline",
  "pipeline_version": "git:8d2a91e",
  "schema_version": "event-schema-v14",
  "source_feeds": [
    {
      "provider": "provider-A",
      "feed_version": "2026.09.28.1",
      "arrival_window": "19:30:00-19:35:00Z"
    }
  ],
  "quality_metrics": {
    "completeness": 0.998,
    "duplicate_rate": 0.0004,
    "late_event_rate": 0.003,
    "schema_validation": "passed"
  },
  "model_versions": {
    "entity_extraction": "v8.3",
    "highlight_classifier": "v12.1"
  },
  "approval_state": "validated"
}

This is especially important for:

* Reproducing ML training data.

* Explaining an analytics result.

* Auditing partner reports.

* Investigating a data-quality incident.

* Rolling back a problematic model or transformation.

* Comparing data before and after source corrections.

Data-versioning practices often use immutable snapshots plus lightweight
metadata pointers precisely to make reproducibility and rollback practical
without duplicating large objects.[atlan
[https://atlan.com/know/llm-training-data-versioning-strategies/]][thedataops
[https://www.thedataops.org/data-version-control-dvc/]]

10. GARBAGE COLLECTION AND RETENTION

Because objects are immutable and shared among commits, deletion must be
careful.

MARK-AND-SWEEP PROCESS

1. Start from all protected references:
    
    * Branch heads.
    
    * Tags.
    
    * Legal holds.
    
    * Active workspaces.
    
    * Retention-policy-protected commits.
    
    * Published reports or model versions.

2. Traverse manifests and mark reachable object hashes.

3. Identify unreferenced objects older than a safety threshold.

4. Soft-delete or move to a quarantine tier.

5. Permanently delete after the recovery window expires.

I would never run immediate deletion after a branch deletion because:

* Clients may have cached metadata.

* A commit may be referenced by an external model or report.

* Cross-region replication may still be in progress.

* Users may need recovery from accidental deletion.

11. SECURITY AND COMPLIANCE

The system should provide:

* Tenant and repository isolation.

* Role-based and attribute-based access control.

* Read, write, branch, merge, delete, and admin permissions.

* Encryption in transit and at rest.

* Per-tenant or per-repository encryption keys where needed.

* Immutable audit logs.

* PII classification and policy enforcement.

* Legal hold and retention policies.

* Masked or tokenized views for sensitive datasets.

* Approval workflows for production-branch merges.

* Signed commits or workload identities for automated pipelines.

For a media analytics system, I would also consider access boundaries for raw
audio/video, rights-managed content, personally identifiable information, and
partner-provided data.

12. APIS

CORE VERSIONING APIS

POST   /repos
POST   /repos/{repo}/branches
GET    /repos/{repo}/branches/{branch}
POST   /repos/{repo}/commits
GET    /repos/{repo}/commits/{commitId}
POST   /repos/{repo}/merge
POST   /repos/{repo}/tags
POST   /repos/{repo}/rollback
GET    /repos/{repo}/diff
GET    /repos/{repo}/lineage
POST   /repos/{repo}/validate

EXAMPLE: CREATE COMMIT

POST /repos/sports-media-analytics/commits

{
  "branch": "main",
  "expected_head": "C1042",
  "root_manifest_hash": "sha256:abc123...",
  "message": "Add corrected live-event feed for 2026-09-28",
  "schema_version": "event-schema-v14",
  "quality_report_id": "QR-32891",
  "pipeline_run_id": "airflow:live-ingest:20260928-1930"
}

EXAMPLE: SUCCESSFUL RESPONSE

{
  "commit_id": "C1043",
  "branch": "main",
  "previous_head": "C1042",
  "new_head": "C1043",
  "status": "committed",
  "created_at": "2026-09-28T23:31:14Z"
}

13. KEY TRADEOFFS

DecisionChoiceWhyData storageImmutable content-addressed objectsDeduplication,
integrity, sharing, reproducibilityMetadata storageTransactional control
planeAtomic commit publication and protected branch operationsConsistencyStrong
for references; eventual for immutable replicasCorrect version visibility
without unnecessary global write latencyVersion granularityPartition/file by
default; row-level only where requiredControls metadata and merge
complexityStreaming commitsMicro-batches plus correction commitsBalances low
latency with manageable metadata volumeConflict approachOptimistic concurrency
and three-way mergeScales collaboration without global lockingQuality
enforcementCommit-time validation gatesPrevents low-quality versions from
becoming production referencesDeletionReachability-based garbage
collectionAvoids deleting shared historical data

14. IMPORTANT FAILURE MODES

OBJECT UPLOADED, BUT COMMIT NEVER COMPLETES

This creates an orphaned immutable object. It is safe because no branch
references it. Garbage collection removes it after the grace period.

COMMIT METADATA CREATED, BUT BRANCH-HEAD UPDATE FAILS

The commit is immutable but unreachable from the target branch. The client can
rebase and retry, create a new branch, or the system can retain it temporarily
for recovery.

TWO WRITERS COMMIT CONCURRENTLY

The first conditional branch-head update succeeds. The second receives a
conflict, refreshes the latest branch head, performs a merge/rebase, and
retries.

REGION FAILS AFTER COMMIT PUBLICATION

Readers use the immutable object hash to fetch from another replica or the
primary region. The control plane preserves the commit identity and branch
mapping.

BAD SOURCE DATA REACHES A COMMIT

The version is never overwritten. The team creates a corrected commit, rolls the
production branch back or forward to the corrected version, records the
incident, and preserves lineage for audit.

15. STRONG INTERVIEW CLOSING

A concise way to conclude is:

> “The core of my design is immutable, content-addressed data combined with a
> small strongly consistent metadata layer. A commit is a reproducible snapshot
> represented by a Merkle-tree manifest; branches are mutable pointers to
> commits; and the system uses atomic compare-and-swap to publish changes
> safely. This allows petabyte-scale data versioning without copying unchanged
> data, while supporting collaboration, streaming micro-batches, rollback,
> lineage, quality gates, and high-confidence reporting. I would then tailor the
> consistency model, merge granularity, and retention policies to the data types
> and business requirements.”

Design a system for managing a distributed data versioning system.

Practice this question with AI

Go Premium

Community Answers

Unlock Full Access

Practice More Questions