Deduplicating identical nodes¶

When a store is opened with dedupe=True, running a step that is content-identical to one already in the store does not create a second node. create_node reuses the existing node and rebinds your with ... as node variable onto it.

Two nodes are content-identical when they share:

• the same step_type,
• the same parent,
• the same user metadata, and
• byte-identical artifacts.

Volatile fields (node_id, timestamp, duration_s, size_mb) and provenance (user, platform, git state) are ignored, so re-running the same step on a different day still deduplicates. The content hash is only a fast lookup — a candidate is byte-verified before reuse, so the saved data really is identical.

This notebook walks through three cases: identical nodes (reused), near-identical nodes (kept separate), and completely different nodes (kept separate).

1. Set up a deduplicating store¶

In [1]:

import shutil
from pathlib import Path

import numpy as np

import ancestree

# Start from a clean store so the notebook is reproducible on re-run.
STORE_DIR = Path("dedupe_demo")
shutil.rmtree(STORE_DIR, ignore_errors=True)

# dedupe=True is the only thing that turns deduplication on.
store = ancestree.LineageStore(STORE_DIR, dedupe=True)


def show(store):
    # List the node directories currently on disk.
    ids = sorted(d.name for d in store.root.iterdir() if d.is_dir())
    print(f"{len(ids)} node(s) on disk: {ids}")

import shutil from pathlib import Path import numpy as np import ancestree # Start from a clean store so the notebook is reproducible on re-run. STORE_DIR = Path("dedupe_demo") shutil.rmtree(STORE_DIR, ignore_errors=True) # dedupe=True is the only thing that turns deduplication on. store = ancestree.LineageStore(STORE_DIR, dedupe=True) def show(store): # List the node directories currently on disk. ids = sorted(d.name for d in store.root.iterdir() if d.is_dir()) print(f"{len(ids)} node(s) on disk: {ids}")

2. Identical nodes → reused¶

We wrap a step in a function so we can run it twice with exactly the same inputs. The same seed produces the same array, which np.save writes as byte-identical files, and the metadata is the same — so the second run is content-identical to the first.

In [2]:

def run_ingest(store, seed, rows):
    rng = np.random.default_rng(seed)
    with store.create_node(step_type="ingest") as node:
        np.save(node / "samples.npy", rng.integers(0, 100, size=(rows, 3)))
        node.add_meta("rows", rows)
        node.add_meta("seed", seed)
    return node


first = run_ingest(store, seed=42, rows=1000)
store.generate_web_graph()

def run_ingest(store, seed, rows): rng = np.random.default_rng(seed) with store.create_node(step_type="ingest") as node: np.save(node / "samples.npy", rng.integers(0, 100, size=(rows, 3))) node.add_meta("rows", rows) node.add_meta("seed", seed) return node first = run_ingest(store, seed=42, rows=1000) store.generate_web_graph()

Out[2]:

PosixPath('dedupe_demo/interactive_pipeline.html')

In [3]:

second = run_ingest(store, seed=42, rows=1000)  # same inputs, run a moment later
store.generate_web_graph()

print("first  node_id:", first.node_id)
print("second node_id:", second.node_id)
print("reused the existing node?", first.node_id == second.node_id)
show(store)

second = run_ingest(store, seed=42, rows=1000) # same inputs, run a moment later store.generate_web_graph() print("first node_id:", first.node_id) print("second node_id:", second.node_id) print("reused the existing node?", first.node_id == second.node_id) show(store)

first  node_id: b0c09bf1
second node_id: b0c09bf1
reused the existing node? True
1 node(s) on disk: ['b0c09bf1']

The two runs happened at different times — their timestamps differ — yet they share a node_id and only one directory exists on disk. Timestamps and provenance are not part of a node's identity, only its content is.

3. Near-identical nodes → kept separate¶

A single difference is enough to make the content hash diverge. The store re-hashes, finds no match, and creates a new node.

3a. One byte of the artifact changes¶

In [4]:

with store.create_node(step_type="ingest") as node:
    rng = np.random.default_rng(42)
    data = rng.integers(0, 100, size=(1000, 3))
    data[0, 0] += 1  # flip a single value -> different bytes on disk
    np.save(node / "samples.npy", data)
    node.add_meta("rows", 1000)
    node.add_meta("seed", 42)  # metadata is identical to the first run
byte_diff = node

print("one byte changed -> node_id:", byte_diff.node_id)
print("merged with the identical run?", byte_diff.node_id == first.node_id)
show(store)

store.generate_web_graph()

with store.create_node(step_type="ingest") as node: rng = np.random.default_rng(42) data = rng.integers(0, 100, size=(1000, 3)) data[0, 0] += 1 # flip a single value -> different bytes on disk np.save(node / "samples.npy", data) node.add_meta("rows", 1000) node.add_meta("seed", 42) # metadata is identical to the first run byte_diff = node print("one byte changed -> node_id:", byte_diff.node_id) print("merged with the identical run?", byte_diff.node_id == first.node_id) show(store) store.generate_web_graph()

one byte changed -> node_id: 9104e550
merged with the identical run? False
2 node(s) on disk: ['9104e550', 'b0c09bf1']

Out[4]:

PosixPath('dedupe_demo/interactive_pipeline.html')

3b. Artifacts identical, one metadata value changes¶

In [5]:

with store.create_node(step_type="ingest") as node:
    rng = np.random.default_rng(42)
    np.save(node / "samples.npy", rng.integers(0, 100, size=(1000, 3)))  # same bytes
    node.add_meta("rows", 1000)
    node.add_meta("seed", 99)  # only this differs from the first run
meta_diff = node

print("one metadata value changed -> node_id:", meta_diff.node_id)
print("merged with the identical run?", meta_diff.node_id == first.node_id)
show(store)

store.generate_web_graph()

with store.create_node(step_type="ingest") as node: rng = np.random.default_rng(42) np.save(node / "samples.npy", rng.integers(0, 100, size=(1000, 3))) # same bytes node.add_meta("rows", 1000) node.add_meta("seed", 99) # only this differs from the first run meta_diff = node print("one metadata value changed -> node_id:", meta_diff.node_id) print("merged with the identical run?", meta_diff.node_id == first.node_id) show(store) store.generate_web_graph()

one metadata value changed -> node_id: aa2e228a
merged with the identical run? False
3 node(s) on disk: ['9104e550', 'aa2e228a', 'b0c09bf1']

Out[5]:

PosixPath('dedupe_demo/interactive_pipeline.html')

4. Nothing alike → kept separate¶

A different step type, different metadata and a different artifact obviously produce a distinct node.

In [6]:

with store.create_node(step_type="report") as node:
    (node / "summary.txt").write_text("Quarterly revenue up 12%.")
    node.add_meta("author", "analytics-team")
    node.add_meta("quarter", "Q2")
different = node

print("completely different -> node_id:", different.node_id)
show(store)

store.generate_web_graph()

with store.create_node(step_type="report") as node: (node / "summary.txt").write_text("Quarterly revenue up 12%.") node.add_meta("author", "analytics-team") node.add_meta("quarter", "Q2") different = node print("completely different -> node_id:", different.node_id) show(store) store.generate_web_graph()

completely different -> node_id: aaa602ab
4 node(s) on disk: ['9104e550', 'aa2e228a', 'aaa602ab', 'b0c09bf1']

Out[6]:

PosixPath('dedupe_demo/interactive_pipeline.html')

5. Re-running a whole pipeline¶

Because reuse keeps the lineage a tree, re-running an entire identical pipeline reuses every node: the root is reused, so its child now has a matching parent and is reused too, all the way down.

In [9]:

def build_pipeline(store):
    rng = np.random.default_rng(0)
    with store.create_node(step_type="raw") as raw:
        np.save(raw / "raw.npy", rng.integers(0, 10, size=(50,)))
    with store.create_node(step_type="clean", parent=raw) as clean:
        (clean / "clean.csv").write_text("a,b\n1,2\n")
        clean.add_meta("dropped", 3)
    with store.create_node(step_type="model", parent=clean) as model:
        (model / "model.pkl").write_text("weights")
        model.add_meta("accuracy", 0.91)
    return raw, clean, model


run1 = build_pipeline(store)
store.generate_web_graph()

def build_pipeline(store): rng = np.random.default_rng(0) with store.create_node(step_type="raw") as raw: np.save(raw / "raw.npy", rng.integers(0, 10, size=(50,))) with store.create_node(step_type="clean", parent=raw) as clean: (clean / "clean.csv").write_text("a,b\n1,2\n") clean.add_meta("dropped", 3) with store.create_node(step_type="model", parent=clean) as model: (model / "model.pkl").write_text("weights") model.add_meta("accuracy", 0.91) return raw, clean, model run1 = build_pipeline(store) store.generate_web_graph()

Out[9]:

PosixPath('dedupe_demo/interactive_pipeline.html')

In [14]:

run2 = build_pipeline(store)  # identical second run

print("run 1:", [n.node_id for n in run1])
print("run 2:", [n.node_id for n in run2])
print("entire pipeline reused?", [n.node_id for n in run1] == [n.node_id for n in run2])
show(store)

run2 = build_pipeline(store) # identical second run print("run 1:", [n.node_id for n in run1]) print("run 2:", [n.node_id for n in run2]) print("entire pipeline reused?", [n.node_id for n in run1] == [n.node_id for n in run2]) show(store)

run 1: ['2800e644', '8b84f0a8', '66c9fdf4']
run 2: ['2800e644', '8b84f0a8', '66c9fdf4']
entire pipeline reused? True
7 node(s) on disk: ['2800e644', '66c9fdf4', '8b84f0a8', '9104e550', 'aa2e228a', 'aaa602ab', 'b0c09bf1']

6. Contrast: the same operations with dedupe=False¶

Deduplication is opt-in. With the default store, identical runs pile up as separate nodes.

In [7]:

PLAIN_DIR = Path("dedupe_off_demo")
shutil.rmtree(PLAIN_DIR, ignore_errors=True)
plain = ancestree.LineageStore(PLAIN_DIR)  # dedupe defaults to False

for _ in range(3):
    run_ingest(plain, seed=42, rows=1000)  # identical every time

show(plain)  # three separate nodes

PLAIN_DIR = Path("dedupe_off_demo") shutil.rmtree(PLAIN_DIR, ignore_errors=True) plain = ancestree.LineageStore(PLAIN_DIR) # dedupe defaults to False for _ in range(3): run_ingest(plain, seed=42, rows=1000) # identical every time show(plain) # three separate nodes

3 node(s) on disk: ['048d3908', 'e8688f33', 'f5c4434e']

7. Visualise the deduplicated store¶

In [8]:

store.generate_web_graph()

store.generate_web_graph()

Out[8]:

PosixPath('dedupe_demo/interactive_pipeline.html')