Getting started with ancestree¶

ancestree records the lineage of the artifacts a pipeline produces: every step is saved as a node that knows which node it came from, what files it wrote and what metadata describes it.

This notebook covers the essentials — defining a ruleset, creating a store, writing a node, looking it back up and rendering the result — using nothing more than a single synthetic array.

1. Imports¶

In [1]:

import numpy as np
import pandas as pd

import ancestree

import numpy as np import pandas as pd import ancestree

2. Define a ruleset¶

The ruleset describes how a pipeline is allowed to progress. Each key is a step type and its value lists the step types it may follow, where [None] marks a valid starting point. Separately, trigger names the step(s) that begin a new generation — a fresh run of the pipeline.

You only need to declare this once: it is written to .lineage_config.json inside the store and read back automatically whenever the store is reopened by name.

In [2]:

RULES = {
    "import data": [None],  # a pipeline starts by importing data
}

RULES = { "import data": [None], # a pipeline starts by importing data }

3. Create the store¶

The store lives in a folder of its own. Its name can be a str or a Path. Passing the rules and gen_triggers here writes the configuration to disk; from then on ancestree.LineageStore("basic_usage") is enough to reopen it.

In [3]:

store = ancestree.LineageStore("basic_usage", rules=RULES)

store = ancestree.LineageStore("basic_usage", rules=RULES)

4. Write a node¶

store.create_node(...) opens a node as a context manager. Inside the block, writing files uses familiar syntax — the store handles pathing via node / "filename" — and node.add_meta(...) attaches searchable, displayable metadata. Here a pandas DataFrame is stored as metadata (it renders as a table in the web graph) alongside a saved array.

In [4]:

with store.create_node(step_type="import data") as node:
    # Save a file
    X = np.random.randint(0, 100, (5, 5))
    np.save(node / "samples.npy", X)

    # Add some metadata
    node.add_meta("something", pd.DataFrame({"key": [1, 2, 3], "other": [4, 5, 6]}))

with store.create_node(step_type="import data") as node: # Save a file X = np.random.randint(0, 100, (5, 5)) np.save(node / "samples.npy", X) # Add some metadata node.add_meta("something", pd.DataFrame({"key": [1, 2, 3], "other": [4, 5, 6]}))

5. Look the node back up¶

get_most_recent_node() returns the last node written to the store.

In [5]:

n = store.get_most_recent_node()
n

n = store.get_most_recent_node() n

Out[5]:

Node = b4143d71, path = b4143d71, generation = 0

6. Visualise the store¶

generate_web_graph() writes a self-contained, interactive HTML page showing every node, its lineage and its metadata.

In [6]:

store.generate_web_graph()

store.generate_web_graph()

Graph generated at basic_usage/interactive_pipeline.html

Out[6]:

PosixPath('basic_usage/interactive_pipeline.html')