Component-Level Versioning for Data Pipelines

Most data pipeline tools version pipelines as monolithic units. When you change a single transform in a dbt project, the entire project gets a new version tag. When you update one task in an Airflow DAG, the whole DAG file is re-parsed and re-deployed. Cupel takes a fundamentally different approach: every component in a pipeline — sources, transforms, quality gates, compliance steps, destinations — is versioned independently. This post explains why we made this decision (ADR-006), how the Component SDK enforces execution contracts, and what component-level versioning enables for teams working at scale.

The Problem with Monolithic Pipeline Versioning

Consider a financial services organization with a pipeline that extracts transaction data from a PostgreSQL source, applies KYC compliance checks, runs quality validation, transforms the data through several aggregation steps, and loads the results into Snowflake. This pipeline has been stable in production for three months.

Now the team needs to update the quality validation threshold from 95% to 98%. In a monolithic versioning model, this change requires a new version of the entire pipeline. The deployment process re-validates everything: the source connector, the compliance checks, the transforms, the Snowflake loader. Every component is re-tested, re-deployed, and re-certified as a unit.

This creates several problems.

Change Amplification

A one-line threshold change triggers a full pipeline deployment cycle. In regulated environments, this means the compliance team must re-review and re-approve the entire pipeline, not just the changed component. Deployment velocity drops because trivial changes carry the same review overhead as major refactors.

Version Coupling

If two teams share a common transform component and that component gets updated in one pipeline, the other pipeline is unaffected in practice but must eventually adopt the new version through its own full re-deployment cycle. There is no mechanism to update the shared component independently.

Rollback Granularity

If a production issue is caused by a single component update, rolling back requires reverting the entire pipeline to a prior version. This reverts all changes, including unrelated improvements to other components that were working correctly.

Cupel's Component-Level Versioning Model

In Cupel, each pipeline component is a first-class entity with its own version, configuration schema, execution contract, and lifecycle. Components are assembled on the React Flow canvas, but they are stored, versioned, and managed independently in the component registry.

The Component Registry

The registry is a versioned catalog of all available components. Each entry contains the component definition, its version history, and its execution contract:

@dataclass(frozen=True)
class ComponentVersion:
    """Immutable snapshot of a component at a specific version."""
    component_id: str
    version: str            # Semantic versioning: major.minor.patch
    component_type: str     # source, transform, quality_gate, compliance, destination
    config_schema: dict     # JSON Schema for component configuration
    input_contract: dict    # Expected input format and columns
    output_contract: dict   # Guaranteed output format and columns
    execution_fn: str       # Reference to the activity function
    default_timeout: int    # Default timeout in seconds
    default_failure_strategy: str
    changelog: str          # What changed in this version
    created_at: datetime
    deprecated: bool
    deprecation_message: str | None

Versions are immutable. Once version 1.2.0 of a transform component is published, it cannot be modified. Fixes and improvements are published as new versions. This guarantees reproducibility: a pipeline built with transform-v1.2.0 today will produce the same results when executed six months from now.

The BaseComponent Abstract Class

All components implement the BaseComponent abstract class, which defines the execution contract every component must satisfy:

from abc import ABC, abstractmethod
from dataclasses import dataclass

@dataclass
class ComponentInput:
    """Standardized input passed to every component."""
    data_uri: str           # URI to input data (S3, GCS, ADLS path)
    config: dict            # Component-specific configuration
    context: ExecutionContext  # Pipeline context (org_id, run_id, etc.)

@dataclass
class ComponentOutput:
    """Standardized output returned by every component."""
    data_uri: str           # URI to output data
    row_count: int          # Number of rows processed
    schema: dict            # Output schema
    quality_metrics: dict   # Component-level quality metrics
    lineage: LineageRecord  # Column-level lineage for this component

class BaseComponent(ABC):
    """Abstract base class for all pipeline components.

    Every component — source, transform, quality gate, compliance step,
    destination — implements this interface. The pipeline compiler relies
    on this contract to generate correct Temporal activity invocations.
    """

    @abstractmethod
    def validate_config(self, config: dict) -> list[str]:
        """Validate component configuration before execution.

        Returns a list of validation error messages. An empty list
        indicates valid configuration.
        """
        ...

    @abstractmethod
    async def execute(self, input: ComponentInput) -> ComponentOutput:
        """Execute the component's core logic.

        This method is invoked by the Temporal activity worker.
        It receives standardized input and must return standardized
        output, regardless of the component type.
        """
        ...

    @abstractmethod
    def get_lineage(self, config: dict) -> LineageMapping:
        """Declare the column-level lineage mapping for this component.

        Called by the pipeline compiler during compilation to build
        the lineage graph. Does not execute the component.
        """
        ...

Version Pinning in Pipelines

When a component is placed on the canvas, the pipeline definition records the exact version:

{
  "nodes": [
    {
      "id": "node_1",
      "type": "source",
      "data": {
        "component_id": "connector-postgresql",
        "version": "2.1.0",
        "config": {
          "connection_string_ref": "vault://pg-prod",
          "table": "transactions",
          "incremental_column": "updated_at"
        }
      }
    },
    {
      "id": "node_2",
      "type": "quality_gate",
      "data": {
        "component_id": "quality-gate-financial",
        "version": "1.4.2",
        "config": {
          "threshold": 0.98,
          "checks": ["schema_validation", "null_ratio", "uniqueness"]
        }
      }
    },
    {
      "id": "node_3",
      "type": "transform",
      "data": {
        "component_id": "transform-aggregate",
        "version": "3.0.1",
        "config": {
          "group_by": ["account_id", "transaction_date"],
          "aggregations": [
            {"column": "amount", "function": "sum", "alias": "total_amount"}
          ]
        }
      }
    }
  ]
}

Each node pins to a specific version. Updating the quality gate from 1.4.2 to 1.5.0 does not affect the PostgreSQL source or the aggregate transform. The pipeline compiler resolves each component independently, and only the changed component goes through validation and testing.

Updating a Single Component

The workflow for updating a component in a running pipeline is straightforward:

1. The UI notifies the pipeline builder that a newer version of a component is available (e.g., quality-gate-financial has v1.5.0 with improved null detection).
2. The builder reviews the changelog and decides to upgrade.
3. The builder selects the component on the canvas and chooses the new version from a dropdown.
4. The pipeline compiler re-validates only the changed component and its downstream connections. Upstream components are unaffected.
5. The updated pipeline is deployed. Only the quality gate activity uses the new version. All other activities remain on their pinned versions.

def validate_version_upgrade(
    pipeline_ir: PipelineIR,
    node_id: str,
    new_version: str,
    registry: ComponentRegistry,
) -> list[str]:
    """Validate that a component version upgrade is compatible.

    Checks that the new version's output contract is compatible
    with the input contracts of all downstream components. This
    prevents breaking changes from propagating through the pipeline.

    Args:
        pipeline_ir: Current pipeline intermediate representation.
        node_id: The node being upgraded.
        new_version: The target version string.
        registry: Component registry for fetching version definitions.

    Returns:
        List of compatibility warnings. Empty list means safe to upgrade.
    """
    current_stage = pipeline_ir.get_stage(node_id)
    new_component = registry.get(current_stage.component.component_id, new_version)

    warnings = []

    # Check output contract compatibility with downstream consumers
    downstream_stages = pipeline_ir.get_downstream(node_id)
    for ds in downstream_stages:
        if not is_schema_compatible(new_component.output_contract, ds.component.input_contract):
            warnings.append(
                f"Output schema of {new_component.component_id}@{new_version} "
                f"is incompatible with input of {ds.component.component_id}@{ds.component.version}"
            )

    return warnings

Contrast with Monolithic Tools

dbt: Project-Level Versioning

dbt versions the entire project as a Git repository. Changing one model requires committing, pushing, and deploying the full project. While dbt recently introduced model versioning for breaking changes in published models, the deployment unit remains the project. You cannot deploy a single model update without deploying every other changed model in the same commit.

Airflow: DAG-Level Versioning

Airflow DAGs are Python files that the scheduler re-parses on every heartbeat. There is no formal versioning mechanism for individual tasks within a DAG. When the DAG file changes, all tasks in the DAG pick up the new definition. Running DAG instances may behave unpredictably if the DAG structure changes mid-execution.

Cupel: Component-Level Independence

Cupel's model is closer to how software libraries work. A pipeline is like an application that depends on specific library versions. Updating one dependency does not require updating all others. Pinning ensures reproducibility. Changelogs communicate what changed. And compatibility checks prevent breaking changes from reaching production.

The Component Marketplace

Component-level versioning is the foundation for Cupel's cross-team component marketplace (Phase 2). When components are independently versioned with clear execution contracts, they become shareable:

• Team A builds a high-quality KYC compliance component and publishes compliance-kyc@2.0.0 to the organization's component library.
• Team B discovers this component in the marketplace, reviews its changelog and quality metrics, and adds it to their pipeline pinned at v2.0.0.
• When Team A publishes v2.1.0 with improved sanctions list matching, Team B is notified but continues running v2.0.0 until they choose to upgrade.

Without independent versioning, sharing components across teams would require coordinating deployment schedules and risking unintended side effects when one team's update affects another team's pipeline.

Execution Contract Guarantees

The BaseComponent abstract class enforces a contract that makes component-level versioning practical. Every component, regardless of what it does internally, must:

1. Accept standardized input (data URI, configuration, execution context).
2. Return standardized output (data URI, row count, schema, quality metrics, lineage).
3. Declare its lineage mapping at compile time (not runtime).
4. Validate its configuration before execution.

This contract means the pipeline compiler can treat all components uniformly. It does not need to know whether a node is a PostgreSQL source or a machine learning transform. It only needs to verify that the output contract of one component is compatible with the input contract of the next.

What This Means for Cupel Users

Component-level versioning gives pipeline builders the confidence to update individual pieces of their pipelines without risking the stability of the whole. Teams in regulated industries can re-certify a single component change rather than re-certifying an entire pipeline. Platform teams can publish improved components to the organization library knowing that consumers will adopt them on their own schedule. And when something goes wrong, operators can roll back a single component to its prior version in seconds, without reverting unrelated changes that were working correctly.