Skip to content
Duodata
Open formats

Four layers. Never conflated.

Keeping intent, observed structure, exchange and execution separate lets a model survive schema changes and move between tools.

Layers

  1. 01

    Business and mapping model

    What a measure means, who owns it, and which business terms bind to which columns. Authored and reviewed by people.

  2. 02

    Observed schema · schema.yaml

    What physically exists: tables, columns, types and keys, with provenance for each key. Not a semantic model.

  3. 03

    Apache Ossie

    An open format for exchanging semantic models between tools. Separate from schema.yaml and from platform-native output.

  4. 04

    Platform-native output

    Semantic views in Snowflake; Unity Catalog metric views in Databricks. The layer your platform executes.

Observed schema

Schema YAML v1.0: a canonical format example.

Schema YAML captures observed physical structure. Business ownership and definition status are separate facts. schema.yaml is not the Ossie semantic-model format.

schema.yaml · CRM.PUBLICyaml
format: schema.yaml
version: "1.0"
schemas:
  - source:
      database: CRM
      schema: PUBLIC
    tables:
      CUSTOMERS:
        columns:
          BALANCE: {type: "NUMBER(38,2)", is: [], default: "0"}
          CUSTOMER_ID: {type: NUMBER, is: [pk, notnull]}
          NAME: {type: VARCHAR(200), is: [notnull]}
        primary_key: CUSTOMER_ID
        provenance:
          pk_source: inferred
          pk_confidence: 0.74
          has_ddl: false

Key provenance

ddl
Declared constraint from the platform's DDL.
inferred
Inferred from metadata; not yet checked against rows.
verified
Checked against the rows themselves.
pinned
Primary key pinned by a person.

In this example the key on CUSTOMER_ID is inferred with confidence 0.74 and no DDL; it is not labelled verified.

Platform-native output

One fictional revenue model, two native formats

Net revenue = gross revenue − discounts − credits, over a quarterly gold view. The Snowflake database DEMO and the Databricks catalog demo are different runtime objects on different platforms.

Observed gold schema · DEMO.GOLD (Snowflake)yaml
format: schema.yaml
version: "1.0"
schemas:
  - source:
      database: DEMO
      schema: GOLD
      type: snowflake
    tables:
      REVENUE_BY_QUARTER:
        columns:
          CREDITS: {type: "NUMBER(18,2)", is: [notnull]}
          DISCOUNTS: {type: "NUMBER(18,2)", is: [notnull]}
          GROSS_REVENUE: {type: "NUMBER(18,2)", is: [notnull]}
          QUARTER: {type: DATE, is: [notnull]}
        kind: view
Snowflake semantic view · revenue_semantics.yamlyaml
name: revenue_semantics
description: Fictional quarterly revenue example.
tables:
  - name: revenue
    base_table:
      database: DEMO
      schema: GOLD
      table: REVENUE_BY_QUARTER
    time_dimensions:
      - name: quarter
        expr: QUARTER
        data_type: DATE
    facts:
      - name: gross_revenue
        expr: GROSS_REVENUE
        data_type: NUMBER
      - name: discounts
        expr: DISCOUNTS
        data_type: NUMBER
      - name: credits
        expr: CREDITS
        data_type: NUMBER
    metrics:
      - name: net_revenue
        expr: SUM(gross_revenue) - SUM(discounts) - SUM(credits)
Databricks metric view · YAML bodyyaml
version: "1.1"
comment: Fictional quarterly revenue example.
source: demo.gold.revenue_by_quarter
fields:
  - name: quarter
    expr: source.quarter
measures:
  - name: net_revenue
    expr: SUM(source.gross_revenue) - SUM(source.discounts) - SUM(source.credits)
Databricks metric view · demo.gold.revenue_semanticssql
CREATE OR REPLACE VIEW demo.gold.revenue_semantics
WITH METRICS
LANGUAGE YAML
AS $$
version: "1.1"
comment: Fictional quarterly revenue example.
source: demo.gold.revenue_by_quarter
fields:
  - name: quarter
    expr: source.quarter
measures:
  - name: net_revenue
    expr: SUM(source.gross_revenue) - SUM(source.discounts) - SUM(source.credits)
$$;

In Snowflake, facts map physical columns to logical names and the metric aggregates those facts. In Databricks, fields is the current term (dimensions remains an accepted synonym) and the measure defines the aggregate directly.

Fictional example

Scoped proof of concept

Bring the model you want to deploy

We start with one business question, the person who owns its meaning, the data behind it and the result you expect. A scoped, paid proof of concept puts that meaning to work in your platform.

  • 01One business question
  • 02One owner of the definition
  • 03The data it depends on
  • 04The result you expect
  • Bring your use case