The Columna Framework

A Manual for the Column-Operator-Frame substrate — the grammar layer beneath the tools, where silent failure has no syntax.

Sixth Edition — reconciled through ADR-035 (the column-processing foundation: relating columns is transport in the engine, the backend delivers columns and never combines them, fan-out is inexpressible, and a many-to-many aggregate-across is a Clarify rather than a defaulted membership). Renamed (framework → Columna, the data-space object → Manifold, the query language → Frame-QL) and repositioned through ADR-029. Reconciled forward from ADR-031 through ADR-032 (the Columna Engine as the runtime unit; the planner/Combiner/Cache(r) register), ADR-033 (vocabulary reconciliation + the concordance), ADR-034 (the definition-language scope), and ADR-035 (the query surface).

Preface

There are two activities hidden inside every act of analytics, and almost every tool ever built has forced them together.

The first activity is saying what you want. “Revenue by region, last quarter.” That is the whole of the analytical intent — a dozen words, perfectly clear, and almost never the source of any error. Nobody misunderstands what “revenue by region” means.

The second activity is writing the procedure that computes it. Which tables to read, how to join them, in what order to filter and group, how to avoid counting a customer’s revenue twice because they appear in three rows of a joined table, which reducer is safe along which axis. This is fifty lines where the first was twelve words. It carries nearly all of the effort, and — this is the part that matters — it carries nearly all of the error. The silent failures that haunt analytics — the dashboard that has been wrong by a third for two years, the board metric that disagrees with the operating metric for reasons no one can locate, the “average” that is secretly an average of averages — are not failures of intent. They are failures of procedure. People know what they want. They misimplement it.

Columna begins from the conviction that these two activities can and should be separated. That an analyst should be able to say what they want — to express a need, clearly and verifiably — without also having to author the procedure that fulfills it. That the procedural knowledge can be captured once, by someone who understands the data, encoded into the structure of a logical data space, verified against reality, and then reused by everyone who queries that space, so that no individual query ever has to carry it again.

When that separation is achieved, something changes about what analytics is. The analyst stops being a programmer of data manipulations and becomes a requester of data. The query stops being a procedure and becomes a specification — a clear statement of what is wanted, which the system is responsible for fulfilling correctly, or, where the statement does not determine an answer, for asking rather than guessing. And the silent errors largely vanish — not because the system refuses to compute, but because there is nothing left for them to hide in: the procedure cannot diverge from the question, the data conditions are verified or honestly tagged, and whatever risk remains travels on the number’s face. You cannot make a procedural mistake in a procedure you are not writing.

Closely paired with this is a second commitment that the framework takes equally seriously: the data behind any analysis must itself satisfy certain conditions, or no analysis on it can be correct. These conditions — functional dependencies that hold, anchors that uniquely identify, hierarchies that compose, universes that are coherent — are not Columna’s inventions. They are properties that any correct analysis would require, regardless of what tool performs it. Conventional tools simply don’t check them, and so produce wrong answers silently when they fail. Columna checks them, names the defects when they are present, and makes sure no number that depends on them ever travels without its condition on its face: where a sound route around a defect exists, the framework takes it and says so; where none exists, the producible number is served carrying the defect — named, quantified, directional where determinable — as its permanent companion. The framework’s contribution to integrity is not adding rules; it is uncovering rules that have always been there, and disclosing, on every result, how the data stands against them.

These two commitments together produce a consequence that has become decisive at this moment in analytics: Columna is the analytical substrate AI agents need to serve business questions reliably. A language model that authors SQL against a conventional warehouse is authoring a procedure — making the same procedural choices that produce silent errors when humans make them, with no substrate that can verify the choices were correct. A Columna agent authors a specification against a substrate that validates it before execution and disclosure-binds everything it serves. Every query the agent proposes can be checked end-to-end without touching data — EXPLAIN returns the framework’s exact reading: the canonical form, the plan, the dependency cone with its verdicts, the complete would-be annotation; underdetermined proposals come back as structured clarifications the agent can act on; and every executed query returns (result, annotation) — a number that is either sound under conditions the framework verified, or carries its exact risk, quantified, on its face. The agent is bounded by the framework’s verifications and disclosures rather than by the limits of its own pattern-matching, and the improvement over text-to-SQL is not a better model but a structurally different substrate. The framework’s design did not bolt this on for AI agents; AI agents simply are the kind of analytical consumer that benefits most directly from the separation of specification from procedure and the verification of data conditions — both of which Columna was building toward for its own reasons, and both of which turn out to be exactly what makes an AI-driven analytical agent trustworthy.

This is the central idea of Columna, and this manual is an account of how it is achieved. The book is opinionated, because the design is. Where there are choices, it will tell you what Columna chose and why. Where conventional practice disagrees, it will say so. The goal is that you finish understanding how the framework thinks — well enough to use it, and well enough to argue with it when you disagree.

A word on who this is for: data engineers and analytics engineers who will author Manifolds, and the more technically inclined analysts who will query them. It assumes you know the working vocabulary of analytics — tables, columns, aggregation, joins, dimensions, measures — but not type theory, not category theory, and not the academic literature on data warehousing. Where Columna rests on a deeper idea than industry tooling usually invokes, the manual builds that idea from the ground up. The formal query language has its own companion reference; this book is about concepts, not syntax. A companion Columna Reference Manual specifies the framework’s contracts in detail; this book is about the ideas those contracts rest on.

One sentence carries the whole book, and it is worth pinning before the first chapter — exactly as strong as the system can honestly make it, and no stronger: a number from Columna is either sound under conditions the framework has verified, or it carries its exact risk on its face — never silently wrong, because the question you ask and the procedure that answers it are one object. Where verification can reach, it reaches; where it cannot, the assumption is tagged and the dependence disclosed; and the one unconditional promise in the system, the banner everything beneath it serves, is four words long: never a silent number.

Part I — Foundations

Chapter 1. What Data Is

Most thinking about data begins, without noticing it has made a choice, by treating data as stuff. Values in cells. Records in tables. Digital artifacts that exist in their own right, that can be stored, moved, counted, and joined. Under this view a table is the natural container and a row is the natural atom, and a column is merely a vertical slice of the container — as arbitrary a thing to single out as a vertical strip cut from a photograph.

Columna rejects this starting point, and the rejection is the root from which everything else grows. So it is worth stating the alternative plainly, as a claim about the nature of data itself:

Data is not stuff. Data is about something.

A datum is never a free-floating value. The number 300 is not a datum. “Customer 42’s revenue on January 1st was 300” is a datum. The “aboutness” — that this is revenue, of a customer, on a day — is not decoration added to the number. It is what makes the number mean anything at all. Strip the aboutness away and you do not have a simpler, purer datum; you have a number with no meaning, which is not a datum at all. A datum is irreducibly a property predicated of a referent — a value that is about some specific thing.

This single claim can be framed two ways, and both will matter:

The first framing is the one a database engineer reaches for: the most primitive form of a datum is a key–value pair. The key says what the datum is about; the value says what is so about it. You cannot have less than this and still have something that means anything — a value with no key is meaningless, a key with no value is empty. The key-value pair is the irreducible minimum of a datum.

The second framing is the one a scientist or statistician reaches for: a datum is a property tied to an anchor. The anchor is the referent — the thing the datum is about. The property is what is asserted of it. This is the same structure as key–value, in different words, and Columna uses the word anchor throughout, so this is the framing that bridges the philosophy to the machinery.

These two framings are one idea seen from two sides — the database side and the measurement side — and Columna’s foundation is legible from both. Whichever one you find natural, the conclusion is the same: the atom of data is the property-at-an-anchor, the value tied to what it is about.

From the atom to the column

Once you take seriously that every datum is a property-at-an-anchor, a question answers itself: what naturally groups data together?

Data points group when they are the same kind of assertion about the same kind of referent. All the points of the form “revenue of a customer” — revenue as the property, a customer as the anchor — belong together. They share an anchor type (customer) and a property type (revenue). That natural grouping is what Columna calls a column.

So a column is not a slice chosen by convention. It is the equivalence class that data points fall into once you grant that each point is a property-at-an-anchor: the collection of all data points sharing the same anchor type and the same property type. Put one more way — and this is the form that makes the machinery clean — a column is a function from anchor instances to values. The anchor type (customer) is the function’s domain; the individual anchors (customer 42, customer 43, …) are the points of that domain; the property (their revenue) is the value the function returns at each point. Same anchor type means same domain; same property type means same kind of value; and the column is the function that ties them.

This is why, in Columna, the column is the atom. Not because tables happen to be awkward to work with — though they are — but because the column is what data naturally is once you see data as property-at-anchor. The table, it turns out, was the artifact all along.

Where the table went

If the column is a function from anchors to values, what is a row? A row is “all the different properties that happen to be asserted of the same single anchor” — customer 42’s revenue, and customer 42’s region, and customer 42’s signup date, bundled side by side. It is a horizontal cross-section: many different properties sharing one anchor instance.

There is nothing wrong with a row as a way to present data. But notice it is not primitive. It is assembled — gathered from several columns that happen to share an anchor, bundled at the moment you want to look at them together. In Columna this assembled cross-section has a name: a frame. The row is a frame. The table is a frame. They are the output of bringing columns together, not the raw material columns are cut from. The atom is the column; the frame is what you build from columns when you want to see several properties at once.

This is the inversion at the heart of the framework, and it is worth letting it land: in the world you are used to, columns are slices of tables. In Columna, frames are assemblies of columns. The arrow of construction runs the other way, because the ontology runs the other way — data is property-at-anchor, the column is the natural unit, and everything tabular is downstream.

The aboutness is recursive

One more feature of the property-at-anchor view, because it will return many times. The anchor of a datum is itself usually identified by other data. “Revenue of customer 42” is anchored on a customer — but “customer 42” is itself a value, the value of a customer property on some finer thing (a transaction, a registry entry). One column’s value can be another column’s anchor. The aboutness chains: a datum is about an anchor, the anchor is named by other data which is about its anchor, and so on down — until you reach base events that are, in effect, about themselves (a transaction record is anchored on the transaction it is). This recursion is not a problem to be resolved. It is the structure that lets anchors be built from columns, lets dimensions be just columns playing a particular role, and lets the whole edifice rest on a single kind of thing — the property-at-an-anchor — all the way down.

From anchor to point: the second half of aboutness

One refinement completes the founding claim, and the book devotes a chapter to it (Chapter 5) once the lattice is in hand; it is stated here so the reader carries the full principle from the start. The something a datum is about is, precisely, a point — this customer on this day, this sensor in this hour — and points do not float free: they belong to universes, declared point sets with an account of where their points come from (an entity registry, a calendar spine, a product of these, or the projection of observed events themselves). The anchor, it will turn out, was never the referent itself. An anchor is a coordinate signature — “addressed by (customer, day)” — and a signature does not determine its inhabitants: two columns can share an anchor and be about different point sets under the same addressing scheme. The complete description of where data lives is therefore a pair, (anchor, universe), and the founding principle in its full form reads: data is property at a point; the anchor names the coordinates, the universe defines the points. Addressing is the anchor’s job; existence is the universe’s. Until Chapter 5 the manual will mostly let the second component ride silently, as analytical practice always has — but nothing said in between depends on ignoring it, and Chapter 5’s work is to stop pretending it is not there.

Why this licenses everything that follows

There is a payoff to beginning here rather than with software convenience, and it connects directly to what the framework is for. If data is fundamentally property-at-anchor, then how the data is physically stored — which table, which row order, which file format — is entirely incidental to what the data is. The fact “customer 42’s revenue was 300” is the same fact whether it sits in a Postgres row, a Parquet column, or a key–value store. Physical storage is an encoding of the property-at-anchor fact; it is not the fact. And so a logical layer that operates directly on the property-at-anchor facts — on columns as functions from anchors to values — is operating on what the data actually is, while the physical tables are merely one of its encodings. This is the deepest justification for everything in the next chapter and the next part: Columna can hold the physical layer at arm’s length not as a convenience but because the column view is closer to the truth of the data than the tables are. The tables encode; the columns are what is encoded.

Chapter 2. The Substrate Beneath the Tools

The previous chapter made an ontological claim. This chapter turns it into an architectural one, because the two are tightly bound: because data is property-at-anchor and physical storage is mere encoding, Columna can operate as the logical substrate on the encoded truth while leaving the encoding entirely to others. This is what distinguishes Columna from the tools it is most often compared to, and the distinction is worth drawing with precision, because it is easy to mistake for a difference that other tools also claim.

Declarative, all the way down

SQL has always marketed itself as declarative: you say what you want, not how to compute it, and the optimizer figures out the execution. This is true, but it is only half true, and the half that is false is exactly where the pain lives. SQL is declarative about execution — you do not choose the join algorithm. But it is thoroughly imperative about semantics — you must still specify which tables to join, on what keys, in what grouping, at what grain, and you must hand-guard against the ways those choices go wrong. The procedure is still yours to write. SQL moved the declarative boundary up to cover execution and stopped there.

Columna pushes the boundary further: it is declarative about semantics too. You do not specify the joins, because the relationships between things are encoded once, in the structure of the data space, and verified. You do not hand-manage the grain, because the operations are typed by what they do to it. You do not guard against double-counting, because the structure makes double-counting unrepresentable. What is left for you to specify is reduced to what the question actually is. Everything beneath that — the entire procedure — has become the responsibility of the layer.

Not a semantic layer’s separation

This is the point at which a knowledgeable reader objects, correctly, that the separation of authoring from consuming is not new. The semantic-layer tools — the lineage that runs through Looker’s LookML, through Cube, through dbt’s metrics layer — have pursued exactly this consumer/author split for years. An expert authors metric definitions once; analysts reuse them. So what, the reader asks, is different?

The difference is what the abstraction is made of and what it compiles to, and it is a difference of kind, not degree.

A semantic layer is, architecturally, a SQL generator. You declare metrics and dimensions; when a query arrives, the layer generates SQL and runs it against the physical warehouse tables. Its output is SQL — procedural code against physical tables. The abstraction is a generation convenience: it spares the analyst from hand-writing the joins, but the joins still happen, as SQL, against tables, every time. And because its currency is SQL and its target is physical tables, the abstraction is porous by construction. The moment a question falls outside what the metric definitions anticipated — a slightly different grain, an unplanned cohort cut, an ad-hoc filter the author did not expose — the analyst drops to SQL. The semantic layer is a convenience over SQL that you keep falling out of. You are always one unanticipated question away from writing the procedure again, against the tables again.

Columna is not a SQL generator with nicer inputs. It is a closed algebra over a logical data space, and this changes the abstraction from generation to insulation. A Columna query does not have “SQL against tables” as its meaning. Its meaning is an expression in the operator algebra — a composition of typed operations over columns. That expression is eventually executed, and execution may involve querying the backend — but the backend query is an execution detail produced by the planner, below the line, invisible. The user’s query is the algebraic expression; the SQL, if any, is generated beneath the insulation boundary and never surfaces. This is the difference between a compiler that targets assembly (you write the high-level language, never touch assembly, assembly is purely an implementation target) and a macro system that expands to C (you are still writing C, and you fall back to raw C constantly). Semantic layers are the macro system. Columna is the compiler.

And the objects the algebra operates on are not physical. In a semantic layer, the model’s objects point at tables — a view is a table with annotations. In Columna, columns and dimension families and metric families are logical constructs defined by their structure and relationships, not by their storage. A column is defined by its anchor, its family, its place in the algebra — not by “the amount field in the transactions table.” The binding to physical storage exists, because the data is somewhere, but it lives in a separate source-binding layer that queries cannot address. The query operates on revenue, the logical column; that revenue happens to be sourced from sum(amount) in some table is a binding the planner resolves, below the line, that the query is blind to.

This is what it means to say the insulation is architectural, not conventional. A semantic layer would like to insulate you from the physical model, but its architecture — SQL generation against tables — prevents it. Columna insulates because there is no path from a query to a physical table that runs through the user’s hands. The architecture forecloses the leak.

The exhaustive role of SQL

It is worth being exact about this, because the claim invites a fair challenge: surely SQL is in here somewhere. It is, and only here. In every planning and execution path Columna uses, SQL appears in exactly one role — as the wire protocol to the backend data layer — and for exactly three purposes:

1. Data extraction — pulling the column-wise data the engine needs to operate on (with whatever filtering or pre-reduction the backend can do efficiently);
2. Metadata extraction — reading schema, types, and value sets to bind columns and build the catalog;
3. Verification — running the integrity checks (functional-dependency validity, uniqueness, coverage, restriction satisfaction) against the data, which are at bottom counting-and-comparing queries.

That is the complete list. SQL is the extraction-and-verification mechanism at the physical boundary. It is never the semantic model, never authored by or shown to the user, never the representation of analytical intent. Everything analytical — the operators, the aggregation, the family logic, the integrity model — happens above this boundary, in the algebra and the engine, where no SQL exists.

There is one honest subtlety. For performance, the column engine will push some work into the backend SQL — a filter, a pre-aggregation — because dragging a billion raw rows up to the engine to sum them would be absurd when the warehouse can sum them in place. So the generated SQL is not always trivial; it can contain aggregation. But that pushed-down work is an execution optimization decided by the column engine, below the line — never an expression of intent by the user, and never a semantic choice — and, this is the point, it is always an operation the planner has already validated as correct above the line. The decision splits cleanly: the planner decides the semantics and validates the reduction; the column engine, beneath the plan, decides whether and how to push that validated reduction into the backend, and pushes only when the backend’s semantics provably match — falling back to in-engine compute otherwise. So when SQL does aggregate, it is executing a fragment of a plan the algebra already owns and verified; the semantics were decided above the line. The role of SQL is extraction and verification; the content of an extraction may include validated backend reduction; but the reduction is the column engine’s optimization of an above-the-line plan, never a user-authored analytical operation. And there is a hard floor beneath the pushdown: the backend is asked to deliver columns — to extract, filter, or pre-reduce a single source — but never to combine them. No join is generated and no multi-table SQL is emitted at query time; relating two columns that live in different tables happens in the engine, by transport along a verified relationship (Chapter 8), not by handing the backend a join to run. The pushdown shrinks one column’s delivery; it never crosses a relationship. That boundary — deliver, never relate — is what keeps Columna a substrate rather than a SQL generator with a governance wrapper around it.

The positioning that follows is precise: Columna is not another tool in the analytics stack; it is the substrate beneath it. The conventional tools — BI, the semantic layer, the SQL an agent writes — all operate on top of a query language that can already express a wrong answer; Columna is the level below, where analytical meaning is expressed and verified before any tool acts on it. It does not manage, replace, or own the physical data layer — the data continues to live in and be managed by whatever data engine the organization runs. Columna’s only contact with the physical layer is SQL used narrowly to extract column-wise data and metadata and to verify integrity facts. All analytical meaning lives in this substrate, where no SQL is authored, exposed, or required — so the surfaces built on it, a query interface, an AI agent, an analytical service, operate on verified answers rather than regenerating the procedure, and its risks, themselves. Because the analytical meaning never descends into SQL, the user never has to think in SQL to understand or to debug an analysis. And because queries are expressed against the logical algebra rather than physical SQL, the same query is portable across backends unchanged: move the data from one engine to another and only the source bindings change, below the line. The logical model is genuinely independent of where the data physically lives.

Where these ideas come from

A reader with a long memory will recognize ancestors for nearly every idea in this book, and the book is better for saying so plainly. The framework’s individual components are not inventions; most are decades old, several are battle-tested at industrial scale, and one is a theorem from the statistics literature. Naming the lineage is partly honesty and partly sharpening: it locates exactly what is new here, which is not any single idea but a specific assembly of them — and it tells the skeptical reader that the load-bearing parts of Columna rest on ideas that have already survived contact with reality.

Semi-additivity is OLAP’s oldest hard lesson. Dimensional modeling has known since the Kimball era that a balance does not sum over time, and the multidimensional engines of the 1990s and 2000s — Essbase, SSAS and their kin — carried per-measure aggregation rules for exactly this reason. What that tradition had was the label: a measure tagged additive, semi-additive, or non-additive, with an aggregation function per dimension in the better tools. What it did not have was precision or verification. The label was coarser than the fact (semi-additive along which axes, under which reducer?), it lived in documentation and cube configuration rather than in a checked declaration, and nothing ever tested it against the data. The B-anchor is that lesson made precise — a per-(column, reducer) set of blocked dimension families — and made consequential: consulted at every plan, checked for consistency at every refresh, its violations disclosed rather than silently computed.

Monoid aggregation and mergeable sketches are proven engineering, not Columna’s discovery. The classification of aggregates by whether they can be combined from partial results goes back at least to the original data-cube paper’s distributive/algebraic/holistic taxonomy, and it became industrial practice in libraries like Twitter’s Algebird and Apache DataSketches: model an aggregation as a commutative monoid and it parallelizes, distributes, and re-aggregates for free; where the exact operation is not a monoid (distinct counts, quantiles), substitute a sketch — HyperLogLog, t-digest — whose merge is one. The fertile/mule distinction is this engineering folklore promoted to a type discipline: not a library a programmer may choose to use correctly, but a property the framework declares per reducer, enforces in the planner, and uses to decide what may found a cached family at all. The sketches enter the type system for the same reason — they are the carriers that make otherwise-sterile reductions fertile — rather than remaining an optimization a pipeline author happens to know about.

The materialization lattice is a 1990s result. The insight that group-bys form a lattice ordered by refinement, and that choosing which lattice points to materialize is the central cost/freshness trade of analytical serving, is the classic data-cube-implementation literature (Harinarayan, Rajaraman, and Ullman’s line of work and its successors). The anchor lattice is that structure, and families-as-cache is that materialization decision — with two refinements the original setting did not need: rooting is per-column rather than cube-wide (different columns natively live at different lattice positions), and traversal is typed — only fertile reducers may reuse cached coarse images, only sound hierarchy edges may be climbed, and the legality of every move is checked against declarations the data has verified rather than assumed from the schema.

The fan trap has a thirty-year mitigation industry, and the closest living relatives are in it. BI tools have fought join-multiplication errors since BI tools existed; Looker’s symmetric aggregates and, more thoroughly, Malloy’s dimensional query semantics compute each aggregate at its own source grain so that joining cannot inflate it. Malloy deserves singling out as the nearest relative on the query-language side: it shares the conviction that the query language itself, not author discipline, should make grain errors hard. The difference is scope and architecture. Malloy is a better language compiling to SQL against the physical tables; its protections are the compiler’s good behavior. Columna removes the join from the language entirely — combining columns at different anchors is not a guarded operation but a non-operation, replaced by explicit ascription to a common anchor — the operation this manual names transport (Chapter 8): the relating happens in the engine, along a verified functional relationship, never as a backend join — and surrounds the language with the parts a language alone cannot carry: the missingness model, coverage and universe integrity, the certificate, and the disclosure contract.

Property-at-anchor has a formal ancestor in fact-based modeling. The ontology of Chapter 1 — data as elementary assertions about identified things, with tables as derived assemblies rather than primitives — is the founding intuition of the fact-based modeling tradition (NIAM and Object-Role Modeling): model the elementary facts; let table shapes be a derived, secondary concern. That tradition demonstrated the ontology’s soundness and then remained a schema-design methodology — a way to arrive at good relational schemas, after which the operational world was tables and SQL again. Columna keeps the ontology operational: the column-as-function-from-anchors-to-values is not a design-time discipline that compiles away but the runtime object the algebra acts on. The categorical view of databases (schemas as categories, instances as functors, columns as functions composed along declared maps) reaches a similar place from the mathematical side, and readers fluent in it will recognize the anchor lattice and roll-up maps as familiar structure; that literature supplies elegance and assurance that the foundations compose, while Columna supplies the concerns the category alone does not speak to — additivity, missingness, coverage, and the verification of declared structure against actual data.

The missing-data machinery is imported from statistics, deliberately. MCAR, MAR, and MNAR are Rubin’s mechanisms, half a century old; the non-identifiability of MNAR from observed data is a theorem of that literature, not a Columna finding, and the manual cites it as such. The framework’s contribution here is wiring: making the mechanism a declared, partially falsifiable property of a column (the M-anchor), deriving the default treatment of every aggregation from it through one small behavior vocabulary, and binding the honesty obligations — what was estimated, what cannot be corrected, what the certificate may and may not say about an asserted mechanism — into the planner and the annotation rather than leaving them to the analyst’s statistical conscience.

Semantic layers are treated in this chapter already (the generator-versus-compiler distinction), and the reader should hold both halves of that comparison: the consumer/author split that semantic layers pioneered is genuinely the right split, and Columna inherits it gratefully; the disagreement is solely about what the abstraction is made of and what it compiles to.

The wedge, stated against the lineage.

So what, against all this ancestry, is actually new? Not the semi-additive lesson, the monoid discipline, the lattice, the grain-safe language, the fact ontology, or the missingness theory — each has an owner. What none of the ancestors has, and what Columna is, is the assembly: the soundness-determining facts of analytics — applicability, scaling, rollability, missingness mechanism, coverage, hierarchy validity, defining boundaries — represented explicitly in one declared model, verified against the data wherever verification is possible, honestly tagged where it is not, governing a closed operator algebra through a planner that consults them on every operation, and accounted for in a certificate and a disclosure that travel with every number. Each ancestor carried one fact or one mechanism in one layer — a cube’s measure label, a library’s monoid, a compiler’s grain rule, a methodology’s ontology, a literature’s taxonomy. None carried representation, verification, and consequence as a single system whose output is a number with its dependencies on its face.

This is also why the lineage is a strength claim rather than a confession. The component ideas being old means they are true — proven separately, at scale, over decades. Columna’s bet is not on any of them; it is on their integration being worth more than their sum, and on integration being the thing that none of the existing tools, each anchored in its own layer, is positioned to do.

(Bibliographic entries to be set at copy-editing, per ADR-008: the data-cube paper (Gray et al.); the cube-materialization lattice (Harinarayan, Rajaraman, Ullman); Kimball’s dimensional-modeling texts; Algebird and Apache DataSketches; Looker symmetric aggregates and the Malloy documentation; Halpin’s Object-Role Modeling; Spivak’s functorial data migration; Rubin’s missing-data work and Little & Rubin for MNAR non-identifiability.)

Chapter 3. The Column

The column is the atom, established as a function from anchor instances to values, the natural grouping of property-at-anchor data. But a Columna column carries far more than a SQL column or a dataframe series. It is a typed assertion about a measurable quantity over a domain, and its components organize into three natural concerns: what it is, where and whether its values exist, and how its values compose.

Identity

The first concern is the simplest. A column has a name, by which it is referenced, and a value type — numeric, string, boolean, date, and so on — which is the familiar notion of data type. These say what the column is called and what kind of values it holds, and nothing more.

Value determination

The second concern answers: over what domain is this column defined, and where within that domain do its values actually exist? Two components together specify this, and they belong together because they are two faces of one idea — the column as a possibly-partial function over its domain.

The V-anchor (value anchor) is the domain. It is the set of dimension coordinates the column is a function of — the key-space over which values are defined. revenue anchored on {customer, day} is a function that, given a customer and a day, returns a value. The V-anchor says what determines which value you are talking about: fix the coordinates, and you have fixed the point’s address. Which points exist under that addressing — whether a customer-day with no row is a point whose value is absent or no point at all — is not the V-anchor’s to say: it is the content of the column’s universe, the declared point set the V-anchor addresses (Chapter 5); the complete statement of where a column lives is the pair (anchor, universe). (“Anchor” by itself is the shared primitive — a named set of dimension coordinates governing one aspect of a column. A column carries three of them; the V-anchor is the first. The bare word is qualified by a letter precisely because more than one of a column’s anchors would otherwise lay claim to it.)

The M-anchor (missingness anchor) says where within that domain the function is actually defined, and what governs its absence. A function over a domain may be total (defined everywhere) or partial (defined at some points, absent at others), and when it is partial, the absence has a mechanism — a reason it is absent here and present there. The M-anchor characterizes that mechanism: it names the coordinates that determine missingness. When missingness depends on nothing (it is completely at random), the M-anchor is empty. When it depends on observed coordinates, the M-anchor names them. When it depends on the unobserved value itself — the hardest case, where a value is missing because of what it would have been — the M-anchor records that too. (Part II returns to what each of these means for computation.)

The V-anchor is the domain; the M-anchor is the partiality of the function over that domain and the mechanism behind it. Together they are value determination: the complete account of where the column’s values exist and what fixes their identity. There is a third facet of value determination — degeneracy, the question of whether the column covers the full universe of its anchor or only a declared subset — but degeneracy is written by an operation (subsetting) and governed by the integrity machinery, so it is treated where those are, in Chapters 7 and 11.

Composition

The third concern answers: how do this column’s values combine when you reduce them? This is where Columna carries the most information, and where its precision goes furthest beyond its ancestors — the OLAP tradition knew the lesson (Chapter 2’s lineage names it); the family set is that lesson made precise, checked, and consequential.

A column does not have a reducer. It has a family set — a set of families, each of which is the column viewed through a particular reducer, together with the properties of that reduction. A family is not a thing the column belongs to; it is the column under a reducer. revenue·SUM, revenue·MAX, revenue·MIN are three families of the one revenue column, each a distinct way the column composes. To name a family is to name a reducer and the behavior of reducing the column that way.

Each family in the set carries, as part of itself, the axes along which its reducer is blocked from rolling up — its B-anchor (blocked anchor). This is the seat of additivity, and it is treated fully in Chapter 7. The B-anchor is the third of a column’s three anchors, but it differs from the V- and M-anchor in two ways worth stating plainly. First, it is per-family, not per-column: it lives inside each family, not floating at the level of the column, because applicability is a property of a reducer applied to a column, and reducers live inside families — revenue·SUM has its B-anchor, inventory_balance·SUM has a different one, and “the column’s B-anchor” is unambiguous only for the default family. Second, it is prohibitive where the V- and M-anchor are locative: the V- and M-anchor point at coordinates (where the value lives, where its missingness is determined), while the B-anchor forbids an operation over coordinates — it names the axes the reducer may not be rolled up across, never where the value is blocked from existing.

Among the families in the set, one is marked the default family — the one a bare reference to the column resolves to. Write revenue with no reducer named, and you get the default family’s reducer. Write MAX(revenue) and you select the max family explicitly. The default is a declared convenience for the common case; the family set is the full truth.

(Readers of the prior edition will note that an earlier “semantic kind” component — flow, stock, rate, and so on — is gone. It is unnecessary: everything it asserted about additivity is captured, more precisely and along specific axes, by the B-anchors of the family set, and everything it asserted about correct reduction is captured by which families the set contains. A human-readable classification can still be derived from the B-anchor pattern for documentation, but it is a derived view, not a declared component, and so it can never contradict the families it summarizes.)

The column, assembled

A column, then, is: a name and value type (identity); a V-anchor, the universe its points belong to (declared where declaration is needed, derived where the data settles it — Chapter 5), and an M-anchor (value determination — where and whether values exist); and a family set with a default family (composition — how values reduce, each family carrying its B-anchor). Three concerns, cleanly separated: what it is, where its values are, how they compose. Said in one line for the common case: a Columna column is a value plus three anchors — V, M, and B — where it lives, where it leaks, and where it blocks — with the V-anchor’s universe defining which points the living happens at. (The three are not symmetric: the V- and M-anchor are value-determination facts that live on the column, while the B-anchor is a composition fact that lives per-family — the unity is that all three name coordinates with the same anchor primitive, not that they sit at the same level.) Everything the framework does to columns acts on one of these three concerns, and keeping them distinct is what keeps the rest of the system clear.

Chapter 4. The Anchor and the Operations

Two columns can be combined only when their anchors are compatible. This single requirement — explicit, checked, never assumed — is what replaces the silent positional alignment of dataframes, and it is the reason a whole class of misalignment errors cannot occur in Columna. This chapter develops the anchor as a structure and the operations defined over it.

The anchor lattice

Anchors are ordered by refinement. {customer, day} is finer than {customer}, which is finer than {}, the empty anchor — which in the context of a specific Manifold denotes the Manifold’s defining boundaries collapsed to one point, treated in Chapter 9. The relation is containment of grain: every value at {customer, day} collapses into one value at {customer} (combining across days), which collapses into one at {} (combining across customers). This partial order is the anchor lattice, and it is the structural backbone of analytical thinking in Columna. Moving up it — finer to coarser — combines values. Moving down it — coarser to finer — replicates them. “Grain,” the word analysts reach for, is just a column’s position in this lattice.

A subtlety worth noting up front: the lattice itself is a single shared structure, built from the Manifold’s dimensions and their hierarchies. What differs across columns is where in the lattice each column natively lives — its root, the finest anchor at which the column is observed. Different columns may root at different lattice positions: revenue may root at {transaction}, while inventory_balance roots at {warehouse, day}, and customer_segment roots at {customer} and goes no finer because there is no finer place for it to live. The lattice is shared; the rooting positions are per-column. Operations move each column from its own root to wherever a query needs.

And one sentence, redeemed in full in Chapter 5, states the lattice’s complete type: the lattice orders (anchor, universe) pairs, not bare anchors — and along a fixed anchor, universes themselves order by containment, the observed events projection sitting inside the expected spine or registry product it is drawn from. Everything in this chapter survives unchanged on the pair; the second component is the next chapter’s subject.

There are exactly two structural ways to move up the lattice, and they differ enough to name separately. A drop removes a coordinate entirely: {customer, store} to {store} eliminates the customer coordinate. A climb replaces a coordinate with a coarser one in the same dimension family: {customer, day} to {customer, month} substitutes day’s parent, month, via a roll-up within the time family. The two have different requirements, developed in Chapter 8.

What an operation can do to an anchor

Every operation Columna performs on columns is classified by what it does to the anchor, and there turn out to be exactly three things it can do:

It can preserve the anchor — same domain in, same domain out, the values transformed point by point. This is map.

It can coarsen the anchor — fine domain in, coarser domain out, multiple fine values combined into one coarse value by a reducer. This is aggregate.

It can refine the anchor — coarse domain in, finer domain out, each coarse value replicated across the finer points beneath it. This is broadcast.

These three — preserve, coarsen, refine — are the complete set of anchor-actions, because they exhaust what can happen to a domain under an operation: it stays the same, it gets smaller, or it gets larger. Aggregate and broadcast are duals: aggregate climbs the lattice by combining, broadcast descends it by replicating. (Broadcast has a single semantics — copy the coarse value unchanged to each finer point — whether the measure is intensive or extensive; Chapter 8 takes up why it never apportions a total.) One question aggregation raises is deferred to Chapter 5: when a destination bucket has no input points at all, whether what renders is a true zero, an absent value, or no point belongs to the destination’s universe, by the empty-bucket rule stated there.

The orthogonal axis: order

Cutting across these three anchor-actions is a second, independent property: whether the operation depends on an ordering of the data.

Most maps do not. square(x), x + y, log(x) produce a value at each point that depends only on the inputs at that point — order is irrelevant. But some anchor-preserving operations produce a value at each point that depends on an order over the domain and on neighboring values: cumsum, lag, rank, rolling_mean. These are still anchor-preserving — same domain in, same domain out — so they are maps; they are simply order-dependent maps. The familiar name for an order-dependent map is a scan or window operation, and Columna keeps that name because it is what you will reach for, but structurally a scan is not a separate kind of operation. It is a map with the order property switched on.

The same axis cuts across aggregation. Most reducers do not depend on order: SUM, COUNT, MAX give the same answer regardless of the order their inputs arrive in. But some do: LAST and FIRST are meaningless without an order to be last or first of. These are order-dependent reducers, and they are exactly what the time axis of a stock measure needs (the right way to collapse an account balance over time is LAST — the closing balance — not SUM).

So the operations are not a flat list of four or five. They are three anchor-actions — map, aggregate, broadcast — crossed with one orthogonal order axis. Order-dependent map is called a scan; order-dependent aggregate is an ordered reducer. And there is a single, uniform well-formedness rule covering both: any order-dependent operation requires an explicit ordering to be well-formed. A scan or an ordered reducer that does not say what to order by is not a query with a sensible default — it is ill-formed, and the framework asks for the missing order rather than guessing. Order is never assumed; it is supplied, or the operation does not type-check — the outcome is a clarification, never a number.

Calendar intelligence — year-to-date, rank-within-quarter, same-period-last-year — is where the order axis earns its keep, and Columna expresses it through three family-aware scan parameters (reset, within, step), each naming a coarser level of the order axis’s dimension family, reachable by declared, verified roll-up edges: reset restarts a cumulative scan at the level’s boundaries (year-to-date, month-to-date), within confines a positional scan inside each level instance (rank within quarter; a lag that does not cross years), and step makes the level the shift unit of lag/lead so that same-period-last-year — and from it year-over-year — is an ordinary co-anchored map. No new machinery is involved: the planner resolves all three to climb-derived partition-and-order specifications, and the climbed edge’s verdict enters the result’s dependency cone like any climb’s. The full treatment belongs to the query-language manual; the point for this book is that the calendar analytics conventional tools relegate to fragile idioms rides here on declared, verified structure — cleaner, not just safer.

Are these complete?

It is fair to ask whether map, aggregate, and broadcast cover every analytical need. The honest answer is: they are complete with respect to a specific principle — they exhaust what an operation can do to an anchor. Within that principle there is nothing missing. Whether that principle covers every imaginable analysis is an empirical question, not a provable one, and the manual will not overclaim. A useful way to test it: any operation either keeps each value at its own coordinate (map, scan), or combines values across coordinates (aggregate), or spreads a value across finer coordinates (broadcast). If someone proposes an operation that is none of these, it is genuinely novel and worth examining. That is an invitation to find the gap, not an assertion that none exists.

Two things that look like missing operations are not. Combining two columns at different anchors — what a SQL writer would reach for a join to do — is not a primitive operation; it is composition. Bring the columns to a common anchor first (aggregate one, broadcast the other, or both), at which point they are co-anchored, and then map them together. The join dissolves into “aggregate and/or broadcast to a common anchor, then map.” And stacking — concatenating one column’s points onto another’s — is deliberately absent, for a principled reason taken up in Chapter 8: the urge to stack is almost always a sign that an anchor coordinate is missing.

Chapter 5. The Universe: What the Anchor Was Always Standing On

5.1 The assumption that was always there

Every framework rests on assumptions it has not yet said aloud, and the mark of a sound one is that saying them makes it stronger. Here is this framework’s: everything established so far — columns as functions from anchors to values, families conserving their totals across the lattice, ratios formed at shared anchors — has quietly presumed that the anchor-point set we observe is the complete universe of observation. That the customer-days in the revenue column are all the customer-days there are. That the {sensor, hour} combinations with readings are the {sensor, hour} combinations that exist.

Stated for one column in isolation, this is a tautology — the observations are, trivially, the observations. It stops being a tautology, and becomes a real, falsifiable, load-bearing claim, in exactly the three places this framework lives:

Cross-column. margin = revenue − cost at {customer, day} is a pointwise subtraction, and a subtraction needs both operands at the same points. Each column’s observed point set is “the universe” by its own lights; the map silently requires them to be the same universe, and nothing in either column alone asserts that. When the cost feed covers a subset of the revenue feed’s customers, the margin is not slightly wrong — it is a well-formed computation over an incoherent premise.

Cross-anchor. Sum revenue to the grand total through {customer}, and again through {region}. The two paths land on the same number only if every root transaction participates in both climbs. One transaction carrying a null region, and the region path silently sheds it while the customer path keeps it: the same metric, disagreeing with itself at {}. Nothing in the reducer failed; the paths simply did not traverse the same points.

Against the world. “Observed equals universe” is, finally, a claim that capture was complete — an assertion about the recording process, not about the data. The data cannot certify its own completeness, for the same structural reason a column cannot testify about its own absent values.

The job of this chapter is to convert the silent premise into declared, decomposed, checkable structure — explicit where it can be verified, honestly tagged where it can only be asserted, and disclosed wherever a result depends on it.

5.2 Points, universes, and the completion of the founding principle

Chapter 1 said data is something about something else — a property at an anchor. This chapter completes the sentence by looking harder at the something else. The referent of a datum is, precisely, a point: this customer on this day; this sensor in this hour. And points do not float free. They belong to universes — declared point sets, each with an account of where its points come from.

The anchor, it turns out, was never the referent. An anchor is a coordinate signature — “addressed by (sensor, hour)” — and a signature does not determine its inhabitants. Two columns can share an anchor and live on different point sets under the same addressing scheme: the sensor’s expected hourly readings and the alerts that happened to fire are both addressed by (sensor, hour), and they are not about the same points. The complete description of where data lives is therefore the pair — (anchor, universe) — and the founding principle states its full form:

Data is property at a point. The anchor names the coordinates; the universe defines the points. Addressing is the anchor’s job; existence is the universe’s.

Everything in the preceding chapters survives unchanged — and gains a component it had been defaulting silently. The lattice orders (anchor, universe) pairs; along a fixed anchor, universes themselves order by containment, the observed inside the expected, an event projection inside the spine-product it is drawn from. A column is a function from a universe’s points to values. The framework has been operating on the pair since Chapter 1; this chapter merely stops pretending the second component is not there.

5.3 Bases: where points come from

A universe is declared with a basis — its account of point provenance. Four bases suffice:

• Registry — an enumerated entity list: a sensor roster, a customer panel, a store directory. Points exist because the registry says so.
• Spine — a generated sequence: the calendar of days, the hours of a year. Points exist by construction of the generator.
• Product — the cross-product of universes: active sensors × calendar hours. Points exist combinatorially, from the factors.
• Events projection — the point set projected from an event root: the observed transactions; the customer-days on which purchases occurred. Points exist because something happened.

The familiar vocabulary of the missing-data literature maps onto the bases as description, not as additional machinery. A sparse universe is one whose basis is an events projection: “universe ≡ observed,” the old tautology — now converted into an explicit assertion that other columns can be checked against. A dense universe is declared independently of observation — registry, spine, product — and observation is measured against it, which is exactly what makes a legitimate point able to lack a record. Whether a record-less legitimate point can exist at an anchor is not a fact a column carries privately; it is the content of the universe its values land on.

One derivation rule does most of the daily work: a column rooted at its own grain key lands on that root’s events projection, by construction. revenue @ {transaction} is about transactions; the point exists because the event does; there is nothing to declare, and the framework forbids declaring it — declaring the derivable is noise. The consequence reaches further than convenience: for event data, which is most of what commerce sums, the framework’s total equals the source system’s total by default, and reconciliation against the warehouse passes. The estimation machinery of the missing-data chapter fires only where an author has deliberately declared a dense universe and a missingness mechanism — which is precisely where the conventional sum is the number that should not be trusted.

The choice between readings, where the data leaves it open, is a declaration, never an inference — the same physical table at {customer, day} supports both “purchases” (events) and “expected daily readings” (spine product), and the data cannot say which; only intent can. As with reducers, processing code is the best proposer of intent: a calendar-spine join in the feeding pipeline is a dense tell, a bare GROUP BY over an event table a sparse tell, and the Discover stage proposes universes with that evidence attached while the author decides.

5.4 Ascription lands on the destination

The ascription rule of the query language — expr @ a, the value of this expression, materialized at anchor a — gains its full form here: every ascription is implicitly expr @ (a, U), and the universe component belongs to the destination.

Watch what this settles. Alerts are sparse at their own root — {alert}, the event grain, nothing missing, nothing to declare. The moment they are ascribed to buckets — alert_count @ {sensor, hour} — the question which (sensor, hour) points exist is no longer about alerts at all; it is about the destination. And the destination’s basis decides the rendering, by a rule this framework states once and applies everywhere:

The empty-bucket rule. Over a declared universe (registry, spine, product), an empty bucket’s SUM and COUNT are true zeros — counting nothing is a defined act with answer 0 — while its order statistics and means are absent — there is no minimum of nothing. Over an events projection, an empty bucket is not a point, and nothing is rendered.

The rule is small and it retires three standing confusions at a stroke. The fill(0) declaration loses most of its population: “missing alerts means no alerts” was never a fill rule, it was a universe statement, and once the destination is a declared sensor-hour universe the zeros appear by semantics, with no substitution and nothing to apologize for. The empty-group question (what is the average of a group with no members?) stops being a hedge: the group exists or it doesn’t, per the destination, and if it exists its sum is zero and its mean is absent. And the bracket filter’s old ambiguity — does the customer with no east-region transactions appear with zero, with an absent value, or not at all? — resolves into a rule: the bracket restricts the event set; the destination universe decides the rendering.

In the common case the destination universe is never written. The consequence rule supplies it — the contributing columns’ covered universes, brought to the output anchor — so that sparse inputs yield sparse frames and no point is manufactured that the data never asserted. Where the writer wants a different destination — most often a reporting grid over the full spine — the universe is pinned explicitly (ON, in the query language). Default by disclosure, pin by syntax: the resolved (anchor, universe) of every series rides in every result’s annotation, so the defaulted parameter is never a hidden one.

5.5 The four layers, and the jurisdiction rule

With the universe named, the framework’s treatment of “what’s there and what isn’t” factors into four layers, each with its own question, its own declaration, and its own integrity machinery — and the factorization is the chapter’s working skeleton:

1. Defining boundaries (Manifold-level): what is this data space about?
2. Coverage (column-level): which region of the boundaries does this column’s data address?
3. Universe (the new layer): which points exist within the covered region?
4. M-anchor (column-level): why are values absent at existing points?

Two consequences are pinned here because everything downstream leans on them.

The jurisdiction rule (design principle): the M-anchor’s jurisdiction is value-presence at universe points; point-existence belongs to the universe; the two never overlap, and they compose. A transaction that happened with an uncaptured amount is a point that exists with a value that’s absent — real missingness, governed by the M-anchor, with the record count as its denominator. An expected reading that never arrived is likewise a point with an absent value — real missingness, with the spine as its denominator. A purchase that didn’t happen is not a point, and no mechanism may be declared over it: asserting one would claim that the expected revenue of a non-purchase is the average purchase, which is the inflation absurdity this rule exists to make unrepresentable.

N_total is the covered universe. The estimation behaviors of the missing-data chapter — skip-and-rescale above all — require a denominator, and the universe supplies it: the cardinality of the column’s universe within its coverage. Rescaling estimates the total over the covered points, never beyond them; extrapolation past coverage is coverage’s finding to disclose, not the rescale’s job to attempt. And the rescale itself is worth seeing plainly, as an identity (algebraic note, not a theorem): SUM(observed) × N_total/N_obs is exactly SUM(observed) + N_missing × mean(observed) — fill every hole with the observed average, then sum, collapsed into one multiplication. The disclosure can therefore say what was actually done — “4 missing values estimated at the observed mean of 31.20” — and the estimate’s unbiasedness remains conditional on the declared mechanism being true, which is asserted, never verified; the certificate’s safety asymmetry applies to universes exactly as it applies to everything else.

5.6 Co-universality: the other half of co-anchoring

“Bring two columns to a common anchor” was only ever half of the soundness condition for combining them; the other half is bring them to the same points. The full type of a sound combination is same (anchor, universe) — and the universe-integrity machinery, which earlier chapters introduced as drift-checking, here receives its proper object: columns combined at a shared anchor are checked for co-universality. Same declared universe: verified by reference, the strongest and cheapest verdict in the framework. Same events-projection by declaration, differing point sets in fact: the drift finding, now stated against a named object rather than a vague expectation. Different universes outright — an events projection joined to a spine, observation compared against expectation: a legitimate combination, served with its permanent asymmetry disclosed, never silently treated as same-points and never refused for being what it honestly is.

5.7 The conservation invariant

The deepest demand the framework makes of itself can now be stated with its preconditions enumerated — the property earlier chapters meant whenever they said a family is consistent:

The conservation invariant. A fertile family’s total is path-independent — the same metric, ascribed to {} along any path through the lattice, lands on the same value — given three preconditions: (i) path-invariance of the reducer (the monoid property, already enforced by type); (ii) coordinate totality at the root — every root point carries a value for every dimension any path traverses: no nulls, no orphans, every child mapping to exactly one parent; and (iii) one root point set feeding every path — the shared-universe premise of 5.1, now declared via universe references and checked as co-universality rather than assumed in silence.

Precondition (ii) is the chapter’s gift to the integrity layers: a new, fully checkable condition with a precise blast radius — a totality violation invalidates the paths through the deficient dimension and nothing else — and a disclosure that can finally be quantified instead of gestured at: “the region path excludes 312 transactions (0.4% of total) carrying null region; the customer path includes them.” That sentence, attached to the served number, is the difference between a grand total that mysteriously disagrees with itself and a grand total whose disagreement has a name, a size, and a fix.

5.8 What can be known

The universe layer obeys the same epistemics as everything else in this book, and it is worth walking the ladder explicitly. The universe choice — events or expected, this registry or that spine — is asserted in principle: the same physical shape supports both readings, so no data check can settle intent (the reducer argument of the authoring chapters, recurring at a deeper layer). It is, however, corroborable at the code-verified rung: the spine-join and GROUP-BY tells in the feeding pipeline are exactly what source inspection attests, and Discover attaches them as evidence to its proposals. Once chosen, the layer’s facts are checkable in proportion to the basis: an events projection has nothing to attest about itself (universe ≡ observed — the load shifts entirely to the cross-column co-universality checks); a registry basis supports referential checks (every point’s entity exists in the roster, and conversely); spines and products support density measurement (what fraction of declared points carry records), which is also where a declared MCAR earns its falsification tests. And throughout, the asymmetry holds: a positive finding — a drift, a totality violation, a registry orphan — contradicts; a clean pass corroborates, and never silently verifies an assertion about the world.

5.9 One parameter, five cracks

A closing observation, historical rather than technical, for the reader wondering how something this foundational stayed invisible for forty years of analytical practice. It stayed invisible because conventional analytics almost always operates in a regime where one universe is ambient — the rows of the one table being queried — and an ambient assumption never has to be written down. Every place the ambience breaks, practice long ago gave the breakage a separate name and a separate folk remedy: the fill(0) confusion (a universe statement disguised as a value rule), the empty-group hedge (a destination question answered with a shrug), the tie-out failure (an estimation denominator nobody declared), the conservation leak (two paths over secretly different point sets), and population drift between joined columns (the universe-integrity problem, sensed but objectless). These were never five problems. They were one undeclared parameter, leaking through five cracks — and this chapter is nothing more, and nothing less, than the declaration of that parameter.

5.10 Authoring: deciding a universe

The working procedure, one question per step, for every column (the reference manual’s authoring chapters carry the worked examples):

1. Rooted at its own grain key? Events projection, derived, done — and not declarable.
2. Otherwise, ask the one question: if an anchor point has no row, did nothing happen, or did something fail to be captured? Nothing happened → reference (or declare, for reuse) the events projection. Failed capture → declare the dense universe and name its basis: which registry, which spine, what product.
3. Then declare the M-anchor as before — it governs null values at existing points, in either case, with the universe supplying its denominator.
4. Expect verification in proportion to the basis (5.8), and expect Discover to have proposed the answer with code-level evidence before the question was asked.

The question in step 2 is the whole chapter in one sentence, and it is the question conventional tools never let anyone ask — because they never admitted there were two possible answers.

Part II — How Values Compose

Chapter 6. Families, Path-Invariance, and Mules

This is the chapter where additivity — the most error-prone concept in practical analytics — becomes rigorous. The machinery rests on a careful framing, two independent distinctions, and a small biological metaphor that turns out to be exactly what is needed.

A clarifying framing: families are an optimization, not a correctness mechanism

Before any technical detail, a framing point that prevents confusion later. For correctness alone, Columna needs no families at all. The framework could keep, for each column, only its data at its own native root — the finest position at which it is observed — and recompute every query against the root every time. Want monthly revenue? Sum the daily root. Want the grand total? Sum the root. Want anything? Go back to the root and reduce. This is always correct. It is also expensive: recomputing from root on every query, for every column, in every dashboard, is wasteful when the same coarse images are wanted again and again.

Families exist to spare this waste. A family is the column viewed at every grain in the lattice that it can correctly take, with the coarser images derivable from finer cached images rather than recomputed from root each time. Families are a caching optimization: they let the framework reuse intermediate computations instead of going to root every time. The optimization is per column — each column has its own root and its own family, optimized independently — because columns live at different lattice positions and have different cost profiles.

This framing has a precise consequence for what families are. Not every reducer can found a family, because not every reducer admits the cache-and-reuse property the optimization requires. Whether a reducer admits this is exactly the question of path-invariance, treated below. Reducers that do not admit it are not families — they are computed at presentation from fertile sources, never materialized as re-aggregable intermediates. The whole technical machinery of this chapter is essentially a careful account of which reducers can found families and which cannot, and how the ones that cannot are still made useful.

A metric is a family, not a value

With that framing in mind: what is a metric? Daily revenue and monthly revenue are not two metrics; they are one metric seen at two grains, connected by a reducer. Monthly revenue is daily revenue summed over the days of the month. This is the family relationship: every member of the family — revenue at every grain in the lattice — is the image of the column under the family’s reducer, derived consistently from the column’s root. A family is a column plus a reducer that admits the family optimization; its members populate the whole lattice from the column’s root upward. And what “consistently” promises is now exactly statable: it is the conservation invariant of Chapter 5 — the same metric, ascribed to {} along any path through the lattice, lands on the same value — which holds given path-invariance of the reducer (typed, below), coordinate totality at the root (a checkable integrity condition, Chapter 11), and one root point set feeding every path (declared via universe references and checked as co-universality). A family is the cached image of that invariant; the invariant, not the cache, is the consistency being promised.

Two axes that are easy to confuse

Reducers differ along two independent properties, and conflating them is the source of much confusion. Keeping them separate is the key to the whole chapter.

The first axis is order-dependence, met already: does the reducer need an ordering of its inputs to be defined? SUM, AVG, MAX do not. LAST, FIRST do.

The second axis is path-invariance: can the reducer be applied across grains consistently — is the result independent of the path taken up the lattice? Aggregating {customer, day} straight to {customer} should give the same answer as going by way of {customer, month}. For some reducers it does; for others it does not.

These axes are genuinely independent, and the cleanest proof is the average. AVG is order-independent — the order of the inputs does not change the average — but it is not path-invariant: the mean of monthly means is not the overall mean. The mean of {1, 1, 1} and {10}, taken as a mean of two monthly means, is 5.5; taken correctly over all four values, it is 3.25. So AVG sits in the cell that the two axes pull apart: order-independent, yet not path-invariant. The two properties are not the same property, and the framework must track them separately.

Fertile reducers and mules

Path-invariance is the property that licenses the family optimization, because it is exactly the property that lets a coarse cached result be reused safely instead of recomputing from root. And it has a sharp characterization: a reducer is path-invariant precisely when it is a commutative monoid — when its values can be combined associatively and commutatively, in any order and any grouping, to the same result. SUM, COUNT, MAX, MIN — and, mathematically, product — are such monoids. (Columna’s catalog carries no PRODUCT reducer; the point is the algebraic shape, not the operator list.)

A reducer with this property is family-bearing — call it fertile. Its result can be re-aggregated up the lattice forever, generating a consistent lineage of values at every grain. SUM at the day grain breeds SUM at the month grain breeds SUM at the year grain, all consistent — and each can be derived from the next-finer cached image, the family optimization paying off.

A reducer without this property is a mule — sterile. It produces a perfectly valid value at the grain you compute it at, but that value cannot be safely reused as an intermediate — re-aggregating it gives wrong answers. AVG is the canonical mule. A per-customer mean is correct as a per-customer mean, but you cannot average those means to get the overall mean. A mule is not a family. It is computed at presentation, from a fertile source, never materialized as a re-aggregable intermediate. The framework’s response is not a warning; it is the structural fact that a mule does not found a family in the first place. A mule may be produced — it answers a real question at its own grain — but it is computed fresh each time, from fertile sources at root, never reused as a cached intermediate.

How to keep a useful mule

Many of the most useful reducers are mules — means, ratios, weighted averages, variances — and the framework does not banish them. It computes them from fertile sources. A mean is computed as the ratio of two fertile families — sum and count — both of which are families, both cached and re-aggregable freely up the lattice. The division — the step that produces the sterile mean — is deferred to the very end, the moment of presentation, where no further reuse is needed. You compute with the fertile sum and the fertile count, all the way to the grain you want, and only at the final step do you divide to produce the mean. The mule appears only where it can do no harm, computed at the destination, never cached.

The same device — compute the mule at presentation from fertile sources — handles ratios (numerator and denominator as separate fertile families), weighted averages, variances (the fertile sources being count, sum, and sum-of-squares), and rates. Each is a mule made safe by deriving from fertile sources, the sterile result produced only at presentation.

For reducers that are mules with no fertile source decomposition at all — exact distinct count, exact median — the framework offers approximate variants built on sketches (HyperLogLog for distinct count, t-digest for quantiles) that are themselves fertile monoids on the sketch, and so can be re-aggregated. The sketch types are genuine fertile-family carriers: the sketch’s merge operation is associative and commutative, the framework caches sketches at coarser grains, and the estimate is presented only at the end. The exact versions remain available but cannot found a family, because no fertile source decomposition exists; the framework enforces the distinction, computing the exact mule from root each time and permitting the sketch variant to be re-aggregated.

Many families per column

A column has many families, one per fertile reducer it supports — its family set. revenue has its sum family, its max family, its min family, each fertile, each generating its own lattice of cached values. They answer different questions — total versus peak — and the framework requires you to say which you mean when the choice is open, resolving a bare reference to the declared default family.

Chapter 7. Applicability, Missing Values, and Degeneracy

Three matters that all concern when an operation is legitimate, and all resolve through the same principle: absence and inapplicability are permitted only when they are made explicit.

The B-anchor (blocked anchor)

Some reductions are meaningless along some axes, and the meaninglessness is not a matter of taste — it is structural. Revenue sums sensibly across customers, across days, across both. An account balance sums sensibly across customers but not across time: you cannot add Monday’s balance to Tuesday’s; it is the same money counted twice. A temperature sums sensibly across nothing. The physicist’s extensive/intensive distinction is the nearest familiar idea — extensive quantities (mass, revenue) add across the parts of a system; intensive ones (temperature, density, rate) do not — but it is not quite the distinction Columna needs here, and conflating the two is a classic source of error (it is taken apart below). What governs whether a reduction is legitimate along a given axis is captured precisely by the B-anchor, a finer fact than the extensive/intensive label.

Each family carries a B-anchor: the set of dimension families along which its reducer is not applicable for that column. SUM of revenue has an empty B-anchor — it applies everywhere. SUM of an account balance has the time family in its B-anchor — it must not coarsen across time. SUM of a temperature has every family in its B-anchor — it never applies. The B-anchor’s three shapes name the three classical applicability cases directly: a reduction legitimate along every axis has an empty B-anchor; one legitimate along no axis has a full B-anchor; a semi-additive one — legitimate along some axes, not others — has a proper subset. The B-anchor is more precise than any one-word label, because it names which axes, and it is anchor-invariant: stated in terms of dimension families, it does not need recomputing as the column’s anchor changes.

It is worth being exact about why the B-anchor is not the same thing as the extensive/intensive label, because the account balance is precisely the case that pulls them apart. An account balance is extensive — it is money; it scales with the number of accounts; summing it across customers is exactly right — and yet its SUM is semi-additive, carrying the time family in its B-anchor. So “extensive” does not imply “empty B-anchor.” Extensivity is a fact about whether a quantity scales with the size of its input, and it governs how missing values are corrected (the next section); the B-anchor is a fact about which axes a reduction may legitimately coarsen along. These are two independent properties of a (column, reducer) family. They coincide only in the simplest cases — a pure flow like revenue is extensive and has an empty B-anchor; an instantaneous temperature is intensive and has a full B-anchor — and they come apart exactly at the semi-additive measures that cause the most trouble in practice. A third, independent property — whether the reducer is fertile (Chapter 6) — governs whether a family can be cached and rolled up at all; AVG is non-rollable not because of any B-anchor but because it is a mule. Three orthogonal axes, then — applicability (B-anchor), scaling (extensive/intensive), and rollability (fertility) — and the framework tracks each separately because each fails in its own way and the naive collapse of any two of them into one is the source of a distinct family of silent errors.

When a query attempts to coarsen along an axis in the relevant family’s B-anchor — summing a balance over time — the framework catches it before any data is touched. What happens next is the inform-and-serve discipline of Chapter 11: the result, if the caller still wants it, is served carrying a critical disclosure that names the crossing and quantifies the double-count where it can (“summing a daily stock over 30 days counts the same money ×30”) — it is never produced silently, and it is withheld outright only where the Manifold’s author has declared a hard stop (WITHHOLD) on that family. And because the family set contains other families, the disclosure can say what is applicable: the balance has, alongside its SUM family (B-anchor includes time), a LAST family — an ordered reducer applicable over time precisely because it respects time’s order — and an AVG computation derivable from fertile sources. “What do I do across time?” is answered by inspecting which of the column’s families permit the time axis: use LAST, the closing balance. The ordered-reducer category from Chapter 4 is what makes the semi-additive story complete — it is not just that SUM over time is a named, quantified hazard, but that LAST is the right thing instead, and the framework knows it from the family set and offers it as the remedy.

Missing values

A value may be legitimately absent — a customer genuinely had no revenue that day, a measurement failed, a field is inapplicable — and absence-of-value is a coherent state of the world. The M-anchor, introduced in Chapter 3, characterizes why a value is absent, and that characterization determines how the framework handles the absence under reduction. The handling is principled rather than uniform, and it follows the mechanism. The detail belongs with the operator registry (Chapter 10), but the shape is worth previewing: under purely random missingness, an intensive reducer like AVG may simply skip the absent values and remain unbiased, while an extensive reducer like SUM must skip and rescale — multiply by the ratio of the covered universe to observed points (the denominator the universe of Chapter 5 supplies) — to remain an unbiased estimate of the total, because a bare skipped sum systematically undercounts. (This is exactly where extensivity earns its keep as a separate axis from the B-anchor: it is the scaling property, not the applicability property, that decides whether a skip needs a rescale. And the rescale is worth seeing plainly: it is exactly fill-every-hole-with-the-observed-mean, collapsed into one multiplication — which is what its disclosure says.) Where missingness depends on observed coordinates the handling stratifies when those coordinates survive the rollup and propagates the missingness forward when they are coarsened away; where it depends on the unobserved value itself — the MNAR case, which is provably not testable from the data — no automatic correction is sound, and the framework serves the observed-data figure with exactly that fact disclosed, never minting a population estimate it cannot stand behind. The full behavior vocabulary is exactly five — skip · skip-and-rescale · stratified · propagate · fill — and Chapter 10 specifies which the mechanism selects. (Withholding is deliberately not on this list: an author who wants a hard stop declares WITHHOLD, a governance rule, never a missing-data treatment — Chapter 10.) The point here is only that missing values are a real, modelable phenomenon, governed by the M-anchor, and that the framework’s response is determined by the declared mechanism rather than by a single careless default.

Subsetting and degeneracy

A column can be restricted by a predicate — east-region revenue, the subset of points where region is east. This operation, subsetting, is unlike map, aggregate, and broadcast: it does not transform values, change grain, or spread values. It restricts the domain — which anchor points are present — producing a column defined at fewer points. It belongs to the value-determination concern of Chapter 3, alongside the V-anchor and the M-anchor, because what it writes is the column’s extent: which slice of the universe the column actually covers.

A column that covers less than the full universe of its anchor is degenerate, and degeneracy is allowed exactly when it is completely and explicitly specified. This is the same principle that governs missing values (absence permitted when its mechanism is characterized) and the same principle that will govern declared incompleteness at the source level (Chapter 11): Columna permits absence only when it is made explicit. Subsetting is safe precisely because the bracket predicate is, by construction, a complete and explicit specification of the degeneration — revenue[region="east"] carries its own restriction as its identity, so it can never produce an unspecified, unaccountable incompleteness. The degeneration is named the moment it is created.

The same degeneration can arise two ways — declared at the source when a column is bound to data that already covers only part of the universe, or produced at query time by subsetting — and both yield the same kind of object: a fully-specified, restricted column that carries its restriction predicate as part of its identity. Both feed the same downstream machinery: when a degenerate column is later combined with others, the universe-compatibility checks of Chapter 11 consult its restriction, so that east-only revenue combined with all-customer cost is recognized as a universe mismatch rather than silently producing a biased result.

Part III — Dimensions, Relationships, and the Manifold

Chapter 8. Dimensions and Their Relationships

Dimension is a role, not a kind

A column is not intrinsically a dimension or intrinsically a measure. Recall from Chapter 1 that one column’s values can be another column’s anchor — revenue is anchored on customer, and the customers that index revenue are the values of the customer column, which is itself anchored on something finer. So customer is, in its own right, a column; and in relation to revenue, it is a dimension — an anchor component. The same column plays both roles at once, with no contradiction. A column is a dimension whenever it serves as an anchor component for some other column, even though it is not always in that role. Dimensionhood is a role conferred by use, not a category a column is born into.

This resolves cleanly a question that the role/kind confusion makes hard: may a dimension have missing values? The answer follows from what the role does. A dimension’s job is to identify — to say which point a datum is about. A missing identifier identifies nothing; a datum about an unknown anchor is not an incompletely-specified datum but, under the property-at-anchor ontology, not a well-formed datum at all. So: a column may not contain missing values while it is serving in the dimension role. Stated most precisely, the requirement is on the anchor: an anchor’s coordinate values must be present and uniquely identifying. The column itself may have missing values in general — governed by its M-anchor, in its measure role — but the anchor built from it, when it serves as a coordinate, must be missing-free.

The requirement is met not by rejecting data but by naming its absence into existence. Where data appears to have a missing dimension value — an anonymous transaction, an unknown region — the correct modeling is to promote the absence to an explicit, named value: ANONYMOUS, UNKNOWN, NOT_APPLICABLE. This is not a missing value; it is a first-class category that identifies its points (they are the anonymous ones), participates in hierarchies (an unknown customer maps to an unknown region, keeping the roll-up total), groups correctly, and — the practical payoff — reconciles: the unknown bucket is a real group whose total sums into the grand total with all the others, where a silent null group would have made the totals fail to add up. The prohibition on missing dimension values and the promotion of unknowns to named categories are, again, one instance of the framework’s standing principle: absence is permitted only when explicitly named.

Dimension families

Dimensions relate to each other through roll-up maps — day to month to quarter to year; store to city to state. When these maps form a commuting hierarchy — when day → month → quarter gives the same result as day → quarter directly — the dimensions form a dimension family. The commuting property is what lets the framework navigate the hierarchy automatically, by any path, to the same answer.

Dimension families are how Columna replaces the joins that a SQL writer would author by hand — not by hiding them below a line but by retiring them as operations. The relationship customer → region is declared once as an edge in the customer family and verified against the data — every customer must map to exactly one region, a total function, a functional dependency. Once declared and verified, every column anchored on customer is automatically queryable by region: the engine fetches the column at customer, fetches the customer → region edge as itself a column-atom (a key→key mapping), and transports the value along it to region — inside the engine, the backend having only delivered two columns and never related them. Transport is the operation that does the work a join did, on the column foundation: it is defined only along functional edges — the reducing direction, where each value moves to the single point it maps to, without replication — so it carries the meaning faithfully, and the relationship never has to be written again. A base dimension may root more than one family — day may roll up to calendar week and to fiscal_period independently — and where two such hierarchies do not relate to each other, a query must say which it climbs.

When a relationship is not functional

Sometimes the relationship between fine and coarse is genuinely many-to-many: a product belongs to several categories, a session touches several campaigns. This is granted reality. There is nothing wrong with it, and the framework takes no issue with its existence. Most operations on it are entirely unconstrained: querying either side, filtering products by category membership, listing the categories a product belongs to — none of these invoke any assumption the relationship fails to meet, and all are well-formed.

A single operation is the exception. Aggregating a measure across the relationship — revenue, at product grain, summed up to category — has a soundness precondition: that the relationship be functional, so each product’s revenue lands in exactly one category, the partition property that a reconciling aggregation requires. A functional relationship supplies this; a many-to-many one does not. The replicate-and-sum an unguarded join would perform here — each product’s revenue copied into every category it belongs to and the copies added as though they were independent observations — is fan-out, and fan-out is not a computation that is merely risky but one with no valid target. Transport carries a value along a functional edge to the single point it maps to; along a non-functional edge there is no single point, so the transport is undefined and the aggregate-across denotes no one number at all.

That is the line that decides the outcome, and it is worth stating as a principle because it sorts every hazard in the framework. A request that denotes exactly one number is served — with a disclosure if that number is risky, the way a B-anchor crossing (summing a stock over time) is served-and-flagged rather than withheld (Chapter 7): the computation is determinate, so the framework computes it and puts the risk on its face. A request that denotes no single number — that is underdetermined among several legitimate readings — is not served and not guessed. The framework returns no number and clarifies: it names what is underdetermined and what would determine it, and lets the user (or the agent, in-loop) choose. Aggregate-across-a-non-functional-edge is the second kind. Its legitimate neighbours — membership, primary designation, allocation — are different questions with different numbers, and silently picking one would be exactly the procedural decision the framework exists not to make on a user’s behalf. So it clarifies. The earlier reading of this manual, on which a bare aggregate-across defaulted to the membership number with the overlap disclosed, is retired: membership is a real answer, but it is the answer to one of the three questions, to be chosen, not the silent default of a request that did not name it.

The framework’s three responses are three ways of relating to that single precondition — and, equivalently, the three answers the clarification accepts. Membership filtering stays on one side and never invokes the precondition — “revenue of products that are in the Fitness category,” a subset on the product side, totals allowed to overlap. Chosen explicitly, it is served (the overlap is expected, correct, and disclosed as such); it is the one reading that needs no new structure. Primary designation makes the relationship genuinely functional — declare a primary_category attribute, so product → primary_category is a true function, and aggregate-across over it is well-formed for real, transporting faithfully. Allocation [Pro] supplies the missing functionality synthetically — a partition-of-unity weighting that distributes each product’s revenue across its categories, summing to one per product, manufacturing exactly the partition property aggregation needs, so totals reconcile to the whole; the declared split is a Pro capability, so in Core, absent a primary designation, the aggregate-across stays a clarification rather than silently allocating. The relationship is never the problem; the only question is which of these the user means, and the framework asks rather than assumes.

A word on the other direction — broadcast — to forestall a tempting but wrong symmetry. It would seem that if aggregating an extensive measure up a many-to-many needs an allocation, then broadcasting one down the lattice should need one too: spread a coarse total across the finer points by apportioning it. Columna deliberately does not do this. Broadcast has a single semantics — replicate: copy the coarse value unchanged to every finer point — regardless of whether the measure is intensive or extensive. Its purpose is to make a coarse value available at a finer grain for a downstream comparison (a share-of-total fine / coarse, a deviation from a group mean), never to distribute a total into a stored finer measure, which answers no real question. The double-counting hazard — replicate a total, then sum it back up — is real, and the framework forecloses it not by apportioning but by the B-anchor: a replicated extensive value carries a B-anchor on the broadcast dimension, so summing it back up the axis it was spread along is caught by the ordinary applicability check and can never happen silently — it is the same critical-disclosure machinery that guards every B-anchor crossing (Chapter 7). Proportional down-allocation of a total is therefore not an anchor-action in Columna at all. (This is a different thing from the many-to-many allocation just above, which supplies a partition-of-unity to make an aggregate across a non-functional relationship well-formed; that remains, as a Pro capability. Broadcast spreads down the lattice and only replicates; allocation reconciles an aggregate up a non-functional edge.)

Chapter 9. The Manifold

What a Manifold is

A Manifold is not, first of all, “the thing you query,” though it is that. It is, more fundamentally, the abstraction of the complete structure of a data space — the collection of columns together with everything required to make that data space self-contained, queryable, and verifiable. It is the operational unit because it is complete: nothing essential to posing a clear question or trusting an answer lives outside it. Hand someone a Manifold and they have the whole apparatus.

Concretely, a Manifold comprises a small set of components:

The column set — all the columns the Manifold addresses, both the anchors (dimensions) and the values (metrics). Each column is fully declared (V-anchor, universe binding, M-anchor, family set with B-anchors and missing-policies, default family, coverage), the way Chapters 3 and 5 described.

The universe set — the declared point sets, each with its basis (registry, spine, product, events projection — Chapter 5), that the Manifold’s columns and ascriptions land on. Events projections are derived from grain roots, never declared; the declared universes are the Manifold’s account of which points exist, shared and referenced by name so that co-universality between columns is a checkable relation rather than a hope.

The defining boundaries — the constraints that say what this Manifold is about. These are typically temporal (the data the Manifold covers ranges from such-a-date to such-another) and dimension-value-subset (the Manifold covers US customers, or the consumer product line, or internal accounts). The defining boundaries are shared by every column in the Manifold: they describe what scope the entire data space is bounded by. They are not column-level filters; they are constitutive of the Manifold — the Manifold is the data within these boundaries. Two Manifolds with the same column declarations but different defining boundaries are different Manifolds, addressing different data spaces.

The operator subset — which operators from the framework’s catalog the Manifold uses, with any name aliases that give the Manifold its local vocabulary (a finance Manifold may call SUM “total”; a retail Manifold may call COUNT_DISTINCT “unique_visitors”). The aliases are a Manifold property: they shape the language of queries against this specific Manifold without changing the underlying algebra.

Derived columns — additional metrics and dimensions defined as expressions over the column set, included in the Manifold and queried like any other column. Derived columns are computed on demand from their inputs, with the framework propagating B-anchors and other properties as Chapter 10 specifies.

Custom functions and reducers — extensions to the operator set specific to this Manifold (a Pro-tier capability). Where a domain has computations the framework’s stock catalog does not provide, they are registered on the Manifold as custom operators with their full algebraic signatures.

These components are what the Manifold declares. The framework’s machinery (the integrity checks, the certificate, the planner) operates on these declarations. The defining boundaries are the hinge between the abstract declarations and concrete data: the column declarations are templates that could describe any data space; the boundaries specify which data space this Manifold is. Without the boundaries, the declarations are a mere schema; with them, the Manifold is an instantiated, queryable thing.

Two projections of one spec

One spec, but the framework reads it through two projections, and the split is not access hygiene — it is the thing that makes a mis-placed disclosure impossible. The planner sees a vocabulary/shape projection: which columns and anchors exist, the roll-up and relationship topology (enough to typecheck a transport — can this column reach that anchor along functional edges?), the defining expressions of derived columns, and types. It does not see provenance — where a column is physically sourced, what it costs to deliver, which caveats its source carries. The engine sees the resolution projection: the source bindings, the physical realizations, the faithfulness rules, the costs, and the per-source caveats. Because a layer can disclose or refuse only what its projection lets it see, the boundary forces every hazard to be raised by the layer that can actually know it: a static, shape-level hazard — a fan-out, an out-of-universe address — is the planner’s, because it is visible in the shape before any data is touched; a resolution-level hazard — a coverage gap between two sources that nominally share an anchor — is the engine’s, because only resolution reveals it. It is impossible to put a caveat in the wrong place, because the wrong layer lacks the information to fabricate it. This is why “the planner cannot know provenance” is a structural fact rather than a convention.

{} and the defining boundaries

A connection worth surfacing: the empty anchor {}, the lattice’s top, is not a context-free abstract notion. In any specific Manifold, {} is the Manifold’s defining boundaries collapsed to a single anchor point — the totality of what this Manifold is bounded by, reduced to one scalar position. A column queried at {} produces the grand aggregate over the Manifold’s defining boundaries; nothing more, nothing less. {} is therefore Manifold-relative — the {} of a US-customer Manifold is the US-customer total; the {} of a global Manifold is the global total; they are different scalars because they aggregate over different boundaries. The defining boundaries and {} are two ways of saying the same thing: the boundaries are what {} aggregates over, and {} is the boundaries as a lattice point.

Roots are per-column, defining boundaries are Manifold-wide

A subtle pair worth being precise about, because confusion between them is easy. Defining boundaries are Manifold-wide: every column in the Manifold operates within these shared boundaries. Roots are per-column: each column has its own native finest anchor, where it is observed in the data. These are different things at different levels. The boundaries say what universe the Manifold is about; each column’s root says where within that universe the column natively lives. Most columns root finer than {} and coarser than infinitesimal — at {customer, day}, {transaction}, {warehouse, week}, whatever the data supports. The lattice is shared (built from the Manifold’s dimensions); the boundaries are shared (built into the Manifold’s identity); the rooting positions vary by column.

Small Manifolds by design

A Manifold is sized, by design, for a team and a question rather than an organization — a finance Manifold for revenue and cost, a product Manifold for engagement, each owned by the small team that understands its data, each fast to build and confident to certify. The trade-off — that an organization ends up with many Manifolds — is addressed by composition, and composition is made natural by the complete-structure definition: a composed Manifold is still a Manifold because completeness is preserved under composition. Two complete structures reconcile into a larger complete structure; closure under composition is just the statement that completes compose to completes. That is why there is no separate “cross-Manifold query” mode — to query across Manifolds, you compose them into one, and query that.

Inheritance and joining

Composition takes two forms. Inheritance refines a parent: a child Manifold adopts everything the parent declares and adds refinements — a narrower set of defining boundaries, additional derived columns, extra reducers on existing columns. The child may narrow and extend but never contradict: it cannot loosen the parent’s B-anchors or widen the parent’s defining boundaries. A company-wide parent is inherited by team-specific children that add local conventions, and the parent’s guarantees flow to the children automatically.

Joining combines peers: two Manifolds become one containing both their metrics and dimensions, with shared dimensions reconciled. Reconciliation is real work — defining boundaries intersected or verified compatible, shared dimensions checked for compatible hierarchies and value sets, name collisions disambiguated — and the join’s certificate is derived from the parents’ certificates plus the reconciliation checks.

Composition and universes. Universe declarations compose under the same narrow-don’t-contradict rule as everything else. A child inherits the parent’s universes by reference — the same named objects, so co-universality across the generations is verifiable by reference, the strongest and cheapest verdict — and may declare new universes of its own, but may never redefine or widen an inherited one; the child’s narrower defining boundaries narrow each inherited universe’s covered extent implicitly, with nothing to re-declare. Joining requires the universes of shared dimensions to be reconciled exactly as the dimensions themselves are: declared the same is checked against the cross-parent population evidence the coverage machinery already maintains; declared different is preserved, with the asymmetry permanent and disclosed wherever the two are combined; and a join carrying an unreconciled universe reference is refused like any other unresolved reconciliation — composition’s one legitimately hard gate, because an unreconciled reconciliation is not an analytical risk but an unanswered question.

Manifolds are versioned, and composition references parent versions explicitly: a child inherits from parent version 3, not from “whatever the parent currently is,” so that an upgrade upstream does not silently change downstream, and so that a past analysis can be reproduced by checking out the versions used at the time. The Manifolds of an organization form a composition graph — parents and children, peers and joins — and that graph is itself a queryable artifact, for lineage and for impact analysis.

Part IV — Operators, Integrity, and the Agent

Chapter 10. The Operator Registry

Each Manifold carries an operator registry: a typed catalog of the operators available to it. For each operator the registry records the operator-level facts the planner needs — its algebraic signature (operation kind, order-dependence, fertility classification for reducers), its intensive-or-extensive bit (which determines, combined with each column’s M-anchor, the missing-data policy applied), the rule by which B-anchors propagate from inputs to outputs under this operator, and any name aliases that present the operator under a domain-specific name in this Manifold. The registry is operator-level: it does not duplicate information that lives on individual columns (each column’s actual B-anchor, missing-policy choices, and per-(column, operator) overrides live in the column-spec, where they belong as column-specific declarations).

The default missing-data policies in the registry are drawn from a fixed vocabulary of five behaviors — skip · skip-and-rescale · stratified · propagate · fill — and the planner selects among them from two facts: the operator’s intensive-or-extensive bit and the column’s missingness mechanism (MCAR / MAR / MNAR). Under purely random missingness an intensive reducer skips absent values and stays unbiased, while an extensive reducer skips-and-rescales — scaling the observed sum by the ratio of the covered universe to observed points (Chapter 5) — to remain an unbiased estimate of the total rather than silently undercounting. Where missingness depends on observed coordinates the policy stratifies while those coordinates survive and propagates the flagged marginal when they are coarsened away. Where it depends on the unobserved value itself — the MNAR case — no automatic correction is sound, and the framework serves the observed-data figure with an UNTESTABLE-state disclosure stating the unknown-direction bias: it never presents the figure as a population estimate and never mints a correction it cannot stand behind. The selected behavior is recorded in the result’s annotation, so a reader can always see what was done. These defaults are correct under their stated assumptions — they are never silently wrong — which is the standard the framework’s open core holds itself to. Withholding, finally, is not in this vocabulary at all, and the omission is the point: declining to serve is never a missing-data treatment and never the engine’s analytical judgment. An author who wants a hard stop on a (column, reducer) declares WITHHOLD — a governance rule, carried in the definition language, executed by the engine as the author’s rule with the permitted alternatives reported (Chapter 11).

The column-spec may override the default per (column, operator) where a specific column’s situation calls for it. Fill is the constant-substitution behavior — declaring fill with the constant zero for missing revenue, say, when that is the right modeling choice for the data; note that fill substitutes a constant, it does not estimate. Estimation — recovering a plausible value rather than asserting a fixed one — is what skip-and-rescale and stratified do within the framework’s principled set. Sophisticated missing-data methods — model-based imputation, multiple imputation, selection-model corrections — are registered as additional behavior through the same column-spec override mechanism, with Core supporting the constant-fill and the principled rescale/stratify behaviors and Pro adding the sophisticated estimators.

Chapter 11. Integrity

A framing point that matters

The framing of this chapter changes how integrity should be understood, and it must come first: the integrity preconditions are not Columna’s conventions about what makes a Manifold well-formed. They are the data conditions that any correct analysis on the data would require, regardless of what tool performs the analysis. A functional dependency that fails — customer 19847 mapping to two regions in the data — is a defect in the data, and it is present whether the data is queried through Columna, SQL, pandas, or anything else. Any query that climbs that relationship is silently wrong in any tool that doesn’t check. Columna uncovers and reports such defects because it checks; conventional tools do not check, and so produce wrong answers without saying so.

The framework’s contribution is not adding rules; it is verifying rules that have always been there. The integrity preconditions are real properties of data — properties the data must have for any correct analysis to be possible — and Columna’s value is in making the verification systematic, making sure no number that depends on a failed condition ever travels without the failure on its face, and surfacing the defects so they can be fixed at the source.

This is why the language of this chapter avoids the words well-formed and well-formedness in the integrity context — those words carry a connotation of conformance to the system’s own conventions (a connotation that correctly applies to query-language well-formedness, where expressions must conform to a grammar). Integrity is not Columna’s conventions; it is the data’s soundness for analytical use. The words this chapter uses are sound, soundness, valid, validity. A Manifold whose integrity preconditions are satisfied has data soundness for the analyses it supports; one whose preconditions fail has data defects the framework is reporting on.

A Manifold answers only on the record

The deepest commitment of the framework: a Manifold is valid for an analysis only insofar as the integrity preconditions that analysis depends on are satisfied — and what the framework does about a failed precondition is where this edition states its identity plainly. The first editions said fail closed: refuse the dependent query, return a number that doesn’t exist instead of a number that is wrong. That instinct — never a silent wrong number — was exactly right, and its mature form is stronger: inform and serve. Where a sound route around a defect exists, the planner takes it and discloses the routing — recompute the mule from its fertile sources, serve from detail past a contradicted pre-aggregate. Where none exists, the framework serves the producible number with the defect riding on it — named, quantified, directional where determinable — because a tool can never know why a user asked, and withholding a determinate result on the tool’s own analytical judgment is not safety but paternalism. What survives as genuine non-service is exactly the non-analytical set: a question that does not determine an answer (a clarification, asked rather than guessed), data that is not there (an inform), and a human rule — the author’s declared governance (WITHHOLD, quarantine), a caller’s entitlement — reported as whose rule it is and why. The safety asymmetry of the certificate is preserved verbatim; what changed is that the certificate’s verdicts now ride to the human instead of gating the bytes. Nothing fails open: a Columna query that returns a number has returned one that is correct under the data conditions the framework verified, or one that carries the unverified condition on its face — never a silent number.

The layered preconditions

The preconditions are layered by what they govern and what their failure invalidates, so that a single failure voids only what depends on it rather than the whole Manifold:

Anchor soundness — keys uniquely identify their points; dimension coordinates are present and identifying (Chapter 8); values are total functions on their anchors or have explicitly modeled missingness. Failure invalidates the affected column.

Co-anchoring integrity — columns declared co-anchored truly share their anchors. Failure invalidates expressions that combine the mis-aligned columns.

Hierarchy integrity — every declared roll-up edge is a true total function in the data. Failure invalidates climbs along the broken edge.

Coordinate totality — every root point of a fertile family carries a value for every dimension family any aggregation path traverses: no null coordinates at the root, no orphans, every traversed child mapping to exactly one parent. This is the checkable core of the conservation invariant (Chapter 5): with path-invariance typed and one root point set declared, totality is what makes a family’s grand total path-independent. Failure invalidates the paths through the deficient dimension — and nothing else — with a disclosure that can be quantified (“the region path excludes 312 transactions, 0.4% of total, carrying null region; the customer path includes them”).

Applicability integrity — the declared B-anchors are consistent with the data’s behavior. Failure invalidates aggregations that rely on the broken claim.

Universe integrity — the Manifold’s defining boundaries are satisfied by all column data; declared coverage matches the data; and columns combined at a shared anchor are co-universal as declared (Chapter 5): same declared universe, verified by reference; declared-same but drifting in fact, the drift finding against a named object; different universes outright, a legitimate combination carrying its permanent asymmetry. Failure invalidates universe-sensitive combinations.

Each layer corresponds to a class of data conditions that any correct analysis would require. The framework verifies what is verifiable, tags what is asserted, and discloses what is unsupported — withholding only where a human rule or an empty support leaves nothing to serve.

What can be checked, and what is asserted

The framework is scrupulous about a distinction conventional tools blur. Some preconditions are checkable from the data: functional-dependency validity, anchor uniqueness, satisfaction of declared restriction predicates, the consistency of many additivity claims, the coverage of columns against declared universes. Others are asserted, not checkable: whether missingness depends on the unobserved value itself (the MNAR case is, provably, not testable from observed data alone), and the definition of the defining boundaries themselves (the data can be checked for internal consistency with the declared boundaries, but the boundaries themselves are taken on the author’s word). Each precondition in a Manifold’s certificate is tagged by its evidential basis — checkable or asserted — so that the certificate is an honest account of what reality has confirmed and what the author has taken on faith. (This evidential basis — can the data confirm or contradict the precondition at all? — is distinct from the verdict the framework then reaches on it, the four states introduced below.)

Universe integrity and coverage

A subtle, costly, and rarely-checked matter deserves emphasis. Every column in a Manifold implicitly claims to operate within the Manifold’s defining boundaries — to describe the universe the Manifold is bounded by. If a column actually covers less than the defining boundaries (the data behind it covers only EU customers in a US-and-EU Manifold, say), every combination of that column with others is silently biased — each column internally consistent, the error living only in the mismatch of their populations.

Columna’s response: every column carries an explicit coverage state — complete (covers the Manifold’s defining boundaries fully), filtered (covers a declared subset of those boundaries, with the restriction predicate verified and the restricted data declared usable), or quarantined (incompletely covers the boundaries with no specification). Operations involving filtered columns produce results restricted to the relevant intersection, with the universe the answer reflects disclosed; a quarantined column does not participate in cross-column operations — and the framework reports that exclusion for what it is, the author’s standing coverage governance (an inform naming the remediation: declare the coverage, or fix the defect at the source), never the engine’s own analytical judgment. The framework maintains, for each dimension, information about how the columns anchored on it actually cover the boundaries — overlaps, asymmetric differences, coverage scores — and tracks the information across refreshes, so that drift becomes visible: when two columns that historically overlapped by 95% suddenly overlap by 70%, something has changed upstream, and the framework can say so the moment it crosses the threshold. The check, since Chapter 5, has its proper object: what is being verified between two columns at a shared anchor is co-universality — the other half of co-anchoring. This catches a class of slow-onset corruption that conventional warehouses discover, if ever, only quarters later — and exactly because the framework’s integrity machinery treats universe coherence as one of the data conditions correct analysis requires, not as an optional ornament.

The certificate

All of this is recorded in the Manifold’s integrity certificate, which is itself queryable data: what each precondition’s verdict is, on what evidential basis, what is currently failing, and how the state has changed over time. “The number came from this Manifold, at this version, at this refresh, with this certificate state” is a fully reproducible statement — which is what makes a Manifold’s output trustworthy in a way a warehouse query’s is not.

Each precondition carries one of four verdicts — never a pass/fail bit, never a numeric score:

• VERIFIED — the data confirms the precondition: it is checkable, and it checks out.
• CORROBORATED — the precondition is consistent with everything the data can show, but the data cannot positively establish it (an asserted claim the evidence is compatible with, or a checkable claim a null finding has not falsified). The strongest verdict an asserted layer can earn.
• CONTRADICTED — the data exhibits a positive defect against the precondition: a functional dependency that splits, a coverage that drifts, an additivity claim the data violates. A contradicted precondition never travels silently: the dependent result is routed around the defect where a sound route exists (the routing disclosed), and otherwise served carrying the contradiction as a quantified finding.
• UNTESTABLE — the precondition is, in principle, not decidable from the data at hand: the MNAR case, and the defining boundaries themselves. The certificate says so plainly rather than implying a clean bill of health.

These verdicts obey a deliberate safety asymmetry. A positive finding is sufficient to CONTRADICT a declaration — one customer mapping to two regions is enough to break the dependency. A null finding is never sufficient to VERIFY an asserted declaration — the absence of a detected defect is at most CORROBORATED, never a guarantee. The framework will fail a thing on evidence, but it will not bless a thing for lack of evidence. This is why MNAR and the genuinely-untestable boundaries are UNTESTABLE, not VERIFIED: their absence of contradiction is not confirmation.

A consequence worth stating: when an author instructs the framework to proceed past a gate that would otherwise withhold — a declared WITHHOLD, a quarantine, a binding-time requirement — or acknowledges a finding flagged as requiring acknowledgment, the act is recorded in the result’s annotation, the precondition retains the verdict it had earned, and the result is demoted to at most CORROBORATED. An acknowledged or overridden finding is never silently promoted to VERIFIED.

The certificate is honest about what it does and does not vouch for: it can certify that the data is internally consistent with the declared assumptions, but it cannot certify that the declarations capture the world correctly. A Manifold author who declares that a column covers “US customers” is making a claim the framework cannot independently verify; the certificate records the declaration, verifies the data-consistency around it, and is explicit that the declaration itself is asserted. The trust the framework offers is real but bounded: real, because within the scope of what has been declared and verified, the number is right; bounded, because the framework cannot vouch for what only the author can assess.

Chapter 12. The Agent

Talking to your data

A Manifold is queried in a formal language, and that language is the bedrock — what the planner compiles, what the certificate is stated against, what tools integrate with. But most people who need answers from data do not want to write formal queries, and they should not have to. Columna’s agent is the surface that lets them simply ask.

The agent is not a chatbot wrapped around a SQL generator. It is bilingual by design: fluent in ordinary analytical language on one side — the vocabulary of business questions, vague and elliptical and context-laden as real questions are — and fluent in the Manifold’s formal structure on the other, knowing its dimensions, metrics, families, B-anchors, restrictions, coverage, defining boundaries, and certificate. Its work is the bridge between them: to figure out which well-formed query a vague request maps to, and to surface the choices the requester did not realize they were making.

This is why the agent’s job is achievable where text-to-SQL is not, and the reason traces straight back to the philosophy of Chapter 2. If the analyst’s task has been reduced to expressing a specification — because the procedure has been absorbed into the Manifold — then the gap between a person expressing a specification and an agent expressing one is small, because the hard part, the procedure, has been removed for both. Text-to-SQL is hard because the agent must author a procedure against a substrate that represents none of the facts a procedure’s correctness depends on — the gap is the substrate’s, not the model’s. The Columna agent authors a specification against a substrate that validates it — and a substrate that discloses. Every query the agent proposes is validated before it runs: EXPLAIN, the language’s first-class validation statement, returns the framework’s exact reading — the canonical desugared form, the planned decomposition, the dependency cone with its verdicts, and the complete would-be annotation — without touching data. So the agent works in a tight propose-validate-refine loop: propose a formal query, EXPLAIN it, and either run it — presenting the answer together with its annotation and an offer to adjust — or, when the validation surfaces genuine ambiguity rather than a fixable slip, ask the requester to clarify, in their own vocabulary, the choice that needs making.

What makes it trustworthy

When the substrate discloses everything a number depends on, the agent can be more confident when it serves and more honest when it is unsure. A query that runs has passed the Manifold’s full integrity machinery, and whatever that machinery could not verify arrives as findings the agent must carry into its answer: the number is correct under the verified conditions, or its exact risk is in the agent’s hands to present — never silently in neither. And when a request is genuinely ambiguous — a metric that has several families, a relationship that is many-to-many, a grain left unspecified, defining boundaries that do not match — the agent does not produce a plausible guess. It surfaces the choice in business terms and lets the requester decide.

Above all, the agent shows its work. Every answer is inspectable: the formal query as executed (the canonical form, returned in every annotation), the interpretation chosen (which family, which reducer, which grain, which scope, which universe), the certificate state at query time, the coverage of any cross-column combination, the provenance back to source. The agent’s authority is borrowed entirely from the substrate’s verifiability, and surfacing that verifiability is what earns the requester’s trust. The agent never freelances outside the substrate — never invents a metric, never bypasses the engine, never guesses where the substrate asks for clarification, and never drops a finding the annotation carries — because the moment it did, it would forfeit the one thing that distinguishes it from a tool that confidently tells you wrong answers.

The text-to-SQL comparison

The obvious competitor to a Columna agent is the text-to-SQL approach: a language model that takes a natural-language question and generates SQL against a conventional warehouse. The comparison is worth making directly, because the difference is structural and decisive, not a matter of degree.

Text-to-SQL is the harder problem from an AI-correctness standpoint, and it is harder for reasons that better models do not solve. Generating SQL means authoring a procedure: choose the tables, the joins, the join keys, the grouping grain, the filters, the order of operations. Procedures are exactly where the silent errors of analytics live — the fan trap from joining at different grains, the silent double-counting from a many-to-many treated as a hierarchy, the average-of-averages from re-aggregating a mule, the bias from combining columns at mismatched universes. A language model authoring SQL has no way to be confident it has avoided these errors, because the SQL itself does not represent the structural facts that would let anyone — model or human — check. The SQL runs and produces a number; whether the number is correct depends on whether the procedure was right, and there is no second observer who can verify it.

A Columna agent does a different job. It authors a specification against a substrate that verifies it. The hard part — the procedure — has been absorbed into the framework’s algebra and the Manifold’s declarations, where it is reusable and verifiable. The agent’s task is to translate a business question into a well-formed Frame-QL expression; the substrate type-checks the expression, asks for clarification where the expression does not determine one answer, and otherwise produces (result, annotation) — a number bound to the conditions it depends on, each verified or disclosed, with structured findings the agent can act on. The agent does not have to be right about the procedure because there is no procedure to author; the procedure has already been written into the Manifold.

This is not a small difference. It is the difference between an unreliable agent operating on a substrate that cannot help it (text-to-SQL) and a structurally bounded agent operating on a substrate built to support it (Columna). Specifically:

The text-to-SQL agent has no structured feedback signal. A wrong SQL query typically runs and returns a wrong number; the agent learns nothing. The Columna agent has structured signal at every step of the loop: EXPLAIN validates a proposal before any execution cost and returns the framework’s reading of it; clarifications are categorized, scoped, and actionable — missing alias, mule without explicit input anchor, ambiguous hierarchy path, missing order; findings on served results are targeted and quantified — B-anchor crossing, coverage gap with bias direction, contradicted edge, routed around. Every cycle of the propose-validate-refine loop carries real signal.

The text-to-SQL agent cannot inspect its own work meaningfully. A SQL string offers no provenance: which assumptions did this number depend on, which family was used, which path through the lattice was taken, what was the certificate state. None of these facts exist in the SQL execution because they do not exist in the SQL data model. The Columna agent surfaces all of them — they are properties of the query plan against the structured substrate — and the human reading the answer can see exactly what was computed and on what basis.

The text-to-SQL agent operates on a substrate that does not represent the facts needed for correctness. The structural facts that determine whether an analytical computation is sound — fertility, B-anchor, M-anchor, coverage, FD validity, defining boundaries — are not in the SQL schema. They cannot be checked, because they are not declared. Columna represents these facts explicitly; the agent consults them, the substrate enforces them, and the resulting numbers carry the correctness of the declarations behind them.

The improvement Columna offers is not a better model and not better prompts. It is a structurally different substrate. A more capable language model generating cleaner SQL against the same warehouse will still produce wrong numbers in the categories Appendix C catalogs, because the substrate it is writing against does not represent or enforce the facts those numbers would depend on. Columna addresses the substrate, not the model — and that is why the improvement is not marginal but kind-different.

The positioning that follows is direct: Columna is the analytical substrate purpose-built for AI agents to serve business questions reliably. Text-to-SQL agents are limited by the substrate they operate on; their reliability ceiling is the ceiling of the data model beneath them, which is procedural and unchecked. Columna agents are limited by the quality of Manifold authoring, which is verifiable, certifiable, and progressively improvable. The locus of trust has been moved from the per-query procedure (where no consumer’s diligence — human or machine — can be verified) to the per-Manifold declarations (where the framework’s verification, certification, and structured disclosure make trust earnable). This is the framework’s deepest contribution to AI-driven analytics, and the reason its value cannot be matched by improving the language model alone.

One honest caveat completes the comparison. Columna’s correctness guarantees hold given correct authoring — declared B-anchors that match reality, M-anchor characterizations that are honest, defining boundaries that are accurate. Where authoring is sloppy, the framework’s verifications catch what they can verify (FDs, anchor uniqueness, coverage consistency, hierarchy validity) but the asserted-only declarations are taken on faith. The framework’s honesty about what is checkable versus asserted (the certificate’s checkable/asserted tagging) keeps this caveat from undercutting the value claim: even imperfectly-authored Manifolds catch most of the silent errors text-to-SQL produces by default, and the framework’s verifications catch a substantial subset of authoring mistakes too. Bad authoring degrades Columna’s value; bad data against SQL produces wrong numbers either way. The comparison still goes Columna’s way decisively, with the qualifier that Columna’s contribution depends on the substrate being declared, not just queried.

Part V — Open Core and Pro

Chapter 13. What Is Core and What Is Pro

Columna is open-core. The line between the open Core and the commercial Pro is drawn on a single principle, and stating it plainly is the best protection against the line drifting:

Core is complete and correct for an individual or a single team operating a single Manifold at the scale that fits on a machine or a directly-queryable warehouse, with the full experiential value loop. Core is never crippled and never silently wrong. Pro adds three things — reach, sophistication, and operations-at-scale — and every Pro capability is genuine additional engineering that matters when an organization moves from experiencing Columna to running it as infrastructure. Pro is never merely Core-with-a-license.

One commitment sits beneath that principle and must be named, because getting it backwards would invert the whole product: the differentiator ships in Core. The column foundation itself — columns related by transport rather than backend joins, with fan-out and non-reconciling reaggregation made inexpressible rather than caught after the fact — is a Core property, present in the free tier from the first line. It is what makes Columna a different kind of thing than text-to-SQL; if it lived in Pro, then Core would be text-to-SQL with a governance wrapper and Pro would be “the real idea, later” — backwards, and fatal. So Pro’s additions are reach, sophistication, and operations-at-scale (below); they are never the correctness idea itself. “Not optimized” is allowed to mean Core takes the simple correct path — aggregate at the home anchor, then transport — even when slow; it is never allowed to mean Core takes a fast shortcut that relates columns in the backend.

Against that principle, the line falls as follows.

Integrations. Core supports two backends — Polars and the full capabilities of DuckDB, including its reach into Parquet and object storage — which together make a complete, genuinely useful analytics stack that anyone can adopt in an afternoon without infrastructure. Pro opens the enterprise backends, with the connector engineering and the support that running against them in production requires.

Surfaces. Core supports three surfaces — the query interface, the agent, and an MCP server interface so that Core can act as a reliable data substrate inside other tools and agent ecosystems. Putting the agent in Core is deliberate: the agent is what makes the framework feel like magic, and the free experience must include it, or the open Core reduces to “learn the formal language and write queries by hand,” which is no adoption path at all. Pro adds further surfaces and the customization of surfaces — management consoles, authoring workbenches at scale, BI-tool connectors, embedded and white-labeled deployments.

Features. Pro supports custom operators — domain-specific reducers, sophisticated missing-data handling (model-based and multiple imputation, inverse-probability weighting, selection-model corrections), custom allocation rules — registered through the operator registry and column-spec missing-policies. And Pro extends Columna from a query-oriented tool into a data-management one: not just asking questions of data but operating the Manifold as living infrastructure — scheduled refreshes, incremental materialization, refresh orchestration, lineage across refreshes. The integrity story straddles this line precisely: integrity checking at build and query time is Core, because Core must never ship a silently-broken Manifold; integrity operations — continuous re-certification, maintained coverage matrices, drift detection and alerting over time — are Pro, because they are the operational machinery of running a Manifold in production rather than the correctness of building one.

Multi-Manifold management. The challenges that only exist at scale — the composition graph across many Manifolds, cross-Manifold lineage and impact analysis, governance, reconciliation spaces for organization-level reporting, version management across a fleet — are Pro. They are also the answer to the natural worry about a many-Manifolds world: at small scale there is nothing to govern and Core suffices; at large scale the governance machinery is exactly what Pro provides.

Optimization. Core’s query processing is correct and reasonable for its target scale. Pro adds the optimization that scale demands — materialization-frontier management, workload-driven caching, derivation of coarse results from cached finer ones, cost-based routing — and the tuning and customization of the agent (org-specific dialogue, learned interaction patterns, longer-lived context).

Support and hosting. Pro includes professional support for custom deployment, operation, and development — which functions both as genuine expertise (most valuably with the adoption-critical work of authoring Manifolds well) and as the accountable vendor relationship that enterprise procurement requires before new infrastructure touches production data. And Pro includes a cloud-hosted service, in which Columna operates the entire stack — planner, artifact store, integrity monitoring, refresh orchestration, agent infrastructure — as a managed subscription. The cloud service is the natural home for the data-management and operational capabilities, the operated form of the very features that self-hosted Pro provides as software, and for most organizations it is the form they will actually buy: the thing that removes the operational burden entirely. Together, self-hosted license, cloud subscription, and professional services give Pro three delivery models for three buyer situations.

The principle behind all of it is one sentence: does this add reach, sophistication, or operational scale beyond what a single team needs to experience correct value? Then Pro. Is it necessary for that single team to get correct value at all? Then Core. Core is the complete, correct, magical-feeling experience, given away. Pro is what it takes to run that experience as enterprise infrastructure, sold.

Appendix A — Glossary

Anchor. The shared primitive: a named set of dimension coordinates attached to a column to govern one aspect of it. A column carries three — the V-anchor, the M-anchor, and (per-family) the B-anchor.

V-anchor (value anchor). The domain of a column: the set of dimension coordinates over which the column is defined as a function — the coordinates that address its points. Which points exist under that addressing is the content of the column’s universe; the complete grain is the pair (anchor, universe). Part of value determination. (The manual’s earlier bare “anchor” for the domain; qualified by V because the M- and B-anchor are anchors too.)

(Anchor, universe). The complete grain: the anchor names the coordinates, the universe defines the points. Every ascription is implicitly expr @ (a, U), with an unwritten U supplied by the consequence rule and always disclosed in the annotation.

Anchor lattice. The partial order of (anchor, universe) pairs under refinement; along a fixed anchor, universes order by containment. Aggregation and broadcast move up and down it. Shared across columns of a Manifold; each column has its own native root within it.

Basis. A universe’s account of where its points come from. Four bases: registry (an enumerated entity list), spine (a generated sequence such as a calendar), product (a cross-product of universes), and events projection (the point set projected from an event root — “universe ≡ observed,” stated explicitly). Verification is proportional to the basis: referential checks for registries, density measurement for spines and products, co-universality checks carrying the load for events projections.

Broadcast. The operation that refines an anchor by spreading a coarse value across finer points; the dual of aggregate. It is replicate-only — it copies the coarse value unchanged to each finer point, whether the value is intensive or extensive — and never apportions a total. A broadcast extensive value carries a B-anchor on the broadcast dimension so it cannot be re-summed (the double-count guard is that B-anchor, not an allocation step). Its purpose is to make a coarse value available at finer grain for downstream comparison (a share-of-total, say), not to distribute the total. (The manual’s earlier fillup is this operation; the many-to-many allocation of one column across another remains a separate, distinct operation — Chapter 8.)

Certificate. The queryable record of a Manifold’s integrity. Each precondition carries an evidential basis — checkable or asserted — and one of four verdicts: VERIFIED (the data confirms it), CORROBORATED (consistent with all the data can show, but not positively established — the strongest an asserted claim can earn), CONTRADICTED (a positive defect; never travels silently — the dependent result is routed around the defect or served carrying the finding), or UNTESTABLE (not decidable from the data — the MNAR case, the defining boundaries). A positive finding is sufficient to CONTRADICT; a null finding is never sufficient to VERIFY.

Climb. Moving up the lattice by replacing a coordinate with its parent in a dimension family.

Columna. The abstraction of the complete structure of a data space: the columns together with the universe set, the defining boundaries, the operator subset, derived columns, and any custom functions and reducers that constitute the data space. The operational unit of the framework.

Conservation invariant. A fertile family’s total is path-independent — the same metric ascribed to {} along any path lands on the same value — given (i) path-invariance of the reducer (typed), (ii) coordinate totality at the root (checkable, per-path blast radius), and (iii) one root point set feeding every path (declared via universe references, checked as co-universality).

Consequence rule. The default destination universe of an ascription: the contributing columns’ covered universes, brought to the output anchor — sparse in, sparse out; no manufactured points. Overridden by pinning a destination explicitly (ON); resolved either way in the annotation.

Co-universality. The other half of co-anchoring: columns combined at a shared anchor are checked for the same (anchor, universe). Same declared universe — verified by reference; declared-same but drifting in fact — the drift finding against a named object; different universes — a legitimate combination served with its permanent asymmetry disclosed.

Column. The atom of Columna: a function from anchor instances to values; the natural grouping of property-at-anchor data sharing an anchor type and a property type.

Composition. Building a new Manifold from existing ones, by inheritance or joining. Closed: Manifolds compose to Manifolds.

Default family. The family a bare column reference resolves to.

Defining boundaries. The constraints that say what a Manifold is about: typically a temporal range and a dimension-value-subset (US customers, the consumer line, etc.). Shared by every column in the Manifold; constitutive of the Manifold rather than applied to it.

Degeneracy. A column covering less than the Manifold’s defining boundaries; permitted only when the restriction is explicitly declared.

Dimension. A column in the role of an anchor component for some other column. A role, not a kind.

Dimension family. Dimensions linked by commuting, functional roll-up maps.

Drop. Moving up the lattice by removing a coordinate entirely.

Empty-bucket rule. Over a declared universe (registry, spine, product), an empty bucket’s SUM and COUNT are true zeros and its order statistics and means are absent; over an events projection, the empty bucket is not a point and nothing is rendered. The destination universe decides; the resolved (anchor, universe) is disclosed.

EXPLAIN. The query language’s first-class validation statement: everything short of execution — the canonical desugared form, the planner’s atom decomposition, the dependency cone with verdicts, and the complete would-be annotation — returned without touching data. The validation verb of the propose-validate-refine loop.

Extensive / intensive. A scaling property of a reducer’s value: an extensive value scales with the size of the input set (a SUM, a COUNT), an intensive one does not (a rate, a ratio, an AVG). This bit, combined with each column’s missingness mechanism, selects the missing-data policy — extensive under MCAR skips-and-rescales, intensive under MCAR merely skips. It is one of three orthogonal axes — applicability (B-anchor), scaling (extensive/intensive), and rollability (fertility) — none of which implies another.

Family. A column under a fertile reducer, the cached image of the column at every grain in the lattice. Path-invariance is what licenses the caching. A column has a family set; one is the default. Mules are not families — they are computed at presentation from fertile sources.

Fertile / mule. A path-invariant (commutative-monoid) reducer is fertile — its results can found a family, re-aggregable from cached intermediates. A non-path-invariant reducer is a mule — sterile, valid at its grain but not re-aggregable; computed at presentation from fertile sources.

Frame. A derived assembly of columns at a shared anchor. The output of operations, not their input. Rows and tables are frames.

Inform-and-serve. The framework’s outcome discipline (the maturation of the earlier fail-closed posture): analytical risk is disclosed on a served result, or dissolved by a disclosed route-around — never the engine’s grounds for withholding. A result is withheld only where no determinate answer exists (clarify), no data exists (inform), or a human rule forbids (the author’s governance, a caller’s entitlement — reported as whose rule it is).

Jurisdiction rule. The M-anchor governs value-presence at universe points; point-existence belongs to the universe. The two never overlap, and they compose; no mechanism may be declared over the point-existence of an events projection.

Macro binding. The query language’s WITH <name> = <expression> form: purely syntactic textual expansion before desugaring and type-checking — de-duplication with no new powers.

Map. The operation that preserves an anchor, transforming values point by point. Its order-dependent form is a scan (window operation).

M-anchor (missingness anchor). The coordinates that determine a column’s missingness mechanism (formerly the missing-determinant anchor, MDA). The other half of value determination.

B-anchor (blocked anchor). The dimension families along which a family’s reducer is blocked from legitimately rolling up — not applicable for its column — the seat of additivity, and the axis the planner consults to catch an illegitimate coarsening — served, if at all, only with the critical crossing disclosure (formerly the non-applicability anchor, NAA). Per-family, not per-column, and prohibitive rather than locative (it forbids a roll-up across the named axes, it does not point at where values live). Three shapes: empty (applicable along every axis), full (applicable along none), or a proper subset (semi-additive — applicable along some axes but not others). This is an applicability axis, distinct from the extensive/intensive (scaling) axis: an account balance is extensive yet its SUM carries time in its B-anchor, so “extensive” does not imply “empty B-anchor.” It is also distinct from fertility — a mule like AVG is non-rollable by fertility, not by B-anchor.

Missing-behavior vocabulary. The five author-pinnable treatments of missing values: skip · skip-and-rescale · stratified · propagate · fill (with default deferring to the registry’s mechanism-derived map). Withholding is deliberately not among them — it is governance (WITHHOLD), never a missing-data treatment.

Ordered reducer. An order-dependent reducer (LAST, FIRST); requires an explicit order to be well-formed (the missing order is a clarification).

Point. The referent of a datum: this customer on this day, this sensor in this hour. Points belong to universes; the anchor addresses them.

Path-invariance. The property that a reduction’s result is independent of the path up the lattice; equivalent to being a commutative monoid; the property that makes a reducer fertile and licenses the family-caching optimization.

Reducer. The function that combines many values into one under the aggregate operation. Classified along two independent axes — order-dependence and path-invariance (fertility).

Root (per-column). The native finest anchor at which a column is observed in the data. Each column has its own root within the shared lattice; different columns may root at different positions.

Soundness (of data). The property of satisfying the integrity preconditions any correct analysis on the data requires. The framework’s integrity machinery verifies soundness; failures are defects in the data, not Columna’s conventions.

Universe. A declared point set with a basis — the account of where its points come from. The second component of the complete grain (anchor, universe): the anchor names the coordinates, the universe defines the points. “Sparse” and “dense” survive as teaching vocabulary: a sparse universe has an events-projection basis; a dense one is declared independently of observation, against which observation is measured.

Value determination. The concern of where and whether a column’s values exist: V-anchor (coordinates), universe (which points exist), M-anchor (partiality and its mechanism), and degeneracy (coverage of the Manifold’s defining boundaries).

WITHHOLD. The author’s declared hard stop — a governance rule on a family entry (forbidding a (column, reducer)) or a column (forbidding it under any reducer). Executed by the engine as an inform reporting the author’s rule and the permitted alternatives; never the engine’s analytical judgment.

{} (the empty anchor). The lattice’s top within a Manifold. In context, {} is the Manifold’s defining boundaries collapsed to a single anchor point — the grand aggregate position. Columna-relative: different Manifolds have different {}.

Appendix B — Common Failure Modes

In every row, the conventional tool’s behavior is silent error — a wrong number returned with no warning, because the underlying fact was never represented or checked. Columna’s behavior is inform and serve: the producible number arrives with its exact condition on its face — routed around the defect where a sound route exists, quantified where it does not — and nothing is withheld except where no determinate answer exists (a clarification), no data exists (an inform), or a human rule forbids (the author’s WITHHOLD, a quarantine, a caller’s entitlement). The framework’s contribution is not adding rules but checking the rules that have always been there — and never serving silently against them.

Appendix C — What Columna Uniquely Catches

Appendix B lists the failure modes as a reference table. This appendix takes the argument further: it groups the silent errors of conventional analytics by what kind of structural fact in the data the error depends on, names the Columna mechanism that detects each, and indicates which errors Columna uniquely catches versus where Columna is simply more rigorous than what came before. The purpose is to make precise what is genuinely new about Columna’s contribution to correctness.

The silent errors that haunt analytics — the ones that produce wrong numbers consistently, across teams, across tools, across years — are not random bugs. They fall into a small number of structural patterns, and each pattern corresponds to a specific class of facts about the data that conventional tools either do not represent or do not check. The framework’s contribution is to represent those facts explicitly and to check them systematically.

C.1 Errors from treating mules as families

The most common silent error in working analytics is the average of averages, and it is not an isolated mistake — it is one instance of a broader pattern: treating a mule reduction as if it were a family-bearing one, allowing its result to be re-aggregated when re-aggregation is mathematically wrong. Columna’s fertile/mule distinction is the structural answer; it makes the entire class of these errors impossible by never treating mule results as further-reducible: a mule image is not a family, so there is nothing cached to mis-aggregate — and a request that would re-aggregate one is routed, recomputed from the fertile sources at root with the routing disclosed (RECOMPUTED_FROM_DETAIL).

Several specific cases fall under this pattern. The plain average of averages — averaging monthly averages to get an annual average — is the textbook version. Median of medians, mode of modes, and the various “summary of summaries” patterns share the same structure: a mule result computed at one grain, then incorrectly aggregated to a coarser grain.

A subtler and more dangerous case is the weighted average with a hidden inner grain. A weighted average has two anchor decisions, not one: the obvious target anchor (the grain the result sits at) and a hidden second anchor (the grain at which the score and weight were observed before being combined). The two are usually different, and the choice of the second anchor changes the answer.

Concretely: “average customer satisfaction weighted by revenue” can be computed at least two ways. Per-customer (one satisfaction score and one revenue total per customer, weighted average is straightforward) versus per-transaction (per-transaction satisfaction weighted by per-transaction revenue, with high-volume customers contributing more transaction-observations). Both produce “weighted average satisfaction by revenue”; they give different numbers, sometimes substantially different. The choice is which inner grain the weighted pair was observed at, and in conventional tooling this choice is hidden in the procedure — a JOIN, a GROUP BY, a sequence of CTEs — never surfaced in the result.

Columna’s catch: a weighted average is a mule, computed at presentation from fertile sources (SUM(score × weight) and SUM(weight)). Both fertile sources are reductions; both require explicit input anchors when they are inputs to a mule. The hidden second anchor cannot stay hidden because the language requires it to be written:

SUM( (score * weight) @ {customer, transaction} ) /
SUM( weight @ {customer, transaction} )
AT {customer}

The inner @ {customer, transaction} makes the observation grain explicit. The writer must say what they mean, and different inner grains produce different (and clearly different) queries. This isn’t Columna adding a rule; it’s Columna surfacing a choice that has always existed but that conventional tools let the writer make accidentally.

The same pattern catches ratios with hidden grain (revenue / orders — at what grain were these combined?), rate calculations (conversion, retention, defect rates), and quantile computations (median customer revenue across what?). All are composite mule reductions; all have hidden inner grains in conventional tooling; all are underdetermined in Columna until the inner grain is named — the query comes back as a clarification naming the missing @ {anchor}, and once named, the number is served.

The unifying principle: mules are never path-invariant, so their results depend on the path taken to compute them. In Columna the path must be explicit. In conventional tools the path is implicit and almost always under-specified. The errors that result are silent precisely because no one can see the choice that produced them.

C.2 Errors from treating non-additive reductions as additive

Conventional tools allow SUM(balance) GROUP BY day over a stock measure like account balance — and produce a meaningless number that nobody questions because it is computed by SUM. The error is in summing the same balance across time, when an account balance is not additive along the time axis.

Columna’s B-anchor records this directly: each (column, reducer) family declares the axes along which the reducer is not applicable. A balance’s SUM family has the time family in its B-anchor. Any attempt to coarsen across time using SUM is caught at plan time and served only with a critical disclosure naming the crossing and quantifying the factor, with the column’s LAST family surfaced as the applicable reducer (it respects time’s order and is the right way to collapse a stock over time) — and withheld outright only where the author has declared WITHHOLD.

The deeper observation is that which reducer is correct depends on the axis. Sum is correct across customers; not correct across time. Last is correct across time (the closing balance); not meaningful across customers. Columna represents this as the B-anchor — a fact about the (column, reducer, axis) tuple — and no operation that violates it can produce a silent number. The framework manual’s Chapter 7 specifies the machinery.

Conventional tools have no representation of this fact at all. The semantic-kind concept (extensive vs. intensive) is sometimes documented; it is almost never checked. The result is that semi-additive measures — balances, inventories, headcounts — are routinely summed across time, producing numbers that are wrong by factors of 30 or 365 depending on the time grain, and the wrongness is invisible because the operation is SUM.

C.3 Errors from joining at different grains (fan traps)

When two tables at different grains are joined, the result has a row count determined by the join’s combinatorics, not by either source. A common case: customers joined to transactions joined to support tickets. A customer with 10 transactions and 3 support tickets appears in 30 result rows, and any subsequent aggregation that touches this customer’s data multiplies by 30. Revenue is summed 3 times; ticket counts are multiplied by 10. The totals are silently wrong.

This is the fan trap, and it is the foundational error that all SQL-based analytics is built on top of without solving. BI tools have partial defenses (Looker’s “symmetric aggregates,” AtScale’s grain-aware semantic layer), but each is a workaround for a problem the underlying language structurally creates by treating tables and joins as primitive.

Columna’s structural answer is that there is no join primitive in the language. Combining two columns at different anchors is not a direct operation at all — it is unrepresentable, not forbidden. The writer must bring the columns to a common (anchor, universe) — aggregating or broadcasting one side — before combining them with a map; an expression that leaves the path unstated is a clarification, never a number. The composition is explicit; the grain at which the combination happens is explicit; the fan-trap cannot arise because the operations that would produce it do not exist as primitives.

This is one of the most fundamental architectural differences between Columna and conventional analytics, and it is the reason a class of errors that took thirty years of BI-tool engineering to partially mitigate is structurally absent in Columna from the start.

C.4 Errors from many-to-many aggregation

When a relationship is many-to-many — products in multiple categories, sessions touching multiple campaigns — aggregating a measure from the fine side to the coarse side is silently wrong unless an allocation is supplied. A product belonging to three categories whose revenue is naively summed at the category grain contributes its revenue to all three; the per-category totals exceed the actual total revenue by the amount of the overlap.

Columna catches this at plan time. The aggregate-across operation has a precondition — that the relationship be functional — which a many-to-many does not supply. Because the cross-a-non-functional-edge transport has no single target, the bare aggregate-across denotes no one number at all (the fan-out is inexpressible, not merely risky), so the framework does not serve a default and does not guess: it clarifies, naming the three resolutions the user may choose between — filter to membership (don’t reconcile; accept overlapping totals; served when chosen), designate a primary side (convert the relationship to functional and transport faithfully), or supply an allocation [Pro] (manufacture functionality through partition-of-unity weighting). Membership is a real answer to a real question — but it is one of the three, to be chosen, not the silent default of a request that did not name it.

The unique contribution here is making the precondition visible. Conventional tools treat many-to-many as a kind of join, with no awareness that aggregating across it is mathematically wrong without allocation. The error happens silently and is essentially never caught.

The three faces now execute (0.12). The three resolutions above are no longer only names the clarify offers — each is a declared face on the RELATE that the engine crosses:

• touch — the value reaches every match; deliberately multi-counted, the over-count stated (totals exceed the grand total).
• assign BY <driver> ORDER MIN|MAX — the value single-counts to each member’s top-ranked pair (ORDER is declared, never defaulted — “top” is rank-min for a rank, score-max for a weight). The total reconciles to the grand total; the shadow of unrepresented memberships is stated.
• alloc BY <driver> — the value splits by the per-member-normalized driver. The total reconciles to the cent; a reconciliation badge certifies the commutation. (Normalization is the face’s declared law, applied by the engine — a raw driver is declared, never a pre-normalized one: pre-normalized weights are derived truth.)

Legality is the measure’s output anchor speaking, one law for all three faces: an additive value crosses; a distinct-class measure refuses at every face — its anchor is spent at the frontier grain, so per-member counts cannot be summed, weighted, or routed (a customer buying two products is not 1.6 people under a split, nor 2 under a union). The refusal teaches the two authoring doors: declare the per-member counts as a value measure (its anchors spent by declaration, so every face may serve it), or ask for the crossed-population count directly (the population door — the crossing increment). Every operation in the space has exactly one honest spelling.

C.5 Errors from broken hierarchies (FD violations)

A hierarchy edge — customer to region, employee to department, product to category — claims that the relationship is a functional dependency: each child maps to exactly one parent. The claim is almost never checked in conventional tools. A claim that holds in the data when the schema was authored may fail later when data drifts (a customer moved between regions; an employee transferred between departments).

When the claim fails — when, say, customer 19847 maps to both ‘east’ and ‘west’ in the current data — every climb across that edge silently double-counts. Revenue summed by region, customer counted by region, any analysis that climbs through the broken edge is wrong.

Columna verifies every declared edge against the data at refresh. A failed edge is recorded in the integrity certificate; a climb along it is routed through detail where a sound route exists (served from root past the broken edge, the routing disclosed) and otherwise served carrying the contradiction’s specifics (which child, which conflicting parents) at certificate state CONTRADICTED. The blast radius is scoped — only paths through the broken edge carry the finding; the rest of the Manifold is untouched.

The verification is what’s unique. Conventional tools never check; the defect compounds in the analytics output for as long as it remains undiscovered.

C.6 Errors from universe and coverage mismatches

This is the silent error that almost no conventional tool catches at all, and the one Columna contributes most uniquely. Every column in a Manifold implicitly claims to operate within the Manifold’s defining boundaries. When the claim fails — when one column actually covers fewer customers, or fewer days, or fewer products than another — combining the two at a shared anchor produces a result that is biased by the coverage gap, but each column is internally consistent and no traditional check would catch it.

Concrete example: a finance team has a revenue column covering all transactions, and an engagement_score column covering only customers active in the digital product. Both anchored on {customer}. Combining them — revenue * engagement_score per customer, summed to total — produces a number that is heavily biased by the digital-product population, but each column behaves correctly on its own. No traditional integrity check would flag this. Most teams will never know the bias is there.

Columna maintains per-column coverage information and checks compatibility for every cross-column operation — and the check has, since Chapter 5, a named object: co-universality, the other half of co-anchoring. The combination above is served with the finding quantified: the overlap, the asymmetric gap, the bias direction (“engagement covers 65% of revenue’s customers; the combination is biased toward the digital population”), with restriction to the intersection offered as the remedy. A quarantined column does not participate — its exclusion is the author’s standing coverage governance, reported as an inform, never the engine’s analytical judgment. Drift in coverage is surfaced through the certificate’s event log. The machinery exists because the framework treats universe coherence as one of the data conditions that any correct analysis on combined columns requires.

This is the case where I would say Columna’s contribution is most uniquely valuable in real analytics work. The other errors in this appendix are caught by at least some other tools in at least some configurations. Coverage drift between columns at a shared anchor is essentially unique to Columna’s checking.

C.7 Errors from an undeclared universe: the empty bucket and the conservation leak

Two error classes share a single root — the undeclared second component of the grain — and Chapter 5’s diagnosis applies to both: one parameter, leaking through different cracks.

The empty bucket. A store-day grid with no sales row for Tuesday: is Tuesday’s revenue zero, missing, or simply not a row? Conventional tools answer by accident — the join used, the COALESCE someone remembered, the BI tool’s default — and the same dashboard can render the three answers in three tiles. The confusion is not about the value; it is about whether the point exists, which no column-level rule can settle because it is a property of the destination. Columna makes the answer a rule rather than an accident: the destination universe decides (the empty-bucket rule, Chapter 5). Over a declared universe — the store-day product of the store registry and the calendar spine — the empty bucket’s SUM and COUNT are true zeros and its mean is absent; over an events projection, the bucket is not a point and nothing is rendered. The resolved (anchor, universe) of every series rides in the annotation, so even the defaulted choice is never a hidden one — and most of fill(0)’s former population dissolves into semantics, with no substitution rule to misapply.

The conservation leak. Sum revenue to the grand total through {customer}, and again through {region}, and watch the two numbers disagree — because 312 transactions carry a null region, and the region path silently sheds what the customer path keeps. Nothing in either reducer failed; the paths traversed different point sets, and conventional tools have no representation in which that fact could even be stated. Columna states it as the conservation invariant (Chapter 5) and checks its one checkable precondition, coordinate totality at the root, as an integrity layer with a per-path blast radius (Chapter 11). The served total carries the leak quantified — “the region path excludes 312 transactions (0.4% of total) carrying null region; the customer path includes them” — which is the difference between a grand total that mysteriously disagrees with itself and one whose disagreement has a name, a size, and a fix.

Both traps are seeded classes in the Columna benchmark, deliberately: the framework’s previously implicit assumption is now its most explicit exam question.

C.8 Errors from missing values handled incorrectly

Two distinct cases share a theme.

Missing values in measures, treated as zero. Conventional SUM with NULL-skipping silently produces a total that undercounts by the missing fraction, with no indication. The total appears authoritative but is biased. Columna’s default for extensive sums under MCAR is skip-and-rescale — multiply the observed sum by the ratio of the covered universe to observed points — producing an unbiased estimate of the population total. The default behavior is correct rather than wrong; the annotation carries the rescale finding with its factor, so the consumer knows the result is a population estimate and can reconcile it against a raw source-system sum (and an author or query that wants tie-out to stored values pins skip, with the undercount disclosed instead).

Missing values in dimensions, producing NULL groups. Conventional GROUP BY silently creates a NULL group for rows where the dimension is missing. The grouped totals do not reconcile to the grand total — the NULL group’s contribution is hidden in some reports, included in others, and the total no longer adds up. Columna forbids missing dimension values entirely; unknowns must be promoted to named categories (ANONYMOUS, UNKNOWN, NOT_APPLICABLE) at binding time. Once promoted, the unknown bucket is a legitimate group that reconciles into the total along with the others.

Chapters 7 and 8 specify the mechanisms. The unifying observation: absence is permitted in Columna only when it is explicitly characterized — by an M-anchor for measures, by a named category for dimensions, by a declared universe for points. Silent absence is unrepresentable.

C.9 Errors from stale values

Conventional warehouses serve aggregates from materialized tables that may have been refreshed last week, with no indication that the underlying data has moved on. The number on the dashboard is correct as of the data version that produced it, but the version is invisible. The user reading the number cannot tell whether they are looking at fresh data, day-old data, week-old data, or stale-from-a-failed-pipeline data.

Columna’s certificate tracks data versions per column. Every query result carries provenance: the certificate state, the data versions, the refresh timestamps of the underlying columns. A stale value is still served (Columna doesn’t refuse to answer queries during a refresh window) but is served labeled — the consumer sees the staleness and can decide whether to wait or to proceed.

In Pro deployments, continuous monitoring tracks the freshness across refreshes and alerts when staleness exceeds configured tolerances. The Core/Pro distinction here is consistent with the integrity story: Core surfaces the data state honestly; Pro operates the monitoring around it.

C.10 The structural pattern of all of these

Looking across the sections of this appendix, a single pattern recurs: each silent error of conventional analytics depends on a fact about the data that conventional tools do not represent, and Columna catches the error by representing the fact explicitly.

• Mule errors depend on which reducers admit re-aggregation; Columna represents this as fertility.
• Semi-additive errors depend on which axes a reducer applies along; Columna represents this as the B-anchor.
• Fan traps depend on which columns are at which grains; Columna represents this as anchors.
• Many-to-many errors depend on functionality of relationships; Columna represents this as the dimension-family declaration.
• Hierarchy errors depend on FD validity in the data; Columna verifies this at refresh.
• Coverage errors depend on the region of the boundaries each column addresses; Columna represents this as coverage state.
• Empty-bucket and conservation errors depend on which points exist; Columna represents this as the universe, with its declared basis and the co-universality relation.
• Missing-value errors depend on the mechanism of absence; Columna represents this as the M-anchor.
• Staleness errors depend on data freshness; Columna represents this as data versions in the certificate.

In every case, the fact has always been there in the data — Columna did not invent any of these structural concepts; they exist in the mathematics of analytics independent of any tool. Conventional tools simply do not represent them, so they cannot check them, so the errors they produce remain silent. Columna’s contribution is the representation: declared, verified, propagated, and consulted at every operation. The disclosures are downstream of the representation.

This is why the framework’s value cannot be replicated by adding “more checks” to a conventional tool. The checks Columna performs are not bolted on; they are checks against facts that the framework’s data model represents in the first place. A tool that does not represent the B-anchor cannot check it; a tool that does not represent column coverage cannot verify universe coherence; a tool that does not represent fertility cannot even see mule re-aggregation, let alone route around it. The structural facts Columna represents are what make the checks possible at all.

The locus of trust in conventional analytics is the per-query procedure: the user must author the procedure correctly, and the tool cannot help with what it does not represent. In Columna the locus of trust has moved to the per-Manifold declarations: the author makes the structural facts explicit once, the framework verifies what is verifiable, tags what is asserted, and discloses what is unsupported. The numbers that come out of the framework carry the correctness of the declarations behind them, not the diligence of the analyst writing today’s query. This is the deepest reason the framework is structurally more trustworthy than the tools it replaces: trust has been moved from a place where it cannot be held (each per-query procedure) to a place where it can be (each Manifold’s declared and verified structure).

End of Columna Framework Manual, Sixth Edition.