When you are designing a data platform from scratch
A greenfield data platform is the rare project where the design decisions are all still reversible and all still cheap. That window closes about six weeks in, which is the argument for spending the first two on the model.
The situation
The organisation has decided to build a data platform. The reference architectures all look the same at the diagram level and diverge entirely at the level where the money is spent: how many layers, what the contract between them is, what happens to late-arriving data, where the modelling actually lives.
The failure mode is not choosing wrong. It is choosing implicitly — the first pipeline sets a pattern, and by the time anyone examines it there are forty of them and the pattern is now the architecture.
Modelling it first makes the decisions explicit while they are still decisions. It also produces the artefact the funding conversation needs, which otherwise gets built separately in slides and immediately diverges from the real design.
You are here if
- The design exists as a reference-architecture diagram from a vendor and nothing else.
- Nobody has said what the layers are for in terms specific to your data.
- The cost model is a spreadsheet with no relationship to the architecture.
- Two teams have started building pipelines with different conventions.
How the work gets done
Start from a shape that already exists
Begin from a pipeline or warehouse template rather than an empty page, and edit it down. The value is not the drawing — it is that a known-good shape forces the questions you have not asked yet, in an order that makes sense.
Delete everything you cannot justify. A platform diagram containing components nobody has a reason for is how a platform ends up containing components nobody has a reason for.
Page ▸ Template galleryETL / Data PipelineStar Schema
Model the data, not only the boxes
The layer diagram is the easy half. The half that determines whether the platform works is the model inside the layers — the grain of the facts, what the conformed dimensions are, what is allowed to be a surrogate key.
Model that properly, with the same tooling that will later generate the DDL, so the design and the built thing start from the same source rather than diverging on day one.
Page ▸ Entity / TableData ▸ Subject Areas
Test the design under load before you build it
Run the design through simulation with a workload profile that resembles what you expect. This is the step most data platform projects skip, and it is the one that catches the throughput assumption that turns out to be off by an order of magnitude.
The industry packs give a starting workload shape closer to reality than a guess, and sensitivity analysis tells you which assumption the design is actually fragile to — which is more useful than a single verdict.
Simulate ▸ Guided Setup WizardSimulate ▸ Workload ProfileSimulate ▸ Industry PackSimulate ▸ Sensitivity Analysis
Record the reasoning while it still exists
Every significant choice here has an alternative that someone will propose again in eighteen months. Record the non-functional requirements the design is meeting and the risks it is accepting, attached to the components they concern.
This is cheap now and impossible later. The reasoning behind a design decays faster than the design.
Arch ▸ ViewpointsArch ▸ Repository
What you end up holding
- A layer design you can defend line by line, with everything unjustified removed.
- A data model at the grain the platform will actually be built at, which generates its own DDL.
- Simulation results against a stated workload, including which assumptions the design is sensitive to.
- The non-functional requirements and accepted risks, recorded against the components they belong to.
Not this, if
- You are buying a managed platform whose architecture is fixed by the vendor. Model your data in it, but the platform design is not yours to make.
- The platform already exists and works. This is a design-time argument; the case for modelling an existing platform is documentation and impact analysis, which is a different page.
- You want cost estimates to the penny. The budget view is an architecture-level estimate, not a cloud pricing calculator.