When your analytics estate costs more than it returns
Analytics platforms rarely fail. They get expensive, and slowly, in ways that no single invoice line explains and no single team owns.
The situation
The platform is built and running. The question on the table is not whether it works but what it costs, and the finance conversation has arrived without an architectural answer to give it.
The cost drivers are usually structural rather than wasteful: compute provisioned for a peak that occurs twice a month, analytical queries landing on a transactional primary because no read replica exists, connection pools sized for a different era, and a raw layer with no backup that quietly became the source of truth.
These are visible in the design. Reading them off the bill instead is what makes the conversation take a quarter.
You are here if
- Monthly spend is rising faster than the data volume or the query load.
- Analytical queries are known to affect operational systems.
- Nobody can say what the raw layer’s recovery position is.
- Cluster sizing was set once and has not been revisited.
How the work gets done
Model the batch shape, not a request rate
The data platform pack applies an analytics profile — low request rate, heavy long-running queries — and the targets below. A platform modelled as though it serves requests will produce answers about the wrong thing entirely.
The latency allowance here is orders of magnitude looser than the transactional packs, and the cost target is the constraint that actually bites.
Simulate ▸ Industry PackSimulate ▸ Workload Profile
Find where the queue forms
Connection pool saturation is one of the six checks with a real implementation behind it, and it is the classic failure on large Spark and Hive workloads — the cluster is not short of compute, it is short of connections.
Bottleneck detection covers the rest of the path, which on analytics platforms is usually a stage that everything upstream waits on.
Simulate ▸ Bottleneck DetectionSimulate ▸ Rule Evaluation
Attach the cost to the architecture
Model the cost per component so that the expensive part of the platform is identifiable as a part of the design rather than as a line on a bill. Then run sensitivity analysis to find which assumption the total is actually driven by.
This is normally where the surprise is. The component people expect to dominate the cost frequently does not.
Arch ▸ BudgetSimulate ▸ Sensitivity AnalysisSimulate ▸ Recommendations
Establish what the raw layer is worth
A raw layer with no recovery position is the risk nobody has priced, because it began as a landing area and became irreplaceable without anyone deciding that it should. Record it as a risk against the component, with a recovery position or an explicit acceptance.
Arch ▸ Tech Debt ReportArch ▸ Repository
What the Data Platform pack sets
Analytical and ML infrastructure. Throughput, cost efficiency, and data freshness are primary concerns. Applying the pack switches the workload profile and sets these six targets on the model. It does not change the design and it does not enable any industry-specific check — the targets are what the design is then evaluated against.
| Workload profile | Analytics Batch — Low RPS but heavy queries; long-running jobs dominate resource use. |
|---|---|
| Availability target | 99% |
| Latency target (p95) | 5000 ms |
| Throughput target | 100 requests/sec |
| Cost target | $40,000 / month |
| Recovery time objective | 60 minutes |
| Recovery point objective | 600 seconds |
The policies this pack highlights
- Monthly Cost Within Budget — Estimated monthly cost must not exceed $50,000 without an approved exception.
The risks it names for this industry
These are prompts to work through against your own design, not findings. The pack lists them; nothing evaluates them for you.
- Over-provisioned compute inflating monthly costs
- Missing read replicas causing analytical queries to hit OLTP primary
- No backup for raw data lake — single source of truth at risk
- Connection pool exhaustion on large Spark/Hive workloads
What is still yours to check
The simulation evaluates the architecture itself — single point of failure, deep synchronous dependency chain, shared database across services, connection pool saturation risk, no autoscaling configured and excessive retry configuration — against the targets above, and the policy checks run on every design. The concerns below are specific to data platform and nothing in the product evaluates them. The pack names them so they stay on your list; the review is a human one.
- Compute right-sizing and idle-cluster shutdown against the budget
- Read-replica strategy so analytical queries never hit the OLTP primary
- Backup and retention for the raw data lake
- Data freshness and pipeline SLA per downstream consumer
What you end up holding
- The platform modelled against a batch workload rather than a request rate.
- Named saturation and bottleneck findings, with the pool sizing among them.
- Cost attributed per component, and the assumption the total is most sensitive to.
- An explicit recovery position for the raw layer, or a recorded acceptance that there is none.
Not this, if
- You want per-service cloud billing analysis. The budget view is an architecture-level estimate, not a cost-management product.
- The platform does not exist yet. Designing one is a different job with a different page.
- The waste is in query patterns rather than in the architecture. That is a workload problem and this will not find it.