Overview
Observability, SRE and cost analysis and evolution across a SaaS platform with +15 Amazon EKS clusters, multiple AWS accounts and continuous 24x7 operations.
Context
In environments with many clusters, workloads and AWS accounts, observability is both a critical operational necessity and a material portion of total cost. The platform needed reliable signals for operations and incident investigation, but not all data needed to remain in expensive, high-performance query tools.
Challenge
Preserve the signals needed for operations, incident investigation and cost analysis without collecting or retaining data without clear purpose. Balance granularity, retention, query performance and cost across multiple telemetry layers.
My role
- CloudWatch, Grafana, Loki, Datadog and Instana — dashboards, alerts and analysis
- Log pipelines with Data Firehose, S3 and Athena for long-term investigation
- Kubernetes, networking and AWS service troubleshooting with multi-tool correlation
- Cost analysis: NAT Gateway, data transfer, compute sizing and observability platforms
- SLI/SLO, MTTR analysis and root cause analysis
- Log and metric retention policies based on operational value
Architecture
The observability architecture considered different destinations and retention periods according to operational data value. High-frequency, frequently queried signals remained in fast-access tools (CloudWatch, Grafana). Eventual investigation and historical analysis data were directed to S3 + Athena, with significantly lower cost. This separation maintained operational visibility without compromising costs.
Technical decisions
- Collection granularity driven by actual usage — not all data needs the same resolution
- Retention proportional to operational and compliance need
- Data-driven analysis before optimization — understand what consumes before reducing
- Compute, observability and networking cost impact evaluated together
- Troubleshooting with Kubernetes, infrastructure, application and network correlation
Automation
Repeatable queries and processes supported identification of major cost drivers and comparison between storage and query alternatives. Automated log pipelines reduced operational overhead of data collection and routing.
Engineering challenges
Reducing observability cost without removing essential signals required understanding access, investigation and operation patterns. Data that seemed important at collection was frequently accessed only in specific incidents — and for those, S3 + Athena offered sufficient capacity at much lower cost. The architectural decision to separate retention layers was part of platform engineering, not just financial optimization.
Results
- Greater clarity on cost drivers in observability and infrastructure
- Better balance between operational visibility and retention cost
- Structured log pipelines for long-term investigation
- Optimization decisions with technical context — not just financial