01
The challenge
Production behaviour was invisible. Incidents surfaced through customers, took two to three hours to diagnose, and the few alerts that existed were noisy enough to be ignored. No SLOs were defined, so nobody could say whether the service was healthy. Forty percent of a six-person team's time went to firefighting.
02
Solution architecture
Layers top to bottom. Controls apply at every layer.
03
Approach
- 01Define what healthy means first: SLIs tied to user experience, then SLOs the business could stand behind.
- 02Deploy one coherent stack for metrics, logs and traces rather than three disconnected tools.
- 03Alert on SLO burn rate, not on every symptom, so pages are rare and actionable.
- 04Write the runbook for every alert before turning it on.
- 05Give leadership the same dashboards as engineering, so reliability conversations use one set of numbers.
04
How the delivery ran
- Week 1: incident history review, SLI candidates, SLO targets agreed with product and engineering.
- Week 2–3: Prometheus, Grafana and structured logging deployed as code; services instrumented.
- Week 4–5: burn-rate alerting, runbooks and on-call rotation; alert tuning against real traffic.
- Week 6: dashboards for engineering and leadership, validation, hand-over. 30 days of support followed.
05
What was delivered
- Observability stack as code with dashboards
- SLO catalogue with recording and alert rules
- Runbook per alert and an incident response procedure
- Leadership reliability dashboard
06
Results
| Dimension | Before | After |
|---|---|---|
| Mean time to detect | ~90 minutes | 5 minutes |
| Mean time to resolve | ~180 minutes | 25 minutes |
| Availability | Unmeasured | 99.8% SLO sustained |
| Engineering time on incidents | ~40% | ~20%, redirected to features |
| Alert quality | Noisy, ignored | Burn-rate pages, each with a runbook |
Facing a similar decision? A 30-minute architecture review is the fastest way to find out whether the same approach fits.
Book an architecture reviewMore case studies
Other engagements
Government · Tourism · Azure
A secure Azure platform for a national tourism digital ecosystem, delivered as code.
A high-traffic tourism ecosystem built on Sitecore, .NET and Optimizely needed standardised, observable and resilient infrastructure across development, UAT and production. I owned cloud architecture, security, networking, CI/CD and production application delivery as Cloud Solution Architect and DevOps Engineer.
Food & Wellness · E-commerce · Azure
The Azure platform behind a high-volume digital meal-plan experience, run as code.
PractiCal sells personalised meal plans across the UAE through a Next.js storefront, Laravel/PHP backend services and Azure serverless workloads. I owned the cloud and application platform end to end: infrastructure, edge security, CI/CD and production operations across development, staging and production.
Education · Enterprise · Azure
A secure Azure platform for a large-scale university digital experience on .NET and Umbraco.
A university digital platform serving prospective students, current students, faculty and other stakeholders: programme discovery, admissions, applications and student services on an ASP.NET and Umbraco CMS stack. I owned Azure infrastructure, application delivery, security and production operations.
B2B SaaS · AWS
Kubernetes platform that cut cloud spend by 60% and deploy time from 45 minutes to 5.
A growing SaaS platform was running on hand-built AWS infrastructure with manual deployments and costs rising 40% month on month. I replaced it with a production-grade EKS platform, GitOps delivery and infrastructure as code, migrated with zero downtime.
FinTech · AWS
Zero-trust AWS controls that took a FinTech to SOC 2 Type II in 8 weeks with no audit findings.
Enterprise deals depended on SOC 2 Type II, and the existing estate had shared secrets in code, permissive IAM and no audit trail. I designed and implemented the identity, network, secrets and audit controls, and the external audit closed with zero findings.