Moving production observability off New Relic
I moved production logging and monitoring for our retail trading platforms from New Relic to a self-hosted stack: Filebeat, Elasticsearch and Kibana for logs and Prometheus with Grafana for metrics. Monitoring cost dropped from about $1,225 a month to $400, and I designed the retention rules that keep it there.
The problem
Our trading platforms run on Kubernetes, and we paid New Relic for logs and monitoring. The bill was about $1,225 a month. Production produces around 15 GB of logs a day, so keeping everything in a hosted tool was the expensive path. We needed the same visibility for a fraction of the cost, with retention we controlled.
What I did
- Built the log pipeline: Filebeat collects container and node logs and ships them straight to Elasticsearch, which stores and indexes them, and Kibana is where the team searches and builds views.
- Set up Prometheus to scrape cluster and service metrics, with Grafana dashboards on top.
- Configured alerting on the metrics so the team hears about problems quickly.
- Added Uptime Kuma to check availability of our customer-facing web platforms, with alerts when a check fails.
- Designed index lifecycle (ILM) and snapshot lifecycle (SLM) policies so storage stays bounded as log volume grows.
Retention design
Log storage is where self-hosting gets expensive if nobody plans for it. Staging indices are deleted after 7 days. Production indices stay searchable in Elasticsearch for 30 days, then live on as snapshots until day 150, so we can restore older logs for an investigation without paying to keep them hot. Indices roll over every day or at 5 GB, which keeps each one a predictable size.
# illustrative: production log policy, shape matches the real rules PUT _ilm/policy/prod-logs { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_age": "1d", "max_primary_shard_size": "5gb" } } }, "delete": { "min_age": "30d", "actions": { "wait_for_snapshot": { "policy": "prod-snapshots" }, "delete": {} } } } } } # illustrative: snapshots kept for 150 days PUT _slm/policy/prod-snapshots { "schedule": "0 30 1 * * ?", "name": "<prod-logs-snap-{now/d}>", "repository": "log-snapshots", "retention": { "expire_after": "150d" } }
Result
Monitoring cost went from about $1,225 a month to $400, roughly 67% lower and about $10K saved a year. The team kept log search, dashboards and alerting, and we now control how long data is kept. I own the stack day to day: pipeline changes, dashboards, alert rules and retention.
What I'd do next
As traffic grows, I would add Elasticsearch replicas and more nodes for high availability. I would move alerting toward SLOs, so alerts fire on user-facing error budgets instead of raw thresholds. I would also look at OpenTelemetry to add traces and keep instrumentation portable as the company moves to AWS EKS.