All workCase study 03 / 05
Professional · Alfa-Safety · 2024 – 2025
Collecting and analyzing 2 TB of logs a week
The Run team's log stack sat on one server, lost data at every peak and kept two weeks of logs. I proposed, learned and built a clustered ELK platform that keeps a year of logs, at no extra cost.
- Role
- Project lead, engineering apprentice
- Built for
- The Run team
- Duration
- 12 months, alongside my studies
- Stack
- Elasticsearch · Logstash · Kibana
2 TB
of logs a week, on a stack sized for up to 1 TB a day
−30%
time to analyze an incident
−70%
storage, thanks to lifecycle rules
01 · The need
- The Run team watches 1,000+ servers and services, the network, and a reverse proxy in front of 5,000+ web apps. Its log stack ran on a single server: it went down at every peak and lost the data of the outage, kept only two weeks of logs, and collected only part of the sources.
- Incidents took too long to analyze, and I had never built a SIEM. I proposed to rebuild it anyway.
02 · Constraints and decision
- Constraints
- Volume · 1.5 to 2 TB of logs a week, with sharp peaks; the stack is sized for up to 1 TB a day.
- Retention · a full year of logs, available at any time.
- Budget · no new spending, on the existing infrastructure.
- Security · logs are sensitive; every exchange encrypted, access by role.
- Footprint · no server left idle for months.
- Three designs
- A first design with five dedicated nodes for hot, warm, frozen data and a master: solid, but some nodes would sit idle for months.
- A second with TLS everywhere and light agents instead of a second Logstash: safer and simpler, but still five nodes always on.
- A third, sized to the real volume, with old logs moved to shared NFS storage: validated by the tech team, operations and management.
- Decision
- Collection · Elastic Agents on the servers, Logstash for the network devices and firewalls, every log parsed into one format.
- Storage · an Elasticsearch cluster with lifecycle rules: 30 days hot, then warm, then frozen on NFS storage, one year in total.
- Security · TLS between every component, Kibana behind an authenticated proxy, and access by role.
- Alerting · alerts from Alertmanager stored next to the logs, Grafana and Zabbix to watch the cluster itself.
- Delivery · an iterative V-cycle from a signed specification, servers deployed with Ansible roles reviewed in GitLab merge requests.
03 · Architecture
The platform
Elasticsearch
Logstash
Kibana
Elastic Agent
Ansible
Grafana
Prometheus
Nutanix
04 · Result
- Faster incidents · the Run team analyzes them 30% faster.
- Retention · one year of logs instead of two weeks, on 70% less storage.
- 0 € spent · open-source tools on the existing infrastructure.
- Validated · 32 of 35 test scenarios passed, then a Tech Day with the Run team before production.
- Adopted · the team builds its own dashboards, and checking the logs became a daily reflex.
- Reused · deployed for several clients, which opened new business.
05 · What I learned
- Learn by building · I had never built a SIEM, so I proposed it, studied it and shipped it in iterations.
- Size for the real need · three designs to remove what would sit idle, for the budget and for the planet.
Next case study
Medical AI hackathon, Lyon · 2026One GPU for a 150‑person medical AI hackathon