Building and tuning a SIEM from scratch

Context

A small production environment doesn’t give you many chances to practice detection engineering. This lab was built to do that work end to end: collect logs, write detections, tune out the noise, and make alerts useful to a person.

Build

• SIEM server. Wazuh on an Ubuntu host, hardened against the CIS Benchmarks with UFW, SSH hardening, PAM policy, auditd, and AppArmor. Progress is tracked with Wazuh’s own security configuration assessment.
• Agents. Deployed on Windows and macOS endpoints for log collection and file integrity monitoring (FIM).
• Alerting. High-severity alerts are routed to Slack with a webhook, so the alerts that matter reach a person.
• Dashboards. Grafana on top of the OpenSearch data shows alert severity, trends over time, top-firing rules, and per-agent activity.

Detection engineering

Out of the box, the main problem wasn’t missing alerts. It was too many of the wrong ones. Custom XML rules were written to:

• Bring SSH activity to the surface that the default rules buried.
• Suppress known false positives with narrowly scoped conditions, not by disabling whole rules.
• Re-level noisy Windows authentication events, so routine sign-ins stop drowning out the ones worth a look.

The guiding rule: every change to a rule is narrow and has a written reason, so tuning never quietly hides a real signal.

Key decisions

• Start with conservative thresholds and tighten later. It’s easier to quiet a noisy rule than to discover a missed detection after the fact.
• Harden the SIEM itself. A detection platform is a high-value target, so it got the same hardening it is meant to watch for.

If I rebuild it

The lab is currently offline. If I stand it up again, the next steps would be:

• Sysmon on Windows for richer process and network telemetry.
• Uptime monitoring for the lab’s services.
• A Python daily digest that summarizes the previous day’s alerts in Slack.