How does CyberKC keep LLM costs flat as log volume grows?

Four mechanisms, all of which push work downward. Rules resolve most traffic at zero marginal cost. The batcher collapses duplicate alerts inside 30-second windows. A semantic cache serves near-identical alerts from embeddings. And the template promoter converts repeated LLM verdicts into free Tier 0 templates, so added volume mostly lands in tiers that cost nothing per alert.

What does each ML model in Tier 1 actually do?

XGBoost reads tabular signal such as rule level, event ID and logon type. A Bi-LSTM reads sequence patterns across a host's event window. An autoencoder paired with isolation forest flags anomalies it was never trained on, which is the zero-day path. They fuse with fixed weights of 0.5, 0.3 and 0.2 respectively.

Is CyberKC open source?

No. The repository is private. What is described here comes from the codebase directly: the tier services, the ONNX training pipeline, the committed evaluation reports and the CI configuration. If you want to see it running against your own Wazuh feed, that is a conversation rather than a clone.

What does it cost to run in production?

The project documentation models 50,000 logs per day at roughly $70 to $160 a month all in, dominated by GPU electricity for the local LLM and cloud API spend on the 1.5% of alerts that reach Tier 3. Those are the repository's own projections. The /api/v1/metrics/cost endpoint reports what you are actually spending.

Can this work with my existing Wazuh stack?

Yes, and it does not modify Wazuh. It reads from the Wazuh Indexer over OpenSearch with cursor pagination, optionally receives high-severity alerts via integratord webhook, and executes blocks through the Wazuh Active Response API. Everything else is a separate Docker Compose stack you point at your indexer URL.

Will it block legitimate traffic automatically?

Dry-run mode is on by default, so actions log without executing until you turn it off. Beyond that: infrastructure IPs sit on an allowlist that cannot be blocked, blocks are rate-limited per hour, anything at or above a /24 requires human approval, and every firewall rule carries a TTL with a background worker that reverses it.

How do you know the ML models are not just memorising the labeller?

I checked, and initially they were. A leakage diagnosis found rule_id predicted the label 100% of the time and rule_level 82.93%. Retraining without those features produced 92.3% accuracy at 0.338 macro F1, and both reports are committed. The leakage check now runs as a CI job on pull requests and pushes to main.

CyberKC — Two-Layer AI Threat Detection for Wazuh SIEM — Case Study

Introduction

Wazuh is good at producing alerts and bad at telling you which ones matter. The obvious fix in 2026 is to point an LLM at the alert stream, and the obvious problem with that fix is the bill. I built CyberKC to answer a specific question: can you get genuine AI reasoning on a real SIEM feed without the cost scaling linearly with log volume.

The Challenge

A mid-sized Wazuh deployment produces tens of thousands of alerts a day, and the overwhelming majority are noise that a deterministic rule could dismiss in microseconds. Sending that stream to a cloud model means paying reasoning prices for pattern matching. But the inverse also fails: pure rules and pure ML miss the novel, multi-vector cases that are exactly what a SOC needs help with. The constraint I had to design around was that the cases needing reasoning are rare and unpredictable, while the cases needing speed are constant and boring. Any single-model architecture optimises for one and pays for the other.

The Solution

I built a funnel rather than a pipeline. Every alert enters at Tier 0 and only moves up when the current tier cannot resolve it with enough confidence. Tier 0 is Wazuh rule levels, direct rule ID mapping, MITRE ATT&CK fields, regex for injection and traversal patterns, and LOLBin parent-child process checks. Tier 1 is a two-stage ML pipeline. Tier 2 is a cybersecurity-tuned local model on Ollama. Tier 3 is a cloud model, reserved for novel or multi-vector threats. Above all of it sits a feedback loop that watches what the LLMs decide and hands the repeatable parts back to Tier 0.

Technical Deep Dive

Tier 1 is two stages, not one. A binary gate answers normal-versus-attack first with a 0.45 threshold, and the multi-class ensemble only runs on positives. That means the expensive part of the ML tier is skipped for most of the traffic that reaches it. The ensemble fuses three models with fixed weights: 0.5 tabular from XGBoost, 0.3 sequence from a Bi-LSTM over per-agent event windows, and 0.2 anomaly from an autoencoder paired with isolation forest. Each model has a job the others cannot do: XGBoost reads tabular signal like event ID and logon type, the LSTM sees ordering across a host's event stream, and the autoencoder catches shapes it has never been trained on.

All three Tier 1 models are exported to ONNX and served through onnxruntime rather than their training frameworks. Training uses PyTorch, XGBoost, LightGBM and CatBoost with Optuna for tuning, but none of that is in the serving path, so the runtime image stays small and inference latency stays predictable. The canonical 16-feature order is pinned in code so the enrichment engine cannot silently feed the models a different vector than they were trained on, and when ONNX artifacts are absent the tier degrades to a heuristic classifier instead of failing. Feature enrichment comes from a rolling window engine computing per-IP counts over 5-minute, 15-minute, 1-hour and 24-hour windows plus Shannon entropy on login hours and event types.

The escalation thresholds are the actual cost control. Tier 0 must reach 85% confidence to terminate. Tier 1 escalates to the local LLM when confidence drops below 0.70 or rule severity is 10 or higher. Tier 2 escalates to cloud when its own confidence falls below 0.70, when a critical alert comes back uncertain, or when the local model fails outright on a circuit-breaker trip or timeout. Those thresholds determine the tier split, and they are configuration, not architecture, so a customer with different risk tolerance moves the money around without a rewrite.

The template promoter is what makes the economics improve rather than hold. When a rule_id plus category combination has been classified the same way by an LLM five or more times, each at 80% confidence or better, it is promoted to a Tier 0 template stored in Redis with a 24-hour TTL. From then on that pattern resolves for free. The LLM tiers effectively pay a one-time tuition fee per pattern instead of a per-alert rent.

Two more layers sit in front of the models. An alert batcher groups alerts sharing rule_id, source_ip and category inside a 30-second window into one representative alert, which matters most during attack bursts when duplicate volume spikes. A semantic cache then does a tiered lookup before any LLM call: exact hash, then cosine similarity above 0.85 using all-MiniLM-L6-v2 embeddings, then a 0.70 to 0.85 near-match band where the cached response is adapted rather than regenerated.

Cost is a first-class observable. app/core/observability/cost_tracker.py exports Prometheus histograms and counters for USD per request labelled by tier and model, token histograms labelled by tier and direction, plus rolling hourly and daily gauges and a budget-exceeded counter. A /api/v1/metrics/cost endpoint returns the breakdown by model and tier. I also wrote scripts/cost_analysis.py, which takes a measured Tier 3 call (749 input tokens, 3,800 output tokens) and projects monthly spend across models at the documented 1.5% Tier 3 volume.

Tier 3 goes through a PII tokenizer before it leaves the network. IPs, emails, Windows domain and user pairs, home directory paths and SIEM hostname patterns are replaced with typed tokens such as IP_EXTERNAL_001 or HOST_001, and the response is de-tokenized so analysts see real values. Each call gets a fresh mapping, so nothing correlates across calls. The same tokenizer redacts the few-shot example bank, which means analyst-corrected examples can be injected into cloud prompts safely.

Key Features

Four-tier escalation funnel

Rules, ML, local LLM and cloud LLM in ascending cost order, with every alert entering at the bottom. The project documentation models the split as roughly 65% Tier 0, 28% Tier 1, 5.5% Tier 2 and 1.5% Tier 3.

Self-shrinking LLM surface

The template promoter converts consistent LLM verdicts into free Tier 0 templates, the batcher collapses duplicate bursts, and the semantic cache absorbs near-identical alerts. Three independent mechanisms all push work downward.

Durable ingestion

Hybrid pull and push: cursor-paginated polling of the Wazuh Indexer plus an integratord webhook for level 10 and above. Both land in Redis Streams with consumer groups, three XPENDING/XCLAIM retries and a dead-letter stream.

Guardrailed action engine

IP blocks via Wazuh Active Response, AD lockout via LDAP, Slack approve/deny/escalate buttons. Constrained by an infrastructure allowlist, an hourly block rate limit, a subnet blast-radius check, TTL auto-reversal and dry-run mode on by default.

Cost and carbon observability

Prometheus metrics for USD and tokens by tier and model, Grafana dashboards for operations and the ML pipeline, Langfuse and OpenLIT tracing, and CodeCarbon energy tracking. Spend is a dashboard panel, not a surprise.

Model registry with shadow deploys

New models register, deploy to shadow, run alongside production with results logged but not acted on, and only promote after reaching 90% agreement with the production model over at least 100 shadow predictions. Rollback is an API call.

Results & Impact

✓The four-tier funnel, the template promoter, the alert batcher and the semantic cache all ship together, which is what makes the economics flat: as log volume grows, the added volume is absorbed almost entirely by the tiers that cost nothing per alert.
✓The repository documents 46 API endpoints across 8 Flask blueprints and a 75-test endpoint suite that runs without Redis, PostgreSQL or Ollama present, because Redis degrades gracefully, SQLite substitutes for PostgreSQL, and ML falls back to a heuristic classifier when ONNX models are not loaded.
✓A committed Tier 3 evaluation against claude-haiku-4-5 over 12 curated alerts recorded full category-match and schema-validity rates, 0.875 MITRE technique recall, and an average measured cost of $0.0040 per call. That harness runs as a CI job whenever an API key is configured.
✓I ran a label-leakage diagnosis on the 58,469-row training set and found rule_id leaking the label 100% of the time and rule_level 82.93%. Retraining without those features gives 92.3% accuracy but only 0.338 macro F1, which is the honest number and is committed to the repo alongside the optimistic one.
✓The whole stack runs from a single docker-compose up: API, two pipeline worker replicas, a dedicated LLM worker, Ollama, TimescaleDB, Redis, Prometheus, Alertmanager, Grafana and Langfuse.

Lessons Learned

"The cheapest LLM call is the one a rule already answered. Most of the engineering value in this build was not in prompting, it was in the four mechanisms that stop an alert before it reaches a model. Teams that start by choosing a model and end by adding caching get the ordering backwards."

"Publishing the leakage report was uncomfortable and correct. The first training run looked excellent because rule_id encoded the label perfectly. A model that has memorised the labeller is worthless in production, and the only way to know is to run the diagnosis and keep the bad number where reviewers can see it. I made the leakage check a CI job."

"Cost tracking has to be instrumentation, not estimation. A README table is a hypothesis. Prometheus histograms labelled by tier and model, plus a rolling daily gauge and a budget-exceeded counter, are how you actually find out that one noisy rule has started routing a chunk of your traffic to the cloud tier."

"Reinforcement learning belongs in shadow mode until it has earned otherwise. The bandit that picks routing tiers is genuinely useful, but I scoped it so its worst outcome is an alert analysed at the wrong tier. It never chooses a response action. Autonomy on analysis is cheap to get wrong; autonomy on blocking is not."

Conclusion

CyberKC is what AI on a SIEM feed looks like when the budget is a design constraint instead of an afterthought. The reasoning is real and reserved for the alerts that need it, and every layer above the rules is instrumented so the spend is visible before it becomes a problem. If you are evaluating AI detection for a Wazuh deployment and the unbounded-bill question is what is stopping you, this is the shape of the answer.

Frequently asked questions

How does CyberKC keep LLM costs flat as log volume grows?: Four mechanisms, all of which push work downward. Rules resolve most traffic at zero marginal cost. The batcher collapses duplicate alerts inside 30-second windows. A semantic cache serves near-identical alerts from embeddings. And the template promoter converts repeated LLM verdicts into free Tier 0 templates, so added volume mostly lands in tiers that cost nothing per alert.
What does each ML model in Tier 1 actually do?: XGBoost reads tabular signal such as rule level, event ID and logon type. A Bi-LSTM reads sequence patterns across a host's event window. An autoencoder paired with isolation forest flags anomalies it was never trained on, which is the zero-day path. They fuse with fixed weights of 0.5, 0.3 and 0.2 respectively.
Is CyberKC open source?: No. The repository is private. What is described here comes from the codebase directly: the tier services, the ONNX training pipeline, the committed evaluation reports and the CI configuration. If you want to see it running against your own Wazuh feed, that is a conversation rather than a clone.
What does it cost to run in production?: The project documentation models 50,000 logs per day at roughly $70 to $160 a month all in, dominated by GPU electricity for the local LLM and cloud API spend on the 1.5% of alerts that reach Tier 3. Those are the repository's own projections. The /api/v1/metrics/cost endpoint reports what you are actually spending.
Can this work with my existing Wazuh stack?: Yes, and it does not modify Wazuh. It reads from the Wazuh Indexer over OpenSearch with cursor pagination, optionally receives high-severity alerts via integratord webhook, and executes blocks through the Wazuh Active Response API. Everything else is a separate Docker Compose stack you point at your indexer URL.
Will it block legitimate traffic automatically?: Dry-run mode is on by default, so actions log without executing until you turn it off. Beyond that: infrastructure IPs sit on an allowlist that cannot be blocked, blocks are rate-limited per hour, anything at or above a /24 requires human approval, and every firewall rule carries a TTL with a background worker that reverses it.
How do you know the ML models are not just memorising the labeller?: I checked, and initially they were. A leakage diagnosis found rule_id predicted the label 100% of the time and rule_level 82.93%. Retraining without those features produced 92.3% accuracy at 0.338 macro F1, and both reports are committed. The leakage check now runs as a CI job on pull requests and pushes to main.

CyberKC — Two-Layer AI Threat Detection for Wazuh SIEM