IDS — Intrusion Detection System
A passive system that monitors copies of network traffic (via tap or SPAN port) and generates alerts when suspicious activity is detected. An IDS never touches live traffic — it cannot block anything. Its value is in visibility and forensic evidence.
- Out-of-band deployment: no risk of disrupting production traffic
- Can analyze all traffic with no performance penalty on the network path
- Alert-only: depends on humans or downstream SIEM to respond
- Best for: forensics, compliance logging, threat hunting
IPS — Intrusion Prevention System
An inline system that sits directly in the traffic path. It can drop packets, reset connections, or shunt suspicious traffic in real time. Because it's inline, a misconfiguration or high false-positive rate can disrupt legitimate traffic.
- Inline deployment: introduces latency (typically 1–5ms for modern appliances)
- Can drop individual packets or terminate connections
- High-availability deployment requires failopen/failclosed decision
- Best for: active threat prevention at choke points
NIDS vs HIDS
Network IDS monitors traffic on a network segment. Host IDS runs on individual endpoints, monitoring system calls, file integrity, log entries, and process behavior. Both are complementary — NIDS sees network-wide patterns; HIDS sees what happens after a connection reaches a host.
- NIDS: Suricata, Zeek, Snort — deploy at network tap points
- HIDS: Wazuh, OSSEC, Falco (containers) — deploy on every host
- HIDS detects encrypted attacks that NIDS cannot decode
- HIDS monitors file integrity (FIM) for rootkits and tampering
False Positive vs. False Negative Tradeoff
Every IDS/IPS operates on a sensitivity dial. More sensitive = catches more attacks (fewer false negatives) but also generates more false positives, causing alert fatigue. The right balance depends on risk tolerance and SOC capacity.
- False positive (FP): legitimate traffic flagged as malicious
- False negative (FN): real attack not detected
- High FP rate → analysts ignore alerts → real attacks missed
- High FN rate → attacks succeed silently
| Attribute | IDS (Detection) | IPS (Prevention) |
|---|---|---|
| Network placement | Out-of-band (SPAN/tap) | Inline (bump-in-the-wire) |
| Response to threats | Alert only | Drop, reset, shunt, alert |
| Risk of disruption | Zero (passive) | High if misconfigured or high FP rate |
| Latency impact | None | 1–5ms typical for hardware appliance |
| Typical deployment | Monitoring, compliance, forensics | Active perimeter defense, critical segments |
| Tuning sensitivity | Can be aggressive — no disruption risk | Must tune carefully before enabling block mode |
Signature-based detection matches network traffic against a database of known attack patterns. It is the most reliable detection method for known threats — high accuracy, low false positive rate — but blind to zero-days, novel attack variants, and encrypted payloads it cannot decode.
Snort — The IDS/IPS Pioneer
Snort (now maintained by Cisco Talos) is the most widely deployed open-source IDS/IPS. Its rule language is the lingua franca of network intrusion detection and has influenced every subsequent system.
- Single-threaded architecture limits throughput on high-speed links
- Snort 3 (2020+) introduced multi-threading and improved performance
- Talos Intelligence: world-class commercial rule set
- Community rules: free, updated daily for Snort 2/3
Suricata — The Modern Standard
Suricata (Open Information Security Foundation / OISF) is multi-threaded, supports YAML-based rules, and handles 10–100 Gbps on commodity hardware. It has largely replaced Snort in high-throughput environments and in cloud-native deployments.
- Multi-threaded: scales linearly with CPU cores
- Native support for Lua scripting for complex detections
- Outputs JSON EVE logs for easy SIEM integration
- AF-PACKET, DPDK, PF_RING for kernel-bypass capture
Rule Sets & Threat Intelligence
IDS rules are only as good as the intelligence behind them. Multiple rule set providers maintain different coverage profiles — commercial sets provide faster response to emerging threats, while open sets provide baseline coverage at no cost.
- ET Open (Proofpoint): free, ~40,000 rules, updated daily
- ET Pro: commercial, includes newer threat coverage
- Cisco Talos: commercial, included with Snort subscription
- Abuse.ch: threat feed integrations (URLhaus, FeodoTracker)
# Snort/Suricata rule anatomy — detect EternalBlue (MS17-010) exploit attempt
# action proto src_ip src_port direction dst_ip dst_port (options)
alert tcp $EXTERNAL_NET any -> $HOME_NET 445 (
msg:"ET EXPLOIT Possible ETERNALBLUE MS17-010 Echo Request (Client)";
flow:established,to_server;
content:"|00 00 00 31|"; depth:4;
content:"|FF|SMB"; within:8;
content:"|72 00 00 00 00 00|"; distance:0;
threshold:type limit, track by_src, count 1, seconds 60;
classtype:attempted-admin;
sid:2024218; rev:4;
)
# Key fields:
# flow: connection direction and state
# content: byte pattern match (hex or ASCII)
# distance: bytes from previous content match
# threshold: rate-limit alerting to reduce noise
# classtype: categorize for prioritization
# sid: unique signature identifierBaseline Profiling
Anomaly detection requires establishing what "normal" looks like first. The system learns typical traffic volumes, protocol distributions, connection patterns, and user behaviors over a baseline period (typically 2–4 weeks), then alerts on deviations.
- Statistical anomaly: deviation from mean + standard deviation
- Protocol anomaly: RFC-violating packets (unusual flags, oversized fields)
- Behavioral anomaly: user accessing resources outside their normal pattern
- Baseline must be re-established after major infrastructure changes
UEBA — User & Entity Behavior Analytics
UEBA platforms build individual behavioral profiles for users, devices, and service accounts. Machine learning identifies outliers — an account downloading 100GB at 3am, a service account logging in interactively, a user accessing HR systems they've never touched.
- Splunk UBA: peer group comparison, Bayesian risk scoring
- Microsoft Sentinel: UEBA built-in, integrates Azure AD signals
- Exabeam: timeline-based behavior analytics
- Effective against insider threats and compromised credentials
DNS Anomaly Detection
DNS traffic reveals attacker behavior that other detection methods miss. Malware communicating via DNS tunnels, DGA (domain generation algorithm) malware generating pseudo-random domains, and beaconing (regular-interval DNS queries to C2 infrastructure) all have distinctive signatures.
- DGA detection: high entropy domain names, NXDOMAIN floods
- DNS tunneling: unusually long TXT/NULL record queries
- Beaconing: identical query intervals (10s, 60s, 300s)
- Tools: Zeek DNS logs + ML classifiers, Cisco Umbrella Investigate
Limitations of Anomaly Detection
Anomaly detection catches what signature detection misses, but it comes with significant operational challenges. The cost of high false positive rates and the time required for reliable baselining mean anomaly detection augments — rather than replaces — signature detection.
- High false positive rate: legitimate new behavior looks anomalous
- Slow baselining: 2–4 weeks before useful detection begins
- Attackers who move slowly ("low and slow") stay within baseline
- ML models require continuous retraining as environment evolves
NTA goes beyond alerting to provide deep visibility into what is actually happening on the network — who is talking to whom, what protocols are in use, what files are being transferred, and what the TLS fingerprints of clients reveal about the software stack.
Zeek (formerly Bro)
Zeek is a powerful network analysis framework that parses protocols and produces structured log files for every connection, DNS query, HTTP request, TLS session, file transfer, and more. It's a platform for writing network security scripts, not just running fixed signatures.
- conn.log: every TCP/UDP/ICMP flow with bytes, duration, state
- dns.log: every DNS query and response with answer
- ssl.log: TLS certificate details, JA3 hash, SNI
- files.log: MD5/SHA of every extracted file for VirusTotal lookup
- Custom scripts in Zeek language for bespoke detections
Encrypted Traffic Analysis (ETA)
Modern attackers use TLS to hide C2 traffic. ETA techniques analyze the observable metadata of encrypted connections — handshake parameters, certificate fields, flow timing, packet sizes — without decrypting content, preserving privacy while enabling detection.
- JA3: MD5 of TLS Client Hello parameters — identifies client software
- JA3S: MD5 of TLS Server Hello — identifies server response profile
- JA4+: 2023 successor to JA3; more robust, versioned, human-readable
- Known malware families have consistent JA3 hashes
NetFlow / IPFIX for Metadata
Flow records (NetFlow v9, IPFIX) provide connection metadata — source, destination, port, protocol, byte/packet counts, duration — without full packet content. Extremely efficient for monitoring high-speed links where full PCAP is too costly to store.
- Exported by routers, switches, firewalls natively
- Collectors: nfdump, ntopng, Elastic, SolarWinds NTA
- 1:1 sampling for security use (vs 1:1000 for capacity planning)
- Combined with IPAM for asset context enrichment
NDR — Network Detection and Response
NDR platforms combine NTA, ML-based anomaly detection, and automated response in a single product. They fill the visibility gap that EDR (endpoint) and SIEM (log-based) leave — specifically, detecting attacks in network traffic without relying on endpoint agents or log sources.
- Darktrace: unsupervised ML, "Enterprise Immune System" model
- Vectra AI: AI-driven threat detection, Attack Signal Intelligence
- ExtraHop Reveal(x): real-time wire data analytics
- Corelight: commercial Zeek + additional detections
Alert Fatigue Kills Detection Programs
An IDS generating thousands of alerts per day trains analysts to ignore them. Effective IDS operation is not about maximum rule coverage — it's about high-confidence, actionable alerts that analysts trust and act on every time. Start with a small, well-tuned rule set and expand deliberately.
Reducing False Positives
Systematic FP reduction is the most important operational task for a new IDS deployment. The goal is not zero FPs (impossible) but a manageable rate where every alert warrants investigation.
- Suppress rules that fire only on known-safe infrastructure
- Threshold tuning: require 10 hits in 60 seconds before alerting
- IP whitelisting: internal scanners, pentest IPs, CDN ranges
- Run in detect-only mode for 2–4 weeks before enabling blocking
Alert Triage Workflow
A documented triage workflow ensures alerts are handled consistently. Every alert should have a defined first responder action, escalation path, and closure criteria.
- Tier 1: initial triage — determine if alert is FP or requires investigation
- Tier 2: investigation — correlate with other data sources, determine scope
- Tier 3: incident response — contain, eradicate, recover
- Document FP determinations to build institutional knowledge
| Metric | Target Value | Tuning Lever |
|---|---|---|
| False positive rate | < 5% of all alerts | Suppress, threshold, whitelist specific rules |
| Mean time to triage (MTTT) | < 15 minutes | SIEM correlation, enrichment automation, playbooks |
| Alert volume per analyst/day | < 50 actionable alerts | Alert consolidation, deduplication, severity tuning |
| Detection coverage (MITRE ATT&CK) | > 60% of relevant techniques | Add targeted rules for gaps in coverage |
| Rule update latency | < 24h for critical CVEs | Automate rule set pulls; CI/CD for rule deployment |
| Blocking mode coverage | High-confidence rules only | Separate high/low confidence rule categories |
| Solution | Type | Strengths | Limitations |
|---|---|---|---|
| Suricata + Zeek | Open source | Free, flexible, community-supported, high performance | Requires ops expertise; no built-in GUI or SIEM |
| Wazuh (HIDS) | Open source | Free, SIEM included, agent-based, FIM built-in | Network-level detection limited; resource intensive |
| Darktrace | Commercial NDR | AI/ML, no rules to write, rapid deployment | Expensive; "black box" ML can be hard to explain |
| Vectra AI | Commercial NDR | Attack Signal Intelligence, strong lateral movement detection | High cost; cloud-centric model |
| ExtraHop Reveal(x) | Commercial NDR | Real-time wire data, strong for OT/IoT environments | Cost; requires significant network tap infrastructure |