IDPS System with Snort / Suricata

Summary

This documentation describes the setup, configuration and testing of an Intrusion Detecetion Prevention System (IDPS) used in a unviersity project.

The system was built in VMware and uses open-source tools such as Snort, Suricata, iptables and NFQUEUE. The main goal was to detect and block malicious traffic in a controlled lab environment.

The project follows the same structure for each attack:

• No IDS / IPS: attack runs without protection
• IDS mode: attack is detected and logged
• IPS mode: attack is detected and blocked

The main attacks tested were:

• SSH brute force
• TCP SYN port scanning
• Spoofed ICMP ping flood
• SQL injection
• HTTP flood

Requirements

Virtual Machines

The lab uses three main virtual machines:

• Attacker VM (Kali Linux)
• IDPS VM (Ubuntu Server)
• Victim VM (Ubuntu Server)

Tools

• VMware Workstation Pro
• Snort
• Suricata
• iptables
• NFQUEUE
• Apache2
• PHP
• SQLite
• Nmap
• hping3
• wrk
• Postfix
• tcpdump

Network Setup

The lab lab uses VMWare as its virtualization platform, for easy of use and more customization options. It also allows snapshots and each member to work independently on the VMs. The setup uses two isolated internal networks: one for the victim and one for the attacker. The virtual networks were created in the VMware Virtual Network Editor, whereby the internal networks were mapped to host-only networks and the external one for internet access was a NAT network, sharing the IP of the host machine.

Topology

Note: In the real topology, vmbr0 corresponds to ens33, vmbr1 to ens37 and vmbr2 to ens38.

Network Plan

Network	Subnet	Purpose
VictimNet	192.168.10.0/24	Victim side
AttackerNet	192.168.20.0/24	Attacker side

IP Plan

Machine	IP Address	Role
Victim	192.168.10.10	Target machine
Attacker	192.168.20.10	Attack machine
IDPS victim side	192.168.10.254	Gateway for VictimNet
IDPS attacker side	192.168.20.254	Gateway for AttackerNet

The IDPS machine acts as the gateway between both networks. This forces attacker to victim traffic through the IDPS, which allows packet inspection, attack detection and blocking.

IP Forwarding

IP forwarding must be enabled on the IDPS machine, which essentially enables an internet connection, since it forwards packets to external networks:

sudo sysctl -w net.ipv4.ip_forward=1

To make it permanent, open the /etc/sysctl.conf file and add or uncomment the line below:

net.ipv4.ip_forward=1

To check if it is enabled, do:

cat /proc/sys/net/ipv4/ip_forward

IDS and IPS Concept

IDS Mode

IDS mode only detects and logs suspicious traffic.

→ Traffic passes normally
→ IDS observes traffic
→ Alert is generated
→ Attack still reaches victim

IPS Mode

IPS mode runs inline. This means the packet must pass through the security tool before it reaches the victim.

→ Packet arrives
→ iptables sends it to NFQUEUE
→ Snort / Suricata inspects it
→ Packet is accepted or dropped

This is why drop rules only work when the tool is actually inline through NFQUEUE.

Snort Setup

Snort was used mainly for the network-based attacks, such as SSH brute force, port scanning and ICMP ping flood.

Snort works with rules. A rule describes what traffic should be detected or blocked. In IDS mode, Snort uses `alert` rules. In IPS mode, the rule action can be changed to `drop`.

The main Snort configuration file is:

/etc/snort/snort.conf

Important network variables were configured inside this file:

ipvar HOME_NET 192.168.10.0/24
ipvar EXTERNAL_NET 192.168.20.0/24

In this project:

• `HOME_NET` = victim network
• `EXTERNAL_NET` = attacker network

Local custom rules were added in:

/etc/snort/rules/local.rules

The local rules file must be included in `snort.conf`:

include $RULE_PATH/local.rules

Before running Snort, the configuration was tested with:

sudo snort -T -c /etc/snort/snort.conf

Snort alerts were saved under:

/var/log/snort/alert

Suricata Setup

Suricata was used for the application-layer attacks, mainly SQL injection and HTTP flood.

Suricata can inspect normal packet data, but it can also understand protocols such as HTTP. This makes it useful for detecting attacks inside URLs, HTTP methods and web requests.

The main Suricata configuration file is:

/etc/suricata/suricata.yaml

Custom rules were added in:

/etc/suricata/rules/local.rules

The local rules file must be enabled in `suricata.yaml` under the rule files section:

rule-files:
  - local.rules

Before starting Suricata, the configuration was tested with:

sudo suricata -T -c /etc/suricata/suricata.yaml

Suricata logs were checked mainly in:

/var/log/suricata/suricata.log
/var/log/suricata/fast.log

For IDS mode, Suricata was run on the attacker-side interface:

sudo suricata -c /etc/suricata/suricata.yaml -i ens38 -l /var/log/suricata

For IPS mode, Suricata was run inline with NFQUEUE:

sudo suricata -c /etc/suricata/suricata.yaml -q 0 -l /var/log/suricata

What is NFQUEUE and Why Is It Needed?

NFQUEUE is used when Snort or Suricata should work as an IPS.

Normally, traffic is forwarded by the Linux kernel without waiting for Snort or Suricata. In that case, the tools can detect attacks, but they cannot stop the packets.

NFQUEUE changes this behavior.

Packet arrives at IDPS
→ iptables sends packet to NFQUEUE
→ Snort / Suricata inspects it
→ packet is accepted or dropped

This means the packet waits for a decision before it reaches the victim.

Without NFQUEUE:

• `alert` rules can generate logs
• `drop` rules do not really block forwarded traffic
• the attack still reaches the victim

With NFQUEUE:

• traffic is inspected inline
• `drop` rules can actually block packets
• the IDPS works as a prevention system

Attack 1: SSH Brute Force

Description

The SSH brute force attack was used to test login attack detection and prevention.

A weak test user was created on the victim machine. The attacker then used Hydra to try many passwords against SSH.

This represents a common real-world attack where an attacker tries to guess login credentials.

Attack Command

hydra -l testuser -P customlist.txt ssh://192.168.10.10

No IDPS

Without IDPS protection:

• The attack runs normally
• Hydra can keep trying passwords
• No real-time alert is generated by the IDPS
• The only evidence is in the victim logs

Victim log:

/var/log/auth.log

Snort IDS

In IDS mode, Snort was used to detect SSH connection attempts.

Example rule:

alert tcp any any -> 192.168.10.10 22
(msg:"SSH Brute Force Attempt"; flags:S; sid:1000004;)

Rule explanation:

• `alert` means Snort generates an alert, but does not block
• `tcp` means the rule checks TCP traffic
• `any any` means any source IP and any source port
• `192.168.10.10 22` means the victim SSH service
• `flags:S` matches TCP SYN packets, which are used when starting a TCP connection
• `msg` is the alert text shown in the logs
• `sid` is the unique rule ID

Result:

• SSH attempts are detected
• Alerts are logged
• The brute force attack can still continue

Snort IPS

In IPS mode, the rule was changed to `drop` and traffic was sent through NFQUEUE.

NFQUEUE rules:

iptables -I FORWARD -p tcp --dport 22 -j NFQUEUE --queue-num 0
iptables -I FORWARD -p tcp --sport 22 -j NFQUEUE --queue-num 0

Blocking rule:

drop tcp any any -> 192.168.10.10 22
(msg:"IPS BLOCKED: SSH Scan Attempt";
threshold:type both, track by_src, count 5, seconds 60;
sid:1000002;)

Rule explanation:

• `drop` means Snort blocks matching packets in inline mode
• destination port `22` targets SSH
• `threshold` avoids blocking every single SSH packet immediately
• `track by_src` counts attempts from each source IP
• `count 5, seconds 60` triggers after 5 attempts in 60 seconds

Result:

• Repeated SSH attempts are detected
• Snort drops matching packets
• The brute force attack is interrupted

Attack 2: TCP SYN Port Scanning

Description

Port scanning was used to simulate reconnaissance.

Before launching a specific attack, an attacker often scans the target to discover open ports and services.

In this project, the victim had several services open to make the scan result more realistic.

Victim Services

Port	Service
21	FTP
22	SSH
80	HTTP / Apache
8000	Demo HTTP service
9000	Demo TCP service

Attack Command

sudo nmap -sS 192.168.10.10

The `-sS` option performs a TCP SYN scan. This type of scan sends SYN packets to check which ports respond.

No IDPS

Without protection:

• Nmap discovers open ports
• Services are visible to the attacker
• No alerts are generated
• No packets are blocked

This gives the attacker useful information for future attacks.

Snort IDS

Snort was first used in IDS mode.

alert tcp 192.168.20.0/24 any -> 192.168.10.10 any
(msg:"NMAP SYN Port Scan Detected"; flags:S;
threshold:type both, track by_src, count 10, seconds 5;
sid:1000005; rev:1;)

Rule explanation:

• `alert` means detect only
• `tcp` checks TCP traffic
• source network is the attacker network
• destination is the victim
• `flags:S` matches SYN packets
• `threshold` triggers only after multiple packets
• `track by_src` counts packets per attacker IP
• `count 10, seconds 5` means 10 SYN packets within 5 seconds
• `msg` is the alert message

Result:

• Port scan is detected
• Alert is saved under `/var/log/snort/alert`
• Scan still reaches the victim

Snort IPS

For IPS mode, the rule action was changed to `drop`.

drop tcp 192.168.20.0/24 any -> 192.168.10.10 any
(msg:"PORT SCAN BLOCKED"; flags:S;
threshold:type both, track by_src, count 20, seconds 5;
sid:1000011; rev:1;)

Rule explanation:

• `drop` blocks matching packets in inline mode
• the rule still watches for SYN packets
• the threshold avoids blocking normal single connections
• when the scan rate becomes suspicious, Snort starts dropping packets

NFQUEUE setup:

iptables -I FORWARD 1 -s 192.168.20.0/24 -d 192.168.10.0/24 -p tcp -j NFQUEUE --queue-num 0

Run Snort inline:

snort -Q --daq nfq --daq-var queue=0 -c /etc/snort/snort.conf -A console

Result:

• Scan traffic is detected
• Matching packets are dropped
• Nmap receives fewer useful results

Email Notification System

Email notifications were added so the administrator does not need to watch the Snort terminal all the time.

Setup

Postfix was installed on the IDPS machine and configured to send mail through Gmail SMTP.

Setup summary:

• Installed Postfix and mail tools
• Configured Gmail SMTP relay
• Created a Gmail App Password
• Used the App Password for authentication

A normal Gmail password was not used because Gmail blocks simple password authentication from scripts and Linux mail tools.

Notification Flow

Snort live output
→ wrapper script reads alert/drop messages
→ mail command sends summary
→ Postfix forwards mail through Gmail SMTP
→ administrator receives email

The wrapper script runs Snort in IPS mode and reads the live console output line by line. It shows the output in the terminal, saves it into a temporary buffer, and sends the collected output by email every 30 seconds. A lock file is used to prevent the script from sending too many emails at the same time. This way, the administrator receives a short email summary of Snort alerts instead of manually watching the terminal.

EMAIL_TO="censored@gmail.com" 
HOSTNAME=$(hostname)
BUFFER="/tmp/snort-live-output-buffer.txt"
LOCK="/tmp/snort-live-output-lock"
SEND_INTERVAL=30

SNORT_CMD="snort -Q --daq nfq --daq-var queue=0 -c /etc/snort/snort.conf -A console"

> "$BUFFER"
rm -f "$LOCK"

sudo stdbuf -oL -eL $SNORT_CMD 2>&1 | while IFS= read -r line; do
    # Show Snort output live in terminal
    echo "$line"

    # Save every Snort output line to buffer
    echo "$line" >> "$BUFFER"

    # Start delayed email sender if not already running
    if [ ! -f "$LOCK" ]; then
        touch "$LOCK"

        (
            sleep "$SEND_INTERVAL"

            if [ -s "$BUFFER" ]; then
                {
                    echo "Snort live output from $HOSTNAME"
                    echo
                    echo "Time: $(date)"
                    echo
                    echo "Last $SEND_INTERVAL seconds of Snort output:"
                    echo
                    cat "$BUFFER"
                } | mail -s "Snort Live Output from $HOSTNAME" "$EMAIL_TO"

                > "$BUFFER"
            fi

            rm -f "$LOCK"
        ) &
    fi
done

Result:

• Admin receives alerts by email
• Monitoring is easier
• The IDPS behaves more like a real monitoring system

Attack 3: Spoofed ICMP Ping Flood

Description

The spoofed ping flood attack sends a high number of ICMP Echo Request packets to the victim.

The attacker also randomizes the source IP address. This makes detection harder because the attack does not appear to come from one fixed IP.

Attack Command

hping3 -1 --flood -V --rand-source 192.168.10.10

Command explanation:

• `-1` uses ICMP mode
• `--flood` sends packets as fast as possible
• `-V` enables verbose output
• `--rand-source` randomizes the source IP
• `192.168.10.10` is the victim

Monitoring

On the victim:

sudo tcpdump -ni any 'icmp and dst host 192.168.10.10'

From the gateway:

ping 192.168.10.10

The `tcpdump` output shows the incoming ICMP packets. The gateway ping shows whether the victim becomes less responsive.

No IDPS

Result:

• Victim receives many ICMP packets
• Source IPs are random/spoofed
• Gateway ping becomes less stable
• No alerts are generated
• No packets are blocked

Snort IDS

Because the source IP is spoofed, the rule tracks by destination instead of source.

alert icmp any any -> 192.168.10.10 any
(msg:"SPOOFED ICMP PING FLOOD DETECTED";
itype:8; detection_filter:track by_dst, count 100, seconds 1;
sid:1000501; rev:1;)

Rule explanation:

• `alert` means detection only
• `icmp` checks ICMP traffic
• `any any` is used because the source IP is spoofed
• destination is the victim
• `itype:8` matches ICMP Echo Requests
• `track by_dst` counts traffic going to the victim
• `count 100, seconds 1` triggers after 100 requests in 1 second

Result:

• Flood is detected
• Alert is logged
• Flood still reaches the victim

Snort IPS

The same idea was used in IPS mode, but with `drop`.

drop icmp any any -> 192.168.10.10 any
(msg:"SPOOFED ICMP PING FLOOD BLOCKED";
itype:8; detection_filter:track by_dst, count 100, seconds 1;
sid:1000502; rev:1;)

Rule explanation:

• `drop` blocks the matching ICMP packets
• source is still `any` because it is spoofed
• detection is based on high traffic rate to the victim
• once the threshold is reached, packets are dropped

NFQUEUE setup:

sudo iptables -F FORWARD
sudo iptables -I FORWARD 1 -i ens38 -o ens37 -p icmp -j NFQUEUE --queue-num 0

Run Snort inline:

sudo snort -Q --daq nfq --daq-var queue=0 -c /etc/snort/snort.conf -A console

Result:

• Flood is detected
• ICMP packets are dropped
• Less traffic reaches the victim
• Gateway ping becomes more stable

Attack 4: SQL Injection

Description

SQL injection was used to test an application-layer attack.

For this scenario, a simple vulnerable PHP product lookup website was created on the victim. It was hosted on Apache and used SQLite as the database.

The attacker entered the malicious input directly in Firefox.

Website Setup

The vulnerable page was:

/sqli-demo.php?id=1

<?php
$db = new SQLite3('/var/www/html/products.db');

$id = $_GET['id'] ?? '1';

$sql = "SELECT id, name, price FROM products WHERE id = $id";

$result = @$db->query($sql);
?>

<!DOCTYPE html>
<html>
<head>
    <title>Product Lookup</title>
</head>
<body>
    <h1>Product Lookup Demo</h1>
    <hr>

    <?php
    if ($result) {
        while ($row = $result->fetchArray(SQLITE3_ASSOC)) {
            echo "<p>";
            echo "<strong>ID:</strong> " . htmlspecialchars($row['id']) . "<br>";
            echo "<strong>Name:</strong> " . htmlspecialchars($row['name']) . "<br>";
            echo "<strong>Price:</strong> " . htmlspecialchars($row['price']);
            echo "</p><hr>";
        }
    } else {
        echo "<p>Query failed.</p>";
    }
    ?>
</body>
</html>

The database contained a simple product table:

ID	Product	Price
1	Laptop	999 EUR
2	Phone	699 EUR
3	Tablet	399 EUR
4	Headphones	149 EUR

Vulnerable PHP Query

The PHP file reads the ID value from the URL:

$id = $_GET['id'] ?? '1';

Then it directly inserts the value into the SQL query:

$sql = "SELECT id, name, price FROM products WHERE id = $id";

This is vulnerable because the application trusts user input. The input becomes part of the SQL command.

Original Query

Normal request:

/sqli-demo.php?id=1

SQL query:

SELECT id, name, price FROM products WHERE id = 1;

Result:

• only product ID 1 is returned

Malicious Query

Malicious request:

/sqli-demo.php?id=1 OR 1=1

SQL query:

SELECT id, name, price FROM products WHERE id = 1 OR 1=1;

Since `1=1` is always true, all rows in the product table are returned.

Why This Query Is Vulnerable

The input is not filtered or parameterized.

Filtered input means checking the value before using it. For example, a product ID should only contain numbers.

Parameterized input means the SQL structure is fixed, and user input is passed separately as data. This prevents the input from becoming SQL code.

In this demo, the input was directly added into the SQL string, so the attacker could change the query logic.

No IDPS

The attacker used Firefox:

http://192.168.10.10/sqli-demo.php?id=1 OR 1=1

Result:

• Request reaches the PHP page
• SQL query is manipulated
• All products are returned
• No alert is generated
• No traffic is blocked

Suricata IDS

Suricata was used to inspect the HTTP URI.

alert http 192.168.20.0/24 any -> 192.168.10.10 80
(msg:"SQL Injection Attempt Detected - Boolean OR Pattern";
flow:established,to_server;
http.uri;
pcre:"/sqli-demo\.php\?id=[^&]*\bOR\b[^&]*1=1/Ui";
sid:2001001; rev:1;)

Rule explanation:

• `alert` means detect and log only
• `http` tells Suricata to inspect HTTP traffic
• source is the attacker network
• destination is the victim web server
• `flow:established,to_server` checks established requests going to the server
• `http.uri` tells Suricata to inspect the URL
• the `pcre` searches for the vulnerable page and the `OR 1=1` pattern
• `sid` is the rule ID

Result:

• SQL injection is detected
• Alert is logged
• Injection still succeeds

Logs:

/var/log/suricata/suricata.log

Suricata IPS

The IDS rule was changed from `alert` to `drop`.

drop http 192.168.20.0/24 any -> 192.168.10.10 80
(msg:"SQL Injection Attempt BLOCKED - Boolean OR Pattern";
flow:established,to_server;
http.uri;
pcre:"/sqli-demo\.php\?id=[^&]*\bOR\b[^&]*1=1/Ui";
sid:2001001; rev:2;)

Rule explanation:

• `drop` blocks the malicious HTTP request
• the same URI pattern is used
• when the SQL injection is detected, the packet is dropped before the app processes it

NFQUEUE for HTTP

For Suricata IPS, HTTP traffic had to be queued in both directions:

sudo iptables -I FORWARD 1 -i ens38 -o ens37 -p tcp --dport 80 -j NFQUEUE --queue-num 0
sudo iptables -I FORWARD 2 -i ens37 -o ens38 -p tcp --sport 80 -j NFQUEUE --queue-num 0

Run Suricata inline:

sudo suricata -c /etc/suricata/suricata.yaml -q 0 -l /var/log/suricata

Why Both Directions Were Needed

Suricata needs enough TCP and HTTP flow information to inspect the request correctly.

The SQL injection rule uses:

flow:established,to_server;
http.uri;

This means Suricata must understand that the request belongs to an established HTTP flow and that it is going to the server.

At first, only attacker-to-victim traffic was queued. The rule did not match correctly.

After queueing both directions, Suricata could understand the full HTTP flow and block the malicious request.

Result:

• SQL injection pattern is detected
• Request is dropped
• Web app does not return all products

Attack 5: HTTP Flood

Description

The HTTP flood targeted the Apache web server by sending a large number of HTTP GET requests.

This is a Layer 7 DoS-style attack. It does not exploit SQL logic. Instead, it tries to overload the web server by forcing it to handle many requests.

Attack Command

wrk -t12 -c400 -d30s http://192.168.10.10/

Command explanation:

• `wrk` is an HTTP benchmarking tool used to generate a high load, especially on multi core CPUs. More about wrk can be found in the References section
• `-t12` uses 12 threads
• `-c400` opens 400 connections
• `-d30s` runs the test for 30 seconds
• target is the Apache server on the victim

Monitoring

On the victim:

sudo tail -f /var/log/apache2/access.log

This shows how many requests are reaching Apache.

No IDPS

Result:

• Apache receives many GET requests
• Access log fills quickly with 200 OK entries
• No alerts are generated
• No blocking is applied

Suricata IDS

alert http any any -> 192.168.10.10 80
(msg:"HTTP Flood Detected - High Request Rate";
flow:to_server,established;
threshold:type threshold, track by_src, count 100, seconds 5;
sid:9000001; rev:1;)

Rule explanation:

• `alert` means Suricata only logs the attack
• `http` inspects HTTP traffic
• destination is the victim web server
• `flow:to_server,established` checks established requests to Apache
• `threshold` detects a high request rate
• `track by_src` counts requests from each source IP
• `count 100, seconds 5` triggers when the source sends 100 requests within 5 seconds

Result:

• HTTP flood is detected
• Alert is generated
• Apache still receives the requests

Suricata IPS

drop http any any -> 192.168.10.10 80
(msg:"HTTP Flood Blocked - High Request Rate";
flow:to_server,established;
threshold:type threshold, track by_src, count 100, seconds 5;
sid:9000001; rev:2;)

Rule explanation:

• `drop` blocks traffic after the threshold is reached
• the rule does not block normal browsing immediately
• it reacts when the request rate becomes suspicious

NFQUEUE:

sudo iptables -I FORWARD 1 -i ens38 -o ens37 -p tcp --dport 80 -j NFQUEUE --queue-num 0
sudo iptables -I FORWARD 2 -i ens37 -o ens38 -p tcp --sport 80 -j NFQUEUE --queue-num 0

Run Suricata inline:

sudo suricata -c /etc/suricata/suricata.yaml -q 0 -l /var/log/suricata

Result:

• High request rate is detected
• Excessive HTTP requests are dropped
• Apache receives fewer successful requests
• The web server stays more stable

Results Summary

Attack	Tool	IDS Result	IPS Result
SSH Brute Force	Snort	Login attempts detected	Repeated attempts blocked
Port Scanning	Snort	SYN scan detected	Scan packets dropped
Ping Flood	Snort	ICMP flood detected	Flood packets dropped
SQL Injection	Suricata	Malicious URI detected	HTTP request dropped
HTTP Flood	Suricata	High request rate detected	Excessive requests dropped

Main Lessons Learned

• IDS mode gives visibility, but does not stop the attack.
• IPS mode is needed when traffic should be blocked.
• NFQUEUE is required for inline blocking.
• Snort worked well for network-based attacks.
• Suricata worked well for HTTP and application-layer attacks.
• For spoofed traffic, tracking by destination can be better than tracking by source.
• For HTTP inspection, Suricata needs enough flow visibility.
• Email notifications make alerts easier to monitor.

References

• https://www.snort.org/documents
• https://docs.suricata.io/en/suricata-8.0.5/
• https://www.netfilter.org/
• https://labex.io/tutorials/hydra-brute-force-ssh-in-hydra-549926
• https://nmap.org/book/synscan.html
• https://ubuntu.com/server/docs/how-to/mail-services/install-postfix/
• https://owasp.org/www-community/attacks/SQL_Injection
• https://clouddocs.f5.com/training/community/firewall/html/class2/ddos2.html
• https://github.com/wg/wrk