51 System Hardening – Fail2Ban
Aden Hilderbrand and Mathew J. Heath Van Horn, PhD
Fail2ban is a free, open-source intrusion prevention framework that monitors system log files for repeated authentication failures and automatically blocks offending IP addresses using the host firewall. It is widely used to protect SSH, FTP, HTTP, and other services against automated brute-force and credential-stuffing attacks. While platforms like Wazuh detect and log SSH brute-force attempts, detection alone does not stop an attack in progress. Fail2ban addresses this by automatically modifying iptables firewall rules to drop traffic from an attacker the moment an attempt threshold is exceeded, shifting the defensive posture from observation to prevention. In this lab, you will build a three-machine GNS3 network consisting of an Ubuntu Server running Fail2ban and SSH, an Ubuntu Desktop acting as a legitimate SSH client, and a Kali Linux machine acting as a brute-force attacker. You will observe the difference in how Fail2ban treats the two. The legitimate client connects freely, while the attacker is automatically banned after exceeding the failure threshold.
SSH is one of the most frequently attacked services on the Internet. Automated bots continuously scan for systems with port 22 open and attempt thousands of username-password combinations per minute. Without protection, a server with a weak password is highly vulnerable to compromise. Fail2ban acts as a log-watching daemon. It reads /var/log/auth.log on a timer and applies configurable rules called jails to count failed login attempts per IP address within a rolling time window. Once that count exceeds a threshold, Fail2ban inserts a DROP rule into iptables to silently discard all further traffic from that IP for a defined duration. Critically, Fail2ban acts on individual IP addresses. A ban on the attacker’s IP has no effect on any other machine on the network.
Estimated time for completion: unknown
Phase 0 – Professional Alignment
Protecting enterprise systems requires more than monitoring suspicious activity—it also requires responding to attacks as they occur. In this chapter, you will configure Fail2ban to detect repeated authentication failures, automatically block malicious hosts, and protect network services from brute-force attacks. These activities demonstrate how automated security controls strengthen enterprise defenses while reducing the administrative effort required to respond to common threats.
DCWF Work Roles
The knowledge and skills developed in this chapter align with the following Department of Defense Cyber Workforce Framework (DCWF) work roles:
- 451 – System Security Analyst
- 531 – Systems Administrator
- 541 – Cyber Defense Analyst
NICE Work Roles
This chapter supports competencies associated with the following NICE Workforce Framework for Cybersecurity work roles:
- Systems Administrator
- Cyber Defense Infrastructure Support Specialist
- Cyber Defense Analyst
Professional Skills
By completing this chapter, you will begin developing the ability to:
- Deploy and configure Fail2ban to protect enterprise Linux services.
- Monitor authentication logs for indicators of brute-force attacks.
- Configure automated blocking rules for repeated authentication failures.
- Validate host-based security controls through controlled testing.
- Analyze security logs to identify unauthorized access attempts.
- Troubleshoot Fail2ban configuration and service integration issues.
What You’ll Be Able to Do
After completing this chapter, you should be able to:
- Install and configure Fail2ban on a Linux server.
- Configure jails to protect services such as SSH from brute-force attacks.
- Verify that repeated authentication failures trigger automated blocking actions.
- Examine Fail2ban logs to identify and investigate unauthorized access attempts.
- Modify Fail2ban settings to balance security with legitimate administrative access.
- Explain how automated intrusion prevention enhances enterprise system security by reducing the impact of password-guessing attacks.
LEARNING OBJECTIVES
- Configure a three-machine GNS3 environment for defensive testing
- Install and configure Fail2ban on a Linux server
- Simulate an SSH brute-force attack and observe Fail2ban ban the attacker in real time
- Demonstrate that legitimate client traffic is unaffected while an attacker is banned
- Inspect and interpret Fail2ban jail status output and log entries
PREREQUISITES
DELIVERABLES
- 5 Screenshots:
- GNS3 topology showing all three VMs running and connected
- Ubuntu Desktop successfully logged in to the Ubuntu Server via SSH
- Fail2ban jail status showing Kali’s IP banned, with the Desktop IP absent from the ban list
- Relevant entries from /var/log/fail2ban.log showing the ban event
- Hydra output showing the transition from login failures to connection refused errors
RESOURCES
CONTRIBUTORS AND TESTERS
- TBD
Phase I – Building the GNS3 topology for this lab
NOTE: Working on this lab should mean you have familiarity with what things should look like. Screenshots are less frequent than in other labs.
This section will walk you through making our defensive test environment. For this example, we will be using the following machines.
| Machine | IP Address | Role |
|---|---|---|
| Ubuntu Server | 10.0.0.1 | SSH Target /fail2ban host |
| Ubuntu Desktop | 10.0.0.2 | Legitimate client |
| Kali Linux | 10.0.0.3 | Attacker |

- Open Oracle VirtualBox
- Take a snapshot or clone of each VM used in this lab
- Kali Linux
- Ubuntu Desktop
- Ubuntu Server
- Ubuntu Server Configuration
- Ensure the VM network adapter is attached to NAT or NAT Network
- Start the VM
- Change the hostname
- Check current hostname
hostnamectl
- Change the hostname
sudo hostnamectl set-hostname server
- Check current hostname
- Verify SSH
- Ensure SSH is present and running
sudo systemctl status ssh
- If it is not running, try
sudo systemctl enable --now ssh
- Check the status again
- Ensure SSH is present and running
- Install fail2ban
- Update the system
sudo apt update
- Upgrade the system
sudo apt upgrade
- Install fail2ban
sudo apt install fail2ban -y
- Verify installation
fail2ban-client -version | less
use q to quit
- Enable fail2ban to start at boot
sudo systemctl enable fail2ban
- Update the system
- Shutdown the VM
- In VirtualBox, change the network adapter to Generic Driver
- Start the VM
- Verify that fail2ban is running
sudo systemctl status fail2ban

Figure 2 – fail2ban installed and running on the server - Close the VM
- Open GNS3 to build the environment
- Import the three devices into the GNS3 environment
- Drag a GNS3 generic, unmanaged switch to the workspace
- Drag the three devices to the workspace
- Connect cables from the switch to each device
- Start all devices
- It should look similar to Figure 1
- Navigate to the Ubuntu Desktop VM and open a terminal
- Stop the network manager so your IP address will stick
sudo systemctl stop NetworkManager
- Look up the NIC name
ip -a
- It should be enp0s3, eth0, or something similar, but it may vary. Adjust your commands as needed
- Change the IP address by typing
sudo ip addr add 10.0.0.2/24 dev enp0s3
- Check that the IP address was set by typing
ip -a
NOTE: Some testers had to change the IP address twice before it took effect.
- Stop the network manager so your IP address will stick
- Repeat this process for the other two devices
- Kali Linux should be 10.0.0.3/24
- Ubuntu server should be 10.0.0.1/24
- Ping between the devices to ensure connectivity. An example command is:
ping -c 4 10.0.0.1
- Navigate to the server
- Password authentication must be enabled for this lab. Fail2ban detects failed logins by reading /var/log/auth.log. If password auth is disabled, Hydra’s attempts are rejected at the protocol level before any log entry is written, and Fail2ban will never trigger!!!
- View/edit the SSH configuration file
sudo vi /etc/ssh/sshd_config
- Ensure that the PasswordAuthentication line is not commented out with a # and is set to yes. It should look like the following:

Figure 3 – SSH configuration file - Save and close the file
:wq
- Verify SSH is up and running
sudo systemctl status ssh
- Reset if necessary,
sudo systemctl enable --now ssh
- Navigate to the desktop and try to SSH into the server
- Open a terminal
- Type the command to SSH (use your appropriate username and IP address for the server)
ssh student@10.0.0.1
- We haven’t uploaded a key, so we will get a warning. Just type ‘yes’ at the warning
- You should see your prompt change from “Student@desktop” to “Student@ubuntuserver,” showing that you are now logged into the server from the desktop

Figure 4 – prompt changed
Phase II – Configuring fail2ban
In this phase, you will configure Fail2ban on the Ubuntu Server and create a custom jail to protect the SSH service. Before making changes, let’s see how Fail2ban separates its configuration:
| Command/Parameter | Meaning |
|---|---|
| /etc/fail2ban/jail.conf | The package default. Don’t edit this file directly; it is overwritten on upgrades. |
| /etc/fail2ban/jail.local | Your custom overrides. Does not exist by default. Settings here take precedence over jail.conf. |
| /etc/fail2ban/jail.d/ | Drop-in directory for focused per-service override files. This is what you will use in this lab. |
- Navigate to the server and configure the SSH Jail. The three most important jail parameters are:
Command/Parameter Meaning maxretry Number of failures allowed before a ban is issued findtime The rolling time window (in seconds) in which failures are counted bantime How long (n seconds) the ban lasts; -1 means permanent - Create a new override file by typing
sudo vi /etc/fail2ban/jail.d/sshd-custom.conf
- This will open a blank configuration file. Type the following EXACTLY

Figure 5 – setting jail settings Command/Parameter Meaning [sshd] The settings below belong to the SSH daemon jail enabled = true Activates this jail. Without this, the jail is defined but never enforced. port = ssh Applies the ban to TCP port 22, the SSH default. filter = sshd Points to the regex filter at /etc/fail2ban/filter.d/sshd.conf that parses log lines. logpath = /var/log/auth.log The log file Fail2ban monitors for authentication failures. maxretry = 5 An IP is banned after 5 failed attempts within the findtime window. findtime = 300 The rolling 5-minute window in which failures are counted. bantime = 600 The offending IP is banned for 10 minutes (600 seconds). action = iptablesmultiport[name=sshd, port=”ssh”, protocol=tcp] The action taken when an IP needs to be banned. NOTE: These values are intentionally low so the ban triggers quickly during the lab. In production, you would increase findtime and bantime significantly, and consider setting bantime = -1 to more permanently ban repeat offenders.
- Save and close the configuration file by pressing escape, followed by
:wq
- Add your custom overrides to allow fail2ban to use systemd for its backend for faster recognition of changes in the auth.log file
sudo vi /etc/fail2ban/jail.local

Figure 6 – setting the custom overrides - est the configuration; if there is no output, there are no errors. Otherwise, read the error message and resolve the issue before moving on
sudo fail2ban-client -t
NOTE: If you are having a difficult time resolving the issue, you can view the exact error in the logs
sudo journalctl -u fail2ban -f
NOTE: if you are really stuck, you can delete the configuration files and begin again
sudo rm /etc/fail2ban/jail.local
sudo rm /etc/fail2ban/jail.d/sshd-custom.conf
- Restart fail2ban to load the new configuration
sudo systemctl restart fail2ban
- Verify the SSHD jail is active
sudo fail2ban-client status

Figure 7 – verify the sshd jail is active - Inspect the detailed status of the SSHD jail
sudo fail2ban-client status sshd

Figure 8 – verify the detailed status of the sshd jail
- Create a new override file by typing
Phase III – Simulating the Attack
In this phase, you will use rockyou.txt wordlist and Hydra on the Kali machine to simulate an automated SSH brute-force attack. You will simultaneously keep an SSH session open from the Ubuntu Desktop to demonstrate that the legitimate client is completely unaffected by the ban.
- Establish a legitimate SSH session from the Ubuntu Desktop to the Ubuntu Server. This session will remain open throughout the attack to demonstrate that the ban only affects the attacker’s IP. This session will remain open throughout the attack to demonstrate that the ban only affects the attacker’s IP
- Navigate to the Ubuntu Desktop VM
- Establish the SSH connection
ssh student@10.0.0.1
- Enter the password for the server when prompted
- Leave this session open and visible. You will be monitoring this connection

Figure 9 – successful ssh from desktop to server – note the change in prompt
- Prepare the rockyou.txt Wordlist. This is the most widely used real-world password list. It contains about 14 million real-world passwords from 2009 and works very well today. Trust me, your password is not that unique
- Navigate to Kali Linux VM
- Verify the rockyou wordlist is loaded
ls -lh /usr/share/wordlists
- Decompress the wordlist
sudo gunzip /usr/share/wordlist/rockyou.txt.gz
- Confirm the txt file is now present and check its size (~134MB)

Figure 10 – Rockyou unzipped successfully
- Launch the brute force attack
- Staying on the Kali VM, launch Hydra targeting the Ubuntu Server’s SSH service (replace user name and IP address for your server as necessary)
hydra -l student -P /usr/share/wordlists/rockyou.txt ssh://10.0.0.1 -V -w 1
Command/Parameter Meaning hydra A network login cracker -l student Use this login username to make login attempts -P /usr/share/wordlists/rockyou.txt the password list to cycle through (use captial P) ssh://10.0.0.1 Target protocol and IP address -V verbose mode (captial V) : print each attempt to the terminal -w 1 wait one second between attempts - Watch the Hydra output on the Kali VM. After enough failed attempts, Fail2Ban on the server will ban Kali’s IP address (10.0.0.3 in our example)

Figure11 – Kali VM’s IP address has been banned by the target
- Staying on the Kali VM, launch Hydra targeting the Ubuntu Server’s SSH service (replace user name and IP address for your server as necessary)
Phase IV – Verifying the Defense
In this phase, you will use Fail2ban’s built-in status commands and the log files on the Ubuntu Server to confirm that the ban was correctly applied and to understand the full sequence of events.
- Navigate to the Ubuntu Server
- Check the sshd jail status:
sudo fail2ban-client status sshd

Figure12 – Kali VM is on the ban list - If you want to run the test again without waiting for the jail timer, you can run the command on the server:
sudo fail2ban-client set sshd unbanip 10.0.0.3
Career connection
Brute-force attacks against remote services are a constant threat in enterprise environments. Systems administrators and cybersecurity professionals routinely deploy tools such as Fail2ban to monitor authentication logs, identify repeated login failures, and automatically block malicious hosts before they gain unauthorized access. The skills developed in this chapter reflect common enterprise security practices where automated detection and response improve system resilience, reduce administrative workload, and provide an additional layer of defense for Internet-facing services.
end of Lab
Deliverables
Four (4) screenshots are required to complete this exercise:
- GNS3 Topology – all 3 VMs running and labeled (similar to figure 1)
- Legitimate Client SSH session (similar to Figure 9)
- Jail Status with ban (similar to Figure 12)
- Hydra Output (similar to Figure 11
Homework 1 – Proactive log analysis and whitelist management
OVERVIEW
In an enterprise environment, misconfigured administrative scripts or forgetful system administrators can easily lock themselves out of critical infrastructure. Conversely, sophisticated attackers will rotate IP addresses to bypass standard brute-force detection thresholds. In this assignment, you will modify your existing Fail2Ban deployment to protect against accidental administrative lockouts using IP white listing, and analyze log trends to proactively identify stealthy malicious behavior.
LEARNING OBJECTIVES
-
Implement IP whitelisting (
ignoreip) within a Fail2Ban configuration. -
Manually inspect and parse
/var/log/auth.logand/var/log/fail2ban.logusing standard Linux CLI utilities. -
Differentiate between standard automated noise and stealthy, distributed credential-stuffing attacks.
Phase I – implementing an administrative safeguard
-
Navigate to your Ubuntu Server (10.0.0.1) terminal.
-
Edit your custom configuration file:
sudo vi /etc/fail2ban/jail.d/sshd-custom.conf -
Add the
ignoreipparameter to the global or[sshd]section to permanently whitelist the Ubuntu Desktop (10.0.0.2), ensuring that legitimate administrative tasks are never interrupted. -
Save the file and restart the service:
sudo systemctl restart fail2ban -
Navigate to your Ubuntu Desktop (10.0.0.2) and purposefully attempt to trigger a ban by failing a login 6 times in rapid succession. Document whether the ban succeeds or fails.
Phase II – log parsing and forensics
-
Return to the Ubuntu Server.
-
Use commands like
grep,awk, ortailto isolate actions in the auth log. -
Write a command string that isolates only the unique IP addresses that have registered a
Failed passwordattempt within/var/log/auth.log.
DELIVERABLES
-
Screenshot 1: The updated contents of
/etc/fail2ban/jail.d/sshd-custom.confshowing the activeignoreipsyntax. -
Screenshot 2: The terminal output from the Ubuntu Desktop showing failed SSH connection attempts that did not result in an iptables DROP rule (proving the whitelist functions).
-
Command Text: Provide the exact
grep/awkcommand string you used to parse out the unique failed IP addresses from your logs.
Homework 2: Advanced defense: custom port knocking and ban escalation
OVERVIEW
Security through obscurity is not a standalone defense, but when combined with dynamic active response systems, it creates layers of friction for an attacker. In this assignment, you will alter the attack surface of your server by migrating the SSH service to a non-standard port, configuring Fail2Ban to monitor this custom interface, and modifying the jail to execute an aggressive, permanent ban policy for repeat offenders.
LEARNING OBJECTIVES
-
Modify network service behavior by changing default service listening ports.
-
Adapt Fail2Ban filters and rules to parse custom ports.
-
Configure permanent ban thresholds (
bantime = -1) to defend against persistent threat actors.
Phase I – altering the attack surface
-
Navigate to the Ubuntu Server and open the SSH configuration file:
sudo vi /etc/ssh/sshd_config -
Locate the line specifying
Port 22. Change this value to a non-standard port (e.g.,Port 2222). -
Apply the changes:
sudo systemctl restart ssh -
Update your
sshd-custom.conffile inside/etc/fail2ban/jail.d/so that theportparameter matches your new custom selection.
Phase II – Hardening the jail policies
-
Real-world threat actors utilize automated tools that pause between attempts to slip under a 5-minute rolling window. You must harden your defenses.
-
Edit your jail overrides so that if an attacker triggers a ban, they are banned permanently rather than for just 10 minutes.
-
Re-run your Hydra attack from the Kali Linux machine, adjusting your target syntax string to specify the new custom port destination.
DELIVERABLES
-
Screenshot 1: The output of
sudo ss -tlnpon the server, showing the SSH daemon listening explicitly on your chosen non-standard port. -
Screenshot 2: The updated configuration file displaying the updated port parameter and the permanent ban duration value.
-
Screenshot 3: The output of
sudo fail2ban-client status sshddemonstrating that the Kali Linux machine has been added to the “Banned IP list” under the new port rules.
