The Essential Guide to Installing and Configuring ModSecurity on Nginx

ModSecurity is one of the most powerful open source web application firewalls available today. In this comprehensive, 4,000+ word guide, we‘ll dive deep on everything you need to know to download, compile, install, configure, customize, and optimize ModSecurity on Nginx servers.

Whether you‘re a seasoned Linux admin or an enthusiastic geek looking to take your Nginx security to the next level, I‘ll share my insider tips to help you master ModSecurity and better protect your web apps. Let‘s get started!

What is ModSecurity and Why Use It?

ModSecurity is an open source, cross-platform web application firewall (WAF) that enables monitoring, logging, and real-time filtering of HTTP traffic. It can block common application layer attacks like cross-site scripting, SQL injection, remote file inclusion, and more.

Originally created in 2002 by Ivan Ristic, ModSecurity has grown to become one of the most widely deployed WAF solutions:

Over 30% of the world‘s busiest sites use ModSecurity, including Bank of America, Microsoft, Amazon, and more.
It comes bundled in popular web servers like Nginx, Apache, IIS, and CPanel.
Major cloud providers like Microsoft Azure, Google Cloud, and Amazon Web Services offer ModSecurity integration.

But what makes ModSecurity unique compared to other firewalls?

Key Benefits and Capabilities

Robust rule engine – ModSecurity operates based on an extensive set of security rules that closely analyze HTTP traffic. Rules can be tuned to minimize false positives and customize protection.
Real-time threat monitoring – Logs provide granular visibility into both legitimate and malicious traffic. Dashboards allow easy monitoring of ongoing attacks.
Flexible deployment – ModSecurity can function as an external proxy, embedded library, or summary module integrated directly into web servers like Nginx.
Highly customizable – Open source nature allows admins to modify rules, write new plugins, and adapt ModSecurity to their environment.
Portal for rules and data – The OWASP ModSecurity Core Rule Set contains over 200 attack detection rules continuously updated by security researchers.
Active community – As one of the oldest and most popular WAFs, questions and support for ModSecurity can easily be found online.

For Nginx administrators in particular, adding ModSecurity takes your security posture to the next level. It‘s a critical open source component alongside SSH keys, firewalls, and other layers of protection.

Prerequisites Before Getting Started

Let‘s briefly recap what‘s required before we can install and configure ModSecurity.

On your Ubuntu, Debian, CentOS, RHEL, or other Linux distribution, you‘ll need:

Nginx – We‘ll be downloading and compiling the latest Nginx source to integrate the ModSecurity module.
GCC & Make – These compiler tools are required for building Nginx and ModSecurity from source.
PCRE – The Perl Compatible Regular Expressions library used by Nginx and ModSecurity for processing rules.
libxml2 – XML parsing library leveraged by ModSecurity.
Curl & libcurl – Used for fetching resources like the latest software releases.
OpenSSL – cryptographic library used by Nginx for SSL functionality.
Zlib – Standard compression library used by Nginx modules.

Install them on Debian/Ubuntu with:

apt install build-essential libpcre3 libpcre3-dev libxml2 libxml2-dev curl zlib1g zlib1g-dev libcurl4-openssl-dev libssl-dev

Or on RHEL/CentOS with:

yum install gcc make pcre-devel libxml2 libxml2-devel curl curl-devel openssl-devel zlib-devel

With the dependencies in place, we‘re ready to continue.

Step 1 – Downloading the Latest Nginx and ModSecurity

Compiling from source allows us to fully customize and optimize our installation, versus using packages which offer less flexibility.

Let‘s grab the latest stable versions of Nginx and ModSecurity:

# Nginx  
wget http://nginx.org/download/nginx-1.19.2.tar.gz

# ModSecurity
wget https://github.com/SpiderLabs/ModSecurity/releases/download/v3.0.5/modsecurity-v3.0.5.tar.gz

Alternatively, browse to their official sites for the newest releases:

I prefer using wget for a quick download directly on my server. Next, extract each archive:

tar xzvf nginx-1.19.2.tar.gz
tar xzvf modsecurity-v3.0.5.tar.gz

This will create nginx-1.19.2 and modsecurity-3.0.5 source directories for the compilation process.

Step 2 – Compiling and Installing Nginx with the ModSecurity Module

Now for the fun part – we‘ll build Nginx from source code with the ModSecurity web application firewall embedded!

First change to the extracted ModSecurity directory:

cd modsecurity-3.0.5

Run the configuration script with:

./configure --enable-standalone-module

This prepares ModSecurity as a standalone module. Compile it with:

make

Once complete, we can shift to the Nginx source and run the configure script there:

cd ../nginx-1.19.2
./configure --add-module=../modsecurity-3.0.5/nginx/modsecurity

This adds the module path for ModSecurity. Now compile Nginx:

make

And finally install the customized Nginx server:

make install

The compiled Nginx executable now has the ModSecurity WAF capabilities baked in!

Step 3 – Configuring ModSecurity in Nginx

Our Nginx + ModSecurity compilation is complete. The final step is configuration to load the ModSecurity module.

Copy the recommended modsecurity.conf file:

cp modsecurity-3.0.5/modsecurity.conf-recommended /path/to/nginx/modsecurity.conf

In Nginx‘s nginx.conf, load the module shared object:
```
load_module /path/to/modsecurity/modsecurity.so; 
```

Enable ModSecurity on your site‘s server block:

modsecurity on;
modsecurity_rules_file /path/to/modsecurity.conf;

Restart Nginx for the changes to take effect.

That‘s it! ModSecurity will now be protecting your web application traffic. You can confirm it loaded properly by checking Nginx‘s logs – you should see startup messages like:

ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"  
ModSecurity: PCRE compiled version="8.44 "; loaded version="8.44 2019-09-27"
ModSecurity: LUA compiled version="Lua 5.1" 
ModSecurity: LIBXML compiled version="2.9.10"

Fine-Tuning ModSecurity for Your Application

The default modsecurity.conf gives you a starting point, but tuning and customization is needed for your specific app. Here are my tips:

Include the OWASP Core Rule Set – This provides over 200 rules defending against common attacks. Grab the latest from GitHub.
Adjust rule thresholds – The defaults may cause false positives. Raise scores and remove disruptive rules.
Set rules to log vs deny – Have most rules log only, and selectively deny the highest risk attacks.
Create exemptions – Exclude trusted IP ranges and user agents like site scrapers to reduce false alarms.
Test extensively – Load test your app and ensure ModSecurity rules don‘t break legitimate functionality.
Review logs regularly – Logs offer visibility into attacks against your application. Monitor dashboards for ongoing issues.

Also consider setting up ModSecurity in detection-only mode during initial testing. This avoids site disruption while you tune the rules before blocking attacks.

Optimizing Performance Impact and Overhead

A downside of ModSecurity is it can negatively impact performance if not properly tuned. Here are some tips to minimize overhead:

Only enable relevant rules – Disable rules unlikely to apply to your application.
Set SecRuleEngine DetectionOnly where possible – Use lighter monitoring before blocking.
Adjust SecRequestBodyLimit and SecResponseBodyLimit – Increase as needed beyond default 131072.
Increase SecRequestBodyInMemoryLimit – Raise from default 131072 if needed.
Compile Nginx and ModSecurity with optimization flags – --with-cc-opt="-O3 -march=native" helps.
Consider ModSecurity on a dedicated server – Removes competition for resources.
Load test extensively – Quantify overhead and iterate on optimizations.

With prudent performance tuning, ModSecurity‘s impact can be negligible for most applications. The benefits of robust protection outweigh the small overhead cost.

Conclusion and Next Steps

Congratulations – at this point you‘ve installed Nginx with the ModSecurity web application firewall module for hardened security!

Make sure to spend time customizing the OWASP rules for your specific application logic. ModSecurity takes some initial tuning, but provides invaluable protection of your apps from attackers.

Here are some recommended next steps:

Set up ModSecurity dashboards and log analysis tools like Splunk. This allows easy monitoring of inbound threats.
For complete coverage, add a WAF like Cloudflare in front of Nginx. Use ModSecurity for application logic awareness, and the external WAF for complete requests.
Check out the ModSecurity Handbook for advanced configuration topics.
Engage with and contribute to the open source ModSecurity community.

I hope you found this guide helpful! Let me know in the comments if you have any other tips for tuning ModSecurity with Nginx. Stay safe out there.