Mitigating Denial of Service Attacks with mod_qos

[elboriyorker]

Denial of service (DoS) attacks, whereby an attacker or group of attackers attempts to make a service on the Internet unavailable, are a growing threat. Chief among the rising DoS threats are those linked to a ransom, where the attackers threaten to attack an organisation’ s systems unless they make a large payment to the attackers.


If your organisation is targeted in such a ransom-based DoS attack, the evidence is clear — do not pay the ransom. There are two major reasons why one should never pay such a ransom — the attackers could continue to demand more ransom payments in the future, knowing that you are a target that will pay. The other major reason you should never pay a ransom to avoid a DoS attack is that there are many measures you can put in place to mitigate or prevent such attacks entirely. We discuss one such mitigation measure in this concise article — blocking, banning, and throttling abusive traffic with mod_qos for Apache.


mod_qos is billed by its author as a quality of service module, meaning it is designed to prioritise requests and traffic to ensure that the most important traffic can get through with good performance and speed.


The documentation (http://mod-qos.sourceforge.net/) lists a number of “control mechanisms” that the module is capable of facilitating:




The maximum number of concurrent requests to a location/resource (URL) or virtual host.
Limitation of the bandwidth such as the maximum allowed number of requests per second to an URL or the maximum/minimum of downloaded kbytes per second.
Limits the number of request events per second (special request conditions).
Limits the number of request events within a defined period of time.
It can also detect very important persons (VIP) which may access the web server without or with fewer restrictions.
Generic request line and header filter to deny unauthorized operations.
Request body data limitation and filtering (requires mod_parp ).
Limits the number of request events for individual clients (IP).
Limitations on the TCP connection level, e.g., the maximum number of allowed connections from a single IP source address or dynamic keep-alive control.
Prefers known IP addresses when server runs out of free TCP connections.





In the example to follow, we will configure a number of the above control mechanisms, particularly those that are most helpful with public web application use cases.


To keep this article brief, we will not cover installation and configuration of Apache itself, or set up of Apache virtual hosts (sites). These instructions assume the server is running Apache HTTP Server on “enterprise Linux” (Red Hat, Oracle, CentOS, AlmaLinux, Rocky, etc.), but they can easily be adapted for other Linux distributions like Ubuntu, Debian, or SuSE.


First, install mod_qos.

In enterprise Linux distributions, mod_qos is in the EPEL repositories. So if that is not already set up on the machine in question, set it up like so:

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

Then install the mod_qos package from EPEL:

yum install mod_qos


Edit the configuration file

Next, edit /etc/httpd/conf.modules.d/10-mod_qos.conf in your favourite text editor, and paste the following:

LoadModule qos_module modules/mod_qos.so

# mod_qos DoS Mitigation

# HTTP response code to send to clients that breach mod_qos DoS mitigation rules
QS_ErrorResponseCode 429

# IP addresses excluded from request limits (office IP address)
QS_ClientEventBlockExcludeIP 9.9.9.9

# Maximum number of clients that mod_qos can track at any one time (not itself a limit on maximum clients allowed).
QS_ClientEntries 200000

# Envrionment variables
SetEnvIf Remote_Host "(.*)" QS_EventRequest=$1
SetEnvIf Remote_Host "(.*)" QS_Event=$1
SetEnvIf Remote_Host "(.*)" QS_Block=$1

# Maximum *concurrent* requests allowed per IP address
QS_ClientEventRequestLimit 150

# Maximum requests allowed to / per second, per IP address
QS_ClientEventPerSecLimit 150

# Maximum requests allowed to / per ip address, per x seconds (number of requests, number of seconds)
QS_ClientEventBlockCount 2000 120

# IP addresses excluded from connection limits (office IP address)
QS_SrvMaxConnExcludeIP 9.9.9.9

# Maximum number of connections allowed per IP, to server across all virtual hosts
QS_SrvMaxConnPerIP 12


You will likely need to add to and adjust this example configuration, but it is a sane place to start. The IP address I’ve set for QS_ClientEventBlockExcludeIP is actually the address for Quad9, a free public DNS service, which we are only using as a place-holder in the example.



Configuration explanation

The number of options for mod_qos is massive, and one can develop quite sophisticated defenses with it alone. The developer really deserves credit and our gratitude for writing such comprehensive and effective software. Given there are so very many options, we are covering an essential set of features here. See the complete documentation (http://mod-qos.sourceforge.net/#configuration) for many more options and details.


LoadModule qos_module modules/mod_qos.so simply loads the mod_qos module when Apache starts. This is all that is part of the default configuration of mod_qos. When you edit 10-mod_qos.conf for the first time, you will likely see this line already present.


QS_ErrorResponseCode is the HTTP status code sent back to offending clients that breach the rules defined by mod_qos. In the example above, I’ve set that to 429 — “too many requests”. You can set this response to anything at all, but 429 is probably the correct response for most use cases. Other 400-level status codes may apply, depending on what sort of abusive traffic your systems are most likely to be blocking. Fun fact: Twitter used to use the unofficial status code of 420 — “enhance your calm”, to tell clients they were making too many requests in a given period of time (they now apparently use 429). You are, of course, welcome to use 420 (or, again, any status code) as well. 🙂 Regardless, I suggest using some 400-level code, because “too many requests” is definitely a client-caused issue, and client-side issues are what 400 codes are for.


QS_ClientEventBlockExcludeIP is where you may put a space delimited list of IP addresses that are excluded from “client event” rules, such as rules that limit how many requests a given host can make. This means that hosts listed in this parameter can make as many requests to the sites as they want, without being throttled, denied, or blocked. We typically put the IP addresses of our clients’ offices in this list, because many users are likely connecting from those IP addresses due to network address translation (NAT) — an entire office with many people in it often has only one IP address.


QS_ClientEntries sets the size of the list of clients that mod_qos may keep track of at any given time. This number can be set as high as you would like. Just bear in mind that the list does consume memory, albeit not a tonne of memory. Each entry in the QS_ClientEntries list consumes 150 bytes on a 64 bit operating system. I often set this parameter to ‘1000000âà ¢â€šÂ¬Ã¢â€žÂ¢ (one million), which consumes up to 150 megabytes of memory. The values stored in QS_ClientEntries survive a graceful restart (systemctl reload httpd), but not a “normalâ� �¢â€šÂ¬Ã‚ restart or stop / start operation.


This next bit, where we set some environment variables, took a bit of thinking to work out. It employs Apache’s own environment variable functions, which are incredibly useful. Let’s break down the first environment variable definition in the example.


SetEnvIf Remote_Host "(.)" QS_EventRequest=$1: Here, ‘SetEnvIfâ� �¢â€šÂ¬Ã¢â€žÂ¢ defines an Apache environment variable, ‘Remote_Hostà¢Ã¢â€šÂ¬Ã¢â€žÂ¢ is the header we are selecting for (and stands for the client IP address), the “(.)ââ ‚¬Â is a regular expression which basically means “match anything”, and QA_EventRequest is the variable name. As you can see, the other two environment variable definitions follow the same syntax.


QS_ClientEventRequestLimit sets the maximum concurrent requests per client.


QS_ClientEventPerSecLimit sets the maximum number of requests allowed per client, per second.


QS_ClientEventBlockCount sets the maximum number of requests allowed per client per x number of seconds. In the example, we set a limit of 2000 requests per client IP address, over 120 seconds (two minutes, of course). We set this limit, as well as the QA_ClientEventRequestLimit, because QA_ClientEventRequestLimit can not be very high in many cases — that’s because we have to allow clients to make somewhat larger numbers of requests in bursts. At the same time, making more than 2000 requests in a two minute time span would be very unlikely to be due to legitimate use in most cases. So we set this as a criterion for a ban.


QS_SrvMaxConnExcludeIP sets which hosts are excluded from connection limit rules.


QS_SrvMaxConnPerIP sets the maximum number of connections allowed per client IP address. Our example puts that at 12, which is more connections than what all modern browsers would open at any single moment in time.


Other rules must be set per virtual host. Most or all of the rules used in our example above must be defined outside of a virtual host directive.



With that, one already has very good defenses against DoS attacks. A lot more can be done to make for more complete protection against such threats, however. Other lines of defense include network firewalls, web application firewalls, web server configuration parameters, and application-level parameters.


Did you like this post on mod_qos? Are you looking for more fabulous articles on amazing things that can be done with Apache? Then check out this article on load balancing with Apache (https://ionica.ca/how-to-set-up-an-apache-load-balancer/)!​