Symfony, reverse proxies and IP protection

Cet article est aussi disponible en 🇫🇷 Français : Symfony, reverse proxies et protection par IP.

Following an issue encountered on one of my projects, I had to dive into how IP protection works in Symfony applications when at least one reverse proxy is in front of them. After some research, trial and error, I thought it was the perfect opportunity to go back to the basics, then explain how to find the origin of the problem and solve it. But first, let’s put it in some context.

Section intitulée contextContext ✍️

The project we write about here is composed of several Symfony applications. Locally, we use a Docker infrastructure which is driven by some Fabric tasks to simplify our DX (this was the premise of our docker-starter – 7 years ago). Our goal was to have a local infrastructure for development looking like the future production infrastructure. Here is what it looks like:

As our applications are based on the old Symfony architecture, we still use the front controllers web/app.php, web/app_dev.php. The app_dev.php is even deployed in production – to simplify remote debugging – but is only accessible from our office IPs.

<?php

use Symfony\Component\ErrorHandler\Debug;
use Symfony\Component\HttpFoundation\Request;

require __DIR__ . '/../app/autoload.php';

// …

$request = Request::createFromGlobals();

if (!\Symfony\Component\HttpFoundation\IpUtils::checkIp($request->getClientIp(), [
    'XXX.XXX.XXX.XXX', // IP JoliCode
    '127.0.0.1', // localhost
    'fe80::1',
    '::1',
])) {
    exit();
}

$kernel = new AppKernel(dev, true);
$response = $kernel->handle($request);
$response->send();
$kernel->terminate($request, $response);

Since last week, our continuous integration has turned red 🔴 and all functional tests failed with the following error:

The request has both a trusted "FORWARDED" header and a trusted "X_FORWARDED_FOR" header, conflicting with each other. You should either configure your proxy to remove one of them, or configure your project to distrust the offending one.

Roughly speaking, with the stack trace, we understand that the $request->getClientIp() method – called in our app_dev.php – raises an error because it detects several conflicting headers. For the moment, this error is still a bit unclear. Let’s start by understanding the logic for how to find client’s IP 🤓.

Section intitulée retrieve-client-s-ipRetrieve client’s IP 🪧

Retrieving the IP of the client that calls our PHP app may seem like a trivial issue. Not at all!

Section intitulée the-problem-with-proxiesThe problem with proxies

To get client’s IP in our application, the PHP documentation informs us that we just have to get the $_SERVER['REMOTE_ADDR'] variable which is created by the web server.

But here, the client of our web server is not the user, it is the varnish proxy. And this is often the case in production: the application server can be located behind one or more proxies (load balancer, HTTP cache, CDN or other).

To overcome this, there is a standard header that reverse proxies can transmit, namely the header FORWARDED, to forward the user’s IP. To simplify, each reverse proxy will therefore check if the request provides a FORWARDED header. If the header is absent, it will create this header with the client’s IP inside. If the header is already present, it just needs to pass it as is. Thus, the first reverse proxy of the chain will create the header, and the following ones will only transmit this header. On the PHP application side, this means reading the $_SERVER['HTTP_FORWARDED'] variable instead of $_SERVER['REMOTE_ADDR'] to get the correct client ip.

Unfortunately, we can’t stop there and have blind trust in this header 🙎.

Section intitulée which-proxy-should-we-trustWhich proxy should we trust?

Indeed, the Forwarded header is a simple HTTP header. That means that any client can pass this header with the value he wants. The risk? That an attacker 🦹‍♀️ pretends to be an IP authorized to access some services, normally secured. In our case, it would be possible to impersonate the IP of our offices and thus access the web/app_dev.php.

To avoid this, most reverse proxies and HTTP applications have introduced a notion of “trusted proxy” (as a set of ip). Before reading the value of the Forwarded header, the service will check if their client has an IP that is part of the list of known proxies 🕵️. If it is the case, then it will be able to take into account the header it received. Otherwise, the header will simply be ignored.

So we are good, now? Spoiler: no ✨.

Section intitulée what-data-should-be-forwardedWhat data should be forwarded?

It is not uncommon for a proxy to call the next service (another reverse proxy or our application) by changing the host or at least the protocol. In our case, HAProxy is responsible for the SSL termination. It receives HTTPS requests and will forward them in HTTP to the next components of our infrastructure.

At the end, our application receives an HTTP request. But it needs to know that the original request was in HTTPS, especially so that it can generate URLs with the right scheme. That’s why the Forwarded header has several pieces of information, not just the client IP. It is structured like this:

Forwarded: by=<identifier>;for=<identifier>;host=<host>;proto=<http|https>

We can note among other things the client identifier, their IP (in the for attribute) as well as the host and protocol initially requested.

Section intitulée forwarded-or-x-forwardedForwarded or X-Forwarded-*?

Since the beginning of this article, I’ve been talking about the Forwarded header – which is the standard – for forwarding information from the original client that would normally be lost. In reality, there are other headers that have become de-facto standards because they were implemented by many software, long before Forwarded was standardized.

So I introduce you to the small family of X-Forwarded-* headers, each of which provides dedicated information:

• X-Forwarded-For contains only the client IP;
• X-Forwarded-Proto is dedicated to the initial protocol;
• X-Forwarded-Host gives the initial requested host.

There are others, to retrieve the port, the forwarded service identifier, etc.

The question that may come naturally is this:

Which header should I read to get the initial information?

And the answer is :

It depends! 🤷

Ideally, your service should only receive either the Forwarded header, or the X-Forwarded-* headers. If it receives both, then you will have to change the configuration of the previous components in the infrastructure, to remove one or the other header.

Section intitulée implementation-in-symfonyImplementation in Symfony 👨‍💻

Now that we know all this, let’s see how to implement – in a Symfony project – an access restriction to our application based on the user’s IP.

Section intitulée ip-based-restrictionIP-based restriction

There are several options to implement this restriction. The first one, more global, is to protect the whole application. This is what we do in our app_dev.php :

if (!\Symfony\Component\HttpFoundation\IpUtils::checkIp($request->getClientIp(), [
	'XXX.XXX.XXX.XXX', // IP JoliCode
	'127.0.0.1', // localhost
	'fe80::1',
	'::1',
])) {
	exit();
}

The second option can be to protect only some URLs. To do this, go the security configuration file:

# config/packages/security.yaml

security:
	# ...
	access_control:
    	-
   		 path: ^/secure-api
   		 role: IS_AUTHENTICATED_ANONYMOUSLY
   		 ips:
       		 - 127.0.0.1 # localhost
       		 - ::1
       		 - XXX.XXX.XXX.XXX # IP JoliCode
    	-
   		 path: ^/secure-api
        	role: ROLE_NO_ACCESS

The second option is the one that should be preferred to follow states of the art.

Now that we have seen how to configure IP protection, let’s see how Symfony looks for the client’s IP.

Section intitulée how-it-works-in-symfonyHow it works in Symfony

If we take a closer look at the Symfony Request class, we notice that it contains all the logic to get the right information in the right headers, as previously explained:

class Request
{
    /**
     * Returns the client IP addresses.
     *
     * In the returned array the most trusted IP address is first, and the
     * least trusted one last. The "real" client IP address is the last one,
     * but this is also the least trusted one. Trusted proxies are stripped.
     *
     * Use this method carefully; you should use getClientIp() instead.
     *
     * @see getClientIp()
     */
    public function getClientIps(): array
    {
        $ip = $this->server->get('REMOTE_ADDR');

        if (!$this->isFromTrustedProxy()) {
        	return [$ip];
        }

        return $this->getTrustedValues(self::HEADER_X_FORWARDED_FOR, $ip) ?: [$ip];
    }
}

If the request does not come from a trusted proxy, the IP of the original client is found in the REMOTE_ADDR variable. Otherwise, we look for it in the forwarded headers.

The same logic exists for the isSecure() method, which determines whether the original request was made using the HTTPs protocol.

It is the getTrustedValues() method that fetches the information (IP, protocol, etc) either from the Forwarded header, or from the equivalent X-Forwarded-* header. And this is the same method that raises the error mentioned at the beginning of this article 💁.

Section intitulée configuring-symfonyConfiguring Symfony

As Symfony does everything for us to find the right IP, the only thing we have to do is to let it know what our trusted proxies are. The method has evolved a bit over the years but the documentation gives us all the necessary information:

# .env.local
TRUSTED_PROXIES=XXX,XXX,XXX,XXX

# config/packages/framework.yaml
framework:
    trusted_proxies: '%env(TRUSTED_PROXIES)%'
    trusted_headers: ['forwarded', 'x-forwarded-for', 'x-forwarded-proto']

By the way, we also configure Symfony to only trust some forwarded headers. This is the role of the trusted_headers configuration.

Note: It is recommended not to trust the X-Forwarded-Host header to avoid being vulnerable to an attack on the HTTP host. In general, we recommend, when possible, to explicitly force the host and protocol to be used by your Symfony application (https in all cases for example) to not depend on the Forwarded header which could be misconfigured by an element of your infrastructure.

Note: before Symfony 5.2, instead of the yaml configuration, you had to manually call the Request::setTrustedProxies($ips, $headers) method in the public/index.php front controller, where $headers was a bit field that allowed to specify any configuration:

• Forwarded but also X-Forwarded-For : Request::HEADER_FORWARDED | Request::HEADER_X_FORWARDED_FOR ;
• Only the Forwarded header: Request::HEADER_FORWARDED;
• All X-Forwarded-* headers except X-Forwaded-Host: Request::HEADER_X_FORWARDED_ALL ^ Request::HEADER_X_FORWARDED_HOST.

Section intitulée reverse-proxies-in-dockerReverse proxies in Docker

If, like us, your project is running locally in a Docker stack, you need to configure your application – and all the proxies in the infrastructure – to trust all the HTTP services that run in this stack.

By default, container IPs are not fixed and will change every time you restart the project. So it is not possible to specify an IP for each as a trusted proxy. After some research, it seems that Docker uses – by default – IPs contained in the range 172.17.xxx.xxx – 172.31.xxx.xxx and 192.168.xxx.xxx. It is possible to change this in Docker’s configuration if necessary, but we will not take this into account for the following, free to adapt what follows 😉.

We will configure Symfony (and all the reverse proxies in the stack) to accept those IPs:

# trust localhost and local docker containers
TRUSTED_PROXIES=127.0.0.1,172.17.0.0/16,172.18.0.0/15,172.20.0.0/14,172.24.0.0/13,192.168.0.0/16

Protip: to convert a range of IPs to CIDR notation, it is possible to use online tools like https://www.ipaddressguide.com/cidr.

Section intitulée back-to-the-problemBack to the problem 🔎

Remember the error mentioned at the beginning of this article?

The request has both a trusted "FORWARDED" header and a trusted "X_FORWARDED_FOR" header, conflicting with each other. You should either configure your proxy to remove one of them, or configure your project to distrust the offending one.

The $request->getTrustedValues() method raises this error because it detects that our application is receiving these 2 headers. So we have to search, in the infrastructure, who is sending these 2 headers.

Section intitulée looking-for-the-culpritLooking for the culprit

To find the culprit, we will use httpbin. This service returns all the properties of the HTTP requests it receives, in a JSON format. So we’ll add it to our docker stack, and insert it between the different proxies that are in front of the container that serves our application.

First of all, we add httpbin in our docker-compose.yml :

services:
    # …

    httpbin:
        image: kennethreitz/httpbin

We start by modifying the Varnish configuration so that it forwards requests to httpbin instead of the container serving our application, then we rebuild the Docker stack. The idea is first to confirm that httpbin is working as expected and that it receives several forwarded headers, as Symfony complains.

To do this, we open the domain name to which our application was responding before. If httpbin is responding, everything is fine.

We can then look at the url http://<host of the local application>/headers?show_env=1 and we get this:

{
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip,deflate,br",
    "Accept-Language": "en-US,en;q=0.5",
    "Cookie": "<redacted>",
    "Date": "Wed, 05 Apr 2023 20:25:26 GMT",
    "Forwarded": "by=redirectionio-proxy/dev-docker-server;for=172.19.0.1;proto=http",
    "Host": "<host de l'application locale>",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0",
    "Via": "1.1 redirectionio-proxy/dev-docker-server",
    "X-Forwarded-By": "redirectionio-proxy/dev-docker-server",
    "X-Forwarded-For": "172.19.0.1, 172.19.0.1, 172.19.0.35",
    "X-Forwarded-Host": "<host de l'application locale>",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https,http",
    "X-Forwarded-Server": "c1bb13649082",
    "X-Real-Ip": "172.19.0.1",
    "X-Varnish": "65547"
  }
}

Protip: remember to add the undocumented ?show_env=1 option in the url, otherwise httpbin doesn’t show all the info, especially the Forwarded, X-Forwarded-For headers. It wasted a lot of time by letting me think, wrongly, that these headers were not transmitted 🤬.

Here, we can see that the 2 contradictory headers are present. Symfony did not lie:

{
  "headers": {
	"Forwarded": "by=redirectionio-proxy/dev-docker-server;for=172.19.0.1;proto=http",
	"X-Forwarded-For": "172.19.0.1, 172.19.0.1, 172.19.0.35"
  }
}

We can now go upstream, by testing each proxy, one after the other. This time we will modify the previous reverse proxy to transfer to httpbin instead of Varnish. In this case, it is the redirection.io agent. We rebuild the Docker stack and we refresh the url. The 2 headers are still there. The problem must exist before Varnish.

We go to the previous proxy. This time, it is HAProxy that we will modify to point to httpbin instead of the agent redirection.io. Rebuild the stack, CTRL + F5 in the browser. This time we have only one header, the X-Forwarded-For.

The culprit is identified ! It is the redirection.io agent which adds the Forwarded header – in addition to passing the X-Forwarded-* headers it receives.

Section intitulée explanation-and-resolutionExplanation and resolution

After talking to our friends at redirection.io, it turns out that the bug was known in version 2.6.0 of the agent. It was indeed a bad handling of the Forwarded header in case the agent receives an X-Forwarded-For.

As we were using the official Docker image in its latest version, the CI was using the freshly released version 2.6.0 while the development stack was using the local image in 2.5.4, without the bug. Impossible to reproduce locally without removing the corresponding Docker image.

While waiting for a fix to be available – it took only a few hours with the release of version 2.6.1 – we simply fixed the version used in the Dockerfile to the 2.5.4 – rather than using the latest one – to confirm the correct operation and find a green CI 🎉.

Section intitulée conclusionConclusion

A bug is often an opportunity to dig into how the code or software involved works. Here, an error raised by Symfony will have allowed to find a malfunction in our Docker infrastructure used in development, to deepen how the retrieval of the IP address of a user works as well as to learn the difference between Forwarded and X-Forwarded-* headers. It was also an opportunity to discover httpbin, a very useful tool to debug an HTTP infrastructure.

It’s also a great opportunity to shape your research and share them as an article. It allows me to give some tips that can help save time if other people encounter a similar problem.

Finally, before ending this article, I wanted to share a piece of information that I came across during my research, which is not directly related to this article, but which I thought was interesting to know. Indeed, one of the problems of the Forwarded header is that it is part of the HTTP protocol. Therefore, this header – and the information it contains – are only available in layer 6/7 of the [OSI] model (https://www.imperva.com/learn/application-security/osi-model/) and are therefore not accessible to lower layer proxies (aka “dumb proxies”). A protocol has therefore been proposed – the PROXY protocol – to transmit the initial client information in a new format, directly at the TCP level (i.e. in layer 4 of the OSI model).