You are here
TurnKey Linux appliances behind a reverse proxy
A reverse proxy is a server which sits between one or more "real" servers serving the desired content and another network, often the internet. It can be used to provide high availability (e.g. fall over servers if your main server crashes) and/or a way to host multiple servers via a single (usually public) IP, often including SSL/TLS termination. Leveraging a single IP to host multiple server is fairly common when "self hosting" (as many TurnKey users do).
The network setup looks a bit like this:
backend server 1
/
internet/external network <-> reverse proxy <-> backend server 2
\
backend server 3
Context and overview
There are a vast array of purpose built reverse proxies, plus common webservers, such as Apache, Nginx and LigHTTPd can all be configured to act as a reverse proxy. As such, this post doesn't seek to provide any software recommendation, nor exhaustively cover software specific configuration. When it does mention config, it will generally be limited to Apache and/or Nginx - as the most popular webservers. It will touch on some specific config for limited use cases, but will generally take a higher level look at general network design and considerations. The primary purpose of this page is to examine and explain the different options of connection between the reverse proxy and the backend servers. It will primarily focus on the configuration/design of the connection between the reverse proxy and the backend servers.
Historically, the default way to configure a reverse proxy is to do TLS/SSL (i.e. https) termination at the reverse proxy and just use vanilla http (i.e. unencrypted, "plain text" traffic) between the reverse proxy and backend servers. That works well and if the backend network is isolated from any potentially malicious actors or compromised devices, it can also be secure enough.
However, these days that sort of set up is generally considered insecure. That is because using plain http allows anything/anyone with access the network to snoop network traffic and intercept sensitive data such as passwords. Also as there is no mechanism to confirm the identity of a server via http, a MITM attack is trivial.
As such, generally it is recommended to use https communication between your reverse proxy and your backend servers. But even that can be configured in multiple different ways, each providing it's own set of advantages and disadvantages. Regardless, let's have a quick look at the options. Hopefully all the important and common ones are mentioned, but this won't be an exhaustive list.
The options
All options make compromises between security and effort required. I've already provided a general and higher level overview of the options, but let's get a bit more specific and break them down a bit more. Here are the common options that I see, in a vague order of less secure to more so (some of the rankings are debatable). We'll then dive into each a little more noting effort required.
- http - unencrypted, plain http
- https - self signed cert; cert ignored by reverse proxy
- https - self signed cert & CA; cert verified by reverse proxy
- https - "proper" CA signed wildcard cert; cert verified by reverse proxy
- https - "proper" CA signed specific cert; cert verified by reverse proxy
http - unencrypted, plain http
As noted above, this is the historical "default" set up. Once configured, it generally "just works" and continues to do so "forever" without any intervention required. Once upon a time it was also super easy to set up; as most servers were often designed out of the box to host via http.
However, as the traffic is being transmitted unencrypted, any malicious actor/device on the same network as the backend server would easily be able to intercept traffic (e.g .steal passwords, etc, perhaps worse). Also as there is no validation that the backend server, making MITM attacks trivial.
With regard to effort, if you are building all your own servers from scratch, then this probably requires the least effort. Although if you're using TurnKey, it will depend on the specific appliance. Many of our appliances redirect to https, at least for log in (to reduce risk of password snooping). Many of them also require being able to connect via a specific pre-set domain, so will redirect to that FQDN by default. The domain redirection probably won't be an issue and generally your reverse proxy will only allow connection via domain so that shouldn't be an issue. However, obviously redirecting http to https (when only http connection is supported) will cause issues.
In summary, whilst it is not a particularly secure set up, assuming that you trust all people & devices that can connect to the network where the backend servers reside, I think that is completely legitimate configuration.
How to
To make this set up work with a TurnKey backend server (or another server which redirects http to https), you'll need to disable the https redirect. In fact, it is probably reasonable to disable https altogether. In Apache config, look for lines that start with "Rewrite" or "Redirect" in the port 80 virtual host. In Nginx, again look for redirects in the port 80 virtual server config; look for lines starting "return 30x" (where x is number) or 'redirect".
https - self signed cert; cert ignored by reverse proxy
Using this option, your backend servers are accessed by the reverse proxy via https, but they just use self signed certificates (the ones that give scary warnings in your web browser). This configuration model is a security upgrade as it means that the traffic is encrypted, so snooping is blocked. However, as there is no validation that the backend server is what you think it is, MITM attacks are still viable (and as easy as with plain http).
This option will require a bit more work on your reverse proxy config and there is a possibility that some specific reverse proxy software may not support this approach. Having said that, most reverse proxies should support this option, all the main webservers certainly do. If using TurnKey as your backend server(s), this should require zero effort on your backend server and should "just work" - although if there is a redirect to the domain, you'll need to ensure that you don't also do that on the font end - that may cause a recursive redirect loop.
How to
All the config required to make this work needs to be done on the reverse proxy. AFAIK Nginx defaults to this behavior - i.e. it ignores the fact that certs are self signed certs and connects anyway. For more info re Nginx config, check the directive docs.
Apache needs some additional config. I won't go into specifics, but will require using SSLProxy* directives, such as SSLProxyCheckPeerCN and SSLProxyCheckPeerName.
https - self signed cert & CA; cert verified by reverse proxy
This option is a further security upgrade, as beyond the previous option, validation also occurs. Using this option means that both snooping and MITM attacks are not viable. The downside is that it takes a bit of extra effort as you need to generate a CA (certificate authority) key and (re)generate self signed certs, signed by your CA key. The CA (public) key also needs to be stored on the reverse proxy.
The risks are that if the CA private key can be found by a malicious actor, then they can also generate their own certs that will work with your reverse proxy. This can be mitigated by generating and storing the required certs somewhere else (e.g. a third server) and not allowing any access to the private CA key. Another mitigation is to ensure that unlocking the CA key pair requires a passphrase as well. Then an attacker would also need to know the passphrase, as well as having access to the private key. With those mitigations in place, this option is probably as secure as you can get. As you control the CA, certificates can be extremely long lived if you wish (meaning it rarely would need to be updated). Or you could set up the third server so it pushes updated certs to the backend server automatically (although reduces the value of a passphrase).
How to
I know that this option is available for both Apache and Nginx. Unfortunately I don't have the specific directives required handy, but they will almost certainly contain "ssl" and likely "ca" (and probably "proxy) in their name(s). As noted above, as Nginx doesn't authenticate certs by default, you'd want to enable that.
https - "proper" CA signed wildcard cert; cert verified by reverse proxy
A "wildcard" certificate is a special TLS/SSL cert that is valid for multiple subdomains. E.g. a wildcard certificate for *.example.com would be valid for www.example.com, docs.example.com and blog.example.com (etc). Using wildcard certs on your backend servers has a similar security profile to the immediate above and below options, although the vaildation is generic, rather than specific. I.e. whilst the cert will be validated, you can't be sure which backend server you are connecting to. It should still be pretty secure, although if an attacker gains access to one of your backend servers, they could steal the wildcard cert and use it to make some other server/service "valid".
How to
Beyond a mechanism to get and distribute new certificates when they are updated, there should be no additional config required. Although as I noted above, Nginx doesn't authenticate certs by default, so you'll want to enable that.
https - "proper" CA signed specific cert; cert verified by reverse proxy
In modern times, where we have services such as Let's Encrypt, IMO this is the easiest and best option. You get the benefits of a reverse proxy, with the security of a "standalone" server with it's own legitimate CA signed cert. This is a particularly good option if there is any chance that this server might be moved to be directly publicly facing. You could allow direct public access with zero changes and it should "just work" (and still be secure).
As your backend server will have it's own legitimate CA signed cert, the https connections will be valid and should "just work". You also then get the added security of a fully authenticated encrypted connection for "free".
How to
For TurnKey backend server, you can leverage Confconsole's Advanced menu to get a "proper" Let's Encrypt certificate. The default (HTTP-01) validation requires access to port 80 as well, but that shouldn't require too much additional config on your reverse proxy. Domain authority validation can also be done via DNS (DNS-01) which is an alternative option (which also supports DNS-01), although IMO HTTP is easier.