Why Docker exposes my private services to the world?
If you are running Docker services relying only on a firewall like UFW or firewalld, you could have a high chance to expose your container services to the world.
Some days ago, I received a notification from one of my container services which runs on a cloud VPS instance. Briefly, this VPS is configured to host multiple applications relying on docker and docker compose. Those applications are not intended to be published directly to the internet, but they need to be accessed through NGINX configured as a proxy, which forwards incoming requests to the proper application. Unfortunately, this VPS instance isn’t provided by an external firewall service, so I’ve configured iptables through UFW, which is a frontend for iptables that simplifies firewall management. The configuration is pretty easy, all incoming connections are dropped as default policy and incoming traffic is only allowed from from 80 and 443 ports (and even 22, in order to manage the VPS through ssh).
I was quite sure that this machine was properly configured, unless a received a notification about a bug in a web application developed within a docker container. So I first decided to investigate by inspecting my NGINX proxy logs, without finding any requests made at the same time when the error occurred. Then I’ve inspected the docker logs and I found an incoming request made by a crawler or someone attempting to hack my service. So I understood that my container services could be exposed directly to the public, without the protection of my NGINX proxy. I did a nmap scan on my VPS and I found that all the docker containers which expose a port were listening on the public network, regardless my UFW configurations. Even declaring a custom UFW deny role on the container port does not resolve the problem: the UFW configuration is totally ignored by Docker.
Understanding iptables chains
I’m not the only one who faces this issue: Jeff Jerling (the author of many ansible-roles and other great stuff) has described the same issue in his blog. There are also issues opened by the community on github (see here and here, for example) which have the same problems in commons: it’s difficult to configure UFW in order to isolate docker containers from the public network. The reason is the way Docker manipulates iptables, which is the software used to configure the packet filtering at Kernel level (ie. the firewall).
Briefly, iptables is configured in rules and chains. A network packet can fall in a certain rule if had certains features (for example the physical interface where is received, the incoming/outcoming address including ports or the internet protocol used) and within this rule the packet could be dropped, it could reach the destination address or it could be directed to another rule. Rules could be grouped in chains relying on their scope, so you could group all your rules related to the INPUT traffic, for example. The order of rules in a chain matters: if you drop all your packets after they arrive at your machine, you wouldn’t be able to reach your services (including SSH) and other iptables rules will never be applied. However iptables chains can be called in any order, for example the DOCKER-USER chain (we will discuss about it later) is applied before any others docker chains, even if it’s declared after any other docker chains. Managing iptables effectively is an hard task and for such reason debian-based distributions offer UFW (Uncomplicated FireWall) while fedora-based distribution use firewalld, with the intention of simplifying network management by providing simple command lines to configure hard stuff.
In order to grant container isolation and manage subnetworks and bridges, docker related rules are applied on the top of iptables chain, and packets which fall in the docker rules exit immediately after them are applied. In such way all the rules applied after docker rules will not be evaluated, and more precisely all the rules we can define with UFW will be ignored. Those docker rules are required by docker in order to provide network isolation between containers, as described by the Docker documentation. Here are such rules for a fresh docker installation without any container running:
Chain DOCKER (1 references)
num target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num target prot opt source destination
1 DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
2 RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
num target prot opt source destination
1 DROP all -- anywhere anywhere
2 RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
num target prot opt source destination
1 RETURN all -- anywhere anywhereWhen you start a new container, Docker will manage iptables properly in order to provide network access from/to containers and localhost/any other address. You could use Docker network features to provide isolation between applications without worrying about forwarding packets through containers: Docker will add the proper iptable rule in the correct chain. For example, take a look at Docker chains after starting a simple httpd process with docker run -d --name httpd -p 8080:80 httpd:alpine:
Chain DOCKER (1 references)
num target prot opt source destination
1 ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:http
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num target prot opt source destination
1 DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
2 RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
num target prot opt source destination
1 DROP all -- anywhere anywhere
2 RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
num target prot opt source destination
1 RETURN all -- anywhere anywhereBy default docker doesn’t publish a service on the outside world, you need to add the --publish (or -p) option to make a port available to services outside of a Docker container, like in the example before. In the previous case docker adds a rule in the DOCKER chain in which our container ip address (172.17.0.2) is listening for all incoming traffic, even from the public interface. This require understanding how iptables works with docker in order to prevent unauthorized access to containers or other services running on your host. The following tips wouldn’t be necessary if you don’t plan to expose your container service to the public (for example a container database which need to be connected only to another docker application). However if you plan to publish a service, even behind a proxy, you have to understand better the picture (or better, buy an external firewall service which can be easier to configure).
Iptables and DOCKER-USER chain
You shouldn’t attempt to modify Docker chains manually or otherwise you wouldn’t be able to work correctly with container networks. However docker provide the DOCKER-USER chain that is intended to be customized in order to prevent unauthorized access. The most restrictive rule you can apply to this chain is to drop all packets coming from the external interface, as described by docker documentation and Jeff Jerling:
$ iptables -I DOCKER-USER ! -s localhost -i eth0 -j DROPthe -I option before DOCKER-USER chain means inserting a rule at the top of the chain, the ! -s localhost means all packets which the source isn’t localhost (the !before inverts the statement). The -i eth0is my public interface (you need to change this according your needs) and the last -j DROP option simply drops all my packets, so connection is interrupted after this rule. Since this rule applies only on DOCKER-USER chain, which is used only by docker containers, only the lasts are affected, while proxies or other services will continue to work as before. This solution could be enough for you if you plan to expose your services behind a proxy and to provide access to them only on localhost. However, despite this solution is very simple, it can’t be applied in all situations: for example, preventing traffic from outside means also that you couldn’t build a container from a docker file by executing updates or by installing the required software packages: if you require a dependency that is outside a docker container you will require to accept traffic from outside. A solution to this problem could be enabling routing to DOCKER-USER chain:
$ iptables -I DOCKER-USER -i eth0 -o docker0 -j ACCEPTWith this commands you will enable traffic forwarding from outside to docker interface only, which is used by docker as default network. This is sufficient to let docker build containers by installing external dependencies, but has the side effect to restore the public access to docker containers which use the default docker network. This could work if you plan to build your dependencies with docker build or docker-compose build, and then run your services on custom networks (see docker and docker-compose documentation about custom networks). However, it is possible to build containers and granting them connections without exposing ports to the public including the default docker network?
UFW and docker
It’s important to understand that all the modifications we made until now with iptables on DOCKER-USER chain are transient: we will lost them after a reboot unless we save them in a file which can be used to restore our configuration (see this tutorial on iptables and iptables-restore for example). But how to deal with UFW? the scope of such software is to simplify iptables management, we can’t save simply a snapshot of our configurations, since we want to continue managing firewall with UFW and we don’t need to store all the UFW managed chains in our configuration file. The solution could be to modify the UFW /etc/ufw/after.rules configuration file, which is written in a format compatible with iptables and it’s applied as it is after UFW initialization. In such way we can continue to use UFW as we did before but we can apply our custom DOCKER-CHAIN configuration immediately after UFW restore its iptables configuration section.
There’s a project on GitHub which describe all the Docker-UFW related problems we discuss before and provide a script which modifies the UFW /etc/ufw/after.rules configuration file in order to isolate docker containers from public network solving all the problems we raised until now:
By following its documentation, we can modify our DOCKER-USER chain by calling a simple script. There’s only one consideration to do, in our new UFW configuration script we have a section like this:
-A DOCKER-USER -j RETURN -s 10.0.0.0/8
-A DOCKER-USER -j RETURN -s 172.16.0.0/12
-A DOCKER-USER -j RETURN -s 192.168.0.0/16These lines cause a packet to exit from DOCKER-USER chains if the source is one of those subnets (which are the default ip addresses used by docker network interfaces) so the following filters are not applied. The 172.16.0.0/12 subnet is the default network interface used by docker bridge interface, if you want to enable firewalling even in custom docker network you need to declare a bigger network subnet like 172.0.0.0/8: this will be sufficient to protect all your docker container from the outside using any bridge network you can declare with docker.
Conclusion
We discuss about issues related to configuring firewall with UFW and docker. Of course, this is a complicated aspect of docker configuration and errors within this configuration could expose our internal applications to the public. If you plan to expose docker services to the public, you should take in consideration to buy a dedicated external firewall service which could be easily managed using a GUI and prevent you from exposing unwanted docker services to the public.