Docker isn't really designed for the cloud.
Ok, inflamitory thesis has been stated. Now, let's talk about docker and kubernetes and orchestration and containers, and VMs and elastic compute.
I am assuming right now that we're talking about agroup operating at scale. What you do to spin up your website or get a wiki to use with your friends is up to you. But as soon as you need to write documenation, or figure out scaling to meet demand, we enter the realm I'm talking about.
Scale
So, the first thing we need to discuss is scaling. There are a few kinds of scaling that happens. The obvious ones are scale and compute. Others exist, but we'll be dealing with these.
People
Every ops team that I have been on was less than 10% the size of the development team. Managing your compute resources is probably reserved for a small number of people, and so making those people effective matters. Therefore, centralizing configuration and minimizing number of tools is key. I suspect you are using an infrastructure management tool (terraform, CloudFormation, etc), a configuration management tool (chef, puppet, ansible), and monitoring stack (something to alert, something to observe, something to collect logs). If you are doing those things, what is docker bringing to the table? You already have your config manager defining your services on your servers, so why put that configuration in a container definition instead? It's another tool in your DevOps stack.
One thing I see a lot is that developers want to be able to ship artifacts, or want to manage containers for local development. If your developers want a local service encapsulated in docker, I get it. Luckily you can make docker containers via config management tools! I prefer ansible so it's the one I know. Since you definitely need something to put docker and your containers on the system, why not just define a server once and let your developers make use of that definition? If your developers are being empowered to do their own containers, give them control of the definitions that builds tests containers/deploys to servers.
Using fewer points of configuration reduces number of cognitive switches, and lets your ops team become experts in that. Fewer tools means your team scales faster.
Compute
Ok, this is the harder sell. Obviously, this really depends on so many questions, and there are setups optimized well for docker, and people running docker in ways that leverage it a lot. But I am going to talk about docker as I see it used.
Firstly, it's not that I want one copy of the userland and dependencies for multiple services. I want multiple computers/VMs/whatnot. Docker feels like it was written to solve problems from the 90s where a group would have one computer which dragged up several services without isolation, and then ran them all dependently. But you shouldn't be building systems like this.
You should be using an infrastructure definition tool (I know terraform best) if you are in the cloud, and you should be making many small servers. Having these small servers lets you spin them up and down as needed on demand to meet scaling demands. This could be autoscaling groups and so on, but I've worked in groups that just turn on servers to meet need (since it's faster than deploying the servers on demand, and demand bursts are short lived if we can meet them quickly). This is similar to how kubernetes would help you scale, but it doesn't incur the costs of keeping large servers up.
This also helps your monitors, since it lets you easily see when individiual peices are being overwhlemed, rather than when the entire cluster is being slowed by load. Being able to say "half our API workers are slow" is very different than saying "This region is experiencing abnormal load". In theory you can do per-process montoring and per-container monitoring, but it doesn't speak to the more aggressively shared resources, like kernel scheduling and RAM channels. If processes have their own compute attached, scheduling now belongs to your hypervisor. Your cloud provider should definitely have a better scheduler on their VMs than you can write, and your load should be spread through a datacenter, so a load spike to you shouldn't kill the entire worker farm. If you separate by compute resources, you can expect your monitoring to reflect the usage of those compute resources.
In the end, to scale and to monitor, you want lots of resources that you can resize on the fly, and monitor effectively. You don't want the scheduling problems that adding multiple services to the same box brings.
Security
Containers are secure! That's a thing I hear a lot. Let's talk threat models, updates, and encapsulation, the three places where I think containers interact with security.
Threat Models
Ok, this is the gimme, everyone has a threat model, whether they acknowledge it or not. But not everyone's threat model matches with reality. when people say that containers are secure, they are saying that they are isolated. If there are four containers running on a box, then each one can't touch the other. If I get a webshell on your API worker, I can only take over your DB if there's a secondary connection between your DB and your API endpoint listener. Which, unfortunately there is).
The two threat models I see peolpe interact with for their web application are data exfiltration & corruption (commonly editting data they shouldn't be able to access), and server control (typically using the server to send spam or mine bitcoin these days). And these would be wonderful if docker did anything for either of these things.
Docker containers are isolated from their hosts, but are definitely able to talk to other parts of your application. Any service the container hosts can access/edit, can be accessed/editted by an attacker. Hopefully, that's obvious, but it also means that putting the service in a docker container doesn't decrease the surface, assuming the host is running nothing else.
Most containers I see also contain their entire userland, therefore anything else (such as a bitcoin miner or spambot) that can access the container can also be bootstrapped into the container. Obviously, small containers will help this, but as long as Alpine Linux is the standard small container, there's enough there to use the server maliciously (and a package manager just in case).
I personally put my monitoring endopoints outside my containers, running directly on the host, and then monitor the container contents. This means that my SIEM is able to see actions inside my containers, while the network accessible daemons are isolated, which is nice. But it also means my SIEM has less access to my container's packages and versions, but that can be worked around. However, I tend to use external monitors and external verification anyway, and having on host monitors is only a nice to have.
Updates
Leading nicely into update management, I like to be able to know my daemon versions and have them matched against CVEs on running workers. I also like to know when my server images are out of date. I love how Quay.io manages this for containers.
But this is yet another point of failure. I need to push new containers, but new servers, and test multiple kinds of updates, just to patch. Typically it means twice as many integration cycles in order to do a single patch update (update containers and update server image) which is bad. And, if I follow 1 service, 1 server, I still can switch versions for different components.
Encapsulation
I touched on this a lot in threat modelling, but I want to differentiate between isolation and encapsulation as I see them. When a contain encapsulates a service, it holds that service and all of its pieces. This is basically a statically linked binary as far as management goes. All the pieces are oneoffs and only useful for that thing. It also will have its own disks and areas to write to the filesystem. This is great, as long as encapsulation is never broken. Passing things out of an encapsulated service exposes risk.
Isolation is when a serive is put into a sandbox, something it can break or interact with freely, while being managed by above services. This is the tradition of a Solaris Zone or chroot jail. The process is executed by the host, but given a small area to open files or sockets. Again, breaking this isolation will cause problems, but the dependencies can be managed safely, as can configuration. In fact, in a docker container, all the process configuration files must be in the isolated zone, whereas in a chroot, the configuration might not be inside the isolate zone. Obviously, this isn't perfect (if you put RAM management in the container/chroot it's gonna be a bad time, if you have other services use that data, or if you just network connect).
Basically, docker is giving you some encapsulation and security, but it's not giving you much, because of how many binaries the standard docker image contains. It's better than nothing, but a lot worse than running one service on one VM and then using chroots.
In Summation
Docker solves problems with the archtecture of the 1990s, if you have one giant server to manage everything on. It doesn't solve the problems facing a company operating in the cloud. If you want to scale, manage small servers, ones that will properly enclose a server, and don't create larger servers that are hard to use on demand. Keep your configuration in as few spots as possible, and make sure you don't have multiple places to monitor for the same thing. This helps companies scale, helps your people scale, and makes sure you aren't missing things.
Docker adds complexity. The cloud adds complexity. But the benefits of the cloud obviates the benefits of docker in the environments I've worked with it.
A Note on Container First Services
Do it. If you can get scaling, monitoring, and pricing value out of those things. Use them. But make sure to compare pricing, infinite on demand scaling is expensive, and never having non-ephemeral state is expensive. Personally, I want a lot of my monitor scripts to go into Lambda so we can run it a lot in random regions, and then feed that data to a centralized graphing and alerting box. If I could safely put my services into ephemeral containers and send all traffic there I would, but that gives up caches and other long term state which I need to not kill my DB servers. But, if your load fits that and it's cheaper, do it!
The Development/Testing Argument
Lots of people I've seen claim docker is bad for ops but good for developers because you can have containers that they can run locally for testing strange builds. Firstly, build containers using your config manager as I already mentioned. Once you have the scaffolding, use it twice. Secondly, it's a question of whose time you want to waste, and where you want to put risk. Operations time is definitely going to be spent on prod, but also probably rarer than developer time. Obviously, if you have a 10:1 operations to dev, having ops waste some time on releases and updates for devs to have a smoother and more understandable tests might be worth it. However, placing the risks on prod servers so testing servers are faster to make is a risk only your company can decide on.
