From Docker Swarm through k3s and Nomad to no orchestrator

.

낙엽1124·November 10, 2026

페미위키:운영팀/개발팀/기술 블로그/feed

© Billkaboo, CC BY-SA 4.0

Femiwiki runs only a handful of containers: a web server (Caddy), php-fpm to run PHP, a background job runner, and a cache (memcached). Even so, from 2018 Femiwiki spent seven years on Docker Swarm and then Nomad, two orchestrators (tools that start containers, restart them when they die and roll them over to new versions), trying k3s along the way, and only in 2025 did it start running containers without an orchestrator. This post looks back, in order, at what happened while we used orchestrators on small servers, and at how we got zero-downtime deploys back without one.

Docker Swarm

At first Femiwiki managed even its production server with Vagrant, a tool for building development environments. The Vagrant AWS plugin started a t2.micro instance, and to deploy, a developer uploaded files from their computer with rsync and then had Vagrant rerun the setup script. Since the same server was kept and patched rather than replaced, some files ended up existing only on the server and not in the repository.

The Vagrant approach only had one developer and one computer in mind, so in autumn 2018, to let several people deploy the same way, we packaged MediaWiki as a Docker image and started deploying it with Docker Swarm (the orchestrator built into Docker). We also tried scaling the MediaWiki instances up and down with EC2 Auto Scaling at the time, but soon dropped it, and ran two instances: one for MediaWiki and one for the database and bots. Then in November 2019, to cut costs, we merged the two into a single t3a.micro and put the database and cache in the same Swarm stack as well.

On Swarm, visitors' IP addresses were the problem. Passing through Swarm's internal network made every visitor appear to connect from the internal address 10.255.0.2, so automatic blocks ended up hitting the wrong people, among other problems. Letting the containers use the server's network directly would fix it, but Swarm stack files ignored that setting (network_mode), and when we tried docker compose instead, the containers could no longer find each other by name. In the end we decided to solve this by moving to Kubernetes, and that same day the development team, saying "we should move to Kubernetes, the industry standard", started looking at k3s, a lightweight distribution of Kubernetes. About ten days later came the news that Docker's enterprise business had been sold to Mirantis, which seemed to mean Swarm was being wound down.

k3s

We tested k3s on an instance separate from the production server, and in the end never used it in production. k3s itself installed quickly, but tying it to AWS took a lot of time. We wanted Kubernetes to create and attach EBS volumes when needed, but the cloud controller manager could not do that, so we had to install aws-ebs-csi-driver separately. For certificates, cert-manager fetched them automatically from Let's Encrypt. Starting at midnight on 7 December 2019, getting this far took 78 hours.

So that the cluster could carry on even if the k3s server was rebuilt, in February 2020 we tried putting the directory where k3s stores the cluster state (/var/lib/rancher/k3s) on a separate EBS volume. But when a new instance loaded that state, the cluster fell into a broken state, with some pods showing twice, once terminating and once running. In the end we gave up on keeping the cluster state, and instead of creating EBS volumes on demand we used the local storage that comes with k3s. Service data such as the database and uploaded files stayed under the /srv directory, on a whole EBS volume attached as in the Swarm days.

The problem we never solved was the very reason we wanted to leave Swarm: visitors' IP addresses. On k3s too, requests that passed through traefik kept only an internal cluster address instead of the visitor's, so MediaWiki saw every visitor as the same person. A known fix was to set the service's externalTrafficPolicy to Local, but the development team was busy at the time and never tried it. In August 2020 the team decided to look for another orchestrator instead of k3s, so the production server stayed on Swarm without ever moving to k3s.

Nomad

The next candidate was Nomad. Nomad is an orchestrator from HashiCorp, the maker of Terraform, and it advertises having fewer components than Kubernetes and being able to run programs other than containers. In 2020, though, letting containers find each other meant running Consul, from the same company, alongside it. Some organisations, such as Roblox, use it at large scale, but far fewer use it than Kubernetes, so there is also less to go on when something goes wrong. Why we chose Nomad is not recorded anywhere. In October 2020 we first ran Femiwiki with Nomad on a test server.

The test server (a t3a.small with 2 GiB of memory) was slow and often returned 502 errors. In December 2020 we found an option that sets a normal memory limit and a separate ceiling that can be exceeded briefly (memory_hard_limit), and speed improved a lot, but occasional 502 errors continued. They happened because the PHP container hit its memory limit and was killed, and in January 2021 raising the memory limit of the Envoy proxy that ran alongside it helped (Envoy is the proxy that Consul Connect, Consul's service mesh feature, attaches to each service; the test server used it to connect the containers). Containers started by Nomad could not use swap either, so in March 2021 we even built Nomad ourselves after removing the part of its Docker driver that turns swap off. The line just above it, however, set the combined memory-and-swap limit equal to the memory limit, so removing that part alone most likely still left swap unusable (how this attempt ended is not recorded).

On 31 March 2021 we sent the visitor traffic bound for the production server to the Nomad test server, and this time the CPU was the problem. A t-series instance spends credits it has saved up whenever it uses more CPU than its baseline (20% for a t3a.small), and once the server took traffic, CPU usage rose to 20–40% and the credits kept falling. PHP used about as much CPU as on the Swarm server, so it looked like the extra share used by Nomad and the Envoy proxies pushed it past the baseline. So that same day we removed Consul and let the containers use the host network directly. Visitors' IP addresses now showed up correctly, but since two containers using the same port could not run at once, we still could not do zero-downtime deploys by starting the new version beside the old one (on Swarm, too, the site paused briefly on every deploy). Even after Consul was removed, the credits kept falling for several days (the cause is not recorded), and on 9 April we moved production back to Swarm, which brought the IP address problem back with it. We also wrote about our thinking at the time in "Rethinking our infrastructure" (in Korean), posted on this blog on 1 April 2021.

The plan when we went back to Swarm was to cache pages and reduce the CPU that PHP used. What we built for that was caddy-mwcache, a Caddy cache module Femiwiki wrote itself (introduced in "A MediaWiki cache built on Caddy: caddy-mwcache"). On 12 April we moved production to an Arm-based t4g.small instance, which we had listed as an alternative for the credit problem, running Nomad and caddy-mwcache together, and this time we used the host network without Consul from the start, so visitors' IP addresses showed up correctly. For the first few days things looked smooth. Soon, though, the disk (EBS) ran out of performance credits and PHP and MySQL kept being killed at their memory limits, so the site was intermittently unreachable. On 24 April we switched the disk to gp3 volumes, which have no performance credits, and most of the remaining problems were solved by June. For the three and a half years after that, we ran on a single Nomad server, with the site pausing briefly on every deploy. So we usually deployed at around 6 a.m. Korean time, when there were few visitors. In October 2024 we moved production to a Nomad and Consul cluster of three t4g.small instances; why is not recorded. It seems that using Consul Connect again at this point also made zero-downtime deploys possible. This cluster stayed in use until May 2025, which the next section covers.

Removing the orchestrator

In May 2025 Femiwiki removed Nomad and cut down to a single t4g.medium server. The reason given at the time was "all we need is a few containers (four, for now), and Nomad or any other orchestrator is too complex". At first the containers were started with docker compose. The compose configuration lived in a script that runs once when the instance first starts (user data), so changing a container's settings meant logging into the server or starting a new instance. Since August 2025 we have used Terraform's Docker provider to create the containers directly on the server's Docker daemon. Why we switched is not recorded, but it seems to have been this hassle. Now, when container settings change, we check the planned changes (terraform plan) in a pull request and apply them just like any other infrastructure change (the Docker daemon is not exposed; we reach it through AWS SSM port forwarding).

Zero-downtime deploys

Removing the orchestrator meant we could no longer do zero-downtime deploys. Since September 2026 we do them again, by starting the new container beside the old one on the same server. Each deploy raises a number called the generation by one, and the container names and ports are set from it. For example, the php-fpm container that runs PHP starts as fastcgi-194 after fastcgi-193. When a new image is released, a bot opens a pull request that raises this number. Terraform normally deletes an existing resource before creating its replacement, but we turned on the setting that reverses the order (create_before_destroy), and it deletes the old container only after the new one's health check succeeds.

The php-fpm container alternates between ports 9000 and 9001 depending on whether the generation is even or odd. The containers still use the host network, but because two consecutive generations use different ports, they can run side by side. The web server, Caddy, has both ports registered as backends and calls a health-check PHP file every second, so as soon as the new container is ready it starts sharing the requests. When the old container receives the stop signal, the container's start script first creates an empty file that marks it as shutting down, which makes the health check fail. Caddy then stops sending it requests, and only after waiting five seconds does the script tell php-fpm to stop, letting it finish the requests it is handling. php-fpm keeps accepting new connections while it shuts down and then drops them unhandled, so without this step Caddy returns 502 errors.

Replacing the web server itself is a little trickier. There is only one port 443 for visitors, so the new and old Caddy have to share it for a moment. Linux has the SO_REUSEPORT option, which lets several processes open the same port, and Caddy turns it on when it opens ports on Linux. But sometimes the new Caddy took requests as soon as it opened port 443, before it had loaded its certificate, and failed for about 0.15 seconds. So the container's start script first runs Caddy with a temporary configuration that takes no visitors, and only after it succeeds in fetching the main page over HTTPS on an internal port (8443) does it load the full configuration through Caddy's admin API and open port 443. Once the new Caddy is ready, Terraform stops the old one, which finishes the requests it is handling before it goes down.

Femiwiki's servers had until then been in AWS's Tokyo region, and after these changes, on 26 September 2026, they moved to the Seoul region. On the first deploy in Seoul we sent requests every 0.5 seconds. All 218 requests succeeded, and only one, at the moment the new container took over the port, took 5.2 seconds (we don't yet know why). Gaps shorter than 0.5 seconds may slip past this measurement, though, so we are still checking whether every deploy is free of sub-second gaps.

When an orchestrator is not the answer

Based on Femiwiki's experience, here are some cases where an orchestrator may not be the right answer.

The first is when you have one or two servers and only a few containers. The strength of an orchestrator is spreading containers across several servers, and moving them to another server when one dies. With a single server there is nowhere to move to, so you cannot use that strength, and all that is left is the work of running the orchestrator itself. When Femiwiki removed Nomad in 2025, the reason given was likewise that it was "too complex for a few containers".

The next is when you use small instances built for occasional bursts of CPU, such as AWS's t series. An orchestrator's agent keeps running even when there is no traffic. According to our 2021 records, the Nomad process alone used about 5% CPU, which on a t3a.small with a 20% baseline was a quarter of the headroom on its own. With a service mesh, the proxies use more CPU on top of that.

The last is when all you want from an orchestrator is zero-downtime deploys. As shown above, starting a new container beside the old one and handing over requests with health checks can be done within a single server. For Femiwiki, Terraform's create_before_destroy, ports that alternate with each generation, and the web server's health checks were enough.

On the other hand, if you have several servers, the service has to keep going without anyone stepping in when one of them dies, and you have people who can handle the complexity, an orchestrator is probably the right answer. For a site run without revenue like Femiwiki, though, adding servers is a burden in itself. In 2021, too, the idea of building a cluster from several small servers came up, but we judged it hard to afford.

Closing

Today Femiwiki runs on one app server that runs the containers and one database server. Because the containers use the host network, visitors' IP addresses show up as they are. We will keep checking whether sub-second gaps remain during deploys.

References

Recent posts

• If you use the pecl memcached client in MediaWiki, turn off Nagle's algorithm November 10, 2026
• Introducing terraform-aws-mediawiki-chart-renderer November 8, 2026
• A MediaWiki cache built on Caddy: caddy-mwcache November 7, 2026

All posts (in Korean)