#1235: Nixing the Kube Config
For a few years I had all my shit hosted on a multi-node Kubernetes (K8S) cluster, which I put on Hetzner Cloud. Even wrote a guide about it, although at the time of writing it isn’t back online. None of this is high-stakes enough for me to care about uptime, really, so it just helps that I’ve got something practical to work with when I want to mess around again.
I’ve got a lot of good things to say about K8S and how it’s evolved over the past decade–but not you, YAML, I’m not friends with you. It’s grown into a pretty reliable platform for declaratively rolling your own infra and so many things are just a helm chart or an operator away now.
Still, though, it’s gonna be overkill for many teams and organisations because of the scale it’s primarily designed to work at, and that includes people running it solo like I’ve been up until now. It served its purpose as a learning experience, but that time is now over.
How did I get to this point? It happened the other week, when I poked around the cluster again to do some updates. It had remained largely untouched for a couple of years, running unattended upgrades when needed, but it became quite difficult to fix things that had since become stale. This blog, in particular, relied on a Docker image that was pulled from Docker Hub (and I’d previously forgot to push my own version of it to my private registry), so I couldn’t deploy it any more. I also wanted to try out some CI/CD and GitOps setup (Flux, Tekton), and at that point it felt like I was just creating a burden for myself.
The first time I used Nix (or NixOS) was about ten years ago, when I was enamoured by the idea of running a declarative, immutable setup. I spun up a VM on my Mac and built a Nix-based environment inside it with most of the tools I use, including a nice little setup with XMonad. It was a fun little endeavour but living inside a Virtualbox VM wasn’t great on the battery life at the time.
Skip forward to now and it has my attention again, especially after learning about devenv.sh, which is a Nix-based tool that helps you set up a completely self-contained and reproducible dev environment. And now, a small segue into devenv…
devving up the env
Here is the setup I use for this very website:
{ pkgs, lib, config, inputs, ... }:
{
packages = with pkgs; [
git
emacs-nox
rsync
];
processes.dev.exec = "${lib.getExe pkgs.watchexec} -n -i out -- build";
services = {
nginx = {
enable = true;
httpConfig = ''
server {
listen 8080;
server_name _;
root ${config.git.root}/out;
}
'';
};
};
scripts.build.exec = ''
emacs -x build.el
'';
scripts.deploy.exec = ''
rsync -avz --delete out/ lee@lmchn.xyz:/var/www/www.leemeichin.com/
'';
tasks = {
"blog:build".exec = "build";
"blog:deploy" = {
exec = "deploy";
before = [ "blog:build" ];
};
};
}That’s all it takes for me to spin up a dev environment that runs an http server and rebuilds the site everytime I change something in it, simply by running devenv up, and deployment is just a case of copying the build over to my server with devenv tasks run blog:deploy.
The appeal of this, to me, is removing the dependency on docker and docker compose, because I can simply use native packages and services instead without having to install them globally. Combined with direnv, everything gets set up as soon as I enter the directory, and torn down once I leave. Nice.
fleetwood linux… stevie nix…
It’s best to start small when trying new things, so I did’t bother figuring out how to wire this up with my NixOS host server, which of course is also fully managed by Nix.
There’s a bit of a ritual with spinning up a fresh server. At the bare minimum you’ll want to configure SSH properly, disable password logins, disable root login, and create a separate privileged user for yourself that can use sudo. After that, if you take your system administration more seriously, you’ll think about how to deploy your software with the correct system user accounts and permissions.
Nothing that can’t be done with a collection of shell scripts, really, yet still this can be enough of a faff that you reach out to Ansible, Chef, Puppet, …even K8S, to get things up and running.
Nix itself is another solution to this problem, and you can use Nix Flakes1 to help neatly package up the config for a number of hosts.
A flake is sort of a way of packaging up a nix configuration (aka derivation), and looks a bit like this, which is an abridged version of my own setup:
{
description = "A flake for setting up a server";
inputs = {
nixpkgs = {
url = "github:NixOS/nixpkgs/nixos-unstable";
};
sops-nix = {
url = "github:Mic92/sops-nix";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs = inputs @ { ... }:
flake-parts.lib.mkFlake { inherit inputs; } {
systems = [ "x86_64-linux" ];
flake = {
nixosConfigurations = {
hub = inputs.nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
inputs.sops-nix.nixosModules.sops
./hosts/lmchn.xyz/configuration.nix
];
};
};
};
};
}
From this, you add any dependencies (aka overlays) you need, should the nixpgs repo not contain the packages you want, and then define an output which is, basically, the rest of your config.
At the bare minumum you’d do as I mentioned above: lock down SSH access and enable the firewall. Your configuration.nix is more or less the entry-point to the entire setup:
{ config, pkgs, ... }:
{
imports = [
./hardware-configuration.nix
./disko.nix
./users
./secrets
./services
./apps
];
services.openssh = {
enable = true;
openFirewall = false;
settings = {
PermitRootLogin = "prohibit-password";
PasswordAuthentication = false;
PubkeyAuthentication = true;
};
};
security.sudo = {
enable = true;
wheelNeedsPassword = false;
};
networking.firewall = {
enable = true;
allowedTCPPorts = [80 443];
};
}nix pkg, which is just like apt or yum or perhaps even brew, will have a derivation for almost everything you can think of, and will provide a pretty sensible way of configuring whatever you choose to install.
What is a server, for example, without HTTP support? Let’s install nginx with automatic SSL certificate creation (in my case using DNS verification through Cloudflare):
{ config, pkgs, lib, ... }:
{
services.nginx = {
enable = true;
recommendedProxySettings = true;
recommendedTlsSettings = true;
recommendedOptimisation = true;
recommendedGzipSettings = true;
};
security.acme = {
acceptTerms = true;
defaults = {
email = "<redacted>";
dnsProvider = "cloudflare";
environmentFile = config.sops.secrets.cloudflare_api_token.path;
dnsPropagationCheck = true;
};
};
}Adding new virtual hosts to this is now quite trivial and I can repeat it for whatever I want to host:
{ config, pkgs, lib, ... }:
let
port = 4001;
webRoot = "/var/www/www.leemeichin.com";
in
{
services.nginx.virtualHosts."leemeichin.com" = {
useACMEHost = "www.leemeichin.com";
forceSSL = true;
globalRedirect = "www.leemeichin.com";
};
services.nginx.virtualHosts."www.leemeichin.com" = {
enableACME = true;
forceSSL = true;
root = "${webRoot}";
locations."/" = {
extraConfig = ''
try_files $uri $uri/ =404;
'';
};
};
security.acme.certs."www.leemeichin.com" = {
domain = "www.leemeichin.com";
extraDomainNames = [ "leemeichin.com" ];
};
systemd.tmpfiles.rules = [
"d ${webRoot} 2755 lee nginx -"
];
}Skipping to the end, when I rebuild my flake it will pull all of this together and, when applied to the host server, put everything in the right place. And this pattern is pretty consistent throughout.
Finally, because it’s an immutable setup, it’s trivial to rollback to a previous snapshot if things go wrong.
nixie chicks
If there is any advantage to this switch that I enjoy more than others, it’s the relative simplicity of the tooling.
The language is not difficult to understand–in fact I’d consider it to be a more intuitive version of Terraform that doesn’t couple itself to a directory layout. It doesn’t have to take over the entire system either, as evidenced by me using rsync to copy things into it.
I think the most complicated part of the switch was the bootstrapping stage, which I’ll write about separately. Not many providers offer NixOS out of the box and you might rely on a tool like nixos-anywhere2, which blows away whatever is currently installed on the host and replaces it with Nix. It’s easy to get yourself locked out of this.
Overall though, the end result is much easier for me to grasp and keep in my head and it took me less than a day to put together. It’s all just basic Linux stuff at the end of the day, so if I want to debug something I can just shell in and poke around in the usual places.
Going forward I’ll consider digging into this more, particularly devenv, because I see a lot of promise in that.