The NixOS life

After a long while of withstanding and being a Debian purist, I finally hopped onto the NixOS bandwagon. Apart from all the work, now I can also learn a thing or two about declarative OSs!

What is it

NixOS, in short, is an OS that forgoes traditional installations and configuration in lieu of a system that is reproducible. In theory, you only need one configuration file to “declare” all the configurations, programs, and look & feel of the OS. This one configuration file, in theory, can set everything up down to the shortcuts on your desktop. That is what I find awesome.

After losing several different OS setups to dumb stuff like tinkering or bad dependencies (or just straight up deleting my GRUB), I decided I would try this new and hip thing. But, being the sensible adult that I am, I first tried it in a supervised environment. Luckily, my “media and scripts and shit” VM just gave up on me, so it was the perfect time to try something new! Let me walk you through the setup:

The migration

My old server was based on a Debian system. This means that if I wanted to salvage it, I would need my VPN configs, my sync scripts, and stuff I really wanted to keep. I got those off the server just fine (it is connected to an NFS share). After that, I set up a proper disk size, RAM and CPU for the install, and started configuring.

Config structure

When I started, I thought I’d keep it all in one folder, i.e. /etc/nixos. When I started setting up the services, I found out that having a 10-thousand line config is bullshit. After the first few modules, I found that referencing config files is much better:

/etc/nixos/configuration.nix - The main config file /etc/nixos/modules/ - The folder with separate modules for necessary services /etc/nixos/files/ - Files used in configuration

The .nix files inside the modules folder can be referenced at will. If you’re good, your folder structure will be beautiful and navigable… if you’re like me, you’ll have “nginx3_final_really.nix” in your folder. I know I do.

VPN configs (Wireguard)

In my old server, I had two VPN tunnels running: One for my parents’ house, and one for my own VPN network. Both of these are almost the same, but it keeps the exit nodes separate.

Apparently, you can declare the VPN config in the configuration.nix file, but I wanted everything set up as quickly as possible, so I used a cheat code:

networking.wg-quick.interfaces.<name>.configFile

This one line allowed me to use the Wireguard configs as is, after putting them in /etc/nixos/files/wireguard/wgX.conf. After that, it’s just about adding one line enabling the VPN connection. Server accessible via VPN? Check.

Jellyfin

The one service that I use the most on my VPN is Jellyfin, a self-hosted media server. I store all my DVDs on there, after having ripped them, and I can access them everywhere. This one was the easiest to set up, basically just the following lines:

services.jellyfin = {
    enable = true;
    openFirewall = true;
    user = "jellyfin";
};

All the remaining configs are stored in Jellyfin’s own DB, so it’s just a matter of setting it up in the WebUI. Since this is no longer a container (as it was when I ran it in Docker), it’s easier to manage and all the functionality is exposed to the system itself. It’s all about setting the folder, and you’re done!

Immich

This was a bit of a pain point, and I couldn’t get it to work at first. After some pain with the setup, I tried doing it the “easy” way: Docker. That messed me up as well, so I returned to the original service setup, and it appear that it actually works:

{ config, pkgs, ... }:

{

  services.immich = {
    enable = true;
    port = 2283;
    host = "127.0.0.1";
    user = "immich";
    group = "immich";
    machine-learning.enable = false;
    mediaLocation = "/mnt/data/immich-nixos/";
    database.enable = true;
    database.createDB = true;
    database.host = "/run/postgresql";
    database.name = "immich";
  };
}

After giving the user “immich” access to the folder that’s referenced in mediaLocation, it suddenly started working and allowed me to set everything up (users, access roles, etc.).

The main beauty of this approach is the fact that Immich is not running in a container. Why is that cool? I don’t have to set up the “old-photos” and migrating them into the current system through the WebUI or adding two locations into my docker config, the entire immich ecosystem is exposed right to the shell.

My wife is the one mainly backing up to Immich (I don’t take photos nearly often enough to warrant this). The migration was as simple as running the immich binary and importing all the old photos recursively to the new setup!

immich -u http://127.0.0.1:2283/ -k [API KEY] upload -r /mnt/data/old-immich/library/upload/

That was basically it. Running deduplication took a bit, but after that, with the right API key, my wifey has her entire photo library back! Is it still duplicated? Yes, but storage space is cheap and I didn’t have to muck about with migrating the existing database. Why do that if I can just burn it down and rebuild?

Syncthing

I use syncthing with my Obsidian and Keepass to sync my passwords and notes. I have yet to find something more “file-agnostic,” that is something that just doesn’t give a shit about the content and just pushes it. It mirrors my phone and laptop. Why install it on a server? I need a backup! What if my phone breaks while I’m out on the road and my laptop gets stolen? Both are encrypted (enough), but I still have around 300 passwords that I need. What if I fuck up again and wipe my laptop? Same solution.

Setting up syncthing was stupid simple, just write in services.syncthing.enable = true;, add the syncthing package to the “packages” line and you’re done. That’s it. Best service by far.

Nginx

Well fuck, if it’s not some service to mess my day up. This time, it was the one I expected least: Nginx.

I knew what I needed. I just want a reverse proxy to forward DNS names to specific ports on the services. This way, jellyfin.server.lan translates to my jellyfin instance without me having to remember a dozen port numbers (I do remember them, but still, some are running localhost-only).

After a lot of troubleshooting, I arrived at a config that actually works. Well, for some services. The upside is that they’re in one file. The downside is… they’re all in one file. If I lose this, I’m fucked. On the other hand, if something messes up, I know which file to blame. I will not post the whole config here, just take this one for Immich:

{ config, ...}:

{
  services.nginx = {
    enable = true;
    recommendedGzipSettings = true;
    recommendedOptimisation = true;
  #  recommendedProxySettings = true;
#    recommendedTlsSettings = true;
    # IMMICH
    virtualHosts."immich.server.lan" = {
      locations = {
         "/" = {
           proxyPass = http://127.0.0.1:2283;
           extraConfig = ''
             client_max_body_size 50000M;
             proxy_http_version 1.1;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "upgrade";
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X-Forwarded-Proto $scheme;
             proxy_set_header X-Forwarded-Protocol $scheme;
             proxy_set_header X-Forwarded-Host $http_host;
             '';
        };
      };
    };

If you put this in your configuration.nix, it should work. Please make sure to end all the curly braces properly! I only copied a snippet, and underneath are several similar services. Take this one and run with it!

One thing I learned setting this shit up: Check what headers your service needs. As you can see, Immich needs a specific max body size, HTTP upgrade, and I won’t even get into the HTTPS flags, because I don’t use them at home. Sue me, but LetsEncrypt currently does not support my .lan domain.

Sync scripts

The last big wedgie NixOS gave me is the fact that crontab is not really supported. They allow you to set up systemd services and timers, which I never did. I am more of a crontab -e kinda guy, but I really want my notifications and syncs to work!

Here is the code, if you want to mirror it:

{ config, pkgs, ... }:

{
  systemd.services.jellyfin-sync= {
    serviceConfig = {
      Type = "oneshot";
      User = "m4iler";
    };
    path = with pkgs; [
      bash
      jq
      curl
      rsync
      openssh
      toybox
    ];
    script = ''
      bash /home/m4iler/.scripts/rsync-check.sh
    '';
  };

  systemd.timers.jellyfin-sync = {
    wantedBy = [ "timers.target" ];
    partOf = [ "jellyfin-sync.service" ];
    timerConfig = {
      OnCalendar = "*:0/1";
      Unit = "jellyfin-sync.service";
    };
  };
}

Basically, it’s a fancy way to write * * * * * /home/m4iler/.scripts/rsync-check.sh. Do I find it dumb? Yes. Do I also think that it’s much better than having to open an interactive shell and find the crontab? Also yes.

Note for my future self: The systemd service is basically running as a separate user, so it may not have access to all the programs that you need. That is why path is declared with all the programs I need inside the rsync-check.sh script.

Backups

As I have already shown above, the /etc/nixos/ folder basically encompasses everything I need. Nevertheless, I have made it a bit of a mess by including external scripts. Sure, I could rewrite those and include them in the config folder under “scripts” or something, but I like to live dangerously.

For backups, I do those manually. Basically when I mess about with my configs, I make it a point to backup to an external drive using rsync. The same is set up on my laptop (also running nixOS) to have all the configs in one folder. The Wireguard configs are sensitive, yes, but I really couldn’t give a shit at this point. I store those in 9001 other places, so I’m pretty sure it won’t be an issue if I lose them.

Do I have a github repo with all this shit? No. I don’t trust myself nearly enough to publish the configs wholesale, so you’ll have to make do with snippets. If anything is unclear, please let me know and I’ll do my best to provide a config that may work for you as well, but I’m a beginner at this. Rage and caffeine made this happen.

Next steps

The next step will probably be… using this system and making it look the way I want. So far, it’s been nice, but the amount of pure, unadulterated autism that goes into its development is astounding. I have opened Pandora’s box, and now I can’t really go back. Soon, everything I own will be declarative, stored in a single file, and I will be hopping from laptop to laptop just for fun.

Use case

If you are currently running on 8 laptops at once and need a way to make every change on one migrate to the others, this is the system for you. If you’re like me, you’ll probably never use the full potential. I do enjoy the fact that a few files make up my system, that’s true. It is a good feeling to know that my system can be backed up with rsync that takes seconds, not hours. Is it a pain sometimes to make things happen? Yes. Is it a tinkering gold mine? Yes.

Now that I’m back in writing and learning, I’ll probably write some more about work; it makes me happy, and I hope my stumbles and facepalms make you happy.

If you’re feeling down, read my blog. You’re not the dumbest one around!