From 43384d242f551b4f971aa1c5aafd6278e4f0d30a Mon Sep 17 00:00:00 2001 From: spleenftw <57947900+Spleenftw@users.noreply.github.com> Date: Tue, 9 Jun 2026 20:27:37 +0200 Subject: [PATCH] Add: WIMH #1 (what's in my homelab) --- my-homelab-setup/content/index.md | 535 ++++++++++++++++++++++++++++++ 1 file changed, 535 insertions(+) create mode 100644 my-homelab-setup/content/index.md diff --git a/my-homelab-setup/content/index.md b/my-homelab-setup/content/index.md new file mode 100644 index 0000000..963e8c6 --- /dev/null +++ b/my-homelab-setup/content/index.md @@ -0,0 +1,535 @@ +--- +title: "WIMH #1 (what's in my homelab)" +date: 2026-05-30 +slug: my-homelab-setup +status: +tags: + - homelab + - network + - linux + - docker + - debian + - blog + - series-wimh +category: homelab +--- + +
So you want a tour, huh ?
+I've been writing about bits and pieces of this lab for a while now. I crashed the honey market, I tried (and partially failed) to evade my ISP, I went rootkit shopping and I played with LVM. But I've never actually sat you down and walked you through the whole thing. What's plugged in, what talks to what, and what's actually running in there.
+So here we are. Grab a coffee, this one's gonna be long. Actually, grab the whole pot.
+A quick disclaimer before we start, the same one I always give myself : this is a homelab. Or, as I like to call it, the not-so-home-but-more-like-prod-lab. None of this is "best practice" and none of this is a reference architecture. A good chunk of it exists purely because it was fun to build at 2 AM. That's the whole point. A production-ready homelab is about having fun trying things, and then doing them the right way because you got annoyed the third time it broke.
+We'll do this in three parts, like a proper open house :
+Let's open the door (which has been removed since a few years).
+
Everything lives in a single 24U rack that, against all my hopes and dreams, is already full. I keep telling myself 24U is plenty. It is not. It is never plenty. If you have a rack, you understand. If you don't, congratulations, you still have money.
+Here's the current occupancy :
++Mon petit sapin de Noël à moi 👀 pic.twitter.com/nL2pMo6nYg
— interlope.xyz (@spleenftw) December 26, 2025
Going piece by piece :
+And on the roadmap, because a homelab is never done even tho my wallet would like to :
+All three nodes run Proxmox VE (because, and I cannot stress this enough, fuck broadcom). They're named after flowers, which is also why the cluster itself is called homegarden. Yes, I'm aware. No, I won't apologize.
+The first one, pivoine (a.k.a. pve01), is the one that started it all : it was my single-node lab before this whole thing metastasized into a cluster even though I stopped at 3 nodes.
+The two other nodes, petunia (pve02) and dalia (pve03), share the exact same build :
+And every node also got a dual 1G NIC bolted on, dedicated entirely to the cluster's nervous system. That's the network we're talking about next.
+Add it all up and you've got a cluster with enough cores and RAM that I can be irresponsible with VM counts and still sleep fine. Which, spoiler, is exactly what I do.
+But I wish I could crank it up to 4x24Gb on petunia and dalia because right now, if pivoine fails, they haven't enough RAM to sustain the load.
+
This is the part I'm proudest of, so lads, buckle the f*** up.
+Back when I first clustered these three, I built what I called a ring network : three direct cables forming a triangle between the nodes, no switch involved, dedicated to Corosync and cluster chatter. The idea is sound. Keep the latency-sensitive cluster traffic off the switch entirely and give Corosync a clean, private path.
+The implementation, however, was held together with hope and static routes. I carved out three /29 subnets, one per link, and manually wrote in /etc/network/interfaces : post-up ip route add ... via ... lines so that, say, pivoine could reach dalia by hairpinning through petunia. I also had to enable IP forwarding on every node to make the middle-man routing work.
It looked roughly like this :
+ pivoine
+ / \
+ (1G) / \ (1G)
+ / \
+ petunia ─────────── dalia
+ (1G)
+It worked, I mean, I wrote it and hoped it would work. +Turns out it had two problems.
+The first was cosmic. For reasons I will take to my grave, the 10.10.1.0/29 subnet simply would not route. I sat there until 3 AM rewriting configs, swapping cables, questioning my career, and then I changed it to 10.10.3.0/29, it worked first try, and I never spoke of it again. Goodbye 10.10.1.x, I loved you (I did not, I hate you, you costed me more sleep that you should have).
The second problem was real : static routing doesn't heal. Pull the petunia <=> dalia cable and the static path between those two is just... gone, until I fix it by hand. On a cluster where Corosync gets very opinionated about losing a link, that's not great.
+So I upgraded the whole thing to OpenFabric.
+OpenFabric (running on FRR, which Proxmox VE 9 now wires up natively through its SDN Fabrics feature) turns the triangle into a proper dynamic-routing fabric. Instead of brittle static routes between interface IPs, each node gets a loopback address in a 10.10.10.0/28 subet (10.10.10.1/.2/.3) and the fabric figures out the paths itself using a link-state protocol. Pull any single cable and it just reconverges over the remaining links. No more 3 AM. No more hand-editing. It heals.
[pivoine]
+root@pivoine:~# vtysh -c "show openfabric neighbor"
+Area mesh:
+ System Id Interface L State Holdtime SNPA
+ petunia enp7s0 2 Up 9 2020.2020.2020
+ dalia enp8s0 2 Up 10 2020.2020.2020
+root@pivoine:~# vtysh -c "show openfabric route" | grep 10.10
+ 10.10.10.1/32 IP internal 0 pivoine(4)
+ 10.10.10.2/32 IP TE 20 petunia enp7s0 petunia(4)
+ 10.10.10.3/32 IP TE 20 dalia enp8s0 dalia(4)
+ 10.10.10.1/32 0 - - -
+ 10.10.10.2/32 20 enp7s0 10.10.10.2 -
+ 10.10.10.3/32 20 enp8s0 10.10.10.3 -
+
+[petunia]
+root@petunia:~# vtysh -c "show openfabric neighbor"
+Area mesh:
+ System Id Interface L State Holdtime SNPA
+ pivoine enp8s0 2 Up 10 2020.2020.2020
+ dalia enp9s0 2 Up 10 2020.2020.2020
+root@petunia:~# vtysh -c "show openfabric route" | grep 10.10
+ 10.10.10.2/32 IP internal 0 petunia(4)
+ 10.10.10.1/32 IP TE 20 pivoine enp8s0 pivoine(4)
+ 10.10.10.3/32 IP TE 20 dalia enp9s0 dalia(4)
+ 10.10.10.1/32 20 enp8s0 10.10.10.1 -
+ 10.10.10.2/32 0 - - -
+ 10.10.10.3/32 20 enp9s0 10.10.10.3 -
+
+[dalia]
+root@dalia:~# vtysh -c "show openfabric neighbor"
+Area mesh:
+ System Id Interface L State Holdtime SNPA
+ pivoine enp8s0 2 Up 10 2020.2020.2020
+ petunia enp9s0 2 Up 10 2020.2020.2020
+root@dalia:~# vtysh -c "show openfabric route" | grep 10.10
+ 10.10.10.3/32 IP internal 0 dalia(4)
+ 10.10.10.1/32 IP TE 20 pivoine enp8s0 pivoine(4)
+ 10.10.10.2/32 IP TE 20 petunia enp9s0 petunia(4)
+ 10.10.10.1/32 20 enp8s0 10.10.10.1 -
+ 10.10.10.2/32 20 enp9s0 10.10.10.2 -
+ 10.10.10.3/32 0 - - -
+And then I layered Corosync on top properly :
+[pivoine]
+root@pivoine:~# corosync-cfgtool -n
+Local node ID 1, transport knet
+nodeid: 2 reachable
+ LINK: 0 udp (10.10.10.1->10.10.10.2) enabled connected mtu: 1397
+ LINK: 1 udp (10.100.2.101->10.100.2.102) enabled connected mtu: 1397
+
+nodeid: 3 reachable
+ LINK: 0 udp (10.10.10.1->10.10.10.3) enabled connected mtu: 1397
+ LINK: 1 udp (10.100.2.101->10.100.2.103) enabled connected mtu: 1397
+
+[petunia]
+root@petunia:~# corosync-cfgtool -n
+Local node ID 2, transport knet
+nodeid: 1 reachable
+ LINK: 0 udp (10.10.10.2->10.10.10.1) enabled connected mtu: 1397
+ LINK: 1 udp (10.100.2.102->10.100.2.101) enabled connected mtu: 1397
+
+nodeid: 3 reachable
+ LINK: 0 udp (10.10.10.2->10.10.10.3) enabled connected mtu: 1397
+ LINK: 1 udp (10.100.2.102->10.100.2.103) enabled connected mtu: 1397
+
+[dalia]
+root@dalia:~# corosync-cfgtool -n
+Local node ID 3, transport knet
+nodeid: 1 reachable
+ LINK: 0 udp (10.10.10.3->10.10.10.1) enabled connected mtu: 1397
+ LINK: 1 udp (10.100.2.103->10.100.2.101) enabled connected mtu: 1397
+
+nodeid: 2 reachable
+ LINK: 0 udp (10.10.10.3->10.10.10.2) enabled connected mtu: 1397
+ LINK: 1 udp (10.100.2.103->10.100.2.102) enabled connected mtu: 1397
+So now I can yank a single mesh cable and watch quorum not even flinch. Same physical triangle as before, completely different confidence level. I can now sleep deeply and dream about the UDM BEAST and its 25G interfaces...
+Everything internal lives in RFC 1918 space (the good old 10.0.0.0/8), carved into one /24 per VLAN. I kinda like to think of myself as a rich guy using a whole /24 for 3 vms... The UDM-SE holds the gateway for each one. I'm not going to hand you the exact map (I've written before about not feeding people free metadata, so it'd be a bit rich to then publish my entire addressing plan), but the structure is the interesting part anyway :
+Here's the full map :
+| VLAN | +What lives here | +Zone | +
|---|---|---|
| 1 | +UniFi gear management | +admin | +
| 2 | +Proxmox / node management | +admin | +
| 3 | +Ceph public | +cluster | +
| 4 | +Ceph private | +cluster | +
| 5 | +VPN | +vpn | +
| 10 | +Admin | +admin | +
| 15 | +Backup | +backup | +
| 20 | +Production | +internal | +
| 25 | +DMZ | +dmz | +
| 30 | +IoT | +internal | +
| 40 | +Lab / clients | +internal | +
| 45 | +House | +internal | +
This is where the UDM-SE does most of its work. Rather than writing a thousand VLAN-to-VLAN rules and slowly losing my mind, everything is grouped into firewall zones, and the rules live between zones :
+And the rules, in plain English :
+The Admin zone is not a blanket "see everything" zone. VLANs 1 and 2 are pure management : UniFi gear talks on 1, servers management on 2, and that's it, no internet access and they don't get to roam, at all. It's specifically VLAN 10 that reaches everywhere. So "god mode" is one carefully-watched VLAN rather than a whole zone. If something on VLAN 10 gets popped, that's my bad day, but it's a much smaller attack surface than letting an entire admin block touch anything.
+The Backup vlan used to live with the production stuff, until I realised that was exactly backwards. Your backups are the one thing that has to survive everything else getting owned, so the last place they should sit is the same broadcast domain as the services most likely to get popped. So it got its own zone, fully isolated and with no internet access at all. Nothing initiates a connection out, and nothing reaches in except the scoped flows that actually need to push or pull backups. If ransomware ever has a field day in the internal zone, it shouldn't even be able to see the box holding the copies that'll save me.
+The Internal zone flows freely between its own VLANs, plus a few surgical exceptions : scoped management hops to specific vpn and cluster IPs (this is how Ansible reaches out to configure things), wazuh clients, nginx hosts in another zones, some prometheus scraping or uptime-kuma icmp and access into the DMZ where it's needed.
+The DMZ is the paranoid one. It's walled off from everything, and it runs its own dedicated NTP and DNS so it never has to reach back into my production resolvers. The whole philosophy of a DMZ is "assume this gets compromised", so I don't want a compromised DMZ box doing DNS lookups against the same Pi-hole my house uses. It gets its own everything (I've got opinions about who gets to see my DNS, so no, the sketchy zone doesn't get to use the good resolver)).
+The VPN and Cluster zones are also locked down tight, but they get one exception : they use the production DNS and NTP. They're trusted infrastructure rather than exposed services, so they get lucky and share the nice resolvers.
+On top of all that, the UDM runs IDS/IPS and geoblocking at the edge. The geoblocking alone cuts an absurd amount of noise. If you've ever pointed a honeypot at the open internet (I have, it's a whole post), you know exactly how much of your inbound traffic is just a handful of countries' worth of bots knocking on every door. Block the regions you have no business talking to and the logs get a lot quieter.
+Storage in this lab has had a journey, and the journey is half the fun.
+It started as plain ZFS on a single node. Then, when I went to three nodes and got greedy for High Availability, I went down the LINSTOR + DRBD rabbit hole. I wrote a whole article about building that out, complete with a keepalived VIP and an HA controller. +And It worked ! It was also a lot : DRBD reactor, promoter configs, a replicated controller database, etc. It was a great thing to learn and a great thing to eventually retire. I spent wayyy too much time and nights to fix those freaking DRDB errors when an nvme failed and I had to reboot in SOS mode to rebuild the cluster until the sun came out and I had to go to work.
+Because I've since moved to Ceph, integrated directly into the Proxmox cluster.
+As for why you may ask ? The reason is just fit. Proxmox's Ceph integration is first-class : it's in the GUI, it's in the cluster's bones, and it gives me the shared, self-healing block storage I wanted without me hand-rolling the replication layer. Each node runs 2 OSDs, so the cluster has 6 OSDs total spread across the three flowers, and the whole thing rides the 10G networking we set up earlier :
+[tree]
+root@pivoine:/home/interlope# ceph osd tree
+ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
+-1 10.91574 root default
+-3 3.63858 host dalia
+ 0 nvme 1.81929 osd.0 up 1.00000 1.00000
+ 1 nvme 1.81929 osd.1 up 1.00000 1.00000
+-5 3.63858 host petunia
+ 2 nvme 1.81929 osd.2 up 1.00000 1.00000
+ 3 nvme 1.81929 osd.3 up 1.00000 1.00000
+-7 3.63858 host pivoine
+ 4 nvme 1.81929 osd.4 up 1.00000 1.00000
+ 5 nvme 1.81929 osd.5 up 1.00000 1.00000
+
+[df]
+root@pivoine:/home/interlope# ceph osd df
+ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
+ 0 nvme 1.81929 1.00000 1.8 TiB 294 GiB 293 GiB 869 KiB 1.2 GiB 1.5 TiB 15.80 0.85 60 up
+ 1 nvme 1.81929 1.00000 1.8 TiB 401 GiB 399 GiB 1.2 MiB 1.5 GiB 1.4 TiB 21.51 1.15 69 up
+ 2 nvme 1.81929 1.00000 1.8 TiB 295 GiB 294 GiB 867 KiB 1.3 GiB 1.5 TiB 15.85 0.85 56 up
+ 3 nvme 1.81929 1.00000 1.8 TiB 402 GiB 401 GiB 1.1 MiB 1.5 GiB 1.4 TiB 21.59 1.16 73 up
+ 4 nvme 1.81929 1.00000 1.8 TiB 347 GiB 345 GiB 1018 KiB 1.6 GiB 1.5 TiB 18.60 1.00 66 up
+ 5 nvme 1.81929 1.00000 1.8 TiB 346 GiB 344 GiB 1014 KiB 1.5 GiB 1.5 TiB 18.56 1.00 63 up
+ TOTAL 11 TiB 2.0 TiB 2.0 TiB 6.0 MiB 8.6 GiB 8.9 TiB 18.65
+MIN/MAX VAR: 0.85/1.16 STDDEV: 2.34
+
+[stat]
+root@pivoine:/home/interlope# ceph osd stat
+6 osds: 6 up (since 6d), 6 in (since 6d); epoch: e4248
+Each node is also a monitor, a manager and a metadata server for a fully capable HA storage service. +I also keep a dedicated Ceph pool for ISOs, so my install media and templates sit on the same resilient, every-node-can-see-it storage as the VMs. No more "oh that ISO only exists on pivoine" nonsense, it's just there, cluster-wide.
+[status]
+root@pivoine:/home/interlope# ceph -s
+ cluster:
+ id: 5819dbca-7480-4070-b1cb-aa44b2adfab1
+ health: HEALTH_OK
+
+ services:
+ mon: 3 daemons, quorum pivoine,dalia,petunia (age 6d)
+ mgr: petunia(active, since 6d), standbys: pivoine, dalia
+ mds: 1/1 daemons up, 2 standby
+ osd: 6 osds: 6 up (since 6d), 6 in (since 6d)
+
+ data:
+ volumes: 1/1 healthy
+ pools: 4 pools, 129 pgs
+ objects: 238.03k objects, 813 GiB
+ usage: 2.0 TiB used, 8.9 TiB / 11 TiB avail
+ pgs: 129 active+clean
+
+ io:
+ client: 773 KiB/s wr, 0 op/s rd, 84 op/s wr
+
+[mon]
+root@pivoine:/home/interlope# ceph mon stat
+e3: 3 mons at {dalia=[v2:10.100.3.103:3300/0,v1:10.100.3.103:6789/0],petunia=[v2:10.100.3.102:3300/0,v1:10.100.3.102:6789/0],pivoine=[v2:10.100.3.101:3300/0,v1:10.100.3.101:6789/0]} removed_ranks: {} disallowed_leaders: {}, election epoch 402, leader 0 pivoine, quorum 0,1,2 pivoine,dalia,petunia
+
+[mgr]
+root@pivoine:/home/interlope# ceph mgr stat
+{
+ "epoch": 166,
+ "available": true,
+ "active_name": "petunia",
+ "num_standby": 2
+}
+
+[fs]
+root@pivoine:/home/interlope# ceph fs status
+ceph-iso - 3 clients
+========
+RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
+ 0 active pve03 Reqs: 0 /s 22 23 16 12
+ POOL TYPE USED AVAIL
+ceph-iso_metadata metadata 1414k 2734G
+ ceph-iso_data data 30.2G 2734G
+STANDBY MDS
+ pve02
+ pve01
+MDS version: ceph version 19.2.3 (d74d168b1c80fb01e1a30d5e4ca9a45b12bc145b) squid (stable)
+Is Ceph heavier than LINSTOR was ? Maybe it was, in some ways. Do I care ? No. The live migrations are clean, there's no IO delay that bothers me in practice, and it's one less bespoke stack for future-me to debug. And I sleep better at night. So I'd say it's a win-win for everyone.
+
Now for the storage that isn't trying to be fast, smart and distributed : the TerraMaster U8-450, running TrueNAS. Where Ceph is the fast, shared, self-healing tier the VMs actually live on, the NAS is the slow, big, "just don't lose my stuff" tier.
+TrueNAS gives me ZFS with a proper GUI on top, and I've fed it 32GB of RAM for the ARC (because ZFS without RAM is just sadness with extra steps). It's split into two pools :
+So the flow is : land fast on SSD, then let TrueNAS age it out onto the HDD pool weekly where I keep more history. My PBS datastores points here, so the whole VM fleet ends up safely off-cluster on storage that doesn't care if a node bursts into flames.
+The one cloud on the horizon : free space is getting fierce. Between the bqckups and the films, those 4TB N300s are filling up faster than I'd like, and I've got my eye on some good old 16TB disks to give the HDD pool some breathing room. It's never if you run out of space in a homelab, only when, and apparently my when is now.
+
Everything so far keeps my data safe inside the rack. Ceph survives a dead node, the NAS survives a dead Ceph, PBS survives a dead VM. But all of it sits in the same room, on the same UPS, behind the same door. So if the house floods or burns down, every single copy goes with it. Not great.
+The rule everyone throws around is 3-2-1 : 3 copies of your data, on 2 different types of media, with 1 of them offsite. I had the 3 and the 2 sorted, but I was missing the offsite one. (actually they weren't really sorted out since it was the same NAS on different pools and disks...)
+That's where PBS 4.0 comes in, since it added S3-compatible object storage as a backup target. Instead of building a second physical PBS at a friend's place and maintaining it forever (and explaining to them why their internet dies every night at 2 AM), I can just push my backups to an S3 bucket in the cloud.
+One gotcha : the S3 backend isn't just a dumb dump. PBS keeps a local cache (a dedicated dataset is best) so it's not hammering the S3 API and charging me for every read. So I get the cheap, infinite, somewhere-else storage without my restores crawling.
+So the count finally adds up :
+And since it's landing in someone else's datacenter, it's encrypted before it leaves the house. If you've read any of my other posts you already know I'm not handing a third party my plaintext anything. They get encrypted blobs, nothing else.
+Is it perfect ? No. Cloud egress costs money and a full restore from S3 would be slow and annoying. But that copy is there for the day the whole rack is gone, not for restoring a VM I fat-fingered on a Tuesday. For everything else, the local NAS copy is right there and fast.
+And then there's the dumbest copy of the lot : an external HDD on my desk that I plug in and sync by hand every now and then. No script, no schedule, I just remember to do it (most of the time I don't). The funny part is that a disk sitting unplugged on a shelf is the one thing ransomware physically can't reach, so my laziest backup somehow ended up being my most ransomware-proof one.
+
It looks like this :
+ [ VMs ] ──► [ PBS ] ──► [ TrueNAS ] (local, fast)
+ │
+ └──► [ S3 bucket ] (offsite, encrypted)
+ └──► [ JBOD ] (on my desk, encrypted)
+Right, the part you actually clicked for.
+At the time of writing I've got about 40 VMs and 1 golden debian13 template (the famous LVM-having Debian template, yes, the one I had to fix after years of regret). I'll walk them zone by zone, because that's how I think about them.
+This is my Tailscale corner, and it's a little more elaborate than "install Tailscale, done".
+First, 2 Tailscale signing nodes. The point of these is independence : by running my own signing nodes, node signing happens on infrastructure I own instead of relying fully on Tailscale's control plane for that trust decision. I use the service, but I hold some of my own keys.
+Then 4 Tailscale instances, each with a deliberately different job :
+Four instances sounds like overkill, but each one is a different trust posture, and keeping them as separate nodes means I never have to stop and ask "wait, which policy is this connection under".
+This is where most of the actual stuff lives. It's a lot, so let me group it.
+The data backbone
+[pgs01]
+root@pgs01:~# sudo -u postgres psql -p 5432 -c "SELECT slot_name, slot_type, active FROM pg_replication_slots;"
+ slot_name | slot_type | active
+-----------+-----------+--------
+ replica_1 | physical | t
+(1 row)
+
+root@pgs01:~# sudo -u postgres psql -p 5432 -c "SELECT usename, client_addr, state, sent_lsn, replay_lsn FROM pg_stat_replication;"
+ usename | client_addr | state | sent_lsn | replay_lsn
+-----------+--------------+---------+------------+------------
+ replicata | pgs02 | catchup | 0/F3060000 | 0/F3AD34F8
+(1 row)
+
+[pgs02]
+root@pgs02:~# sudo -u postgres psql -p 5432 -c "SELECT pg_is_in_recovery();"
+ pg_is_in_recovery
+-------------------
+ t
+(1 row)
+
+root@pgs02:~# sudo -u postgres psql -p 5432 -c "SELECT status, sender_host, slot_name, written_lsn, flushed_lsn, latest_end_lsn, last_msg_receipt_time FROM pg_stat_wal_receiver;"
+ status | sender_host | slot_name | written_lsn | flushed_lsn | latest_end_lsn | last_msg_receipt_time
+-----------+--------------+-----------+-------------+-------------+----------------+-------------------------------
+ streaming | pgs01 | replica_1 | | 0/F3AD34F8 | 0/F6EC9EF8 | 2026-05-30 03:49:11.642677+02
+(1 row)
+Identity & access
+Reverse proxy & docs
+host:port services into clean something.hlb1.lan URLs with Let's Encrypt certs.docker-compose.yml in the lab, each on its own branch. My infrastructure's source of truth.Automation
+Semaphore VM : installed from binaries, pgs-backed. The web UI and scheduler in front of all those Ansible playbooks from Gitea. This is how config actually gets pushed out across the fleet (and why the firewall has those scoped management exceptions into other zones). I have a few dedicated categories for my homelab such as :
+apt update playbook every day at 7am.[apt]
+ansible@bastion:~/semaphore/apt$ ls
+apt_add_package.yml apt_full_upgrade.yml apt_update.yml conf set_no_apt_cache.yml
+apt_del_package.yml apt_update_sources.yml apt_upgrade.yml set_apt_cache.yml
+[security]
+ansible@bastion:~/semaphore/security$ ls
+cis_harden.yml cis_vars.yml copyfail-and-dirtyfrag.yml requirements.yml
+[utilities]
+ansible@bastion:~/semaphore/utilities$ ls
+deploy_k8s.yml disk_space.yml linstor_pve_cluster.yml nuke_cluster.yml rsyslog.yml update_initramfs.yml
+disable_ipv6.yml enable_brr.yml linux_version.yml resolv.conf update_dns.yml update_portainer_agent.yml
+[ntp]
+ansible@bastion:~/semaphore/ntp$ ls
+check_ntpsec_status.yml ntp.conf ntp-migration.yml ntpsec_update_conf.yml
+[snmpd]
+ansible@bastion:~/semaphore/snpmd$ ls
+observium_deploy.yml snmpd.conf
+[ssh]
+ansible@bastion:~/semaphore/ssh$ ls
+check_authorized_keys.yml conf remove_ssh_key.yml ssh.yml
+Monitoring & security - the graphs we all secretly love
+
+
+
+The "useful to actual humans" services
+Lab-of-the-lab
+Time
+The sealed-off zone, with its own self-sufficient basics :
+So here's the whole organism, zoomed all the way out :
+Three flowers, one garden, a 10G spine for storage, a self-healing 1G fabric for the cluster's heartbeat, a zone-based firewall keeping everything in its lane, and a small city of VMs doing everything from serving my photos to catching bots to running my house.
+
If you've made it this far : thanks, and also, get a hobby (he says, having just written a small novel about his own).
+None of this happened in a weekend. It's the result of a lot of late nights, a few cursed subnets, one entire storage stack I built and then retired, and a rack that's been full since roughly the day I bought it. Bits of it are over-engineered. Bits of it are held together with post-up and prayer. The bastion is on the wrong VLAN. And every one of those rough edges is something I learned by breaking first.
That, to me, is what a production-ready homelab really is. Not a perfect, finished, reference-grade datacenter, but a living thing you keep poking at because poking at it is fun. The day it's "done" is the day it gets boring, and then I'll just buy another switch I don't have room for and start again.
+Thanks for reading me,
+spleenftw
\ No newline at end of file