++In WIMH #2, I described the VPS entry into the +lab: public traffic lands outside the house, then crosses the overlay to reach my services. The house keeps +its ports closed. That part stays. +
++What changed is the public entrance. The older setup mixed application hosting with Nginx and Anubis on the +same VPS machines. It worked, but moving the entrance onto a dedicated pair of machines gives me somewhere +else to send visitors when one edge fails. +
++So this is the writeup of that migration: two OVH VPS instances, HAProxy on both, an Anubis instance per +service on each, Cloudflare Load Balancing in front, and Semaphore deploying the configuration from Git. The media workload needed its own treatment, because a video client has very little interest in +solving a JavaScript challenge. +
++Grab a coffee. There are health checks inside health checks. +
++ ++Node names in this article are aliases. Public example addresses come from documentation ranges; private +addresses, internal DNS names and infrastructure identifiers are omitted or replaced. Screenshot labels have +been anonymized, with graph legends redrawn where needed; plotted values are unchanged. Configuration +fragments are shortened examples, not a dump of my access policy. No credentials, real client addresses, service hostnames or live service-to-host inventory are included. Application labels in screenshots are generic. +
+I'll call the public entry machines edge-a and edge-b. They live in different OVH
+locations. The existing application VPS machines become origin-a and origin-b, while
+the subnet router at home becomes home-router. Nginx Proxy Manager is still the HTTPS gateway to
+the local applications.
+
+Each edge runs HAProxy and an Anubis instance per configured application. Each Anubis has its own policy and target. The applications +themselves remain on their existing hosts. +
+
+
+
The following are illustrative routing patterns, not a catalogue of the live applications or their host assignments.
+| Example workload | Origin path | Public entrance |
|---|---|---|
| Browser application | Private application host over the overlay | Cloudflare proxy |
| Application with native API clients | Private HTTPS gateway | Cloudflare proxy with application-specific challenge rules |
| Media workload | Private HTTPS gateway | DNS-only load balancing, direct HTTPS |
+For a service at home, the request travels through HAProxy, its selected Anubis, the Tailscale overlay, the +home subnet router, the firewall, and NPM before reaching the application. Between the router and NPM, the +firewall permits HTTPS to that gateway. The edges don't get a general invitation into the LAN. +
++Also, a change from that older WIMH post: this setup uses Headscale as its control plane. The +clients are still Tailscale clients. The article describing the official Tailscale control plane records an +earlier state of the lab. +
+ +This diagram separates visitor traffic from deployment, control and monitoring. Application groups are illustrative; no real service catalogue, hostnames or private addresses are included. Solid arrows show the main flow within each section, while dashed arrows identify secondary paths, checks or control operations.
+
+The Mermaid source describes this generic architecture. It is maintained documentation, not a production configuration export.
+ ++The work started behind the public entrance. The edges needed a usable route to each origin before they could +proxy anything. Joining Headscale alone doesn't make a loopback-only application reachable, and a working +local curl doesn't prove that another node has permission to reach it. +
++One web application was a good example. Its existing port was bound to loopback for the old local reverse proxy. A second +binding on its overlay address let the new edges reach it while the old path stayed available during +migration. The other applications also needed their actual listening addresses checked. There was no reason to +open their application ports to the public Internet. +
++For the home services, the subnet router advertised the route to NPM. The edge clients accepted private routes +without using an exit node. Their ordinary Internet egress stayed their own, which also matters later for +Cloudflare API access. +
++The enrollment command looked like this, with the control hostname and node name replaced: +
+ + +```bash +sudo tailscale up \ + --login-server=https://control.example.net \ + --hostname=edge-a \ + --advertise-tags=tag:edge \ + --accept-routes +``` + + ++The other edge uses its own hostname. Registration and approval happen through Headscale; an enrollment +command is not a substitute for checking the resulting tags, approved routes and connectivity. +
+
+I kept tag:edge as its own role. The access rules describe the application destinations and ports
+the edges need, plus their private peer gateways. Giving these machines a broad management tag would have made
+the initial tests easier and the trust boundary much worse.
+
+One early timeout between the application VPS machines was consistent with that policy. The machines were +online, but their role didn't grant the origin access intended for the edges. The useful test was from a +correctly tagged edge, not from whichever machine happened to have a shell open. +
+ ++This is the part that took the most explaining during the build. Cloudflare and HAProxy make different +decisions. +
++Cloudflare selects a public edge. Both edges are active, using equal endpoint weights and Random steering +inside one pool. With session affinity, this does not mean every other HTTP request lands on the other VPS. A +browser session can stay on one edge, and a long connection can account for much more traffic than a short +visit. +
++HAProxy then selects a path for the requested service. It prefers the local Anubis. If that local path is +unhealthy, it uses a private gateway on the other edge. +
+ + +```text +Normal: +visitor → Cloudflare → edge-a → local Anubis → origin + +Local Anubis unavailable: +visitor → Cloudflare → edge-a → private gateway on edge-b + → edge-b Anubis → origin + +edge-a unavailable: +visitor → Cloudflare → edge-b → local Anubis → origin +``` + + ++The second path is per service. Losing one application's Anubis on edge-a does not send the other applications across +the mesh too. Their healthy local paths keep working. +
++The private gateways bind only to the edges' overlay addresses. Headscale controls access, and HAProxy +additionally checks the source edge and the expected Host. Crucially, a private gateway can reach only its own +local Anubis. It cannot send the request back to the peer. No proxy ping-pong. +
++HAProxy inserts a service-specific routing cookie to keep the chosen path consistent through challenges and +subsequent requests. Cloudflare's affinity cookie operates one level earlier, choosing the public edge. They +solve different problems. +
++None of this moves an already-open WebSocket or video connection between machines. A client has to reconnect. +There is also no extra HAProxy policy here to replay an application POST after receiving a response. +
+ ++Here is a shortened version of the generated backend on edge-a for an example application. The address tokens stand for +private overlay addresses and must be replaced before using this as configuration: +
+ + +```haproxy +backend be_app + cookie EDGE_ROUTE_app insert indirect nocache httponly secure + option httpchk + http-check send meth GET uri /_edge/health ver HTTP/1.1 hdr Host app.example.net + http-check expect status 200 + + server anubis EDGE_A_MESH_IP:18923 source EDGE_A_MESH_IP cookie edge-a check inter 5s fall 2 rise 2 init-state fully-down + server peer EDGE_B_MESH_IP:18923 source EDGE_A_MESH_IP cookie edge-b backup check inter 5s fall 2 rise 2 init-state fully-down +``` + + +
+The server named anubis here points to the local private gateway, which checks the complete local
+path. The peer is a backup server. On edge-b, the generated configuration reverses the addresses
+but keeps cookie values tied to the actual edge aliases.
+
+fall 2 and rise 2 avoid changing state after one isolated probe result. They don't
+promise a ten-second recovery for the whole chain: these gateway checks depend on the separate Anubis and
+origin checks, their own intervals, and connection timeouts. The peer cannot be selected until HAProxy
+considers it healthy.
+
+A session that used the peer can remain there after the local route recovers because its routing cookie still +selects that path. A new session prefers the healthy local server. This is why looking at one browser +immediately after recovery isn't a reliable way to infer where all new traffic goes. +
+ ++A TCP port can be open while the application behind it is dead. Anubis can also return a perfectly good +challenge page while its target is unavailable. So checking for a generic HTTP 200 would tell a rather +optimistic story. +
++The configuration checks Anubis and the origin separately. A local service gateway is ready only when both are +healthy. With inter-edge redundancy enabled, the public readiness route accepts either a healthy local path or +a healthy peer path. +
+| Endpoint | Question it answers |
|---|---|
/_edge/entry-health | Can this edge receive HTTPS and respond through HAProxy? |
/_edge/health on an application Host | Does this service have a usable local or peer path? |
| Private gateway health | Are this edge's own Anubis and origin healthy? |
/_edge/metrics | What states and traffic does HAProxy report? |
+The distinction matters because I ended up using one shared Cloudflare pool with two endpoints. Every public +load balancer reuses it. This kept the configuration within the endpoint allocation shown by my subscription, +rather than duplicating a two-endpoint pool for each service. +
++A shared pool also shares its health verdict. Using one application as its monitor would make its failure +influence all the other domains. Requiring every application to pass would let one failed common origin take +both edges out of rotation. Neither describes the health of the entrance very well. +
+
+The common monitor therefore checks HTTPS ingress only. HAProxy returns edge-ready on a dedicated
+Host and path, after checking the network source and monitor token. The monitor expects a valid certificate,
+HTTP 200 and that exact body.
+
+It deliberately stays green when a shared application is down. The application checks and Prometheus show that +failure separately. Cloudflare's monitor +documentation describes how those probes determine endpoint health. +
++There is a limit to this choice: an edge can keep answering the ingress probe while one service has lost both +its local route and its route through the peer. If the other edge can still serve that service directly, the +common monitor will not steer just that domain away from the broken edge. Separate service pools could make +that decision. This setup relies on the private per-service fallback for that layer. +
+ +
+
+The monitor is HTTPS on port 443, using GET on /_edge/entry-health. Its configured interval is 60
+seconds, timeout five seconds, with two retries. The expected result is HTTP 200 and edge-ready.
+The option to skip certificate verification is unchecked.
+
+These are monitor settings, not a measurement of how long a visitor waits during failure. Probe scheduling, +health decisions and existing connections all affect the result. +
+
++Anubis adds a browser proof-of-work challenge in front of an +application. It does not replace that application's login. In this deployment, its listeners and metrics stay +on loopback; HAProxy is the public HTTP entrance. +
++The policies follow the application. Browser-oriented services receive a challenge. The notification service's API and stream +routes bypass it so notifications and clients can work. The request application's API does too. The media service challenges its +root and web interface, while native API, media and socket routes continue to the application's +authentication. +
+
+An Anubis ALLOW rule means "skip this challenge". It does not mean "give this person access to my
+libraries".
+
+That distinction deserved a real check on the media service. Anonymous requests to the tested private routes, including
+users, sessions, libraries, server configuration and API keys, returned 401 on both edges. The public
+information route remained public, and the public user-list route returned an empty list. HAProxy blocked
+/health and /metrics with 404, while the protected edge readiness route returned 403
+without its secret.
+
+Those results cover the routes tested, not every plugin or account permission in the media service. They do show why +"block the whole API" would have been the wrong fix: the clients need it. +
+
+The URL itself also needs attention before path-based restrictions. For the services with strict path rules,
+HAProxy normalizes supported encodings and path components, and rejects ambiguous forms before applying
+restrictions. Otherwise, protecting /admin while overlooking an equivalent encoded path is an
+expensive way to learn about URL parsing.
+
+The media policy makes the exception explicit. This is the policy structure rendered by the template, +including the configured challenge difficulty: +
+ + +```yaml +bots: + - name: media-browser + path_regex: '(?i)^/(?:web(?:/|$)|$)' + action: CHALLENGE + challenge: + algorithm: fast + difficulty: 5 + + - name: media-native-routes + path_regex: '^/' + action: ALLOW +``` + + ++The web rule precedes the catch-all allowance, and its match is case-insensitive. The path restrictions in +HAProxy run before selecting the local or remote route, so choosing the peer cannot skip those restrictions. +
++The notification client needs a different list: message delivery, stream connections and client/application routes belong to +its normal API use. The request application needs its API routes for the web UI as well as integrations. Reusing the same +blanket browser policy everywhere would have been simpler YAML and broken clients. +
++The containers also have distinct targets and policy directories. A policy-content hash in their labels causes +a recreation when the policy changes. Mounting a directory rather than a single policy file avoids keeping an +old file inode mounted after an atomic replacement. +
+ ++The other web services use proxied Cloudflare load balancers. The media service uses a DNS-only load balancer backed by +the same edge pool. Cloudflare participates in DNS selection and monitoring; the client connects directly to +an edge for HTTPS. The video traffic does not cross Cloudflare's reverse proxy. +
+ + +```text +DNS: media hostname → healthy edge address +HTTPS: client → edge-a or edge-b → Anubis → home NPM → the media service +``` + + ++Cloudflare +documents both proxied and DNS-only load balancing. In the latter case, failover depends on detection and +DNS caching. An ongoing stream does not migrate, and a client can keep using a cached address for a while. +
++This needed a configuration change before the DNS switch. The proxied services accept public application +traffic from Cloudflare sources, plus explicit administrative exceptions. The media service needs to accept direct +clients on its own vhost. Turning the cloud grey without changing that restriction would have produced a very +well-protected 403 page. +
++I first tested from the administration connection, then from a phone outside that exception. The logs +confirmed direct client traffic on the second edge. Login and playback worked. That external test matters: +testing only through an allowlisted administration address can hide a broken public access rule. +
+ +
++Getting it running was half the job. Adding the eighth service shouldn't require remembering every place where +the first seven were listed. +
+
+The exposure configuration now lives in docker/haproxy/vars/services.yml. Each service declares a
+permanent ID, domain, origin, policy profile and any special health or access rules. A shortened, anonymized
+example:
+
+The ID determines the local Anubis, metrics and private gateway ports. It stays stable when entries move +around in the YAML. Removing one service must not silently renumber every other listener. +
++The catalogue generates HAProxy routing, Anubis instances and policies, origin probes, private fallback paths, +and the required managed Headscale rules. A separate plan playbook validates the declarations without +contacting the VPS machines or changing DNS: +
+ + +```bash +ansible-playbook -i localhost, docker/haproxy/plan_services.yml +``` + + ++That plan runs in a separate Semaphore template with a local inventory and no deployment secrets. It catches +declaration errors. It does not prove that an origin answers over the network. +
++Application deployment remains separate. The catalogue does not create a database, configure backups or +install a new service. First deploy the application privately and verify it. Then declare how the edges should +expose it. +
+ +| Listener | Formula | Example for ID 1 | Exposure |
|---|---|---|---|
| Anubis HTTP | 8922 + ID | 8923 | Loopback only |
| Anubis metrics | 9200 + ID | 9201 | Loopback only |
| Private edge gateway | 18922 + ID | 18923 | Overlay address only |
+The compiler checks IDs, origins, supported profiles and derived listener collisions. An entry doesn't get to +refer to an arbitrary undefined origin. A disabled service leaves the active routing and Compose definition, +and managed permissions can be removed on the next deployment. Its data and public DNS are not automatically +deleted. +
++The plan writes a service summary and a Headscale ACL fragment on the runner. That gives me something concrete +to review before a deployment. It also keeps the maintenance path short: edit the catalogue, inspect the plan, +deploy to the edges, check the application, then publish the domain separately. +
++For a new application behind home NPM, I first create its Proxy Host and confirm the internal application +works. NPM needs a valid certificate for the application hostname, and WebSocket support when the service uses +it. From the edge, testing with private resolution keeps TLS verification meaningful: +
+ + +```bash +curl --noproxy '*' --connect-timeout 5 --max-time 15 \ + --resolve app.example.net:443:NPM_PRIVATE_IP \ + -I https://app.example.net/ +``` + + ++Then comes a catalogue entry. This example is illustrative; it is not an eighth deployed service: +
+ + +```yaml + example_app: + id: 8 + domain: app.example.net + origin: + node: npm + port: 443 + scheme: https + profile: browser + health: + path: / + status: '200-399' + dns: manual +``` + + +
+For HTTPS origins, Anubis uses the application hostname while the deployment pins that name to the private
+address. The HAProxy origin probe also sends the right SNI and verifies the certificate against that name.
+Connecting privately is no excuse for adding -k and hoping for the best.
+
+A service with clients that cannot solve a browser challenge may need profile: native and an
+explicit health route returning 200. That choice leaves authentication with the application. A new policy
+behavior outside the existing profiles is a code change, not something the catalogue can guess.
+
+Semaphore runs docker/haproxy/deploy_haproxy.yml from the repository, with the inventory, SSH
+credentials, privilege-escalation settings and secret variable group. The pair deploys serially, one edge at a
+time.
+
+The playbook checks the target and mesh membership, verifies routes and origin responses, obtains the +certificate, pulls the selected images, and validates the candidate HAProxy configuration with the downloaded +image before installing it. After deployment it checks TLS, challenges, aliases, restrictions and readiness. +With peer readiness required, a healthy fallback cannot hide a broken local path: both receive their own +checks. +
+
+The images intentionally use haproxy:lts and Anubis latest. A deployment can
+therefore bring an image update as well as a configuration change. That's part of how this lab is operated,
+not a claim that a moving tag is reproducible.
+
+Four secrets have separate jobs: +
++They are stored as secret extra variables in Semaphore. The playbook also supports environment fallbacks, but +there is no reason to define competing copies in both places. +
++Headscale management runs through delegated Ansible tasks on the control host. It reads the active policy, +updates the rules owned by this deployment, validates a candidate with Headscale, backs up the previous file +and reloads the policy. An adjacent tracking file records which rules it owns so obsolete permissions can be +removed. The rest of the policy stays intact. +
++Cloudflare application load balancers remain configured separately. The playbook manages dedicated DNS-only +edge names, and has an optional single-primary DNS publication mode. That mode points at one edge; it is not +the two-edge load balancer. The migrated services keep manual application publication in the catalogue. +
+ ++The first edge cannot pass a peer check against a machine that hasn't been deployed yet. Bootstrap therefore +happens with peer readiness not required. Once both stacks run, I enable the blocking peer validation and +deploy again: +
+ + +```yaml +haproxy_edge_redundancy_enabled: true +haproxy_edge_require_peer_ready: true +``` + + ++That second setting stays enabled for normal operation. It makes an incomplete peer configuration fail the +deployment instead of quietly shipping a broken fallback. A maintenance window on the peer can make this +validation fail even while the running local path continues serving users; deployment success and current +local availability are different questions. +
++The deployment template targets the edge inventory group. A typical ordinary extra-variable selection, with +aliases substituted, is: +
+ + +```json +{"target_hosts": "vps_haproxy"} +``` + + ++The SSH account must work on each edge and on the host receiving delegated Headscale tasks. Collections also +have to be available to the runner. In the successful deployment, Galaxy installation is skipped because the +execution environment already has its required collections. Skipping that step does not install missing +modules. +
++The migration deployment completed on both edges with no unreachable hosts and no failed tasks. I retain the full job output privately; this article describes the checks without publishing its application inventory. +
++Before completing the run, both edges retried the per-domain HTTPS check once. The eventual checks passed. A +retry message in the middle of this log is therefore not a failed final deployment, but it is still useful +information about startup convergence. +
++The run checked every configured service's public TLS route on the selected node, its browser-policy behavior +where applicable, the four blog-alias redirects, and service readiness. It then checked the local paths +separately and required every configured peer path. That last distinction prevents a working peer from masking a +broken local installation during rollout. +
++The access tests included the blog API and administration paths, an encoded-path case, the media service health and +metrics restrictions, and the metrics route. They verify the specific expected responses in the playbook; they +aren't a full application penetration test. +
++The Headscale candidate was validated during this run. Reloading was skipped because the active policy did not +change. Temporary candidate files also count as changes in the job. A change counter is not a count of application settings altered. +
++The application publication task was skipped, and the final message explicitly left the Cloudflare pools and +affinity unchanged. Semaphore also reported successful delivery of its deployment notification. That is useful +deployment feedback, alongside the separate functional notification tests. +
+ + ++The first edge did not materialize from one flawless green Semaphore run. There were runner dependency and +proxy issues, then SSH user and sudo settings to align with the working templates. Having a valid SSH key is +only useful if the template chooses the account that can use it. +
++Certificate issuance then failed because the Cloudflare token restricted the allowed source addresses. Certbot +runs on the edge, so Cloudflare saw the edge's public egress, not the Semaphore runner. The token needed to +authorize each issuing edge for renewals too. +
++Certificates are obtained locally through DNS-01. The renewal hook assembles the certificate and private key, +checks that they match, validates HAProxy and signals its master to reload. If validation or signalling fails, +it restores the previous PEM on disk. That is a local safeguard, not a full transactional rollback of a +deployment. +
++Later, both Cloudflare endpoints showed a TLS name mismatch even though the local HTTPS probe worked. The fix +was in the monitor headers: +
+ + +```text +Host: edge-health.example.net +X-Edge-Health-Token:
+Those are two headers. The technical hostname is the value of Host; it is not a header name. The
+monitor uses that Host for TLS SNI. Endpoint Host overrides stay empty so they don't replace the intended
+name. Certificate verification stayed enabled. Cloudflare's error
+reference covers this mismatch.
+
+For proxied requests, HAProxy accepts the visitor address in CF-Connecting-IP only when the
+actual connection comes from a configured Cloudflare range. Otherwise, the connection source is the client
+address. It rebuilds forwarding headers before passing traffic downstream.
+
+Without that boundary, anyone able to reach the listener could claim to be the administration address by +supplying a header. Source filters for metrics and monitors use the real network peer, not the reconstructed +visitor address. +
++The Cloudflare ranges are an explicit snapshot in configuration. They need review when Cloudflare changes +them. Anubis stays on loopback, private gateways stay on the overlay, and the unauthenticated local Prometheus +exporter stays on loopback too. +
++HAProxy logs connection and backend information without logging request URLs. The notification service and the media service can carry +credentials in query strings; those URLs do not belong in a log excerpt pasted into a blog post. +
+ ++The existing Prometheus stack collects HAProxy metrics over public HTTPS using dedicated DNS-only names for +each edge. The monitoring host doesn't need to join the mesh. The metrics route requires both an authorized +source address and its own Bearer token. +
++Grafana can then show traffic, sessions, backend errors and health by edge and service. Local Anubis and +origin states remain separate, with the peer path alongside them. A green peer path can keep the application +reachable while a local component needs repair. +
++Traffic share needs some restraint when interpreting it. HAProxy sees requests that reach the edges, not +requests already satisfied by Cloudflare's cache. A video stream and a blog visit aren't comparable units of +work. Equal endpoint weights won't make those graphs perfectly symmetrical. +
++Likewise, HAProxy metrics don't contain Cloudflare's actual pool verdict. That belongs to Cloudflare's monitor +and event view. And a dashboard import doesn't install Prometheus alert rules or configure notification +delivery. The reference rules still need to be loaded and connected to the existing alerting setup. +
+ ++Each target has a dedicated DNS-only name. If I used an application load-balancer name here, I could scrape +whichever edge got selected twice and believe I was observing both. The labels and targets below are +anonymized, and the proxy address stands in for the monitoring stack's existing outbound proxy: +
+ + +```yaml +scrape_configs: + - job_name: haproxy-edges + scrape_interval: 15s + scrape_timeout: 10s + scheme: https + proxy_url: http://monitor-proxy.example.net:3128 + metrics_path: /_edge/metrics + authorization: + type: Bearer + credentials_file: /etc/prometheus/secrets/edge-metrics-token + tls_config: + insecure_skip_verify: false + static_configs: + - targets: ['edge-a.example.net:443'] + labels: + edge: a + - targets: ['edge-b.example.net:443'] + labels: + edge: b +``` + + ++The secret file must be mounted and readable inside Prometheus. The source restriction on HAProxy needs to +match the actual public egress of that scrape path. A correct token coming from another source is still +refused. +
+
+The basic reachability query is up{job="haproxy-edges"}. A value of one means the scrape
+succeeded. It does not mean every application is up, nor that Cloudflare agrees with HAProxy about an
+endpoint's state.
+
++The traffic screenshot shows a larger slice for one edge and a short burst of media response bytes. It is a +snapshot of requests and bytes reaching this layer, not a benchmark or a unique-user count. Equal configured +weights do not make an uneven short window evidence of a routing fault. +
+
+
+There is a naming trap in this dashboard version: the panel titled "Anubis health" displays be_*
+series. With redundancy enabled, those backends represent service paths through the private gateways. I read
+them alongside the probe_* origin series, rather than treating that title as proof of an isolated
+Anubis process state.
+
+The short health drops and error peaks are visible around the same part of the window. To assign a cause, I +would need the corresponding logs and a time-aligned deployment or incident event. These captures don't +establish an end-user outage duration. +
+
++One rolling response-time series rises to roughly 850 ms and then stays flat in this window. That's a reason +to inspect the underlying samples and request volume, not enough information to announce a regression or a +percentile. The panel is a rolling average; it doesn't show a p95 latency, and a quiet service can retain a +value without generating much new traffic. +
+ ++I deployed and tested the edges before changing the public application names. Local resolution overrides let +the browser use the real hostname and certificate while contacting the new machine. The public DNS still +served the older entrance during those tests. +
++The checks went beyond seeing a challenge page: application logins, live updates, notifications and the media service +playback all needed their own tests. Both edges also passed the deployment's local and peer +readiness checks with peer validation required. +
++For the private fallback test, one application's Anubis on edge-a was stopped. Requests entering edge-a used the +peer path, and a WebSocket opened through edge-b. The application origin stayed running. That tested the +fallback layer without deliberately taking the shared application down. +
++The public migration started with one status service. After its load balancer was active, HAProxy on edge-a +was stopped. Cloudflare detected the failed entrance, the other edge served the status application, and edge-a +was restored. This was an actual stop-and-recover test; it wasn't just two green endpoints sitting next to +each other. +
+
+The remaining proxied services followed, with logins and notifications checked after migration. The root
+domain and www were pointed at the blog and their redirects checked. The media service moved separately to
+DNS-only load balancing, followed by direct access and playback tests from the phone.
+
+Finally, the old application vhosts were disabled and the redundant old Anubis containers stopped. The +application containers stayed running: the new edges still use them. Nginx and Certbot also stayed where other +active services needed them, including the Headscale entrance. Disabling those old vhosts means rollback now +requires restoring them as well as changing the public routing. +
+ ++A direct test can target the new edge without changing public DNS. This example address is a documentation +address, not one of my VPS addresses: +
+ + +```bash +curl --noproxy '*' --silent --show-error \ + --connect-timeout 5 --max-time 15 \ + --resolve app.example.net:443:192.0.2.10 \ + --output /dev/null \ + --write-out 'HTTP %{http_code}, TLS %{ssl_verify_result}\n' \ + https://app.example.net/ +``` + + ++The hostname stays in the URL so Host and SNI remain correct. A TLS verification result of zero and a valid +response establish the connection, but HTTP 200 may still be an Anubis challenge. The next step is opening the +application and using it. +
++On the edge itself, the common monitor can be checked over loopback with certificate verification intact: +
+ + +```bash +curl --noproxy '*' --silent --show-error --max-time 10 \ + --resolve edge-health.example.net:443:127.0.0.1 \ + https://edge-health.example.net/_edge/entry-health +``` + + +
+The expected body is edge-ready. Loopback is exempt from the external monitor token requirement.
+That local result cannot prove Cloudflare's external probe source, headers or firewall access are correct.
+
+The two controlled stop tests exercise different layers: +
+| Stopped component | Expected routing | Evidence used |
|---|---|---|
| One service's local Anubis | That service enters the same edge, then uses the peer | Peer backend selection and a working application/WebSocket |
| One edge's HAProxy | Cloudflare marks that entrance unavailable and selects the other | Cloudflare state, surviving edge logs and browser access |
+For the first case, the relevant container name has the shape haproxy-prod-anubis-example_app. For
+the second, it is haproxy-prod. The restoration is starting that same container again and waiting
+for health checks to settle. These were controlled tests during migration, not commands to run casually
+against both entrances at once.
+
+I also kept the public readiness and metrics routes in the negative tests: an unauthenticated request from the +workstation received 403. A monitoring route that returns the expected data only to its authorized caller is +behaving differently from one that happens to answer everyone. +
++The old application CNAME records were retained while the same-name Cloudflare load balancers took over public +routing. That made the DNS side of rollback simpler during migration. It did not preserve the old application +entrance after its Nginx vhost was disabled. +
++The cleanup therefore happened in stages: disable migrated application vhosts, validate and reload Nginx, +check the retained services, then stop the old Anubis containers. The origin applications, the control-plane +entrance and other still-used vhosts remained in place. Certificates weren't removed just because one +application name had migrated. +
++This is also why the old Nginx deployment playbook isn't part of the new edge procedure. Rerunning it during +cleanup could recreate the entrance I had just retired. The two playbooks belong to different architectures. +
+ ++Both entrances use the same application origins. If a single application instance fails, neither edge can +manufacture a replacement application. If the home connection, home gateway or NPM fails, the home-hosted services can lose +both paths at once. +
++The pair also remains with one VPS provider, and Cloudflare is still a dependency for the proxied services. +Different locations reduce some shared failure risks; they don't remove every one. +
++What is deployed now is a redundant public entrance, with per-service private fallback and a catalogue that +makes the next addition less tedious. Application availability still depends on each application's own +storage, backups and redundancy. +
++Seven services migrated, two entrances answering, and a stopped edge no longer being the end of the story. +That's a useful place to leave this round of homelab-ing. +
++Thanks for reading me, +
++spleenftw +
diff --git a/load-balancing-my-homelab/diagrams/README.md b/load-balancing-my-homelab/diagrams/README.md new file mode 100644 index 0000000..b234e22 --- /dev/null +++ b/load-balancing-my-homelab/diagrams/README.md @@ -0,0 +1,114 @@ +# Infrastructure overview + +Generic documentation: no real service inventory, node names or private addresses. + +```mermaid +flowchart TB + subgraph PUBLIC["Public access"] + direction LR + WEB["Browser and API clients"] + MEDIA["Direct media clients"] + DNS["Cloudflare DNS-only LB