Our Open-Source Enterprise Networking case study covers replacing a legacy commercial VPN concentrator with a WireGuard road-warrior mesh for a distributed workforce with no central office to fall back on. The architecture is the headline, but two small operational habits that came out of that rollout are worth a post of their own, because both are the kind of lesson that only shows up after something has already gone quietly wrong.
Lesson one: a working config and a working connection are not the same claim
WireGuard peer setup looks deceptively simple: generate a keypair, put the public half on the server, put the matching private half (and the server’s public key) in the client config, and the tunnel comes up. It’s simple enough that it’s tempting to treat “I added the peer and saved the config” as equivalent to “the peer works” — and for most peers, in most rollouts, it is.
Not always. In one real rollout batch, half the new peers had a silent key mismatch. Not a typo you’d catch reading the file — a genuine mismatch between the public key registered server-side and the private key actually shipped in the client’s configuration. Nothing about that state produces an error anywhere. There’s no failed handshake logged as “key mismatch,” no obvious symptom on the server side. The tunnel just never comes up, and from the user’s side it looks identical to “my internet is being weird” or “the VPN app isn’t working” — which is exactly what got reported, after the rollout had already been marked complete.
The fix going forward wasn’t a smarter deployment script. It was a verification step added before marking any peer done, not after a user reports a problem: diff the public key registered on the server side against the private key actually present in that peer’s own config file, for every single peer, individually. It’s a boring, mechanical check — and it caught the exact bug described above in real time, on a batch where it would otherwise have taken a support ticket (or several) to surface.
The general version of this lesson generalizes past WireGuard: a config that was generated correctly and a config that was verified to match on both ends are different claims, and the gap between them is invisible until someone actually tries to use it.
Lesson two: a saved config and a running config are not the same claim either
The second bug looked completely different on the surface but is really the same lesson from the other direction. A recurring reachability problem across a double VPN tunnel hop presented like a routing bug — packets not making it where they should, intermittently, in a way that pointed at a misconfigured route or a NAT issue somewhere in the path.
It wasn’t a routing bug. It was a firewall alias — a named group of allowed addresses used by an access rule — whose live, in-kernel state table had silently desynced from its own saved configuration on disk. The rule as written was correct. The rule as actually enforced by the running firewall had drifted from that and no longer matched. Nothing flagged this: no error, no alert, no log entry announcing that the two had diverged. The fix, once found, was almost anticlimactic — re-apply the identical, already-correct configuration, which forced the running state back in sync with the saved one.
The diagnostic cost of this bug wasn’t in fixing it. It was in ruling out everything else first, because every other layer looked completely normal: the saved config was correct, the route table was correct, DNS was correct. The only thing that was wrong was invisible from every place you’d normally look — you had to specifically go check whether the live enforcement matched the saved intent, which isn’t most people’s first troubleshooting instinct because most of the time those two things just can’t drift apart on their own.
The habit that covers both
Both bugs share a root shape: a written artifact (a config file, a firewall rule) and the real-world state it’s supposed to produce (a working tunnel, an enforced rule) can separate from each other without any warning. The practical habit that came out of dealing with both, now applied on every engagement:
- Never mark a rollout step “done” based on the config being correct — verify the actual behavior it’s supposed to produce, per unit, not just per batch.
- When something that “should be working” isn’t, check whether the running state matches the saved state before assuming the saved state itself is wrong. They’re not always the same thing, and nothing tells you when they’ve drifted apart.
Neither of these is a sophisticated technique. They’re closer to a checklist item than an architecture decision. But they’re the specific two checklist items that, on a real rollout, caught bugs that would otherwise have shown up as a confused user report days later instead of a five-minute fix during deployment.
The full VPN, firewall, and IDS architecture this rollout is part of is in the Open-Source Enterprise Networking case study.