Operations

Quarterly failover drill#

The whole point of a dual-provider HA setup is that failover works when the primary uplink fails for real. The only way to have confidence in that is to test it deliberately.

Suggested cadence: every quarter, during a low-traffic window, with the operator on console.

Procedure:

1. Announce the drill on the team channel — failover should be invisible, but if something goes wrong, you want to know immediately.
2. Disable the primary uplink at the interface level (not at the provider modem — that simulates the wrong failure mode).
3. Within five seconds, the failover monitor should switch the default route to the secondary uplink.
4. Verify external reachability: a request from inside a zone should still resolve, fetch, and return.
5. Re-enable the primary uplink.
6. The monitor should restore the default route back to the primary within its configured interval.
7. Record the drill in a logbook: date, time-to-detect, time-to-recover, anything unusual.

If failover takes longer than expected, the monitor’s check interval or thresholds need tuning. If failover does not happen at all, the monitor’s config is broken — fix it before you go home. A non-functional failover monitor is worse than no monitor at all; it gives the operator false confidence.

Log review#

Three log streams matter:

• Firewall events. Drop decisions, table additions, NAT state. Loud during reconnaissance scans, quiet under normal load.
• SSL offloading events. Termination errors, certificate validation failures, scanner verdicts. Spikes here mean something changed — a certificate rotation upstream, a new client that does not trust the internal CA root, or a malware pattern that bypassed an earlier filter.
• Failover-monitor events. Link transitions. Should be empty most weeks. Frequent transitions point to a flaky uplink — raise it with the carrier.

Suggested operator habit: a weekly fifteen-minute scan of these three streams. Flag anything unusual, file follow-up tickets for the next quarterly review.

Zone-boundary verification#

The zones are defined in the ruleset. The ruleset can be incorrect. The way to know which one is true is to test.

Suggested cadence: every six months, or after any significant ruleset change.

For each pair of zones (clients ↔ DMZ, clients ↔ VPN, anonymization ↔ everything else, etc.):

1. From a host in zone A, attempt to reach a service in zone B that is not supposed to be reachable.
2. Confirm the connection fails — TCP RST, ICMP unreachable, or a timeout, depending on the rule type.
3. Record the result in the same logbook.

If any expected-blocked path is reachable, that is a rule bug. Fix it the same day.

Blocklist housekeeping#

The blocklist tables grow over time. Two failure modes are common:

• Stale entries. An IP that was abusive six months ago is now a residential DSL line with a new owner. Periodically rotating the blocklist file (see the Abusive HTTP Watch configuration docs) bounds this.
• Whitelist drift. Office and VPN IP ranges change over time. Stale whitelist entries do not cause incidents directly, but they reduce the value of the whitelist as a control. Review the whitelist file every quarter.

Certificate rotation#

The TLS termination layer uses certificates issued by the Sovereign Certificate Authority. Rotation is automatic via ACME, but:

• Confirm rotation at least monthly — openssl s_client against the gateway, verify the certificate’s notAfter is in the future, verify the chain still validates against the internal root.
• Test client trust after any CA-side change — pick a representative client from each zone, browse to an HTTPS service inside the network, confirm no certificate warning.

Failure mode: a rotation succeeds on the CA side but the firewall fails to pick up the new certificate (configuration reload missing, file permissions wrong). The previous cert remains valid for a while, masking the bug. The monthly verification catches this.

Routine housekeeping#

• OS patches. Apply security-relevant patches within the vendor’s recommended window. Reboot — the firewall is not a cattle-style appliance, but uptime past nine months is asking for trouble.
• Dependency review. The Tor, I2P, Squid, and Privoxy daemons each have their own patch cycle. Track upstream releases.
• Capacity check. State-table fill, CPU at peak, memory headroom. The firewall should be running comfortably below half capacity. If it is not, plan the next hardware refresh.

When things break#

The single most useful operator habit: write down what happened.

A short note per incident — when, what you saw, what you changed, what worked, what did not — turns one event into a permanent uplift. The next time the same pattern shows up, either the same person or their successor solves it in minutes instead of hours.

If the incident exceeds your team’s expertise or available time, the Support service is the escalation path.