# OperationsRunning the firewall in production: failover drills, log review, boundary tests, housekeeping.

## Quarterly failover drill

The whole point of a dual-provider HA setup is that failover
works **when the primary uplink fails for real**. The only way to
have confidence in that is to test it deliberately.

Suggested cadence: **every quarter**, during a low-traffic
window, with the operator on console.

Procedure:

1. Announce the drill on the team channel — failover should be
   invisible, but if something goes wrong, you want to know
   immediately.
2. Disable the primary uplink at the interface level (not at the
   provider modem — that simulates the wrong failure mode).
3. Within five seconds, the failover monitor should switch the
   default route to the secondary uplink.
4. Verify external reachability: a request from inside a zone
   should still resolve, fetch, and return.
5. Re-enable the primary uplink.
6. The monitor should restore the default route back to the
   primary within its configured interval.
7. Record the drill in a logbook: date, time-to-detect,
   time-to-recover, anything unusual.

If failover takes longer than expected, the monitor's check
interval or thresholds need tuning. If failover does not happen
at all, the monitor's config is broken — fix it before you go
home. A non-functional failover monitor is worse than no monitor
at all; it gives the operator false confidence.

## Log review

Three log streams matter:

- **Firewall events.** Drop decisions, table additions, NAT
  state. Loud during reconnaissance scans, quiet under normal
  load.
- **SSL offloading events.** Termination errors, certificate
  validation failures, scanner verdicts. Spikes here mean
  something changed — a certificate rotation upstream, a new
  client that does not trust the internal CA root, or a malware
  pattern that bypassed an earlier filter.
- **Failover-monitor events.** Link transitions. Should be empty
  most weeks. Frequent transitions point to a flaky uplink —
  raise it with the carrier.

Suggested operator habit: a weekly fifteen-minute scan of these
three streams. Flag anything unusual, file follow-up tickets for
the next quarterly review.

## Zone-boundary verification

The zones are defined in the ruleset. The ruleset can be
incorrect. The way to know which one is true is to test.

Suggested cadence: **every six months**, or after any
significant ruleset change.

For each pair of zones (clients ↔ DMZ, clients ↔ VPN,
anonymization ↔ everything else, etc.):

1. From a host in zone A, attempt to reach a service in zone B
   that is not supposed to be reachable.
2. Confirm the connection fails — TCP RST, ICMP unreachable, or
   a timeout, depending on the rule type.
3. Record the result in the same logbook.

If any expected-blocked path is reachable, that is a rule bug.
Fix it the same day.

## Blocklist housekeeping

The blocklist tables grow over time. Two failure modes are
common:

- **Stale entries.** An IP that was abusive six months ago is now
  a residential DSL line with a new owner. Periodically rotating
  the blocklist file (see the [Abusive HTTP Watch
  configuration](../../abusive-http-watch/configuration/) docs)
  bounds this.
- **Whitelist drift.** Office and VPN IP ranges change over time.
  Stale whitelist entries do not cause incidents directly, but
  they reduce the value of the whitelist as a control. Review
  the whitelist file every quarter.

## Certificate rotation

The TLS termination layer uses certificates issued by the
[Sovereign Certificate Authority](../../sovereign-certificate-authority/).
Rotation is automatic via ACME, but:

- **Confirm rotation** at least monthly — `openssl s_client`
  against the gateway, verify the certificate's notAfter is in
  the future, verify the chain still validates against the
  internal root.
- **Test client trust** after any CA-side change — pick a
  representative client from each zone, browse to an HTTPS
  service inside the network, confirm no certificate warning.

Failure mode: a rotation succeeds on the CA side but the
firewall fails to pick up the new certificate (configuration
reload missing, file permissions wrong). The previous cert
remains valid for a while, masking the bug. The monthly
verification catches this.

## Routine housekeeping

- **OS patches.** Apply security-relevant patches within the
  vendor's recommended window. Reboot — the firewall is not a
  cattle-style appliance, but uptime past nine months is asking
  for trouble.
- **Dependency review.** The Tor, I2P, Squid, and Privoxy
  daemons each have their own patch cycle. Track upstream
  releases.
- **Capacity check.** State-table fill, CPU at peak, memory
  headroom. The firewall should be running comfortably below
  half capacity. If it is not, plan the next hardware refresh.

## When things break

The single most useful operator habit: **write down what
happened**.

A short note per incident — when, what you saw, what you
changed, what worked, what did not — turns one event into a
permanent uplift. The next time the same pattern shows up,
either the same person or their successor solves it in minutes
instead of hours.

If the incident exceeds your team's expertise or available time,
the [Support](/services/support/) service is the escalation
path.