Pulse HA / Multi-AZ Runbook

For the on-call operator. Assumes a working single-node Pulse deployment you want to flip into active-passive across two AZs, then keep healthy.

1. What HA mode does for you

HA mode runs two Pulse JVMs behind a stateful load-balancer, backed by a Postgres primary + sync replica in a second AZ. One node is leader (writes go there) — the other is standby (read-only, takes over on failover). RTO 30 s for a JVM crash, 45 s for an AZ outage. Availability moves from 99.5 % single-host to 99.9 % HA.

Leader election runs on a Postgres session-scoped advisory lock with a per-promotion epoch counter to fence stale writers.

Failover-latency SLO

Once the advisory lock is free (planned shutdown releases it immediately; unplanned crash frees it when Postgres reclaims the dead session), a standby is promoted within a bound set by the leaseHeartbeat interval (default 5000 ms):

• Expected promotion: leaseHeartbeat × 2 ≈ 10 s at default.
• Enforced ceiling (SLO): leaseHeartbeat × 3 = 15 s.

End-to-end RTO (30 s JVM / 45 s AZ) adds crash detection (LB health check + Postgres reclaiming the dead session). Lowering leaseHeartbeat shrinks the window at the cost of more lock traffic. An integration test measures actual promotion time and fails the build if it exceeds leaseHeartbeat × 3.

2. Prerequisites

• Postgres 14+ (advisory locks on by default). AWS RDS Multi-AZ or self-hosted Patroni. Sync replication is non-negotiable for RPO=0.
• 2 Pulse JVMs, each on its own host in a different AZ.
• The cluster-bridge module on the classpath of every Pulse instance. Auto-registers via SPI — no code change.
• A stateful LB in front (ALB / Caddy / HAProxy). Health check: GET /api/pulse/health × 2 misses → down.

3. Activation

On EACH Pulse host:

1export PULSE_DB_URL='jdbc:postgresql://pg-cluster.internal:5432/pulse?targetServerType=primary'
2export PULSE_DB_USER=pulse
3export PULSE_DB_PASSWORD=<vault>
4export PULSE_NODE_ID=pulse-a            # unique per host
5export PULSE_BIND_HOST=10.0.1.12        # routable address
6export PULSE_BIND_PORT=9090
7export PULSE_REGION_ID=us-east-1a       # or 1b on the standby
8export PULSE_HA_MODE=true               # the toggle
9systemctl restart pulse

Or flip the toggle live without a restart:

curl -X POST https://pulse-a.internal/api/admin/cluster/enable \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"nodeId":"pulse-a","address":"10.0.1.12:9090","regionId":"us-east-1a"}'

Without the bridge module or with PULSE_HA_MODE=false, the coordinator boots in passive mode (single member, always leader, fixed epoch 1).

Expected startup log line:

ClusterCoordinator started: node=pulse-a region=us-east-1a lock=pulse-cluster-leader heartbeat=5000ms
Promoted to LEADER: node=pulse-a epoch=1

4. Reading cluster status

curl -s https://pulse-a.internal/api/admin/cluster/status | jq

1{
2  "self": {"nodeId":"pulse-a","role":"LEADER","epoch":4,"address":"10.0.1.12:9090"},
3  "members": [
4    {"nodeId":"pulse-a","role":"LEADER","epoch":4},
5    {"nodeId":"pulse-b","role":"FOLLOWER","epoch":4}
6  ],
7  "coordinator": "enterprise"
8}

UI: Settings → Cluster shows the same data with role badges. Logs: every promotion/demotion logs one line — grep "Promoted to LEADER\\|transition.*FOLLOWER" /var/log/pulse/pulse.log.

5. Manual failover

Use when the leader is alive enough to hold the lock but degraded (GC stall, hung scheduler). Healthy auto-failover needs no operator action.

1# 1. On the standby — force promotion (admin scope required)
2curl -X POST https://pulse-b.internal/api/admin/cluster/force-promote \
3  -H "Authorization: Bearer $ADMIN_TOKEN"
4# Expected: {"status":"promoted","previousRole":"FOLLOWER","newEpoch":5}
5
6# 2. Verify the advisory lock moved
7psql -h pg-cluster.internal -U pulse -c \
8  "SELECT pid, mode, granted FROM pg_locks WHERE locktype='advisory';"
9# Expected: one row, granted=t, pid matches pulse-b's connection
10
11# 3. Verify the standby's role flipped
12curl -s https://pulse-b.internal/api/admin/cluster/status | jq -r '.self.role'
13# Expected: LEADER
14
15# 4. Drain the old leader — auto-downgrades on the next heartbeat (5s).
16#    To accelerate:
17curl -X POST https://pulse-a.internal/api/admin/cluster/step-down \
18  -H "Authorization: Bearer $ADMIN_TOKEN"
19# Expected log on pulse-a: "transition: LEADER → FOLLOWER"

6. Postgres replica sync verification

1# Connect to the STANDBY postgres replica
2psql -h pg-replica.internal -U pulse -c "
3  SELECT pg_last_wal_receive_lsn() AS received,
4         pg_last_wal_replay_lsn()  AS replayed,
5         pg_wal_lsn_diff(pg_last_wal_receive_lsn(),
6                         pg_last_wal_replay_lsn()) AS lag_bytes;"

Expected: lag_bytes < 16 MB under normal load. > 5 min sustained replay lag is alertable. To force a manual catch-up if a replica is stuck:

1# On the standby Postgres host
2sudo systemctl stop postgresql
3sudo -u postgres pg_basebackup -h pg-primary.internal -D /var/lib/postgresql/14/main \
4  -U replicator -P -R --wal-method=stream
5sudo systemctl start postgresql

Rebuilds the replica from the primary; 5–30 min for a 100 GB cluster. Pulse keeps serving from the primary during this.

7. Restart the primary cleanly

1# 1. On the primary (Pulse A): tray → Quit, OR
2systemctl stop pulse
3# Expected log: "ClusterCoordinator stopped"
4
5# 2. On the standby (Pulse B), watch the auto-promotion
6tail -f /var/log/pulse/pulse.log | grep -E "Promoted|MemberLeft"
7# Expected within 5–10s:
8#   MemberLeft: pulse-a
9#   Promoted to LEADER: node=pulse-b epoch=N+1
10
11# 3. Restart the binary on Pulse A
12systemctl start pulse
13# Expected log: "ClusterCoordinator started: ... role=FOLLOWER"
14# It comes up as FOLLOWER and only takes leadership on the next election.

8. Common failures + recovery

Both nodes think they're leader (split brain)

Symptom: pg_locks shows two granted advisory rows for the same key, OR clients see writes ack'd but missing on read after failover.

In-band mitigation: the coordinator advances epoch on every promotion. The data plane rejects writes whose tagged epoch is lower than the cluster's current epoch — a stale leader's writes are dropped, not committed.

1# 1. Identify the stale node (lower epoch = loser)
2for n in pulse-a pulse-b; do
3  echo "=== $n ==="
4  curl -s https://$n.internal/api/admin/cluster/status | jq '.self | {nodeId, epoch, role}'
5done
6
7# 2. Force-kill the lower-epoch node
8ssh root@<lower-epoch-host> systemctl stop pulse
9
10# 3. Restart it cleanly — it'll join as FOLLOWER on the higher epoch
11ssh root@<lower-epoch-host> systemctl start pulse

Postgres replica lagging > 5 min

Either accept eventual consistency (point the LB read pool at targetServerType=primary until catch-up finishes) or rebuild the replica per §6.

Both nodes can't reach Postgres

Both auto-downgrade to FOLLOWER (no lock holder = no leader). The cluster is read-only until Postgres recovers. Action: fix Postgres connectivity — election restarts automatically on the next heartbeat.

9. Monitoring

The bridge exposes Prometheus metrics on the existing /metrics endpoint. Scrape and alert on:

Values: pulse_cluster_role = 1 (LEADER), 0 (FOLLOWER), -1 (UNAVAILABLE). advisory_lock_held = 1 on the holder, 0 elsewhere.

Suggested PagerDuty rules: leader gap > 60 s = page on-call; replication lag > 30 s = warn channel; epoch flapping = warn channel.

10. Rollback to single-node

1# 1. Stop the standby
2ssh root@pulse-b systemctl stop pulse
3
4# 2. On the primary: disable HA
5curl -X POST https://pulse-a.internal/api/admin/cluster/disable \
6  -H "Authorization: Bearer $ADMIN_TOKEN"
7# OR set PULSE_HA_MODE=false and restart.
8
9# 3. Optional: remove the bridge module from the classpath
10systemctl restart pulse
11# Expected log: "ClusterCoordinator: standalone"

Postgres data stays intact. Pulse resumes as a single-node install against the same DB. The replica can be left idle or torn down — your call.