VPC router applies the VPC source NAT IP to locally-generated traffic on secondary public interfaces, breaking gateway health checks

[prashanthr2]

problem

On a VPC with public IPs from more than one range/VLAN, the virtual router installs an unscoped source NAT rule on every public interface using the single VPC source NAT address. Because the rule has no source match, it also rewrites traffic the router itself originates, so the VR cannot emit packets with the correct source address from any public interface other than the source-NAT one.

Generated in systemvm/debian/opt/cloud/bin/cs/CsAddress.py:

if self.get_type() == "public" and self.config.is_vpc() and method == "add" and not self.config.is_routed():
    if self.address["source_nat"]:
        vpccidr = cmdline.get_vpccidr()
        self.fw.append(["filter", 3, "-A FORWARD -s %s ! -d %s -j ACCEPT" % (vpccidr, vpccidr)])
        self.fw.append(["nat", "", "-A POSTROUTING -j SNAT -o %s --to-source %s" % (self.dev, self.address['public_ip'])])
    elif cmdline.get_source_nat_ip() and not self.is_private_gateway():
        self.fw.append(["nat", "", "-A POSTROUTING -j SNAT -o %s --to-source %s" % (self.dev, cmdline.get_source_nat_ip())])

The if branch pairs its SNAT with a FORWARD -s rule, showing the intent is forwarded guest traffic. The elif branch has no equivalent scoping and no -s match

VPC router r-21-VM — two public ranges on separate VLANs:

eth1  10.1.30.5/24    public range 1 (source NAT IP), gateway 10.1.30.1
eth2  10.1.31.2/24    public range 2,                 gateway 10.1.31.1
eth3  10.0.0.1/24     guest tier
eth4  10.1.1.1/24     guest tier

Resulting NAT rules:

-A POSTROUTING -o eth1 -j SNAT --to-source 10.1.30.5
-A POSTROUTING -o eth2 -j SNAT --to-source 10.1.30.5     ← eth2 is 10.1.31.2; wrong subnet
-A POSTROUTING -s 10.1.1.0/24 -d 10.1.1.52/32 -o eth4 -p tcp -m tcp --dport 80 -j SNAT --to-source 10.1.1.1

/etc/cloudstack/ips.json is correct.

root@r-21-VM:~#  cat /etc/cloudstack/ips.json | grep -iE 'eth|public_ip|source_nat|network|gateway'
  "eth0": [
      "device": "eth0",
      "gateway": "",
      "network": "169.254.0.0/16",
      "public_ip": "169.254.82.31",
      "source_nat": false
  "eth1": [
      "device": "eth1",
      "gateway": "10.1.30.1",
      "is_private_gateway": false,
      "network": "10.1.30.0/24",
      "public_ip": "10.1.30.5",
      "source_nat": true,
  "eth2": [
      "device": "eth2",
      "gateway": "10.1.31.1",
      "is_private_gateway": false,
      "network": "10.1.31.0/24",
      "public_ip": "10.1.31.2",
      "source_nat": false,
  "eth3": [
      "device": "eth3",
      "gateway": "10.0.0.1",
      "network": "10.0.0.0/24",
      "public_ip": "10.0.0.1",
      "source_nat": false
  "eth4": [
      "device": "eth4",
      "gateway": "10.1.1.1",
      "network": "10.1.1.0/24",
      "public_ip": "10.1.1.1",
      "source_nat": false

The eth2 entry has source_nat: false and public_ip: 10.1.31.2 so this is not bad input from the management server; the elif deliberately substitutes the VPC-wide source NAT IP.

Packet capture, VR pinging its own eth2 gateway (10.1.31.1):

root@r-21-VM:~# tcpdump -i any -nnnvvv 'host 10.1.31.1 and icmp'
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
10:01:07.711338 eth2  Out IP (tos 0x0, ttl 64, id 10156, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.30.5 > 10.1.31.1: ICMP echo request, id 6524, seq 0, length 64
10:01:08.712408 eth2  Out IP (tos 0x0, ttl 64, id 10259, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.30.5 > 10.1.31.1: ICMP echo request, id 6524, seq 1, length 64
10:01:09.713031 eth2  Out IP (tos 0x0, ttl 64, id 10338, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.30.5 > 10.1.31.1: ICMP echo request, id 6524, seq 2, length 64

Requests and replies traverse different interfaces. On a permissive network the ping still succeeds via that asymmetry; on any gateway enforcing reverse-path forwarding the packet is dropped at ingress and no reply is generated.

User-visible impact: /root/health_checks/gateways_check.py pings every NIC's gateway, so 10.1.31.1 is reported unreachable despite being perfectly reachable, producing repeated Health checks failed alerts. If router.health.checks.failures.to.recreate.vr includes gateways_check.py, CloudStack will also repeatedly recreate a healthy VR.

More generally, any VR-originated traffic leaving a non-source-NAT public interface carries an address that does not belong on that segment.

Not affected: guest egress. A VPC has a single source NAT IP by design and guest traffic follows the default route out the source-NAT interface, so it is unchanged.

versions

CloudStack : 4.22.1
Hypervisor: KVM
Gateway: pfSense (reverse-path filtering enabled)

CsAddress.py is byte-identical in 4.20.1.0, 4.20.3.0, 4.22.1.0 and main long-standing behaviour, not a regression.

The steps to reproduce the bug

1. Create a VPC with the default VPC offering and at least one tier.
2. Acquire a public IP from a second public range on a different VLAN/subnet and associate it with the VPC (e.g. add a port-forwarding rule), so the VR gains a second public NIC.
3. On the VR, confirm the interface and its SNAT rule:

ip addr show eth2
iptables -t nat -S POSTROUTING | grep eth2

The --to-source value is the VPC source NAT IP (10.1.30.5), not the eth2 address (10.1.31.2).
4. Ping the second range's gateway from the VR while capturing both public interfaces:

tcpdump -lnni eth1 -s 96 'icmp and host 10.1.31.1' > /tmp/eth1.txt 2>&1 &
tcpdump -lnni eth2 -s 96 'icmp and host 10.1.31.1' > /tmp/eth2.txt 2>&1 &
sleep 1; ping -c 3 10.1.31.1; sleep 2; kill %1 %2
cat /tmp/eth1.txt /tmp/eth2.txt 

4. Run the health check directly:
   python3 /root/health_checks/gateways_check.py basic

Expected: requests leave eth2 sourced from 10.1.31.2, replies return on eth2, and the check reports all gateways reachable.
Actual: requests leave eth2 sourced from 10.1.30.5; replies arrive on eth1, or are dropped entirely by a reverse-path-enforcing gateway, and the check reports 10.1.31.1 unreachable.

What to do about it?

Scope the elif rule so it does not match locally-generated packets:

elif cmdline.get_source_nat_ip() and not self.is_private_gateway():
    self.fw.append(["nat", "", "-A POSTROUTING -m addrtype ! --src-type LOCAL -j SNAT -o %s --to-source %s" % (self.dev, cmdline.get_source_nat_ip())])

! --src-type LOCAL excludes packets whose source is an address on the router, while forwarded guest traffic still matches and is SNATed as before.

Verified by replacing the rule manually on the VR:

iptables -t nat -D POSTROUTING -o eth2 -j SNAT --to-source 10.1.30.5
iptables -t nat -A POSTROUTING -o eth2 -m addrtype ! --src-type LOCAL -j SNAT --to-source 10.1.30.5

Behaviour	Before	After
VR → 10.1.31.1	10.1.30.5 > 10.1.31.1, reply on eth1 / dropped	10.1.31.2 > 10.1.31.1, reply on eth2
Guest → internet	SNAT to 10.1.30.5 via eth1	unchanged

A CIDR-based scope (-s ) was considered and rejected: it fails when the public ranges fall inside the VPC CIDR.

Workaround for affected deployments — a manual iptables edit is reverted by CloudStack on any config refresh, so suppress the check instead:

cmk update configuration name=router.health.checks.to.exclude value=gateways_check.py zoneid=<zone-uuid>
cmk list configurations name=router.health.checks.failures.to.recreate.vr   # ensure gateways_check.py is NOT listed