bug: ALB reports an HTTP 503 IP target as healthy despite a 200 health-check matcher

[AndreKurait]

Existing report search

I searched LocalStack's current Discussions and older issue tracker for ALB/IP-target health and unhealthy-target reports. I found no matching current report. Historical Moto issues about stopped, terminated, or deregistered instance targets are a different case; this reproduction uses a reachable IP target serving HTTP 503.

Current and expected behavior

An active application load balancer has a listener forwarding to an IP target group with:

• HTTP on port 8080, health path /healthcheck;
• success matcher 200;
• interval 5 seconds, timeout 2 seconds;
• healthy and unhealthy thresholds both 2.

The target returns HTTP 503, including direct requests from the unmodified LocalStack parent container before and after the observation. Nevertheless, DescribeTargetHealth reports healthy in all six samples at approximately 0, 5, 10, 15, 20 and 25 seconds. The final API readbacks confirm the active ALB, listener association and specified health-check settings.

Expected: repeated unsuccessful checks should produce unhealthy, normally Target.ResponseCodeMismatch for this status-code mismatch, instead of accepting the target as healthy. See the AWS ALB health-check contract.

This is a bounded 25-second health-state observation. It does not qualify request forwarding, automatic recovery, HTTPS/gRPC, instance termination, or every timing/threshold combination. No live AWS comparison was performed. If actual periodic HTTP checks are intentionally unsupported, please classify this as the corresponding missing capability and document the health-state limitation.

Measured versions

LocalStack image	Bundled moto-ext	Result
2026.7.5	5.1.46	Healthy at all six samples through 25.005s; actual HTTP503 before/after
2026.8.1	5.1.48	Healthy at all six samples through 25.004s; actual HTTP503 before/after

Repository digests:

• 2026.7.5: sha256:0d74e1d2d7ce13a3cb25fc64cf15eb225f1c95c762e56e057bc6a9ed0ed29306
• 2026.8.1: sha256:4aef81c531684570d7b3cfd2805afa02194c929d52bbeedabb7d4874798b1572

Measured on 2026-10-08. These are two specific version comparisons, not a claim about the latest release. The identical Python client and target fixture below were used in both final runs.

Both runs use Linux ARM64, the normal vendor entrypoint, dummy AWS credentials and a fresh isolated Docker network. There are no custom extensions, source patches, application packages or Docker socket mounts. The only target is the complete stdlib HTTP fixture below. The API client uses boto3/botocore 1.43.101. Owned API resources, parent, target and network are removed after each run.

The target is explicitly assigned a usable RFC1918 address (10.254.200.10 in the example), outside the first four and last addresses of its modeled subnet. The parent /24 is split into two /25 ALB subnets. Choose an unused nonoverlapping private network on your host. The script refuses reserved-address and too-small-subnet inputs.

Reproduction

Save both Python files below. Use a host Python environment with boto3 installed, Docker, and your normal LocalStack license in LOCALSTACK_AUTH_TOKEN. The token is passed by environment name; do not put it in a report or command transcript.

1. Set a unique name prefix with prefix="ls-alb-health-$(python3 -c 'import uuid; print(uuid.uuid4().hex[:12])')", then create a disposable Docker network with an unused /24, for example:
   docker network create --subnet 10.254.200.0/24 "${prefix}-net".
2. Start the target on that network at .10, without ports or mounts:
   docker run -d --name "${prefix}-target" --network "${prefix}-net" --ip 10.254.200.10 python:3.12-slim python -u -c "$(cat alb_http_503_sidecar.py)".
3. Start an unmodified LocalStack parent on the same network, using the chosen pinned image from the table, with an unused loopback edge port:
   docker run -d --name "${prefix}-parent" --network "${prefix}-net" -p 127.0.0.1:4589:4566 --env LOCALSTACK_AUTH_TOKEN -e DNS_ADDRESS=0 -e EC2_DOWNLOAD_DEFAULT_IMAGES=0 --tmpfs /var/lib/localstack localstack/localstack-pro:2026.7.5.
4. Wait for LocalStack readiness. From inside the parent container, run the connectivity snippet below. It must report HTTP503 and the fixture body before proceeding.
5. Run:
   python3 alb_ip_target_http_health.py --endpoint-url http://127.0.0.1:4589 --target-ip 10.254.200.10 --network-cidr 10.254.200.0/24.
6. Repeat the same parent-to-target connectivity snippet after the client. Inspect its final JSON: configuration_confirmed=true, observed_unhealthy=false, all six states healthy, and every API cleanup entry successful. Exit0 means the observation completed, not that expected health behavior passed.
7. In a shell cleanup trap or finally, remove only these owned containers with docker rm -f "${prefix}-parent" "${prefix}-target", then docker network rm "${prefix}-net". The random prefix keeps cleanup scoped to this invocation. The Python client's finally removes the synthetic API resources; the outer cleanup remains necessary on any setup/client failure.

Connectivity snippet (run before and after):

docker exec -i "${prefix}-parent" /opt/code/localstack/.venv/bin/python - <<'CHECK'
import json
from urllib.error import HTTPError
from urllib.request import ProxyHandler, build_opener
url = "http://10.254.200.10:8080/healthcheck"
try:
    response = build_opener(ProxyHandler({})).open(url, timeout=5)
except HTTPError as error:
    response = error
with response:
    body = response.read().decode()
    print(json.dumps({"http_status": response.code, "body": body}))
    assert response.code == 503 and body == "deliberate-unhealthy-target\n"
CHECK

The expected observed health shape is:

{"Target":{"Id":"10.254.200.10","Port":8080},"HealthCheckPort":"traffic-port","TargetHealth":{"State":"healthy"}}

alb_http_503_sidecar.py

"""Disposable public ALB probe target; run with stdlib Python only."""
from http.server import BaseHTTPRequestHandler, HTTPServer


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        body = b"deliberate-unhealthy-target\n"
        self.send_response(503 if self.path == "/healthcheck" else 404)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass


HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()

alb_ip_target_http_health.py

#!/usr/bin/env python3
"""Observe public ALB IP-target health against a separately verified HTTP 503."""
import argparse
import ipaddress
import json
import time
from urllib.parse import urlsplit

import boto3
from botocore.config import Config


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--endpoint-url", required=True)
    parser.add_argument("--target-ip", required=True)
    parser.add_argument("--network-cidr", required=True)
    args = parser.parse_args()
    endpoint = urlsplit(args.endpoint_url)
    if (
        endpoint.scheme != "http"
        or endpoint.hostname not in {"127.0.0.1", "localhost", "::1"}
        or endpoint.username or endpoint.password or endpoint.query
        or endpoint.fragment or endpoint.path not in {"", "/"}
    ):
        parser.error("use the owned stock parent's loopback HTTP endpoint")
    network = ipaddress.ip_network(args.network_cidr)
    target_ip = ipaddress.ip_address(args.target_ip)
    rfc1918 = tuple(ipaddress.ip_network(cidr) for cidr in (
        "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16",
    ))
    if (
        network.version != 4
        or not any(network.subnet_of(allowed) for allowed in rfc1918)
        or not 16 <= network.prefixlen <= 26
        or target_ip not in network
    ):
        parser.error("use the owned RFC1918 IPv4 Docker network and its target")
    children = list(network.subnets(new_prefix=max(24, network.prefixlen + 1)))
    target_subnet = next(child for child in children if target_ip in child)
    other_subnet = next(child for child in children if child != target_subnet)
    if (
        int(target_ip) - int(target_subnet.network_address) < 4
        or target_ip == target_subnet.broadcast_address
    ):
        parser.error("target must not use the modeled AWS subnet's reserved addresses")
    options = {
        "endpoint_url": args.endpoint_url,
        "region_name": "us-east-1",
        "aws_access_key_id": "test",
        "aws_secret_access_key": "test",
        "config": Config(
            proxies={}, retries={"max_attempts": 0},
            connect_timeout=3, read_timeout=10,
        ),
    }
    ec2 = boto3.client("ec2", **options)
    elb = boto3.client("elbv2", **options)
    vpc = group = lb = listener = security_group = None
    subnets = []
    samples = []
    cleanup = []
    result = {
        "mode": "public loopback API; IP target; no source adapters",
        "expected": "The reachable HTTP 503 target becomes unhealthy with Matcher=200.",
        "target": {"Id": str(target_ip), "Port": 8080},
        "parent_to_target_http_evidence": "Required separately from the stock parent network namespace.",
    }
    try:
        vpc = ec2.create_vpc(CidrBlock=str(network))["Vpc"]["VpcId"]
        for cidr, zone in (
            (target_subnet, "us-east-1a"), (other_subnet, "us-east-1b")
        ):
            subnets.append(ec2.create_subnet(
                VpcId=vpc, CidrBlock=str(cidr), AvailabilityZone=zone,
            )["Subnet"]["SubnetId"])
        security_group = ec2.create_security_group(
            VpcId=vpc, GroupName="ip-health-probe",
            Description="Disposable ALB health probe",
        )["GroupId"]
        lb = elb.create_load_balancer(
            Name="ip-health-probe", Type="application", Scheme="internal",
            Subnets=subnets, SecurityGroups=[security_group],
        )["LoadBalancers"][0]["LoadBalancerArn"]
        group = elb.create_target_group(
            Name="ip-health-probe", Protocol="HTTP", Port=8080,
            VpcId=vpc, TargetType="ip", HealthCheckEnabled=True,
            HealthCheckProtocol="HTTP", HealthCheckPort="traffic-port",
            HealthCheckPath="/healthcheck", HealthCheckIntervalSeconds=5,
            HealthCheckTimeoutSeconds=2, HealthyThresholdCount=2,
            UnhealthyThresholdCount=2, Matcher={"HttpCode": "200"},
        )["TargetGroups"][0]["TargetGroupArn"]
        listener = elb.create_listener(
            LoadBalancerArn=lb, Protocol="HTTP", Port=80,
            DefaultActions=[{"Type": "forward", "TargetGroupArn": group}],
        )["Listeners"][0]["ListenerArn"]
        target = result["target"]
        elb.register_targets(TargetGroupArn=group, Targets=[target])
        started = time.monotonic()
        for offset in (0, 5, 10, 15, 20, 25):
            remaining = started + offset - time.monotonic()
            if remaining > 0:
                time.sleep(remaining)
            health = elb.describe_target_health(
                TargetGroupArn=group, Targets=[target]
            )["TargetHealthDescriptions"]
            sample = {
                "elapsed_seconds": round(time.monotonic() - started, 3),
                "target_health": health,
            }
            samples.append(sample)
            print(json.dumps({"sample": sample}), flush=True)
        result["load_balancer"] = elb.describe_load_balancers(
            LoadBalancerArns=[lb]
        )["LoadBalancers"][0]
        result["listener"] = elb.describe_listeners(
            ListenerArns=[listener]
        )["Listeners"][0]
        result["target_group"] = elb.describe_target_groups(
            TargetGroupArns=[group]
        )["TargetGroups"][0]
        result["samples"] = samples
        config = result["target_group"]
        result["configuration_confirmed"] = (
            result["load_balancer"]["State"]["Code"] == "active"
            and len(result["load_balancer"]["AvailabilityZones"]) == 2
            and result["listener"]["DefaultActions"][0]["TargetGroupArn"] == group
            and config.get("LoadBalancerArns") == [lb]
            and config["TargetType"] == "ip"
            and config["HealthCheckEnabled"] is True
            and config["HealthCheckPath"] == "/healthcheck"
            and config["HealthCheckIntervalSeconds"] == 5
            and config["UnhealthyThresholdCount"] == 2
            and config["Matcher"]["HttpCode"] == "200"
        )
        last = samples[-1]["target_health"]
        result["observed_unhealthy"] = (
            len(last) == 1 and last[0]["TargetHealth"]["State"] == "unhealthy"
        )
        result["interpretation"] = (
            "Interpret only after matching separate parent-to-target 503 evidence "
            "and confirmed active configuration. No instance-lifecycle claim."
        )
    finally:
        # This does not own or stop the stock parent or the target sidecar.
        actions = [
            ("listener", lambda: elb.delete_listener(ListenerArn=listener), listener),
            ("load_balancer", lambda: elb.delete_load_balancer(LoadBalancerArn=lb), lb),
            ("target_group", lambda: elb.delete_target_group(TargetGroupArn=group), group),
        ]
        actions.extend(
            ("subnet", lambda subnet=subnet: ec2.delete_subnet(SubnetId=subnet), subnet)
            for subnet in subnets
        )
        actions.extend([
            ("security_group", lambda: ec2.delete_security_group(GroupId=security_group), security_group),
            ("vpc", lambda: ec2.delete_vpc(VpcId=vpc), vpc),
        ])
        for kind, action, identity in actions:
            if identity:
                try:
                    action()
                    cleanup.append({"type": kind, "id": identity, "deleted": True})
                except Exception as error:
                    cleanup.append({
                        "type": kind, "id": identity, "deleted": False,
                        "error_type": type(error).__name__, "message": str(error),
                    })
        result["cleanup"] = cleanup
        print(json.dumps({"result": result}, indent=2, sort_keys=True, default=str), flush=True)
    # A behavior mismatch is retained in JSON; setup exceptions remain failures.


if __name__ == "__main__":
    main()

[ryan-berke]

Hi @AndreKurait, thanks for this report as well. I have escalated this to my engineers, so I will notify you as soon as a fix is ready. If there is anything else I can do for you, please let me know.