recvmsg: truncated SCM_RIGHTS control data (MSG_CTRUNC) panics in AncillaryDrain and again in Drop on macOS

[xrl]

On macOS, when recvmsg truncates SCM_RIGHTS control data because the RecvAncillaryBuffer is too small, the kernel sets MSG_CTRUNC but leaves each cmsg_len holding its untruncated value. AncillaryDrain::advance then subtracts that length from the buffer's remaining length, which underflows. Drop for RecvAncillaryBuffer drains again, hits the now-corrupt read offset, and the second panic aborts the process.

Linux is unaffected: scm_detach_fds reduces cmsg_len to the number of descriptors it actually copied.

Platform / versions

• macOS 15.6 (Darwin 25.6.0), aarch64-apple-darwin
• rustix 1.1.4 and 1.1.5 (identical src/net/send_recv/msg.rs), libc backend
• rustc 1.98.1

What the kernel hands back

A C probe, sending 16 fds over a socketpair and receiving into CMSG_SPACE(5 * sizeof(int)) = 32 bytes:

recvmsg n=5 msg_flags=0x20 (MSG_CTRUNC) msg_controllen=32 (buffer was 32)
cmsg_len=76 cmsg_level=65535 cmsg_type=1
payload bytes available in buffer: 20 -> 5 whole fds: 21 22 23 24 25
CMSG_NXTHDR = 0x0

cmsg_len is 76 — CMSG_LEN(16 * 4) — against 32 bytes of buffer. msg_controllen correctly reports the 32 bytes that were written, so the initialized region is known; it's cmsg_len that lies.

(Separately, and not rustix's problem: XNU installs all 16 descriptors into the receiver's file table before copying out, so the 11 whose numbers never reach userspace are leaked by the kernel with no way to close them.)

The panic

// src/net/send_recv/msg.rs
fn advance(
    read_and_length: &mut Option<(&'buf mut usize, &'buf mut usize)>,
    msg: &c::cmsghdr,
) -> Option<RecvAncillaryMessage<'buf>> {
    if let Some((read, length)) = read_and_length {
        let msg_len = msg.cmsg_len as usize;
        **read += msg_len;
        **length -= msg_len;   // 36 - 76
    }
    ...

Running a recvmsg of 16 ScmRights into a cmsg_space!(ScmRights(5)) buffer on rustix main (287214b):

thread 'cmsg::test_truncated_scm_rights' panicked at src/net/send_recv/msg.rs:544:13:
attempt to subtract with overflow

thread 'cmsg::test_truncated_scm_rights' panicked at src/net/send_recv/msg.rs:484:63:
range start index 76 out of range for slice of length 36

thread 'cmsg::test_truncated_scm_rights' panicked at library/core/src/panicking.rs:233:5:
panic in a destructor during cleanup
thread caused non-unwinding panic. aborting.

signal: 6, SIGABRT: process abort signal

Line 544 is **length -= msg_len. Line 484 is RecvAncillaryBuffer::drain's &mut self.buffer[self.read..][..self.length], reached from clear (478) from Drop (492) while unwinding the first panic. Hence the abort.

With overflow checks off it's worse

In a release build the subtraction wraps instead of panicking, and cvt_msg goes on to build the payload slice from the same untruncated cmsg_len:

let payload_len = msg.cmsg_len as usize - c::CMSG_LEN(0) as usize;   // 64
let payload: &'buf mut [u8] = slice::from_raw_parts_mut(payload, payload_len);

64 bytes over a 24-byte payload region, read as OwnedFd. Instrumenting the same test in --release to print what it yielded:

yielded 14 fds: [21, 22, 23, 24, 25, 26, 0, 1811145888, 1, 36, 0, 76, 0, -40]

Six real descriptors followed by uninitialized bytes reinterpreted as descriptors — including 0 and 1. Each is an OwnedFd, so dropping them closes the process's stdin and stdout. That is an out-of-bounds read and arbitrary close() reachable from entirely safe code.

Why a user can't work around it

Checking ReturnFlags::CTRUNC before draining doesn't help, because Drop for RecvAncillaryBuffer calls clear → drain regardless. Once recvmsg returns with a truncated SCM_RIGHTS message in the buffer, the abort happens when the buffer goes out of scope. mem::forget on the buffer would dodge the drain but leak every descriptor. Sizing the buffer larger only moves the threshold — a peer can always send more descriptors than the buffer holds, so any process receiving fds from a less-trusted peer is one oversized message away from a remote abort.

Minimal repro

const NUM_FDS: usize = 16;
const NUM_SLOTS: usize = 5;

let (send_sock, recv_sock) =
    socketpair(AddressFamily::UNIX, SocketType::STREAM, SocketFlags::empty(), None).unwrap();

let fds: Vec<OwnedFd> = (0..NUM_FDS)
    .map(|_| socket(AddressFamily::UNIX, SocketType::STREAM, None).unwrap())
    .collect();
let borrowed: Vec<_> = fds.iter().map(AsFd::as_fd).collect();

let mut space = [MaybeUninit::uninit(); rustix::cmsg_space!(ScmRights(NUM_FDS))];
let mut cmsg_buffer = SendAncillaryBuffer::new(space.as_mut_slice());
assert!(cmsg_buffer.push(SendAncillaryMessage::ScmRights(&borrowed)));
sendmsg(&send_sock, &[IoSlice::new(b"hello")], &mut cmsg_buffer, SendFlags::empty()).unwrap();

let mut cmsg_space = [MaybeUninit::uninit(); rustix::cmsg_space!(ScmRights(NUM_SLOTS))];
let mut cmsg_buffer = RecvAncillaryBuffer::new(cmsg_space.as_mut_slice());
let mut buffer = [0_u8; 5];
let result = recvmsg(
    &recv_sock,
    &mut [IoSliceMut::new(&mut buffer)],
    &mut cmsg_buffer,
    RecvFlags::empty(),
)
.unwrap();

assert!(result.flags.contains(ReturnFlags::CTRUNC));

// aborts here on macOS
for _msg in cmsg_buffer.drain() {}

Found from a Unix-socket protocol that caps how many descriptors it will accept per frame and rejects oversized ones — the rejection path is exactly the path that aborts.

Fix

A PR follows: have Messages report the buffer space remaining at each header, clamp cmsg_len to it in advance, and size the payload in cvt_msg from the clamped length, rounding SCM_RIGHTS down to whole descriptors so the ones that did arrive still get closed rather than leaked. Platform-neutral, no new unsafe.

Disclosure: this report, and the change that follows it, were written by an AI coding agent (Claude, Anthropic) at my direction; I reviewed them. Please review with that in mind.

[xrl]

PR: #1684.