[Improvement] Replace Unsafe with Netty direct memory in ChunkedAllocationManager and expose metrics

[loserwang1024]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

The write accumulator uses Arrow's columnar serialization to build record batches. Arrow internally allocates native memory via an AllocationManager.Factory — in our case ChunkedAllocationManager, which packs small allocations into pre-allocated 4MB chunks using a bump-pointer strategy. Large allocations (≥ chunkSize) get their own dedicated memory region.

Before this change, the entire allocation/release stack relied on sun.misc.Unsafe:

• Chunk allocation: Unsafe.allocateMemory(4MB)
• Sub-allocation bump-pointer: offset from chunk's raw address
• Direct (large) allocation: Unsafe.allocateMemory(size)
• All releases: Unsafe.freeMemory(address)

This had several problems:

1. No visibility into memory usage — operators had no way to monitor how much native memory the accumulator was consuming at runtime.
2. Unsafe is a restricted API — sun.misc.Unsafe access is increasingly restricted in newer JDK versions (requires --add-opens, and critical methods were deprecated in JDK 23 via JEP 471).
3. No leak detection — Unsafe.freeMemory() offers no diagnostics. A missed release means native memory is lost until the process dies.
4. OOM is silent — when Unsafe.allocateMemory() fails, the OS may kill the process directly without any Java-level error or diagnostic trace.

Solution

• Expose memory metrics
• Replace Unsafe with Netty direct ByteBuf

No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!