vkproxy

A split Vulkan loader. Lets a Vulkan client run on a GPU whose user-space driver can't be loaded into that client's process — for example because the driver is a Bionic-only blob expecting the AOSP hwvulkan HAL ABI while the client is a glibc process, or because the driver lives on a different machine entirely.

vkproxy splits the Vulkan stack into two halves connected by a small RPC:

            Vulkan client process                     Server process
   ┌──────────────────────────────┐         ┌─────────────────────────────┐
   │  app / engine                │         │                             │
   │     │                        │         │   husky_vkserver            │
   │     ▼                        │         │   (dlopen's the real        │
   │  libvulkan loader            │   UDS   │    Vulkan driver and        │
   │     │                        │◄───────►│    forwards every call)     │
   │     ▼                        │  (or    │                             │
   │  mali_proxy_icd.so           │   TCP)  │                             │
   │  (ICD shim; marshals every   │         │                             │
   │   call onto the wire)        │         │                             │
   └──────────────────────────────┘         └──────────────┬──────────────┘
                                                           │
                                                  ┌────────▼────────┐
                                                  │ real GPU driver │
                                                  │ (e.g. Mali on   │
                                                  │  /dev/mali0)    │
                                                  └─────────────────┘

Every Vulkan call goes:

1. The Vulkan loader on the client side picks mali_proxy_icd.so based on the ICD JSON manifest (mali_proxy_icd.json lives under /usr/local/share/vulkan/icd.d/).
2. The ICD serialises the call's arguments into a wire format and writes them to /tmp/husky-vk.sock (UDS) or a TCP socket if VKPROXY_REMOTE is set.
3. husky_vkserver reads the request, calls the real driver, writes the reply back.

There are two server binaries from the same dispatch core:

• husky_vkserver — Bionic / aarch64. Built with the Android NDK, runs in a Halium-style chroot where the Mali blob's expected libc and binder are available, dlopen's libGLES_mali.so.
• husky_vkserver_pc — host glibc / x86_64. Talks TCP and dlopen's a stock libvulkan.so.1. Useful for development without a Mali host — the wire protocol is the same.

Source tree

vkproxy/
├── Makefile                       # builds server (NDK), ICD (aarch64-gcc), smoke client, PC server
├── codegen/
│   ├── gen.py                     # parses vk.xml; emits opcode enum, client stubs, server dispatch, fn table
│   └── vk.xml                     # the Vulkan registry
├── include/                       # headers shared between client & server
│   ├── vkproxy_proto.h            # wire framing: vkp_cmd_hdr, vkp_reply_hdr, SYNTH opcode enum
│   ├── vkproxy_proto_gen.h        # codegen'd: every Vulkan command -> opcode id
│   ├── transport.h                # client-side socket API (vkp_send_cmd / vkp_call / fd-passing)
│   ├── udp_frame.h                # optional UDP frame-blob channel (for remote presents)
│   └── hwvulkan.h                 # subset of AOSP <hardware/hwvulkan.h> the server needs
├── protocol/                      # Wayland protocol bindings (xdg-shell, linux-dmabuf), generated
├── client/                        # mali_proxy_icd.so — the ICD the Vulkan loader loads
│   ├── mali_proxy_icd.json        # ICD manifest installed to /usr/local/share/vulkan/icd.d/
│   ├── icd_entry.c                # vk_icd* entry points the loader calls first
│   ├── stubs_gen.c                # codegen'd: every Vulkan command -> marshal -> vkp_send_cmd/vkp_call
│   ├── manual_stubs.c             # hand-written overrides for everything codegen can't do
│   │                                (vkCreateInstance, vkCreateDevice, vkGetPhys*Properties,
│   │                                 the actually-complex pipeline/descriptor commands, swapchain,
│   │                                 etc.) plus _vkp_manual_override which vkGetDeviceProcAddr
│   │                                must consult before _vkproxy_lookup
│   ├── dispatch.h                 # vkp_dispatchable / handle-translation helpers
│   ├── transport.c                # AF_UNIX (or TCP via VKPROXY_REMOTE) socket plumbing
│   ├── wl_present.c/.h            # Wayland present path (wl_display, wl_buffer construction)
│   ├── x_wl_surface.c             # xdg-shell shim used when running under Xwayland
│   ├── udp_frame.c                # UDP frame-blob receive plumbing (PC build only)
│   ├── udp_shm_present.c/.h       # SHM presentation when frames arrive as raw bytes
│   ├── decode_h264.c/.h           # H.264 decode path for PC mode (server side encodes)
│   └── udp_frame.h                # local copy of the wire header
├── server/                        # husky_vkserver (Bionic) + husky_vkserver_pc (glibc)
│   ├── main.c                     # vksrv main: AF_UNIX listener, per-client thread, signal handler
│   ├── main_pc.c                  # PC variant: TCP listener, no Bionic/AHB paths
│   ├── vk_init.c / vk_init.h      # dlopen libGLES_mali.so, get hwvulkan_device_t, populate g_pfn
│   ├── vk_init_pc.c               # PC variant: dlopen libvulkan.so.1, vkGetInstance/DeviceProcAddr
│   ├── vk_funcs_gen.c/.h          # codegen'd: g_pfn struct of every Vulkan function pointer + populator
│   ├── dispatch_gen.c             # codegen'd: case OP_vkXxx -> unmarshal args -> call g_pfn.vkXxx -> reply
│   ├── dispatch_manual.c          # hand-written dispatch for OPs that need server-side state
│   │                                (handle table, BC-texture transcode staging, dmabuf alloc via
│   │                                 dma_heap, MMU/registry quirks, etc.) — by far the biggest file
│   ├── handle_table.c/.h          # 64-bit cookie ↔ dispatchable-handle map (locked, growable)
│   ├── encode_h264.c/.h           # PC build only: encode rendered frames for the remote viewer
│   ├── udp_frame.c/.h             # UDP send side of frame chunks
│   ├── logsink_inline.c           # one-line LOG() macro that writes to stderr (journald in chroot)
│   └── pc_stubs/ue_husky.h        # tiny stub of ue-husky for the PC build (no BC transcoder there)
├── smoke/                         # smoke client used to validate the link layer
│   ├── vksmoke.c                  # connect, OP_vkp_hello, print server's "name/api/ext_count"
│   ├── logsink.c                  # same LOG() macro
│   ├── hwvulkan.h                 # local copy
│   └── Makefile
└── tests/                         # standalone Vulkan triangle/cube tests
    ├── mincube.c / mincube2.c     # raw Vulkan + Wayland present (the latter does textured cube)
    ├── mincube2.{vert,frag}       # GLSL source
    ├── mincube2_{vert,frag}_spv.h # the same as SPIR-V byte arrays
    ├── vkp_hello_test.c           # bare vkp_hello round-trip
    └── xdg-shell-{client-protocol.h,protocol.c}   # generated Wayland boilerplate (local copy)

Wire protocol

Both directions use the same simple framing:

client → server                            server → client
┌────────────────────────────────┐        ┌──────────────────────────────────┐
│ struct vkp_cmd_hdr             │        │ struct vkp_cmd_hdr (echo opcode) │
│   uint16_t opcode              │        │   flags |= VKP_FLAG_REPLY        │
│   uint16_t flags               │        ├──────────────────────────────────┤
│   uint32_t len                 │        │ struct vkp_reply_hdr             │
├────────────────────────────────┤        │   int32_t  status (VkResult)     │
│ payload (len bytes)            │        │   uint32_t len                   │
└────────────────────────────────┘        ├──────────────────────────────────┤
                                          │ payload (len bytes)              │
                                          └──────────────────────────────────┘

flags:

bit	name	meaning
0	VKP_FLAG_EXPECTS_REPLY	client wants a synchronous reply (used by vkp_call)
1	VKP_FLAG_REPLY	the frame is a reply (server -> client direction)
2	VKP_FLAG_HAS_FD	one fd is being passed via SCM_RIGHTS on this datagram

Opcode space:

Range	Meaning
0x0000 – 0x7FFF	One-to-one mirror of a real Vulkan command — generated by gen.py
0x8000 – 0xFFFF	Synthesised / non-Vulkan ops (OP_vkp_*) — handled by dispatch_manual.c

The synthesised range covers things that need server-side state or custom marshalling:

• OP_vkp_hello — protocol handshake; returns the GPU's device name, Vulkan API version, and instance extension count.
• OP_vkp_create_* / OP_vkp_destroy_* for instance, device, queue, command pool/buffer, image, buffer, image view, sampler, etc. The manual server side maintains the handle table and BC-transcode staging registry alongside.
• OP_vkp_map_memory / OP_vkp_unmap_memory — coordinate dmabuf-backed vkAllocateMemory, including sending the heap fd back via SCM_RIGHTS so the client can mmap the same memory.
• OP_vkp_cmd_pipeline_barrier, OP_vkp_cmd_copy_buffer_to_image, OP_vkp_cmd_copy_image_to_buffer, etc. — command recording that needs ht_get on the dispatchable cmdbuf handle (codegen doesn't know which arg is dispatchable).
• OP_vkp_get_buffer_device_addr, OP_vkp_get_phys_mem_props, and other "spoofable" property queries.

How a call flows

Take vkCmdBindIndexBuffer as the worked example:

1. The Vulkan loader inside the client process loads libvulkan.so.1, scans /usr/local/share/vulkan/icd.d/, finds mali_proxy_icd.json, dlopens mali_proxy_icd.so, then calls vk_icdGetInstanceProcAddr (in icd_entry.c) for every entry point. icd_entry.c calls _vkproxy_lookup, which walks the codegen'd table in stubs_gen.c. For overrides that need hand work, manual_stubs.c's _vkp_manual_override is consulted first inside the ICD's own vkGetDeviceProcAddr / vkGetInstanceProcAddr.
2. The client records its command-buffer call. The function pointer it got is one of these:
   • the codegen stub in stubs_gen.c — marshals scalar args, sends the opcode via vkp_send_cmd (fire-and-forget), no reply.
   • a manual implementation in manual_stubs.c — same protocol, but handles things gen.py doesn't (variable-length arrays, pNext chains, dispatchable-handle translation, ABI mismatches between spec versions).
3. vkp_send_cmd (in client/transport.c) takes the global socket lock, writes vkp_cmd_hdr + payload to the socket, and returns. For queries that need a reply, vkp_call blocks until the server replies.
4. husky_vkserver's per-client thread reads the header, grows the payload buffer if needed, then dispatches:
   • opcode < VKP_OP_BASE_SYNTH (0x8000): try vkp_dispatch_manual first (it owns a fixed list of opcodes that need handle translation), then fall back to the codegen dispatcher vkp_dispatch in dispatch_gen.c.
   • opcode ≥ 0x8000: vkp_dispatch_manual only (handles all OP_vkp_*).
5. The chosen handler unmarshals into stack-allocated Vulkan structs and calls g_pfn.vkCmdBindIndexBuffer — a real pointer obtained at bring-up by walking the driver's GetInstanceProcAddr. The handle table (handle_table.c) translates the client-side cookie for VkCommandBuffer into the driver's actual dispatchable handle.
6. For commands that produce a reply (vkAllocateMemory, vkCreateBuffer, the vkGet*Properties family), the handler calls vkp_send_reply to ship the returned VkResult + result struct back to the client.

Codegen (codegen/gen.py)

Driven from upstream vk.xml. Walks every <command> element and classifies its args:

• Scalars / by-value handles → emit a 1:1 marshal into a packed struct.
• Pointers to small fixed-size structs → emit a struct-copy.
• Anything else (variable-length arrays, pNext chains, ambiguous unions, dispatchable handles, sType-based polymorphism) → emit a vkp_not_implemented stub. Those commands are picked up by manual_stubs.c / dispatch_manual.c and registered in _vkp_manual_override so the lookup chain finds the hand-written version first.

Outputs:

• include/vkproxy_proto_gen.h — enum vkp_opcodes { OP_vkXxx = N, ... }
• client/stubs_gen.c — every VKAPI_ATTR ... VKAPI_CALL vkXxx(...) plus a name → fn-pointer table _vkp_stubs[] consulted by _vkproxy_lookup.
• server/dispatch_gen.c — vkp_dispatch() is one big switch over opcodes that unmarshals and calls g_pfn.vkXxx.
• server/vk_funcs_gen.c/.h — struct vk_funcs g_pfn { PFN_vk...; } plus a populator that calls GetInstanceProcAddr for every member.

Regenerate: python3 codegen/gen.py.

Build

make                # builds husky_vkserver (Android Bionic), mali_proxy_icd.so (glibc aarch64),
                    # and vkp_hello_test (smoke client). Output in build/.

make pc             # builds husky_vkserver_pc (host x86_64 glibc) against a runtime-dlopen'd
                    # libvulkan.so.1. No ue-husky link — texture substitution is Mali-only.

The Makefile splits compilers:

• CC_AND = aarch64-linux-android29-clang from $NDK/toolchains/... for the Bionic server.
• CC_ARM64 = aarch64-linux-gnu-gcc for the glibc ICD and smoke client.
• CC_HOST = gcc for the host-side PC build.

The ICD links with -Wl,-Bsymbolic so internal references to vk* symbols bind to the ICD's own copies; otherwise the Vulkan loader (which is loaded first into the process) would interpose, cause infinite recursion, and reject the ICD.

Deploy

Target	Path on the GPU-side machine
husky_vkserver	/var/lib/machines/halium/data/local/tmp/husky_vkserver (runs in chroot)
libue_husky.so (sibling)	same dir, beside husky_vkserver (rpath $ORIGIN)
mali_proxy_icd.so	/usr/local/lib/vkproxy/mali_proxy_icd.so
mali_proxy_icd.json	/usr/local/share/vulkan/icd.d/mali_proxy_icd.json

The server is run as a systemd unit (husky-vkserver.service). The unit's ExecStart chroot-execs it and pipes stderr into /tmp/vksrv.log inside the chroot.

A Vulkan client process picks up the ICD via:

export VK_ICD_FILENAMES="/usr/local/share/vulkan/icd.d/mali_proxy_icd.json"
export VK_LOADER_DRIVERS_SELECT="mali_proxy_icd.json"

Runtime knobs

Variable	Side	Effect
VKPROXY_SOCKET	server	UDS path (default /tmp/husky-vk.sock)
VKPROXY_LISTEN	PC	host:port for the TCP variant
VKPROXY_REMOTE	client	If set, dial TCP instead of UDS
UEHUSKY_VERBOSE	server	Turn on verbose op-by-op logging
VKPROXY_SPOOF_SM6	client	Spoof shaderInt64/Float64/Int16 (for SM6 / D3D12). Off by default.
VK_LOADER_DEBUG=all	loader	Khronos loader trace; useful for "why didn't this ICD load"

Debugging

husky_vkserver's SIGSEGV/SIGABRT/SIGBUS/SIGILL handler (main.c):

1. Writes *** FATAL SIGNAL *** and sig=N tid=T code=K addr=A pc=P.
2. Walks the aarch64 frame-pointer chain via the ucontext_t and prints up to 20 caller PCs.
3. Dumps /proc/self/maps.
4. Re-raises with SIG_DFL so systemd still sees a non-zero exit.

To resolve a PC to a library:offset, find the maps line whose VA range covers the PC, then RVA_in_file = pc - va_start + file_offset. Open the .so in IDA at its ImageBase and jump to that RVA.

Known sharp edges (Mali backend)

These apply only when the back-end driver is Mali's Valhall blob. The proxy itself is driver-agnostic; everything below is upstream-driver behaviour worked around in dispatch_manual.c or the kernel module.

• Mali user-space race: libGLES_mali.so:sub_1A65300 walks a registry inside a pthread_mutex_t it doesn't itself create. Two concurrent server threads can observe a partially-NULL'd registry pointer and dereference NULL + 0xC8. Mitigation: serialise all dispatch through a single mutex in main.c. Lower throughput, no crash.
• Mali kbase CONFIG_MALI_DMA_BUF_MAP_ON_DEMAND=y (kernel side): imports do not eager-map dma_bufs into the GPU page table; the fault handler refuses to demand-map and kills the context. Patch the kernel module to #undef CONFIG_MALI_DMA_BUF_MAP_ON_DEMAND in mali_kbase_mem.c and mali_kbase_mem_linux.c.
• -Wl,-Bsymbolic is mandatory on the ICD or the loader rejects it with an infinite-recursion error.
• GS/tess on Mali-G715: advertised but draw-time hangs the MCU. Strip in vkCreateDevice feature filtering on the server.