Repository navigation
check for live devices in unload_drivers_without_physical_devices - #2103
Conversation
unload_drivers_without_physical_devices runs after every vkEnumeratePhysicalDevices call that fills an array. It unloads each driver whose latest enumeration produced no physical devices: it destroys the driver's VkInstance, frees the loader_icd_term through loader_icd_destroy and closes the driver library. loader_icd_destroy also frees every loader_device on the driver's logical_device_list. physical_device_count is recomputed on every enumeration, so a driver that reported a device earlier and reports none now is unloaded while the application still holds a VkDevice created from that device. The VkDevice, its queues and command buffers dispatch through the freed loader_device, so the next device-level call reads the freed dispatch table and calls a function pointer taken from it. Skip drivers that still have logical devices. Such a driver is unloaded by a later enumeration once its devices are destroyed, or at instance destruction. Add DriverUnloadingFromZeroPhysDevs.DriverWithLiveDeviceNotUnloaded. On the unpatched loader AddressSanitizer reports a heap-use-after-free in vkGetDeviceQueue; it passes with the change.
|
User aizu-m not on autobuild list. Waiting for curator authorization before starting CI build. |
1 similar comment
|
User aizu-m not on autobuild list. Waiting for curator authorization before starting CI build. |
charles-lunarg
left a comment
There was a problem hiding this comment.
While real world drivers probably won't do this, it is something that could happen (even if there isn't much support for hot-reloading in vulkan, it isn't explicitly unsupported) so the loader should be robust to changes.
Added suggested change for what should fix the CI errors.
|
CI Vulkan-Loader build queued with queue ID 188632. |
|
CI Vulkan-Loader build # 3979 running. |
|
CI Vulkan-Loader build # 3979 failed. |
|
CI Vulkan-Loader build queued with queue ID 191637. |
|
CI Vulkan-Loader build queued with queue ID 191654. |
|
CI Vulkan-Loader build # 3983 running. |
|
CI Vulkan-Loader build # 3983 passed. |
|
Macos CI failed because the macos-14 runners were deprecated. I have #2105 to fix it, but I'd rather merge this now to avoid needless rerunning of CI. |
AddressSanitizer, macOS Debug build with
LOADER_ENABLE_ADDRESS_SANITIZER=ON, new test run against the unpatched loader:Found while checking what a second
vkEnumeratePhysicalDevicescall frees. The existingDriverUnloadingFromZeroPhysDevstests only cover drivers that never had a device.unload_drivers_without_physical_devicesruns after every enumeration that fills an array. It unloads any driver whose latest enumeration gave no devices:vkDestroyInstanceon the driver,loader_icd_destroy, thendlclose.loader_icd_destroyalso frees everyloader_deviceon the driver'slogical_device_list.physical_device_countis recomputed on each call. A driver that reported a device before and reports none now (an enumeration error, a lost or unplugged GPU) is unloaded while the app still holds aVkDevicefrom it. ThatVkDevice, its queues and its command buffers all dispatch through the freedloader_device. The next device call reads the freed table and calls a function pointer taken from it, into a driver library that has been closed. A plainvkDestroyDeviceis enough to hit it.The fix skips drivers that still have logical devices. Such a driver is unloaded by a later enumeration once its devices are gone, or at instance destruction. The check sits in the unload helper because that is the only place the loader decides a driver is unused.
DriverUnloadingFromZeroPhysDevs.DriverWithLiveDeviceNotUnloadedcreates aVkDeviceon each of two drivers, makes one driver failvkEnumeratePhysicalDevices, re-enumerates and callsvkGetDeviceQueueon both devices. It fails as above before the change and passes after. The full ctest suite passes under ASan, exceptintegration.find_package, which fails the same way on an untouched tree here because of a local SDK issue.One related problem with the same trigger is left alone.
loader_device::phys_dev_termstill points at the terminator object thatsetup_loader_term_phys_devsfrees when the device drops out of enumeration.loader_lookup_device_dispatch_tablereads it onvkGetDeviceProcAddr, so ASan reports a second use-after-free if the test also callsvkGetDeviceProcAddr(dev, "vkCreateBuffer")on the orphaned device. Fixing that means changing the generated code, so it fits better as its own change.