diff --git a/USAGE_android.md b/USAGE_android.md index 79724982ae..41bc5850c3 100644 --- a/USAGE_android.md +++ b/USAGE_android.md @@ -120,14 +120,95 @@ the layer to return `VK_ERROR_INITIALIZATION_FAILED` from its The Vulkan API allows Vulkan memory objects to be mapped by an application for direct modification. To successfully capture an application, the GFXReconstruct layer must be able to -detect when the application modifies the mapped memory. - -The layer can be configured to detect memory modifications by marking the mapped -memory as write protected, triggering an access violation when the application -writes to the memory. -The layer then uses a signal handler to intercept the signal generated by the -access violation, where it removes the write protection, marks the modified -memory page as dirty, and allows the application to continue. +detect if the application modifies the mapped memory in order to dump the changes +in the capture file so that they can be re-applied while replaying. +To achieve this GFXR utilizes four different modes: + +##### 1. `assisted` +This mode expects the application to call `vkFlushMappedMemoryRanges` +after memory is modified; the memory ranges specified to the +`vkFlushMappedMemoryRanges` call will be written to the capture file +during the call. + +##### 2. `unassisted` +This mode writes the full content of mapped memory to the capture file +on calls to `vkUnmapMemory` and `vkQueueSubmit`. It is very inefficient +for performance and it will bloat capture file sizes. May be unusable +with real-world applications that map large amounts of memory. + +##### 3. `page_guard` +`page_guard` tracks modifications to individual memory pages, which are +written to the capture file on calls to `vkFlushMappedMemoryRanges`, +`vkUnmapMemory`, and `vkQueueSubmit`. This method requires allocating +shadow memory for all mapped memory. The way the changes are being tracked +varies depending on the operating system. +- On Windows `Vectored Exception Handling` mechanism is used on the shadow +memories that correspond to the mapped device memory regions. +- On Linux and Android the shadow memory regions are similarly trapped by +changing its access protection to `PROT_NONE`. Every access from the +application will generate a `SIGSEGV` which is handled by the appropriate +signal handler installed by the page guard manager. + +Because a shadow memory is allocated and returned to the application instead +of the actual mapped memory returned by the driver, both reads and writes need +to be tracked. +- Writes need to be dumped to the capture file. +- Reads must trigger a memory copy from the actual mapped memory into the shadow +memory so that the application will read the correct/updated data each time. + +`page_guard` is the most efficient, both performance and capture file size +wise, mechanism. However, as described in +[Conflicts With Crash Detection Libraries](#conflicts-with-crash-detection-libraries), +it has some limitation when capturing applications that install their own +signal handler for handling the `SIGSEGV` signal. This limitation exists +only on Linux and Android applications. To work around this +limitation there is the `uffd` mechanism. + +##### 4. `uffd` +This mode utilizes the userfaultfd mechanism provided by the Linux kernel which +allows user space applications to detect and handle page faults. +Under the hood `uffd` is the same mechanism as `page_guard` but instead of trapping +the shadow memory regions with the `PROT_NONE` + `SIGSEGV` trick, it +registers those memory regions for tracking to the userfaultfd mechanism. + +Shadow memory regions are registered using the +`UFFDIO_REGISTER_MODE_WP | UFFDIO_REGISTER_MODE_MISSING` flags with the +userfaultfd mechanism and a handler thread is started and polls for faults +to trigger. The combination of those flags will trigger a fault in two cases: +- When an unallocated page is accessed with either a write or a read. +- When a page is written. + +This imposes a limitation: When the shadow memory is freshly allocated all +pages will be unallocated, making tracking both reads and writes simple as +both will trigger a fault. However, after the first time the accesses are +tracked and dumped to the capture file, the reads cannot be tracked any longer +as the pages will be already allocated and won't trigger a fault. +To workaround this each time the memory is examined, the dirty regions are +being "reset". This involves unregistering those subregions from userfaultfd, +requesting new pages from the OS to be provided at the same virtual addresses +and then the subregions are registered again for tracking. +This has a performance penalty as in this case both reads and writes need +to be copied from the actual mapped memory into the shadow memory when +detected, while the `page_guard` method requires this only for reads. + +Also there is another limitation. The way the new pages are requested each +time and the regions are unregistered and registered again, makes this +mechanism prone to race conditions when there are multiple threads. If a +thread is accessing a specific page within a region and at the same time +that region is being reset, then the access is not trapped and undefined +behavior occurs. + +In order to work around this a list of the thread ids that access each +region is kept. When that specific region is being reset a signal is +sent to each thread which will force them to enter a signal handler that +GFXR registers for that signal. The signal handler essentially performs a +form of synchronization between the thread that is triggering the reset and +the rest of the threads that potentially are touching pages that are being +reset. The signal used one of the real time signals, the first in the range +[`SIGRTMIN`, `SIGRTMAX`] that has no handler already installed. + +`uffd` is less efficient performance wise than `page_guard` but +should be fast enough for real-world applications and games. ##### Disabling Debug Breaks Triggered by the GFXReconstruct Layer @@ -368,7 +449,7 @@ Log Break on Error | debug.gfxrecon.log_break_on_error | BOOL | Trigger a debug Log File Create New | debug.gfxrecon.log_file_create_new | BOOL | Specifies that log file initialization should overwrite an existing file when true, or append to an existing file when false. Default is: `true` Log File Flush After Write | debug.gfxrecon.log_file_flush_after_write | BOOL | Flush the log file to disk after each write when true. Default is: `false` Log File Keep Open | debug.gfxrecon.log_file_keep_open | BOOL | Keep the log file open between log messages when true, or close and reopen the log file for each message when false. Default is: `true` -Memory Tracking Mode | debug.gfxrecon.memory_tracking_mode | STRING | Specifies the memory tracking mode to use for detecting modifications to mapped Vulkan memory objects. Available options are: `page_guard`, `assisted`, and `unassisted`. Default is `page_guard` +Memory Tracking Mode | debug.gfxrecon.memory_tracking_mode | STRING | Specifies the memory tracking mode to use for detecting modifications to mapped Vulkan memory objects. Available options are: `page_guard`, `userfaultfd`, `assisted`, and `unassisted`. See [Understanding GFXReconstruct Layer Memory Capture](#understanding-gfxreconstruct-layer-memory-capture) for more details. Default is `page_guard`. Page Guard Copy on Map | debug.gfxrecon.page_guard_copy_on_map | BOOL | When the `page_guard` memory tracking mode is enabled, copies the content of the mapped memory to the shadow memory immediately after the memory is mapped. Default is: `true` Page Guard Separate Read Tracking | debug.gfxrecon.page_guard_separate_read | BOOL | When the `page_guard` memory tracking mode is enabled, copies the content of pages accessed for read from mapped memory to shadow memory on each read. Can overwrite unprocessed shadow memory content when an application is reading from and writing to the same page. Default is: `true` Page Guard Persistent Memory | debug.gfxrecon.page_guard_persistent_memory | BOOL | When the `page_guard` memory tracking mode is enabled, this option changes the way that the shadow memory used to detect modifications to mapped memory is allocated. The default behavior is to allocate and copy the mapped memory range on map and free the allocation on unmap. When this option is enabled, an allocation with a size equal to that of the object being mapped is made once on the first map and is not freed until the object is destroyed. This option is intended to be used with applications that frequently map and unmap large memory ranges, to avoid frequent allocation and copy operations that can have a negative impact on performance. This option is ignored when GFXRECON_PAGE_GUARD_EXTERNAL_MEMORY is enabled. Default is `false` @@ -523,13 +604,13 @@ some content may be fast enough using the trigger property may be difficult.) As described in [Understanding GFXReconstruct Layer Memory Capture](#understanding-gfxreconstruct-layer-memory-capture), -the capture layer uses a signal handler to detect modifications to -mapped memory. +the capture layer, when utilizing the `page_guard` mechanism, it uses a signal +handler to detect modifications to mapped memory. Only one signal handler for that signal can be registered at a time, which can lead to a potential conflict with crash detection libraries that will also register a signal handler. -Conflict between the capture layer and crash detection libraries depends on the +Conflict between the `page_guard` mechanism and crash detection libraries depends on the order with which each component registers its signal handler. The capture layer will not register its signal handler until the first call to `vkMapMemory`. @@ -548,10 +629,11 @@ After the crash detection library sets its signal handler, it immediately receives a SIGSEGV event generated by the concurrent write to mapped memory, which it detects as a crash and terminates the application. +`userfaultfd` mechanism was introduced in order to work around such conflicts. #### Memory Tracking Limitations -There is a limitation with the page guard memory tracking method used by the +There is a limitation with the `page_guard` memory tracking method used by the GFXReconstruct capture layer. The logic behind that method is to apply a memory protection to the guarded/shadowed regions so that accesses made by the user to trigger a diff --git a/USAGE_desktop_Vulkan.md b/USAGE_desktop_Vulkan.md index 29dc594713..fcdd30525b 100644 --- a/USAGE_desktop_Vulkan.md +++ b/USAGE_desktop_Vulkan.md @@ -124,6 +124,101 @@ The following command would be executed from the command line to set the export VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_gfxreconstruct ``` +#### Understanding GFXReconstruct Layer Memory Capture + +The Vulkan API allows Vulkan memory objects to be mapped by an application +for direct modification. +To successfully capture an application, the GFXReconstruct layer must be able to +detect if the application modifies the mapped memory in order to dump the changes +in the capture file so that they can be re-applied while replaying. +To achieve this GFXR utilizes four different modes: + +##### 1. `assisted` +This mode expects the application to call `vkFlushMappedMemoryRanges` +after memory is modified; the memory ranges specified to the +`vkFlushMappedMemoryRanges` call will be written to the capture file +during the call. + +##### 2. `unassisted` +This mode writes the full content of mapped memory to the capture file +on calls to `vkUnmapMemory` and `vkQueueSubmit`. It is very inefficient +for performance and it will bloat capture file sizes. May be unusable +with real-world applications that map large amounts of memory. + +##### 3. `page_guard` +`page_guard` tracks modifications to individual memory pages, which are +written to the capture file on calls to `vkFlushMappedMemoryRanges`, +`vkUnmapMemory`, and `vkQueueSubmit`. This method requires allocating +shadow memory for all mapped memory. The way the changes are being tracked +varies depending on the operating system. +- On Windows `Vectored Exception Handling` mechanism is used on the shadow +memories that correspond to the mapped device memory regions. +- On Linux and Android the shadow memory regions are similarly trapped by +changing its access protection to `PROT_NONE`. Every access from the +application will generate a `SIGSEGV` which is handled by the appropriate +signal handler installed by the page guard manager. + +Because a shadow memory is allocated and returned to the application instead +of the actual mapped memory returned by the driver, both reads and writes need +to be tracked. +- Writes need to be dumped to the capture file. +- Reads must trigger a memory copy from the actual mapped memory into the shadow +memory so that the application will read the correct/updated data each time. + +`page_guard` is the most efficient, both performance and capture file size +wise, mechanism. However, as described in +[Conflicts With Crash Detection Libraries](#conflicts-with-crash-detection-libraries), +it has some limitation when capturing applications that install their own +signal handler for handling the `SIGSEGV` signal. This limitation exists +only on Linux and Android applications. To work around this +limitation there is the `uffd` mechanism. + +##### 4. `uffd` +This mode utilizes the userfaultfd mechanism provided by the Linux kernel which +allows user space applications to detect and handle page faults. +Under the hood `uffd` is the same mechanism as `page_guard` but instead of trapping +the shadow memory regions with the `PROT_NONE` + `SIGSEGV` trick, it +registers those memory regions for tracking to the userfaultfd mechanism. + +Shadow memory regions are registered using the +`UFFDIO_REGISTER_MODE_WP | UFFDIO_REGISTER_MODE_MISSING` flags with the +userfaultfd mechanism and a handler thread is started and polls for faults +to trigger. The combination of those flags will trigger a fault in two cases: +- When an unallocated page is accessed with either a write or a read. +- When a page is written. + +This imposes a limitation: When the shadow memory is freshly allocated all +pages will be unallocated, making tracking both reads and writes simple as +both will trigger a fault. However, after the first time the accesses are +tracked and dumped to the capture file, the reads cannot be tracked any longer +as the pages will be already allocated and won't trigger a fault. +To workaround this each time the memory is examined, the dirty regions are +being "reset". This involves unregistering those subregions from userfaultfd, +requesting new pages from the OS to be provided at the same virtual addresses +and then the subregions are registered again for tracking. +This has a performance penalty as in this case both reads and writes need +to be copied from the actual mapped memory into the shadow memory when +detected, while the `page_guard` method requires this only for reads. + +Also there is another limitation. The way the new pages are requested each +time and the regions are unregistered and registered again, makes this +mechanism prone to race conditions when there are multiple threads. If a +thread is accessing a specific page within a region and at the same time +that region is being reset, then the access is not trapped and undefined +behavior occurs. + +In order to work around this a list of the thread ids that access each +region is kept. When that specific region is being reset a signal is +sent to each thread which will force them to enter a signal handler that +GFXR registers for that signal. The signal handler essentially performs a +form of synchronization between the thread that is triggering the reset and +the rest of the threads that potentially are touching pages that are being +reset. The signal used one of the real time signals, the first in the range +[`SIGRTMIN`, `SIGRTMAX`] that has no handler already installed. + +`uffd` is less efficient performance wise than `page_guard` but +should be fast enough for real-world applications and games. + ### Capture Options The GFXReconstruct layer supports several options, which may be enabled @@ -180,7 +275,7 @@ Log File Create New | GFXRECON_LOG_FILE_CREATE_NEW | BOOL | Specifies that log f Log File Flush After Write | GFXRECON_LOG_FILE_FLUSH_AFTER_WRITE | BOOL | Flush the log file to disk after each write when true. Default is: `false` Log File Keep Open | GFXRECON_LOG_FILE_KEEP_OPEN | BOOL | Keep the log file open between log messages when true, or close and reopen the log file for each message when false. Default is: `true` Log Output to Debug Console | GFXRECON_LOG_OUTPUT_TO_OS_DEBUG_STRING | BOOL | Windows only option. Log messages will be written to the Debug Console with `OutputDebugStringA`. Default is: `false` -Memory Tracking Mode | GFXRECON_MEMORY_TRACKING_MODE | STRING | Specifies the memory tracking mode to use for detecting modifications to mapped Vulkan memory objects. Available options are: `page_guard`, `assisted`, and `unassisted`. Default is `page_guard` +Memory Tracking Mode | GFXRECON_MEMORY_TRACKING_MODE | STRING | Specifies the memory tracking mode to use for detecting modifications to mapped Vulkan memory objects. Available options are: `page_guard`, `uffd`, `assisted`, and `unassisted`. See [Understanding GFXReconstruct Layer Memory Capture](#understanding-gfxreconstruct-layer-memory-capture) for more details. Default is `page_guard`. Page Guard Copy on Map | GFXRECON_PAGE_GUARD_COPY_ON_MAP | BOOL | When the `page_guard` memory tracking mode is enabled, copies the content of the mapped memory to the shadow memory immediately after the memory is mapped. Default is: `true` Page Guard Separate Read Tracking | GFXRECON_PAGE_GUARD_SEPARATE_READ | BOOL | When the `page_guard` memory tracking mode is enabled, copies the content of pages accessed for read from mapped memory to shadow memory on each read. Can overwrite unprocessed shadow memory content when an application is reading from and writing to the same page. Default is: `true` Page Guard External Memory | GFXRECON_PAGE_GUARD_EXTERNAL_MEMORY | BOOL | When the `page_guard` memory tracking mode is enabled, use the VK_EXT_external_memory_host extension to eliminate the need for shadow memory allocations. For each memory allocation from a host visible memory type, the capture layer will create an allocation from system memory, which it can monitor for write access, and provide that allocation to vkAllocateMemory as external memory. Only available on Windows. Default is `false` @@ -194,8 +289,49 @@ Queue Zero Only | GFXRECON_QUEUE_ZERO_ONLY | BOOL | Forces to using only QueueFa Allow Pipeline Compile Required | GFXRECON_ALLOW_PIPELINE_COMPILE_REQUIRED | BOOL | The default behaviour forces VK_PIPELINE_COMPILE_REQUIRED to be returned from Create*Pipelines calls which have VK_PIPELINE_CREATE_FAIL_ON_PIPELINE_COMPILE_REQUIRED_BIT set, and skips dispatching and recording the calls. This forces applications to fallback to recompiling pipelines without caching, the Vulkan calls for which will be captured. Enabling this option causes capture to record the application's calls and implementation's return values unmodified, but the resulting captures are fragile to changes in Vulkan implementations if they use pipeline caching. #### Memory Tracking Known Issues -There is a known issue with the page guard memory tracking method. The logic behind that method is to apply a memory protection to the guarded/shadowed regions so that accesses made by the user to trigger a segmentation fault which is handled by GFXReconstruct. -If the access is made by a system call (like `fread()`) then there won't be a segmentation fault generated and the function will fail. As a result the mapped region will not be updated. +### Capture Limitations + +#### Conflicts With Crash Detection Libraries + +As described in +[Understanding GFXReconstruct Layer Memory Capture](#understanding-gfxreconstruct-layer-memory-capture), +the capture layer, when it utilizing the `page_guard` mechanism, it uses a signal +handler to detect modifications to mapped memory. +Only one signal handler for that signal can be registered at a time, which can +lead to a potential conflict with crash detection libraries that will also +register a signal handler. + +Conflict between the `page_guard` mechanism and crash detection libraries depends on the +order with which each component registers its signal handler. +The capture layer will not register its signal handler until the first call to +`vkMapMemory`. +As long as the application initializes the crash detection library before +calling `vkMapMemory`, there should be no conflict. + +The conflict occurs when the application initializes its Vulkan component and +its crash detection library concurrently. +Applications have been observed to initialize Vulkan and begin uploading +resources with one or more threads, while at the same time initializing a crash +detection library from another thread. +For this scenario, the crash detection library sets its signal handler after the +first call to `vkMapMemory`, while a resource upload thread is actively writing +to the mapped memory. +After the crash detection library sets its signal handler, it immediately +receives a SIGSEGV event generated by the concurrent write to mapped memory, +which it detects as a crash and terminates the application. + +`uffd` mechanism was introduced in order to work around such conflicts. + +#### Memory Tracking Limitations + +There is a limitation with the `page_guard` memory tracking method used by the +GFXReconstruct capture layer. +The logic behind that method is to apply a memory protection to the +guarded/shadowed regions so that accesses made by the user to trigger a +segmentation fault which is handled by GFXReconstruct. +If the access is made by a system call (like `fread()`) then there won't be a +segmentation fault generated and the function will fail. +As a result the mapped region will not be updated. #### Settings File diff --git a/android/framework/util/CMakeLists.txt b/android/framework/util/CMakeLists.txt index cdefaa9fdb..fb0e712b0b 100644 --- a/android/framework/util/CMakeLists.txt +++ b/android/framework/util/CMakeLists.txt @@ -30,6 +30,7 @@ target_sources(gfxrecon_util ${GFXRECON_SOURCE_DIR}/framework/util/output_stream.h ${GFXRECON_SOURCE_DIR}/framework/util/page_guard_manager.h ${GFXRECON_SOURCE_DIR}/framework/util/page_guard_manager.cpp + ${GFXRECON_SOURCE_DIR}/framework/util/page_guard_manager_uffd.cpp ${GFXRECON_SOURCE_DIR}/framework/util/page_status_tracker.h ${GFXRECON_SOURCE_DIR}/framework/util/platform.h ${GFXRECON_SOURCE_DIR}/framework/util/settings_loader.h diff --git a/android/layer/build.gradle b/android/layer/build.gradle index 561db4495d..fab5d34728 100644 --- a/android/layer/build.gradle +++ b/android/layer/build.gradle @@ -2,7 +2,7 @@ apply plugin: 'com.android.library' android { compileSdkVersion 33 - ndkVersion '21.3.6528147' + ndkVersion '24.0.8215888' defaultConfig { minSdkVersion 26 targetSdkVersion 33 diff --git a/framework/encode/capture_manager.cpp b/framework/encode/capture_manager.cpp index ca56111d14..770a6f6ce6 100644 --- a/framework/encode/capture_manager.cpp +++ b/framework/encode/capture_manager.cpp @@ -104,7 +104,8 @@ CaptureManager::CaptureManager(format::ApiFamilyId api_family) : CaptureManager::~CaptureManager() { - if (memory_tracking_mode_ == CaptureSettings::MemoryTrackingMode::kPageGuard) + if (memory_tracking_mode_ == CaptureSettings::MemoryTrackingMode::kPageGuard || + memory_tracking_mode_ == CaptureSettings::MemoryTrackingMode::kUserfaultfd) { util::PageGuardManager::Destroy(); } @@ -283,7 +284,7 @@ bool CaptureManager::Initialize(std::string base_filename, const CaptureSettings rv_annotation_info_.descriptor_mask); } - if (memory_tracking_mode_ == CaptureSettings::kPageGuard) + if (memory_tracking_mode_ == CaptureSettings::kPageGuard || memory_tracking_mode_ == CaptureSettings::kUserfaultfd) { page_guard_align_buffer_sizes_ = trace_settings.page_guard_align_buffer_sizes; page_guard_track_ahb_memory_ = trace_settings.page_guard_track_ahb_memory; @@ -410,14 +411,21 @@ bool CaptureManager::Initialize(std::string base_filename, const CaptureSettings if (success) { - if (memory_tracking_mode_ == CaptureSettings::MemoryTrackingMode::kPageGuard) + if (memory_tracking_mode_ == CaptureSettings::MemoryTrackingMode::kPageGuard || + memory_tracking_mode_ == CaptureSettings::MemoryTrackingMode::kUserfaultfd) { + const util::PageGuardManager::MemoryProtectionMode mem_prot_mode = + memory_tracking_mode_ == CaptureSettings::MemoryTrackingMode::kPageGuard + ? util::PageGuardManager::MemoryProtectionMode::kMProtectMode + : util::PageGuardManager::MemoryProtectionMode::kUserFaultFdMode; + util::PageGuardManager::Create(trace_settings.page_guard_copy_on_map, trace_settings.page_guard_separate_read, util::PageGuardManager::kDefaultEnableReadWriteSamePage, trace_settings.page_guard_unblock_sigsegv, trace_settings.page_guard_signal_handler_watcher, - trace_settings.page_guard_signal_handler_watcher_max_restores); + trace_settings.page_guard_signal_handler_watcher_max_restores, + mem_prot_mode); } if ((capture_mode_ & kModeTrack) == kModeTrack) @@ -1106,12 +1114,36 @@ void CaptureManager::WriteCreateHeapAllocationCmd(uint64_t allocation_id, uint64 void CaptureManager::WriteToFile(const void* data, size_t size) { + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) + { + util::PageGuardManager* manager = util::PageGuardManager::Get(); + if (manager) + { + // fwrite hides a lock inside to synchronize writes to files. If a thread is in the middle + // of a write to the capture file and the uffd mechanism interupts it, it will cause + // a deadlock as uffd will also try to write to the capture file as well. For this + // reason RT signal needs to be disabled while writing. + // This can be removed once writing to the capture file(s) is delegated to a separate thread. + manager->UffdBlockRtSignal(); + } + } + file_stream_->Write(data, size); if (force_file_flush_) { file_stream_->Flush(); } + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) + { + util::PageGuardManager* manager = util::PageGuardManager::Get(); + if (manager) + { + // Enable again signal + manager->UffdUnblockRtSignal(); + } + } + // Increment block index auto thread_data = GetThreadData(); assert(thread_data != nullptr); diff --git a/framework/encode/capture_settings.cpp b/framework/encode/capture_settings.cpp index 9a0ace8017..4f8e8a6ffb 100644 --- a/framework/encode/capture_settings.cpp +++ b/framework/encode/capture_settings.cpp @@ -485,8 +485,12 @@ void CaptureSettings::ProcessOptions(OptionsMap* options, CaptureSettings* setti ParseBoolString(FindOption(options, kOptionKeyCaptureFileForceFlush), settings->trace_settings_.force_flush); // Memory tracking options +#if defined(WIN32) || defined(__APPLE__) settings->trace_settings_.memory_tracking_mode = ParseMemoryTrackingModeString( FindOption(options, kOptionKeyMemoryTrackingMode), settings->trace_settings_.memory_tracking_mode); +#else + settings->trace_settings_.memory_tracking_mode = MemoryTrackingMode::kUserfaultfd; +#endif // Trimming options: // Trim frame ranges, trim queue submit ranges, and trim frame hotkey are mutually exclusive. @@ -727,6 +731,10 @@ CaptureSettings::ParseMemoryTrackingModeString(const std::string& { result = MemoryTrackingMode::kPageGuard; } + else if (util::platform::StringCompareNoCase("userfaultfd", value_string.c_str()) == 0) + { + result = MemoryTrackingMode::kUserfaultfd; + } else if (util::platform::StringCompareNoCase("assisted", value_string.c_str()) == 0) { result = MemoryTrackingMode::kAssisted; diff --git a/framework/encode/capture_settings.h b/framework/encode/capture_settings.h index 3268a04170..28dc72e02b 100644 --- a/framework/encode/capture_settings.h +++ b/framework/encode/capture_settings.h @@ -49,9 +49,14 @@ class CaptureSettings // flush. kAssisted = 1, // Use guard pages to determine which regions of memory to write on unmap and queue submit. This mode replaces - // the mapped memory value returned by the driver with a shadow allocation that the capture layer can monitor - // to determine which regions of memory have been modified by the application. - kPageGuard = 2 + // the mapped memory value returned by the driver with a shadow allocation that the capture layer can monitor, + // using the PROT_NONE + SIGSEGV trick, to determine which regions of memory have been modified by the + // application. + kPageGuard = 2, + // Similar mechanism as page guard. The mapper memory returned by the driver is replaced by a shadow + // allocation but in this case the memory is monitored using the userfaultfd mechanism provided by the linux + // kernel. + kUserfaultfd = 3 }; enum RuntimeTriggerState diff --git a/framework/encode/vulkan_capture_manager.cpp b/framework/encode/vulkan_capture_manager.cpp index aa8df6d980..f109ffca56 100644 --- a/framework/encode/vulkan_capture_manager.cpp +++ b/framework/encode/vulkan_capture_manager.cpp @@ -1995,11 +1995,12 @@ void VulkanCaptureManager::PostProcess_vkMapMemory(VkResult result, wrapper->mapped_size = size; } - if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard || + GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd #if defined(VK_USE_PLATFORM_ANDROID_KHR) - // Hardware buffer memory is tracked separately, so VkDeviceMemory mappings should be ignored to avoid - // duplicate memory tracking entries. - && (wrapper->hardware_buffer == nullptr) + // Hardware buffer memory is tracked separately, so VkDeviceMemory mappings should be ignored to + // avoid duplicate memory tracking entries. + && (wrapper->hardware_buffer == nullptr) #endif ) { @@ -2057,7 +2058,8 @@ void VulkanCaptureManager::PostProcess_vkMapMemory(VkResult result, GFXRECON_LOG_WARNING("VkDeviceMemory object with handle = %" PRIx64 " has been mapped more than once", memory); - if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard) + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard || + GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) { assert((wrapper->mapped_offset == offset) && (wrapper->mapped_size == size)); @@ -2083,7 +2085,8 @@ void VulkanCaptureManager::PreProcess_vkFlushMappedMemoryRanges(VkDevice if (pMemoryRanges != nullptr) { - if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard) + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard || + GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) { const DeviceMemoryWrapper* current_memory_wrapper = nullptr; util::PageGuardManager* manager = util::PageGuardManager::Get(); @@ -2152,7 +2155,8 @@ void VulkanCaptureManager::PreProcess_vkUnmapMemory(VkDevice device, VkDeviceMem if (wrapper->mapped_data != nullptr) { - if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard) + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard || + GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) { util::PageGuardManager* manager = util::PageGuardManager::Get(); assert(manager != nullptr); @@ -2217,7 +2221,8 @@ void VulkanCaptureManager::PreProcess_vkFreeMemory(VkDevice if (wrapper->mapped_data != nullptr) { - if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard) + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard || + GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) { util::PageGuardManager* manager = util::PageGuardManager::Get(); assert(manager != nullptr); @@ -2246,7 +2251,8 @@ void VulkanCaptureManager::PostProcess_vkFreeMemory(VkDevice // Destroy external resources. auto wrapper = GetWrapper(memory); - if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard) + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard || + GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) { util::PageGuardManager* manager = util::PageGuardManager::Get(); assert(manager != nullptr); @@ -2422,7 +2428,8 @@ void VulkanCaptureManager::PreProcess_vkQueueSubmit2(VkQueue queue, void VulkanCaptureManager::QueueSubmitWriteFillMemoryCmd() { - if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard) + if (GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard || + GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) { util::PageGuardManager* manager = util::PageGuardManager::Get(); assert(manager != nullptr); @@ -2632,7 +2639,9 @@ void VulkanCaptureManager::OverrideGetPhysicalDeviceSurfacePresentModesKHR(uint3 bool VulkanCaptureManager::CheckBindAlignment(VkDeviceSize memoryOffset) { - if ((GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard) && !GetPageGuardAlignBufferSizes()) + if ((GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kPageGuard || + GetMemoryTrackingMode() == CaptureSettings::MemoryTrackingMode::kUserfaultfd) && + !GetPageGuardAlignBufferSizes()) { return (memoryOffset % util::platform::GetSystemPageSize()) == 0; } diff --git a/framework/util/CMakeLists.txt b/framework/util/CMakeLists.txt index 738559b83c..e9033968d2 100644 --- a/framework/util/CMakeLists.txt +++ b/framework/util/CMakeLists.txt @@ -64,6 +64,7 @@ target_sources(gfxrecon_util ${CMAKE_CURRENT_LIST_DIR}/output_stream.h ${CMAKE_CURRENT_LIST_DIR}/page_guard_manager.h ${CMAKE_CURRENT_LIST_DIR}/page_guard_manager.cpp + ${CMAKE_CURRENT_LIST_DIR}/page_guard_manager_uffd.cpp ${CMAKE_CURRENT_LIST_DIR}/page_status_tracker.h ${CMAKE_CURRENT_LIST_DIR}/platform.h ${CMAKE_CURRENT_LIST_DIR}/settings_loader.h diff --git a/framework/util/defines.h b/framework/util/defines.h index 5aac79542a..b5065eb952 100644 --- a/framework/util/defines.h +++ b/framework/util/defines.h @@ -60,4 +60,11 @@ } \ } +// Useful to avoid sign extension when converting a 32bit pointer to uint64_t +template +static constexpr uint64_t GFXRECON_PTR_TO_UINT64(T ptr) +{ + return static_cast(reinterpret_cast(ptr)); +} + #endif // GFXRECON_UTIL_DEFINES_H diff --git a/framework/util/page_guard_manager.cpp b/framework/util/page_guard_manager.cpp index 9ae518557b..87d9dad670 100644 --- a/framework/util/page_guard_manager.cpp +++ b/framework/util/page_guard_manager.cpp @@ -1,7 +1,7 @@ /* ** Copyright (c) 2016 Advanced Micro Devices, Inc. All rights reserved. -** Copyright (c) 2015-2020 Valve Corporation -** Copyright (c) 2015-2020 LunarG, Inc. +** Copyright (c) 2015-2023 Valve Corporation +** Copyright (c) 2015-2023 LunarG, Inc. ** ** Permission is hereby granted, free of charge, to any person obtaining a ** copy of this software and associated documentation files (the "Software"), @@ -216,38 +216,66 @@ void PageGuardManager::InitializeSystemExceptionContext(void) } PageGuardManager::PageGuardManager() : - exception_handler_(nullptr), exception_handler_count_(0), system_page_size_(util::platform::GetSystemPageSize()), - system_page_pot_shift_(GetSystemPagePotShift()), enable_copy_on_map_(kDefaultEnableCopyOnMap), - enable_separate_read_(kDefaultEnableSeparateRead), unblock_sigsegv_(kDefaultUnblockSIGSEGV), - enable_signal_handler_watcher_(kDefaultEnableSignalHandlerWatcher), - signal_handler_watcher_max_restores_(kDefaultSignalHandlerWatcherMaxRestores), - enable_read_write_same_page_(kDefaultEnableReadWriteSamePage) -{ - InitializeSystemExceptionContext(); -} - -PageGuardManager::PageGuardManager(bool enable_copy_on_map, - bool enable_separate_read, - bool expect_read_write_same_page, - bool unblock_SIGSEGV, - bool enable_signal_handler_watcher, - int signal_handler_watcher_max_restores) : + PageGuardManager(kDefaultEnableCopyOnMap, + kDefaultEnableSeparateRead, + kDefaultUnblockSIGSEGV, + kDefaultEnableSignalHandlerWatcher, + kDefaultSignalHandlerWatcherMaxRestores, + kDefaultEnableReadWriteSamePage, + kDefaultMemoryProtMode) +{} + +PageGuardManager::PageGuardManager(bool enable_copy_on_map, + bool enable_separate_read, + bool expect_read_write_same_page, + bool unblock_SIGSEGV, + bool enable_signal_handler_watcher, + int signal_handler_watcher_max_restores, + MemoryProtectionMode protection_mode) : exception_handler_(nullptr), exception_handler_count_(0), system_page_size_(util::platform::GetSystemPageSize()), system_page_pot_shift_(GetSystemPagePotShift()), enable_copy_on_map_(enable_copy_on_map), enable_separate_read_(enable_separate_read), unblock_sigsegv_(unblock_SIGSEGV), enable_signal_handler_watcher_(enable_signal_handler_watcher), signal_handler_watcher_max_restores_(signal_handler_watcher_max_restores), - enable_read_write_same_page_(expect_read_write_same_page) + enable_read_write_same_page_(expect_read_write_same_page), protection_mode_(protection_mode) { - InitializeSystemExceptionContext(); + if (kUserFaultFdMode == protection_mode_ && !USERFAULTFD_SUPPORTED) + { + GFXRECON_LOG_WARNING("Kernel does not support all features for userfaultfd memory tracking mode. Falling back " + "to mprotect mode."); + + protection_mode_ = kMProtectMode; + } + + if (kMProtectMode == protection_mode_) + { + InitializeSystemExceptionContext(); + } + else + { + if (!InitializeUserFaultFd()) + { + GFXRECON_LOG_ERROR("Userfaultfd initialization failed. Falling back to mprotect memory tracking mode."); + + protection_mode_ = kMProtectMode; + InitializeSystemExceptionContext(); + } + } } PageGuardManager::~PageGuardManager() { - if (exception_handler_ != nullptr) + if (kMProtectMode == protection_mode_) { - ClearExceptionHandler(exception_handler_); + if (exception_handler_ != nullptr) + { + ClearExceptionHandler(exception_handler_); + } + } + else + { + UffdTerminate(); } } @@ -326,12 +354,13 @@ void* PageGuardManager::SignalHandlerWatcher(void* args) } #endif // !defined(WIN32) -void PageGuardManager::Create(bool enable_copy_on_map, - bool enable_separate_read, - bool expect_read_write_same_page, - bool unblock_SIGSEGV, - bool enable_signal_handler_watcher, - int signal_handler_watcher_max_restores) +void PageGuardManager::Create(bool enable_copy_on_map, + bool enable_separate_read, + bool expect_read_write_same_page, + bool unblock_SIGSEGV, + bool enable_signal_handler_watcher, + int signal_handler_watcher_max_restores, + MemoryProtectionMode protection_mode) { if (instance_ == nullptr) { @@ -340,7 +369,8 @@ void PageGuardManager::Create(bool enable_copy_on_map, expect_read_write_same_page, unblock_SIGSEGV, enable_signal_handler_watcher, - signal_handler_watcher_max_restores); + signal_handler_watcher_max_restores, + protection_mode); #if !defined(WIN32) if (enable_signal_handler_watcher && @@ -637,7 +667,7 @@ bool PageGuardManager::SetMemoryProtection(void* protect_address, size_t protect { if (!unblock_sigsegv_) { - GFXRECON_LOG_WARNING("SIGSEGV is blocked while page guard manager expects the signal to be handled. " + GFXRECON_LOG_WARNING("SIGSEGV is blocked while page_guard mechanism expects the signal to be handled. " "Things might fail and/or crash with segmentation fault. To force-enable SIGSEGV " "try setting GFXRECON_PAGE_GUARD_UNBLOCK_SIGSEGV environment variable to 1.\n"); } @@ -711,6 +741,14 @@ void PageGuardManager::ProcessEntry(uint64_t memory_id, const ModifiedMemoryFunc& handle_modified) { assert(memory_info != nullptr); + assert(memory_info->is_modified); + + uint32_t n_threads_to_wait = 0; + + if (protection_mode_ == kUserFaultFdMode) + { + n_threads_to_wait = UffdBlockFaultingThreads(memory_info); + } bool active_range = false; size_t start_index = 0; @@ -740,15 +778,18 @@ void PageGuardManager::ProcessEntry(uint64_t memory_id, // enable_read_write_same_page_ is false. if (memory_info->status_tracker.IsActiveReadBlock(i)) { + memory_info->status_tracker.SetActiveReadBlock(i, false); + assert(memory_info->shadow_memory != nullptr); void* page_address = static_cast(memory_info->aligned_address) + (i << system_page_pot_shift_); size_t segment_size = GetMemorySegmentSize(memory_info, i); - memory_info->status_tracker.SetActiveReadBlock(i, false); - - SetMemoryProtection(page_address, segment_size, kGuardReadWriteProtect); + if (protection_mode_ == kMProtectMode) + { + SetMemoryProtection(page_address, segment_size, kGuardReadWriteProtect); + } } // If the previous pages were modified by a write operation, process the modified range now. @@ -765,6 +806,12 @@ void PageGuardManager::ProcessEntry(uint64_t memory_id, { ProcessActiveRange(memory_id, memory_info, start_index, memory_info->total_pages, handle_modified); } + + // Unblock threads + if (protection_mode_ == kUserFaultFdMode) + { + UffdUnblockFaultingThreads(memory_info, n_threads_to_wait); + } } void PageGuardManager::ProcessActiveRange(uint64_t memory_id, @@ -794,7 +841,15 @@ void PageGuardManager::ProcessActiveRange(uint64_t memory_id, // Page guard was disabled when these pages were accessed. We enable it now for write, to // trap any writes made to the memory while we are performing the copy from shadow memory // to mapped memory. - SetMemoryProtection(guard_address, guard_range, kGuardReadOnlyProtect); + if (kMProtectMode == protection_mode_) + { + SetMemoryProtection(guard_address, guard_range, kGuardReadOnlyProtect); + } + else if (kUserFaultFdMode == protection_mode_) + { + assert(memory_info->aligned_address == memory_info->shadow_memory); + UffdUnregisterMemory(guard_address, page_count << system_page_pot_shift_); + } // Copy from shadow memory to the original mapped memory. if (start_index == 0) @@ -814,8 +869,15 @@ void PageGuardManager::ProcessActiveRange(uint64_t memory_id, // the memory range. handle_modified(memory_id, memory_info->shadow_memory, page_offset, page_range); - // Reset page guard to detect both read and write accesses when using shadow memory. - SetMemoryProtection(guard_address, guard_range, kGuardReadWriteProtect); + if (kMProtectMode == protection_mode_) + { + // Reset page guard to detect both read and write accesses when using shadow memory. + SetMemoryProtection(guard_address, guard_range, kGuardReadWriteProtect); + } + else if (kUserFaultFdMode == protection_mode_) + { + UffdResetRegion(guard_address, page_count << system_page_pot_shift_); + } } else { @@ -823,8 +885,11 @@ void PageGuardManager::ProcessActiveRange(uint64_t memory_id, { void* guard_address = static_cast(memory_info->aligned_address) + page_offset; - // Reset page guard to detect only write accesses when not using shadow memory. - SetMemoryProtection(guard_address, page_range, kGuardReadOnlyProtect); + if (kMProtectMode == protection_mode_) + { + // Reset page guard to detect only write accesses when not using shadow memory. + SetMemoryProtection(guard_address, page_range, kGuardReadOnlyProtect); + } } // Copy directly from the mapped memory. @@ -907,7 +972,7 @@ void* PageGuardManager::AddTrackedMemory(uint64_t memory_id, { aligned_address = shadow_memory; - if (enable_copy_on_map_) + if (enable_copy_on_map_ && kUserFaultFdMode != protection_mode_) { MemoryCopy(shadow_memory, mapped_memory, mapped_range); } @@ -1007,25 +1072,33 @@ void* PageGuardManager::AddTrackedMemory(uint64_t memory_id, start_address = shadow_memory; } - std::lock_guard lock(tracked_memory_lock_); - if (!use_write_watch) { - AddExceptionHandler(); - - // When using shadow memory, enable page guard for read and write operations so that shadow memory can be - // synchronized with the mapped memory on both read and write access. When not using shadow memory, only - // detect write access. - if (use_shadow_memory) + if (kMProtectMode == protection_mode_) { - success = SetMemoryProtection(aligned_address, guard_range, kGuardReadWriteProtect); + AddExceptionHandler(); + + // When using shadow memory, enable page guard for read and write operations so that shadow memory can + // be synchronized with the mapped memory on both read and write access. When not using shadow memory, + // only detect write access. + if (use_shadow_memory) + { + success = SetMemoryProtection(aligned_address, guard_range, kGuardReadWriteProtect); + } + else + { + success = SetMemoryProtection(aligned_address, guard_range, kGuardReadOnlyProtect); + } } else { - success = SetMemoryProtection(aligned_address, guard_range, kGuardReadOnlyProtect); + assert(shadow_memory == aligned_address); + success = UffdRegisterMemory(shadow_memory, shadow_size); } } + std::lock_guard lock(tracked_memory_lock_); + if (success) { assert(memory_info_.find(memory_id) == memory_info_.end()); @@ -1050,8 +1123,15 @@ void* PageGuardManager::AddTrackedMemory(uint64_t memory_id, { if (!use_write_watch) { - RemoveExceptionHandler(); - SetMemoryProtection(aligned_address, guard_range, kGuardNoProtect); + if (kMProtectMode == protection_mode_) + { + RemoveExceptionHandler(); + SetMemoryProtection(aligned_address, guard_range, kGuardNoProtect); + } + else + { + UffdUnregisterMemory(aligned_address, guard_range); + } } if (shadow_memory != nullptr) @@ -1073,26 +1153,35 @@ void* PageGuardManager::AddTrackedMemory(uint64_t memory_id, return (shadow_memory != nullptr) ? shadow_memory : mapped_memory; } -void PageGuardManager::RemoveTrackedMemory(uint64_t memory_id) +void PageGuardManager::ReleaseTrackedMemory(const MemoryInfo* memory_info) { - std::lock_guard lock(tracked_memory_lock_); - - auto entry = memory_info_.find(memory_id); - if (entry != memory_info_.end()) + if (!memory_info->use_write_watch) { - const MemoryInfo& memory_info = entry->second; - - if (!memory_info.use_write_watch) + if (kMProtectMode == protection_mode_) { RemoveExceptionHandler(); SetMemoryProtection( - memory_info.aligned_address, memory_info.mapped_range + memory_info.aligned_offset, kGuardNoProtect); + memory_info->aligned_address, memory_info->mapped_range + memory_info->aligned_offset, kGuardNoProtect); } - - if ((memory_info.shadow_memory != nullptr) && memory_info.own_shadow_memory) + else { - FreeMemory(memory_info.shadow_memory, memory_info.shadow_range); + UffdUnregisterMemory(memory_info->shadow_memory, memory_info->shadow_range); } + } + if ((memory_info->shadow_memory != nullptr) && memory_info->own_shadow_memory) + { + FreeMemory(memory_info->shadow_memory, memory_info->shadow_range); + } +} + +void PageGuardManager::RemoveTrackedMemory(uint64_t memory_id) +{ + std::lock_guard lock(tracked_memory_lock_); + + auto entry = memory_info_.find(memory_id); + if (entry != memory_info_.end()) + { + ReleaseTrackedMemory(&entry->second); memory_info_.erase(entry); } @@ -1174,6 +1263,8 @@ void PageGuardManager::ProcessMemoryEntries(const ModifiedMemoryFunc& handle_mod bool PageGuardManager::HandleGuardPageViolation(void* address, bool is_write, bool clear_guard) { + assert(protection_mode_ == kMProtectMode); + MemoryInfo* memory_info = nullptr; std::lock_guard lock(tracked_memory_lock_); diff --git a/framework/util/page_guard_manager.h b/framework/util/page_guard_manager.h index e7e0d3aa45..233ba55b0f 100644 --- a/framework/util/page_guard_manager.h +++ b/framework/util/page_guard_manager.h @@ -36,23 +36,44 @@ #include #include #include +#include +#include #if !defined(WIN32) #include #endif +#if defined(WIN32) || defined(__APPLE__) +#define USERFAULTFD_SUPPORTED 0 +#else +#include +// Older kernels might not support all features we need from userfaultfd. Check what is available. +#if defined(UFFD_USER_MODE_ONLY) && defined(UFFDIO_WRITEPROTECT) && defined(UFFDIO_WRITEPROTECT_MODE_WP) +#define USERFAULTFD_SUPPORTED 1 +#else +#define USERFAULTFD_SUPPORTED 0 +#endif +#endif + GFXRECON_BEGIN_NAMESPACE(gfxrecon) GFXRECON_BEGIN_NAMESPACE(util) class PageGuardManager { public: - static const bool kDefaultEnableCopyOnMap = true; - static const bool kDefaultEnableSeparateRead = true; - static const bool kDefaultEnableReadWriteSamePage = true; - static const bool kDefaultUnblockSIGSEGV = false; - static const bool kDefaultEnableSignalHandlerWatcher = false; - static const int kDefaultSignalHandlerWatcherMaxRestores = 1; + enum MemoryProtectionMode + { + kMProtectMode, + kUserFaultFdMode + }; + + static const bool kDefaultEnableCopyOnMap = true; + static const bool kDefaultEnableSeparateRead = true; + static const bool kDefaultEnableReadWriteSamePage = true; + static const bool kDefaultUnblockSIGSEGV = false; + static const bool kDefaultEnableSignalHandlerWatcher = false; + static const int kDefaultSignalHandlerWatcherMaxRestores = 1; + static const MemoryProtectionMode kDefaultMemoryProtMode = kMProtectMode; static const uintptr_t kNullShadowHandle = 0; @@ -63,12 +84,13 @@ class PageGuardManager typedef std::function ModifiedMemoryFunc; public: - static void Create(bool enable_copy_on_map, - bool enable_separate_read, - bool expect_read_write_same_page, - bool unblock_SIGSEGV, - bool enable_signal_handler_watcher, - int signal_handler_watcher_max_restores); + static void Create(bool enable_copy_on_map, + bool enable_separate_read, + bool expect_read_write_same_page, + bool unblock_SIGSEGV, + bool enable_signal_handler_watcher, + int signal_handler_watcher_max_restores, + MemoryProtectionMode protection_mode); static void Destroy(); @@ -119,15 +141,20 @@ class PageGuardManager const void* GetMappedMemory(uint64_t memory_id) const; + void UffdBlockRtSignal(); + + void UffdUnblockRtSignal(); + protected: PageGuardManager(); - PageGuardManager(bool enable_copy_on_map, - bool enable_separate_read, - bool expect_read_write_same_page, - bool unblock_SIGSEGV, - bool enable_signal_handler_watcher, - int signal_handler_watcher_max_restores); + PageGuardManager(bool enable_copy_on_map, + bool enable_separate_read, + bool expect_read_write_same_page, + bool unblock_SIGSEGV, + bool enable_signal_handler_watcher, + int signal_handler_watcher_max_restores, + MemoryProtectionMode protection_mode); ~PageGuardManager(); @@ -177,6 +204,11 @@ class PageGuardManager bool is_modified; bool own_shadow_memory; +#if USERFAULTFD_SUPPORTED == 1 + // Keep a list of all the threads that accessed this region. Only useful for Userfaultfd method + std::unordered_set uffd_fault_causing_threads; +#endif + #if defined(WIN32) // Memory for retrieving modified pages with GetWriteWatch. std::unique_ptr modified_addresses; @@ -201,6 +233,7 @@ class PageGuardManager private: size_t GetSystemPagePotShift() const; void InitializeSystemExceptionContext(); + void ReleaseTrackedMemory(const MemoryInfo* memory_info); void AddExceptionHandler(); void RemoveExceptionHandler(); @@ -247,13 +280,46 @@ class PageGuardManager const bool enable_read_write_same_page_; #if !defined(WIN32) - pthread_t signal_handler_watcher_thread_; + pthread_t signal_handler_watcher_thread_; + static uint32_t signal_handler_watcher_restores_; static void* SignalHandlerWatcher(void*); static bool CheckSignalHandler(); void MarkAllTrackedMemoryAsDirty(); +#endif - static uint32_t signal_handler_watcher_restores_; + MemoryProtectionMode protection_mode_; + +#if USERFAULTFD_SUPPORTED == 1 + bool uffd_is_init_; + int uffd_rt_signal_used_; + sigset_t uffd_signal_set_; + int uffd_fd_; + pthread_t uffd_handler_thread_; + static std::atomic is_uffd_handler_thread_running_; + static std::atomic stop_uffd_handler_thread_; + std::unique_ptr uffd_page_size_tmp_buff_; +#endif + + bool InitializeUserFaultFd(); + void UffdTerminate(); + uint32_t UffdBlockFaultingThreads(const MemoryInfo* memory_info); + void UffdUnblockFaultingThreads(MemoryInfo* memory_info, uint32_t n_threads_to_wait); + bool UffdRegisterMemory(const void* address, size_t length); + void UffdUnregisterMemory(const void* address, size_t length); + bool UffdResetRegion(void* guard_address, size_t guard_range); + +#if USERFAULTFD_SUPPORTED == 1 + bool UffdInit(); + bool UffdSetSignalHandler(); + void UffdRemoveSignalHandler(); + bool UffdStartHandlerThread(); + bool UffdHandleFault(uint64_t address, uint64_t flags, bool wake_thread, uint64_t tid); + bool UffdWakeFaultingThread(uint64_t address); + void UffdSignalHandler(int sig); + void* UffdHandlerThread(void* args); + static void* UffdHandlerThreadHelper(void* this_); + static void UffdStaticSignalHandler(int sig); #endif }; diff --git a/framework/util/page_guard_manager_uffd.cpp b/framework/util/page_guard_manager_uffd.cpp new file mode 100644 index 0000000000..6a6a4a22f5 --- /dev/null +++ b/framework/util/page_guard_manager_uffd.cpp @@ -0,0 +1,742 @@ +/* +** Copyright (c) 2023 Valve Corporation +** Copyright (c) 2023 LunarG, Inc. +** +** Permission is hereby granted, free of charge, to any person obtaining a +** copy of this software and associated documentation files (the "Software"), +** to deal in the Software without restriction, including without limitation +** the rights to use, copy, modify, merge, publish, distribute, sublicense, +** and/or sell copies of the Software, and to permit persons to whom the +** Software is furnished to do so, subject to the following conditions: +** +** The above copyright notice and this permission notice shall be included in +** all copies or substantial portions of the Software. +** +** THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +** IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +** FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +** AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +** LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +** FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +** DEALINGS IN THE SOFTWARE. +*/ + +#include "util/page_guard_manager.h" + +#if USERFAULTFD_SUPPORTED == 1 +#include "util/logging.h" +#include "util/platform.h" + +#include +#include +#include +#include +#include +#include +#include + +GFXRECON_BEGIN_NAMESPACE(gfxrecon) +GFXRECON_BEGIN_NAMESPACE(util) + +static pthread_mutex_t reset_regions_mutex = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t reset_regions_cond = PTHREAD_COND_INITIALIZER; +static bool block_accessor_threads = false; + +static pthread_mutex_t wait_for_threads_mutex = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t wait_for_threads_cond = PTHREAD_COND_INITIALIZER; +static uint32_t accessor_threads_in_handler = 0; + +std::atomic PageGuardManager::is_uffd_handler_thread_running_ = { false }; +std::atomic PageGuardManager::stop_uffd_handler_thread_ = { false }; + +void PageGuardManager::UffdStaticSignalHandler(int sig) +{ + assert(instance_); + + instance_->UffdSignalHandler(sig); +} + +void PageGuardManager::UffdSignalHandler(int sig) +{ + assert(sig == uffd_rt_signal_used_); + assert(uffd_rt_signal_used_ != -1); + + if (sig == uffd_rt_signal_used_) + { + // Signal reset regions + { + pthread_mutex_lock(&wait_for_threads_mutex); + ++accessor_threads_in_handler; + pthread_cond_signal(&wait_for_threads_cond); + pthread_mutex_unlock(&wait_for_threads_mutex); + } + + // Wait for reset regions to finish + { + pthread_mutex_lock(&reset_regions_mutex); + while (block_accessor_threads) + { + pthread_cond_wait(&reset_regions_cond, &reset_regions_mutex); + } + pthread_mutex_unlock(&reset_regions_mutex); + } + + // Decrement counter + { + pthread_mutex_lock(&wait_for_threads_mutex); + --accessor_threads_in_handler; + pthread_cond_signal(&wait_for_threads_cond); + pthread_mutex_unlock(&wait_for_threads_mutex); + } + } + else + { + GFXRECON_LOG_ERROR("%s() received signal %d instead of %d", __func__, sig, uffd_rt_signal_used_); + } +} + +bool PageGuardManager::UffdSetSignalHandler() +{ + // Search for a free RT signal + for (int sig = SIGRTMIN; sig < SIGRTMAX + 1; ++sig) + { + struct sigaction current_handler = {}; + if (sigaction(sig, nullptr, ¤t_handler)) + { + GFXRECON_LOG_ERROR("%s(): sigaction query failed: %s", __func__, strerror(errno)); + continue; + } + + if (current_handler.sa_handler == nullptr && current_handler.sa_sigaction == nullptr) + { + uffd_rt_signal_used_ = sig; + break; + } + } + + if (uffd_rt_signal_used_ == -1) + { + GFXRECON_LOG_ERROR( + "Searched through all RT signals [%d, %d] and no free signal was found", SIGRTMIN, SIGRTMAX); + return false; + } + + // Install signal handler for the RT signal + struct sigaction sa = {}; + sa.sa_flags = SA_SIGINFO; + sigemptyset(&sa.sa_mask); + sa.sa_handler = UffdStaticSignalHandler; + + if (sigaction(uffd_rt_signal_used_, &sa, NULL)) + { + GFXRECON_LOG_ERROR("sigaction failed: %s", strerror(errno)); + uffd_rt_signal_used_ = -1; + return false; + } + + // Have uffd_signal_set_ prepared for when we need to block/unblock the signal + sigemptyset(&uffd_signal_set_); + sigaddset(&uffd_signal_set_, uffd_rt_signal_used_); + + return true; +} + +void PageGuardManager::UffdRemoveSignalHandler() +{ + assert(uffd_rt_signal_used_ != -1); + + struct sigaction sa = {}; + sa.sa_flags = SA_SIGINFO; + sigemptyset(&sa.sa_mask); + sa.sa_handler = nullptr; + + if (sigaction(uffd_rt_signal_used_, &sa, NULL)) + { + GFXRECON_LOG_ERROR("%s() sigaction failed: %s", __func__, strerror(errno)); + } + + uffd_rt_signal_used_ = -1; +} + +void PageGuardManager::UffdTerminate() +{ + if (is_uffd_handler_thread_running_) + { + stop_uffd_handler_thread_ = true; + if (pthread_join(uffd_handler_thread_, nullptr)) + { + GFXRECON_LOG_ERROR("%s() pthread_join failed: %s", __func__, strerror(errno)); + } + } + + std::lock_guard lock(tracked_memory_lock_); + + for (auto& entry : memory_info_) + { + ReleaseTrackedMemory(&entry.second); + } + + if (uffd_fd_ != -1 && close(uffd_fd_)) + { + GFXRECON_LOG_ERROR("Error closing uffd_fd: %s", strerror(errno)); + } + + if (uffd_rt_signal_used_ != -1) + { + UffdRemoveSignalHandler(); + } + + uffd_is_init_ = false; + uffd_fd_ = -1; +} + +bool PageGuardManager::UffdInit() +{ + if (uffd_fd_ == -1) + { + // open the userfault fd + uffd_fd_ = syscall(SYS_userfaultfd, UFFD_USER_MODE_ONLY | O_CLOEXEC | O_NONBLOCK); + if (uffd_fd_ == -1) + { + GFXRECON_LOG_ERROR("syscall/userfaultfd: %s", strerror(errno)); + return false; + } + + // enable for api version and check features + struct uffdio_api uffdio_api; + uffdio_api.api = UFFD_API; + uffdio_api.features = UFFD_FEATURE_THREAD_ID; + if (ioctl(uffd_fd_, UFFDIO_API, &uffdio_api) == -1) + { + GFXRECON_LOG_ERROR("ioctl/uffdio_api: %s", strerror(errno)); + + return false; + } + + if (uffdio_api.api != UFFD_API) + { + GFXRECON_LOG_ERROR("Unsupported userfaultfd api"); + + return false; + } + + const uint64_t required_features[] = { UFFD_FEATURE_THREAD_ID }; + for (size_t i = 0; i < sizeof(required_features) / sizeof(required_features[0]); ++i) + { + if ((uffdio_api.features & required_features[i]) != required_features[i]) + { + GFXRECON_LOG_ERROR("Unsupported userfaultfd feature: 0x%" PRIx64 "\n", required_features[i]); + + return false; + } + } + + const uint64_t requested_ioctls[] = { 0x1 << _UFFDIO_REGISTER }; + for (size_t i = 0; i < sizeof(requested_ioctls) / sizeof(requested_ioctls[0]); ++i) + { + if ((uffdio_api.ioctls & requested_ioctls[i]) != requested_ioctls[i]) + { + GFXRECON_LOG_ERROR("Unsupported userfaultfd ioctl: 0x%" PRIx64 "\n", requested_ioctls[i]); + + return false; + } + } + } + + return true; +} + +bool PageGuardManager::UffdHandleFault(uint64_t address, uint64_t flags, bool wake_thread, uint64_t tid) +{ + assert(protection_mode_ == kUserFaultFdMode); + + MemoryInfo* memory_info; + bool found = FindMemory(reinterpret_cast(address), &memory_info); + + if (found) + { + // Pages are not touched before they are registed so only missing page faults are expected to happen + assert((flags & UFFD_PAGEFAULT_FLAG_WP) != UFFD_PAGEFAULT_FLAG_WP); + + memory_info->uffd_fault_causing_threads.insert(tid); + + memory_info->is_modified = true; + + assert((memory_info != nullptr) && (memory_info->aligned_address != nullptr)); + assert(static_cast(address) >= reinterpret_cast(memory_info->aligned_address)); + + // Get the offset from the start of the first protected memory page to the current address. + const size_t start_offset = + reinterpret_cast(address) - static_cast(memory_info->aligned_address); + const size_t page_index = start_offset >> system_page_pot_shift_; + const size_t page_offset = page_index << system_page_pot_shift_; + assert(start_offset == page_offset); + const size_t segment_size = GetMemorySegmentSize(memory_info, page_index); + assert(segment_size <= system_page_size_); + + assert(!(GFXRECON_PTR_TO_UINT64(memory_info->aligned_address) % system_page_size_)); + assert(memory_info->aligned_offset == 0); + + const bool is_write = (flags & UFFD_PAGEFAULT_FLAG_WRITE) == UFFD_PAGEFAULT_FLAG_WRITE; + if (is_write) + { + memory_info->status_tracker.SetActiveWriteBlock(page_index, true); + } + else + { + memory_info->status_tracker.SetActiveReadBlock(page_index, true); + + if (enable_read_write_same_page_) + { + // The page guard has been removed from this page. If we expect both reads and writes to the page, + // it needs to be marked for active write. + memory_info->status_tracker.SetActiveWriteBlock(page_index, true); + } + } + + // Copy from the mapped memory to the shadow memory. + uint8_t* source_address; + if (segment_size < system_page_size_) + { + // This shouldn't happen, at least not often, as we should warn if mapped memory (and therefore shadow + // memory too) is not page aligned. + util::platform::MemoryCopy(uffd_page_size_tmp_buff_.get(), + segment_size, + static_cast(memory_info->mapped_memory) + page_offset, + segment_size); + + source_address = uffd_page_size_tmp_buff_.get(); + } + else + { + source_address = static_cast(memory_info->mapped_memory) + page_offset; + } + + uint8_t* destination_address = static_cast(memory_info->shadow_memory) + page_offset; + + // Pointers need to be properly casted to uint64_t to avoid sign extension in case it is a 32bit build + struct uffdio_copy copy; + copy.dst = GFXRECON_PTR_TO_UINT64(destination_address); + copy.src = GFXRECON_PTR_TO_UINT64(source_address); + copy.len = system_page_size_; + copy.mode = wake_thread ? 0 : UFFDIO_COPY_MODE_DONTWAKE; + + if (ioctl(uffd_fd_, UFFDIO_COPY, ©)) + { + if (errno == EEXIST) + { + // EEXIST is ok and can happen when for example 2 threads try to access the same page + // simultaneously. The first one will allocate the page when served, so the second one + // will generate EEXIST. + return true; + } + else + { + GFXRECON_LOG_ERROR("ioctl/uffdio_copy errno: %d: %s", errno, strerror(errno)); + GFXRECON_LOG_ERROR(" is_write: %d", is_write); + GFXRECON_LOG_ERROR(" flags: 0x%" PRIx64, flags); + GFXRECON_LOG_ERROR(" destination_address: %p", destination_address); + GFXRECON_LOG_ERROR(" source_address: %p", source_address); + GFXRECON_LOG_ERROR(" copy.dst: 0x%" PRIx64, copy.dst); + GFXRECON_LOG_ERROR(" copy.src: 0x%" PRIx64, copy.src); + GFXRECON_LOG_ERROR(" copy.len: %" PRIu64, system_page_size_); + + return false; + } + } + + if (copy.copy != system_page_size_) + { + GFXRECON_LOG_ERROR("Unexpected copy.copy (%" PRId64 " != %zu)", copy.copy, system_page_size_); + return false; + } + + return true; + } + else + { + GFXRECON_LOG_ERROR("Could not find memory entry containing 0x%" PRIx64); + + return false; + } +} + +bool PageGuardManager::UffdWakeFaultingThread(uint64_t address) +{ + struct uffdio_range uffdio_wake; + uffdio_wake.start = address; + uffdio_wake.len = static_cast(system_page_size_); + if (ioctl(uffd_fd_, UFFDIO_WAKE, &uffdio_wake) == -1) + { + GFXRECON_LOG_ERROR("ioctl/uffdio_wake: %s", strerror(errno)); + return false; + } + + return true; +} + +void* PageGuardManager::UffdHandlerThread(void* args) +{ + GFXRECON_UNREFERENCED_PARAMETER(args); + assert(uffd_fd_ != -1); + + is_uffd_handler_thread_running_ = true; + + while (true) + { + struct pollfd pollfd[1]; + pollfd[0].fd = uffd_fd_; + pollfd[0].events = POLLIN; + + int pollres = poll(pollfd, 1, 200); + + if (stop_uffd_handler_thread_) + { + goto exit; + } + + switch (pollres) + { + case -1: + GFXRECON_LOG_ERROR("%s() poll error: %s", __func__, strerror(errno)); + continue; + break; + case 0: + continue; + break; + case 1: + break; + default: + GFXRECON_LOG_ERROR("%s() poll error: got %d fds instead of 1\n", pollres); + goto exit; + } + + if (pollfd[0].revents & POLLERR) + { + GFXRECON_LOG_ERROR("Poll got POLLERR"); + goto exit; + } + + if (!(pollfd[0].revents & POLLIN)) + { + continue; + } + + int64_t readres; + struct uffd_msg msg[16]; + while ((readres = read(uffd_fd_, &msg, sizeof(msg))) < 0) + { + if (errno == EAGAIN) + { + // Try again + continue; + } + else + { + GFXRECON_LOG_ERROR("read/userfaultfd: %s", strerror(errno)); + goto exit; + } + } + + if (readres % sizeof(uffd_msg)) + { + GFXRECON_LOG_ERROR("Unexpected read size\n"); + goto exit; + } + + if (readres == sizeof(msg)) + { + GFXRECON_LOG_ERROR("Messages might have been lost"); + } + + // Lock here instead of inside UffdHandleFault() in order to avoid multiple locks + unlocks + // in case of multiple messages. + tracked_memory_lock_.lock(); + + const unsigned int n_messages = readres / sizeof(struct uffd_msg); + for (unsigned int i = 0; i < n_messages; ++i) + { + if (msg[i].event == UFFD_EVENT_PAGEFAULT) + { + UffdHandleFault(msg[i].arg.pagefault.address, + msg[i].arg.pagefault.flags, + n_messages == 1, + static_cast(msg[i].arg.pagefault.feat.ptid)); + } + } + + // When there are multiple messages from multiple threads deferre waking + // them up to avoid race conditions + if (n_messages > 1) + { + for (unsigned int i = 0; i < n_messages; ++i) + { + if (msg[i].event == UFFD_EVENT_PAGEFAULT) + { + UffdWakeFaultingThread(msg[i].arg.pagefault.address); + } + } + } + + tracked_memory_lock_.unlock(); + } +exit: + + is_uffd_handler_thread_running_ = false; + + return nullptr; +} + +void* PageGuardManager::UffdHandlerThreadHelper(void* this_) +{ + assert(this_); + + return reinterpret_cast(this_)->UffdHandlerThread(nullptr); +} + +bool PageGuardManager::UffdStartHandlerThread() +{ + assert(uffd_fd_ != -1); + + if (pthread_create(&uffd_handler_thread_, nullptr, UffdHandlerThreadHelper, this)) + { + GFXRECON_LOG_ERROR("%s() pthread_create: %s", __func__, strerror(errno)); + return false; + } + + stop_uffd_handler_thread_ = false; + + return true; +} + +bool PageGuardManager::InitializeUserFaultFd() +{ + assert(!uffd_is_init_); + + uffd_is_init_ = false; + uffd_rt_signal_used_ = -1; + uffd_fd_ = -1; + uffd_page_size_tmp_buff_ = std::make_unique(util::platform::GetSystemPageSize()); + + if (!UffdInit()) + { + goto init_failed; + } + + if (!UffdStartHandlerThread()) + { + goto init_failed; + } + + if (!UffdSetSignalHandler()) + { + goto init_failed; + } + + uffd_is_init_ = true; + + return true; + +init_failed: + UffdTerminate(); + return false; +} + +bool PageGuardManager::UffdRegisterMemory(const void* address, size_t length) +{ + assert(uffd_fd_ != -1); + + struct uffdio_register uffdio_register; + uffdio_register.range.start = GFXRECON_PTR_TO_UINT64(address); + uffdio_register.range.len = length; + uffdio_register.mode = UFFDIO_REGISTER_MODE_MISSING; + if (ioctl(uffd_fd_, UFFDIO_REGISTER, &uffdio_register) == -1) + { + GFXRECON_LOG_ERROR("ioctl/uffdio_register: %s", strerror(errno)); + return false; + } + + const uint64_t zero_page_ioctl = (uint64_t)0x1 << _UFFDIO_ZEROPAGE; + const uint64_t copy_ioctl = (uint64_t)0x1 << _UFFDIO_COPY; + const uint64_t expected_ioctls = zero_page_ioctl | copy_ioctl; + if ((uffdio_register.ioctls & expected_ioctls) != expected_ioctls) + { + GFXRECON_LOG_ERROR("Unexpected userfaultfd ioctl set (expected: 0x%llx got: 0x%llx)\n", + expected_ioctls, + uffdio_register.ioctls); + return false; + } + + return true; +} + +void PageGuardManager::UffdUnregisterMemory(const void* address, size_t length) +{ + assert(uffd_fd_ != -1); + + struct uffdio_range uffdio_unregister; + uffdio_unregister.start = GFXRECON_PTR_TO_UINT64(address); + uffdio_unregister.len = static_cast(length); + if (ioctl(uffd_fd_, UFFDIO_UNREGISTER, &uffdio_unregister) == -1) + { + GFXRECON_LOG_ERROR("ioctl/uffdio_unregister: %s", strerror(errno)); + } +} + +bool PageGuardManager::UffdResetRegion(void* guard_address, size_t guard_range) +{ + // Release pages and allocate fresh ones + madvise(guard_address, guard_range, MADV_DONTNEED); + void* shadow_memory = guard_address; + guard_address = util::platform::AllocateRawMemory(guard_range, false, guard_address); + if (!guard_address) + { + GFXRECON_LOG_ERROR("mmap error: %s", strerror(errno)); + + return false; + } + else if (guard_address != shadow_memory) + { + GFXRECON_LOG_ERROR("MAP_FIXED was not honored when remapping memory"); + + return false; + } + + return UffdRegisterMemory(guard_address, guard_range); +} + +uint32_t PageGuardManager::UffdBlockFaultingThreads(const MemoryInfo* memory_info) +{ + assert(!accessor_threads_in_handler); + + // Set to one before sending the signal to the threads + block_accessor_threads = true; + + uint32_t n_threads_to_wait = 0; + const uint64_t this_tid = util::platform::GetCurrentThreadId(); + for (const auto& entry : memory_info->uffd_fault_causing_threads) + { + // Don't send the signal to this thread. + // Threads that do not exist any more do not count. In this case SendSignalToThread() + // will return an error + if (this_tid != entry && util::platform::SendSignalToThread(entry, uffd_rt_signal_used_) == 0) + { + ++n_threads_to_wait; + } + } + + // If the signal was sent to any threads wait to make sure all of them have entered the signal handler + if (n_threads_to_wait) + { + pthread_mutex_lock(&wait_for_threads_mutex); + while (accessor_threads_in_handler < n_threads_to_wait) + { + pthread_cond_wait(&wait_for_threads_cond, &wait_for_threads_mutex); + } + pthread_mutex_unlock(&wait_for_threads_mutex); + } + + return n_threads_to_wait; +} + +void PageGuardManager::UffdUnblockFaultingThreads(MemoryInfo* memory_info, uint32_t n_threads_to_wait) +{ + if (n_threads_to_wait) + { + pthread_mutex_lock(&reset_regions_mutex); + block_accessor_threads = false; + pthread_cond_broadcast(&reset_regions_cond); + pthread_mutex_unlock(&reset_regions_mutex); + + // Wait for all threads to exit the signal handler. Otherwise there's race condition if we try to + // send another batch of threads into the signal handler and the previous one haven't exited yet. + pthread_mutex_lock(&wait_for_threads_mutex); + while (accessor_threads_in_handler) + { + pthread_cond_wait(&wait_for_threads_cond, &wait_for_threads_mutex); + } + pthread_mutex_unlock(&wait_for_threads_mutex); + } + else + { + // No reason to synch. Just set it to false. + block_accessor_threads = false; + } + + // Reset this, otherwise threads might start accumulate and we will have to sent too many signals + memory_info->uffd_fault_causing_threads.clear(); +} + +void PageGuardManager::UffdBlockRtSignal() +{ + if (uffd_rt_signal_used_ != -1) + { + sigprocmask(SIG_BLOCK, &uffd_signal_set_, NULL); + } +} + +void PageGuardManager::UffdUnblockRtSignal() +{ + if (uffd_rt_signal_used_ != -1) + { + sigprocmask(SIG_UNBLOCK, &uffd_signal_set_, NULL); + } +} + +GFXRECON_END_NAMESPACE(gfxrecon) +GFXRECON_END_NAMESPACE(util) + +#else + +GFXRECON_BEGIN_NAMESPACE(gfxrecon) +GFXRECON_BEGIN_NAMESPACE(util) + +bool PageGuardManager::InitializeUserFaultFd() +{ + return false; +} + +void PageGuardManager::UffdTerminate() {} + +bool PageGuardManager::UffdRegisterMemory(const void* address, size_t length) +{ + GFXRECON_UNREFERENCED_PARAMETER(address); + GFXRECON_UNREFERENCED_PARAMETER(length); + + return false; +} + +void PageGuardManager::UffdUnregisterMemory(const void* address, size_t length) +{ + GFXRECON_UNREFERENCED_PARAMETER(address); + GFXRECON_UNREFERENCED_PARAMETER(length); +} + +uint32_t PageGuardManager::UffdBlockFaultingThreads(const MemoryInfo* memory_info) +{ + GFXRECON_UNREFERENCED_PARAMETER(memory_info); + return 0; +} + +void PageGuardManager::UffdUnblockFaultingThreads(MemoryInfo* memory_info, uint32_t n_threads_to_wait) +{ + GFXRECON_UNREFERENCED_PARAMETER(memory_info); + GFXRECON_UNREFERENCED_PARAMETER(n_threads_to_wait); +} + +bool PageGuardManager::UffdResetRegion(void* guard_address, size_t guard_range) +{ + GFXRECON_UNREFERENCED_PARAMETER(guard_address); + GFXRECON_UNREFERENCED_PARAMETER(guard_range); + + return true; +} + +void PageGuardManager::UffdBlockRtSignal() {} + +void PageGuardManager::UffdUnblockRtSignal() {} + +GFXRECON_END_NAMESPACE(gfxrecon) +GFXRECON_END_NAMESPACE(util) + +#endif // USERFAULTFD_SUPPORTED == 1 diff --git a/framework/util/platform.h b/framework/util/platform.h index 78b0c83fd2..1a895eec7c 100644 --- a/framework/util/platform.h +++ b/framework/util/platform.h @@ -223,7 +223,7 @@ inline size_t GetSystemPageSize() return sSysInfo.dwPageSize; } -inline void* AllocateRawMemory(size_t aligned_size, bool use_write_watch = false) +inline void* AllocateRawMemory(size_t aligned_size, bool use_write_watch = false, void* address = nullptr) { assert(aligned_size > 0); @@ -236,7 +236,7 @@ inline void* AllocateRawMemory(size_t aligned_size, bool use_write_watch = false flags |= MEM_WRITE_WATCH; } - return VirtualAlloc(nullptr, aligned_size, flags, PAGE_READWRITE); + return VirtualAlloc(address, aligned_size, flags, PAGE_READWRITE); } return nullptr; @@ -278,6 +278,13 @@ inline uint64_t GetCurrentThreadId() #endif } +#if !defined(__APPLE__) +inline int SendSignalToThread(uint64_t tid, int signal) +{ + return syscall(SYS_tgkill, getpid(), tid, signal); +} +#endif + inline void TriggerDebugBreak() { raise(SIGTRAP); @@ -505,13 +512,14 @@ inline size_t GetSystemPageSize() } // Memory allocation without extra space for private info like with new or malloc -inline void* AllocateRawMemory(size_t aligned_size, bool use_write_watch = false) +inline void* AllocateRawMemory(size_t aligned_size, bool use_write_watch = false, void* address = nullptr) { assert(aligned_size > 0); if (aligned_size > 0) { - void* memory = mmap(nullptr, aligned_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + const int flags = address ? (MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED) : (MAP_PRIVATE | MAP_ANONYMOUS); + void* memory = mmap(address, aligned_size, PROT_READ | PROT_WRITE, flags, -1, 0); if (memory == MAP_FAILED) {