WIP: Tyche support #1
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Adding a placeholder for Tyche I copied the linux-sgx folder and adapted it to compile without the sgx driver. I relied on a placeholder.h file to provide the missing definitions. We will have to figure out how to replace the implementation. We can also use the skeleton folder to guide where we have a look. We could have started from this one but let's try it this way first to speed things up? 2. Infrastructure to install gramine-tyche I modified the meson build to generate a gramine-tyche wrapper script (similar to gramine-direct and gramine-sgx). There might still be some adjustments to be made, but it seems to work on my machine and will speed up testing and dev. 3. Adding the dependencies for Tyche I managed to configure meson to import basically all the driver and sdk libraries we need. It mostly imports headers but add the source for the backend (which is all we care about).
I replaced the add pages logic to copy the enclave content and call the backend_td_register_region. The code now crashes in init_enclave (which should be our commit). I need to backtrack to figure out stacks and shared memory (potentially the dump frames for sgx state too).
Started configuring per-core entry info.
I added the build support and code to generate the page tables for the enclave. So far, we only map the confidential memory regions. Next: figure out where we're at inside the enclave.
Gramine requires FOUR threads ... so we need to make things work with 4 threads.
I had to disable rflag motivations. I hid this behind a compiler flag NO_RFLAGS=1 that gets passed via -D if we are building the tyche backend
Due to TCS not being mapped to cores in SGX, they have a complex scheme to notify the thread exited which is part of the thread reset of the enclave. In our case, we will do things slightly differently and just clear the id upon exit, because we have pinned threads to cores.
The tyche backend now supports switchless calls
WORK IN PROGRESS
We can therefore remove the MAX_ORDER check in the code and simply get one mmap. This simplifies a lot the logic but also gives a neater display of memory regions in info proc mappings.
This requires memory that's not mapped by the driver.
Redis implementation seems to function now thanks to actual mmap/munmap implementation.
The problem with blender is that it needs 64 cores... If we ever get to the point of managing to run "threads" rather than cores, we might be able to handle it.
If we find a solution for threads that would be awesome.
We might need to run domains as sandboxes (if we do not have confidential VMs). We also need to experiment with the quantum for scheduling to debug performance. I added two new toml entries in the manifest to tyche domains. They are both available under sgx blocks e.g.: [sgx] quantum = 10 sandbox = true The default values are sandbox = false and quantum = 10. Quantum must be between 10 and 62, as it is a shift for the vmx timer.
1. Page table bug: The page tables in gramine were bugged. We still assumed that the driver only returned contiguous regions when actually on physoffset is valid only for the start address (that is part of us attempting to make it easy to mmap large regions and automatically handling splits). As a result, now, I need to keep track of whitin the page table region, what are the physical addresses. I reused the dll structure and basically do all that work during initialization of the pt mapper. 2. MMap in shared memory: when allocating shared memory we're trying to do placement to fixed addresses. For some reason the code reads an address with a bitflipped mask from mmap, e.g., 0xfffffffffff40000 instead of 0x40000. Profiling/Debuging showed that mmap does return the right address but I cannot yet figure out where the bitflip happens. For now we patch the result to let things go through.
We add the sqlite source folder and compile the speedtest from source.
The type cast was wrong and lead to 2's complement sign extension.
|
Are there stuff you would like me to look at in particular among the 1M line of new code?^^ |
The interface was updated to support finer-granularity catching of interrupts.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Just creating a PR to easily compare the changes :)