Enable guest tracing

[dblnz]

Description

This PR updates the work @syntactically has done in #103 to latest main and extends it to support tracing the guest for performance.
This effort is part of the #669 issue.
Improvements on the trace file output will be addressed in #704 .

How it works

It defines a TraceBuffer in the guest that holds TraceRecords. These records are only sent to the host when the buffer gets full, or an exit (Out/Halt instructions) is anticipated.
The records are sent to the host using an Out instruction and providing the pointer to an array of TraceRecords.
The host accesses the guest memory, reads these records and writes them to the trace file.

The guest uses the Invariant TSC and rdtsc instruction to read the clock cycles in the guest and calculate the timestamp on the host.
The timestamp is relative to when the TraceInfo started (before creating the Hypervisor Driver in this case).

Two additional crates define the guest logic for tracing. One defines some macros meant to make the annotation of functions simpler to the user

#[hyperlight_guest_tracing_macro::trace_function]
fn function_to_trace() {}

Note: The data processing could be improved (maybe OTEL), I've just used what we had at hand.

Results

Below are some results I have generated based on our hello-world example in the hyperlight repository.

1. My host machine on WSL2 running KVM on Intel(R) Core(TM) Ultra 7 165H

Has Invariant TSC

[338µs    ] BLAKE3 hash of binary is 05e1c8a51d535507af833a5f4e5e4033a42f4e395cdba4cfe7453dcffe3ea76d
[58.197ms ] > entrypoint
[58.294ms ]   > load_gdt
[58.314ms ]   < load_gdt
[58.314ms ]   > load_idt
[58.328ms ]     > init_idt
[58.356ms ]     < init_idt
[58.356ms ]   < load_idt
[58.44ms  ]   hyperlight_main
[58.574ms ] > halt
[59.385ms ] > dispatch_function
[59.386ms ]   > internal_dispatch_function
[59.408ms ]     > try_pop_shared_input_data_into
[59.423ms ]       Start converting buffer
[59.566ms ]       Finish converting buffer
[59.567ms ]     < try_pop_shared_input_data_into
[59.567ms ]     > call_guest_function
[59.567ms ]       Calling guest function
[59.628ms ]       > call_host_function
[59.628ms ]         > call_host_function_without_returning_result
[59.671ms ]           > push_shared_output_data
[59.683ms ]             Start copy of data
[59.683ms ]             Finish copy of data
[59.683ms ]           < push_shared_output_data
[59.683ms ]           > out32
[60.936ms ]           < out32
[60.936ms ]         < call_host_function_without_returning_result
[60.937ms ]         > get_host_return_value
[61.002ms ]           > try_pop_shared_input_data_into
[61.002ms ]             Start converting buffer
[61.046ms ]             Finish converting buffer
[61.046ms ]           < try_pop_shared_input_data_into
[61.046ms ]         < get_host_return_value
[61.046ms ]       < call_host_function
[61.05ms  ]     < call_guest_function
[61.05ms  ]     > push_shared_output_data
[61.05ms  ]       Start copy of data
[61.05ms  ]       Finish copy of data
[61.05ms  ]     < push_shared_output_data
[61.05ms  ]   < internal_dispatch_function
[61.051ms ] > halt

2. Azure VM running mshv3 on AMD EPYC 7763 64-Core Processor

Does not report an invariant TSC. I am currently investigating if I can get an Azure VM that reports supporting an invariant TSC.

[67µs     ] BLAKE3 hash of binary is 56561e9811e4a1907f2883cf34345cab6e48ad444201d8a9678860f46749dd41
[941µs    ] > entrypoint
[943µs    ]   > load_gdt
[943µs    ]   < load_gdt
[943µs    ]   > load_idt
[943µs    ]     > init_idt
[944µs    ]     < init_idt
[944µs    ]   < load_idt
[945µs    ]   hyperlight_main
[956µs    ] > halt
[1.431ms  ] > dispatch_function
[1.446ms  ]   > internal_dispatch_function
[1.446ms  ]     > try_pop_shared_input_data_into
[1.446ms  ]       Start converting buffer
[1.448ms  ]       Finish converting buffer
[1.448ms  ]     < try_pop_shared_input_data_into
[1.448ms  ]     > call_guest_function
[1.449ms  ]       Calling guest function
[1.45ms   ]       > call_host_function
[1.45ms   ]         > call_host_function_without_returning_result
[1.452ms  ]           > push_shared_output_data
[1.453ms  ]             Start copy of data
[1.453ms  ]             Finish copy of data
[1.453ms  ]           < push_shared_output_data
[1.453ms  ]           > out32
[1.908ms  ]           < out32
[1.908ms  ]         < call_host_function_without_returning_result
[1.909ms  ]         > get_host_return_value
[1.909ms  ]           > try_pop_shared_input_data_into
[1.909ms  ]             Start converting buffer
[1.91ms   ]             Finish converting buffer
[1.91ms   ]           < try_pop_shared_input_data_into
[1.91ms   ]         < get_host_return_value
[1.91ms   ]       < call_host_function
[1.912ms  ]     < call_guest_function
[1.912ms  ]     > push_shared_output_data
[1.912ms  ]       Start copy of data
[1.913ms  ]       Finish copy of data
[1.913ms  ]     < push_shared_output_data
[1.913ms  ]   < internal_dispatch_function
[1.913ms  ] > halt

TODO

• For now, the timestamp of the traces is not calculated correctly. It needs to be calculated relative to the start of the guest and base of on the invariant TSC frequency.
• Check the features introduced in the crates to ensure consistency
• Expand dump_trace functionality to visualize the traces
• Investigate invariant TSC not being correctly reported in Azure VMs.

[dblnz]

The output data should be changed to something that other tools can open #704.

[syntactically]

Does not report an invariant TSC. I am currently investigating if I can get an Azure VM that reports supporting an invariant TSC.

Just to document this: it was confirmed that the host under a nested MSHV will incorrectly not see the invariant TSC bit, and this should be ignored; on the azure platforms, at least, the TSC is in fact invariant.

[jsturtevant]

Great work! Could we add a step where the test binaries are executing in the CI pipeline? Right now it looks like none of this code is being exercised since it is all behind feature flag.

[jsturtevant]

Why all the deadcode allows?

[syntactically]

I think these are to get rid of clippy warnings when you build with the basic trace_guest infrastructure support but not with any of the things that use it. Maybe another option here is to replace the feature with a cfg item that is any(features_which_use_this)

[dblnz]

Actually, this is something weird for which I spent some time and I wasn't able to find out why it happened.
The commit in question is : [hyperlight_host/trace] Support collecting guest stacktraces after which all these clippy warnings appear. The weird thing is I couldn't find where this commit actually stopped using these functions.
I'll have another look at it.