Add documentation on how to use the tracing functionality

Signed-off-by: Doru Blânzeanu <[email protected]>

Author: dblnz

docs/hyperlight-metrics-logs-and-traces.md

@@ -91,3 +91,77 @@ NOTE: when running this on windows that this is a linux container, so you will n
 ```
 
 Once the container or the exe is running, the trace output can be viewed in the jaeger UI at [http://localhost:16686/search](http://localhost:16686/search).
+
+## Guest Tracing, Unwinding, and Memory Profiling
+
+Hyperlight provides advanced observability features for guest code running inside micro virtual machines. You can enable guest-side tracing, stack unwinding, and memory profiling using the `trace_guest`, `unwind_guest`, and `mem_profile` features. This section explains how to build, run, and inspect guest traces.
+
+The following features are available for guest tracing:
+- `trace_guest`: Enables tracing for guest code, capturing function calls and execution time.
+- `unwind_guest`: Enables stack unwinding for guest code, allowing you to capture stack traces.
+- `mem_profile`: Enables memory profiling for guest code, capturing memory allocations and usage.
+
+### Building a Guest with Tracing Support
+
+To build a guest with tracing enabled, use the following commands:
+
+```bash
+just build-rust-guests debug trace_guest
+just move-rust-guests debug
+```
+
+This will build the guest binaries with the `trace_guest` feature enabled and move them to the appropriate location for use by the host.
+
+### Running a Hyperlight Example with Guest Tracing
+
+Once the guest is built, you can run a Hyperlight example with guest tracing enabled. For example:
+
+```bash
+cargo run --example hello-world --features trace_guest
+```
+
+This will execute the `hello-world` example, loading the guest with tracing enabled. During execution, trace data will be collected and written to a file in the `trace` directory.
+
+### Inspecting Guest Trace Files
+
+To inspect the trace file generated by the guest, use the `trace_dump` crate. You will need the path to the guest symbols and the trace file. Run the following command:
+
+```bash
+cargo run -p trace_dump <path_to_guest_symbols> <trace_file_path> list_frames
+```
+
+Replace `<path_to_guest_symbols>` with the path to the guest binary or symbol file, and `<trace_file_path>` with the path to the trace file in the `trace` directory.
+
+This command will list the stack frames and tracing information captured during guest execution, allowing you to analyze guest behavior, stack traces, and memory usage.
+
+#### Example
+
+```bash
+cargo run -p trace_dump ./src/tests/rust_guests/bin/debug/simpleguest ./trace/<UUID>.trace list_frames
+```
+
+You can use additional features such as `unwind_guest` and `mem_profile` by enabling them during the build and run steps.
+
+> **Note:** Make sure to follow the build and run steps in order, and ensure that the guest binaries are up to date before running the host example.
+
+## System Prerequisites for `trace_dump`
+
+To build and use the `trace_dump` crate and related guest tracing features, you must have the following system libraries and development tools installed on your system:
+
+- **glib-2.0** development files  
+  - Fedora/RHEL/CentOS:  
+    ```bash
+    sudo dnf install glib2-devel pkgconf-pkg-config
+    ```
+- **cairo** and **cairo-gobject** development files  
+  - Fedora/RHEL/CentOS:  
+    ```bash
+    sudo dnf install cairo-devel cairo-gobject-devel
+    ```
+- **pango** development files  
+  - Fedora/RHEL/CentOS:  
+    ```bash
+    sudo dnf install pango-devel
+    ```
+
+These libraries are required by Rust crates such as `glib-sys`, `cairo-sys-rs`, and `pango-sys`, which are dependencies of the tracing and visualization tools. If you encounter errors about missing `.pc` files (e.g., `glib-2.0.pc`, `cairo.pc`, `pango.pc`), ensure the corresponding `-devel` packages are installed.