Merge pull request #66 from arielb1/decode-no-pollcatch

feat: improve handling of decoding non-pollcatch samples

Author: arielb1

README.md

@@ -81,14 +81,19 @@ The metadata is not used by the agent directly, and only provided to the reporte
 If you want to find long poll times, and you have `RUSTFLAGS="--cfg tokio_unstable"`, you can
 emit `tokio.PollCatchV1` events this way:
 
-```
+```rust
     #[cfg(tokio_unstable)]
     {
         rt.on_before_task_poll(|_| async_profiler_agent::pollcatch::before_poll_hook())
             .on_after_task_poll(|_| async_profiler_agent::pollcatch::after_poll_hook());
     }
 ```
 
+If you can't use `tokio_unstable`, it is possible to wrap your tasks by instrumentation that calls
+`before_poll_hook` before user code runs and `after_poll_hook` after user code runs, but that
+runs the risk of forgetting to instrument the task that is actually causing the high latency,
+and therefore it is strongly recommended to use `on_before_task_poll`/`on_after_task_poll`.
+
 ### Not enabling the AWS SDK / Reqwest default features
 
 The `aws-metadata-no-defaults` and `s3-no-defaults` feature flags do not enable feature flags for the AWS SDK and `reqwest`. 
@@ -139,6 +144,50 @@ The output should look like this
 
 If it does not work, make sure you are using the most recent version of `async-profiler` and that you enabled the pollcatch hooks.
 
+### Missing samples
+
+If you are missing pollcatch samples, you can also run the decoder with `--include-non-pollcatch` and a time
+filter of 0 us to see all samples, as in:
+
+```
+./decoder/target/release/pollcatch-decoder longpolls --include-non-pollcatch --zip profile_WHATEVER_*.zip
+```
+
+#### Samples are still not showing
+
+If the samples are still missing, this is probably a problem with async-profiler or its use. Make sure you are using
+async-profiler correctly.
+
+#### Sample found with "sample with no pollcatch information"
+
+If this discovers the samples as "sample with no pollcatch information", for example:
+
+```
+[87.831540] thread 1878 - sample with no pollcatch information
+ -   1: libc.so.6.clock_nanosleep
+ -  52 more frame(s) (pass --stack-depth=53 to show)
+```
+
+It is likely that you do not have pollcatch enabled correctly. Make sure you are using the [`before_poll_hook`
+and `after_poll_hook` hooks][pollcatch], which normally requires using `--cfg tokio_unstable`. Also make sure that
+you are using async-profiler that supports user events (`v4.0` or higher, if you are using official versions
+from upstream).
+
+[pollcatch]: #pollcatch
+
+### Dealing with `unknown` frames / truncated stack traces
+
+If you are seeing `unknown` frames, this indicates that `async-profiler` has problems finding unwinding
+information. There are 2 normal causes for that:
+
+1. Your code, or the code of some library you are using, is missing debuginfo. To fix that, compile your
+   code with `debuginfo` or at least `debuginfo-lines`.
+2. The running stack frame contains code that is generated by a JIT. Current versions of `async-profiler`
+   support unwinding JVM frames, but frames generated by other JITs will show up as `unknown`.
+
+`async-profiler` is often not capable of unwinding past `unknown` frames, so they will often "truncate"
+the stack trace which will not go past one or two `unknown` frames.
+
 [`jfrs`]: https://docs.rs/jfrs
 
 ## Security
@@ -147,4 +196,4 @@ See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more inform
 
 ## License
 
-This project is licensed under the Apache-2.0 License.
+This project is licensed under the Apache-2.0 License.

decoder/Cargo.lock

@@ -18,3 +18,6 @@ tracing = "0.1"
 tracing-subscriber = { version = "0.3", features = ["fmt"] }
 zip = "2.6"
 serde = "1"
+
+[dev-dependencies]
+test-case = "3"