Add Unsafe Rust Deep Dive (#2806)

Adds the start of an unsafe deep dive to Comprehensive Rust. 

The `unsafe` keyword is easy to type, but hard to master. When used
appropriately, it forms a useful and indeed essential part of the Rust
programming language.

By the end of this deep dive, you'll know how to work with `unsafe` code, 
review others' changes that include the `unsafe` keyword, and produce 
your own.

What you'll learn:

- What the terms undefined behavior, soundness, and safety mean
- Why the `unsafe` keyword exists in the Rust language
- How to write your own code using `unsafe` safely
- How to review `unsafe` code

Here is a tentative outline of a 10h (2 day) treatment:

Day 1: Using and Reviewing Unsafe

- Welcome
- Motivations: explain why the `unsafe` keyword exists
- Foundations: provide background knowledge; what is soundness? what is
undefined behavior? what is validity in respect to pointers?
- Mechanics: what a safe `unsafe` block should look like
- Representations and Interoperability: explore how data is laid out in
memory and how that can be sent across the wire and/or stored on disk.
- Reviewing unsafe
- Patterns for safer unsafe: Encapsulating unsafe code in safe-to-use
abstractions, such as marking a type's constructor as `unsafe` so that
invariants only need to be enforced once by the programmer.

Day 2: Deploying Unsafe to Build Abstractions

- Welcome
- Validity in detail: A refresher. Emphasis on the details of the
invariants that are being upheld by a “typical” unsafe block, such as
aliasing, alignment, data validity, padding.
- Concurrency and thread safety: understanding `Send` and `Sync`,
knowing how to implement them on a user-defined type
- Case study: Small string optimization
- Case study: Zero-copy parsing
- Review

---------

Co-authored-by: Dmitri Gribenko <[email protected]>

Author: timClicks

src/SUMMARY.md

@@ -440,6 +440,23 @@
 
 ---
 
+# Unsafe
+
+- [Welcome](unsafe-deep-dive/welcome.md)
+- [Setup](unsafe-deep-dive/setup.md)
+- [Motivations](unsafe-deep-dive/motivations.md)
+  - [Interoperability](unsafe-deep-dive/motivations/interop.md)
+  - [Data Structures](unsafe-deep-dive/motivations/data-structures.md)
+  - [Performance](unsafe-deep-dive/motivations/performance.md)
+- [Foundations](unsafe-deep-dive/foundations.md)
+  - [What is unsafe?](unsafe-deep-dive/foundations/what-is-unsafe.md)
+  - [When is unsafe used?](unsafe-deep-dive/foundations/when-is-unsafe-used.md)
+  - [Data structures are safe](unsafe-deep-dive/foundations/data-structures-are-safe.md)
+  - [Actions might not be](unsafe-deep-dive/foundations/actions-might-not-be.md)
+  - [Less powerful than it seems](unsafe-deep-dive/foundations/less-powerful.md)
+
+---
+
 # Final Words
 
 - [Thanks!](thanks.md)

src/running-the-course/course-structure.md

@@ -82,6 +82,15 @@ You should be familiar with the material in
 
 {{%course outline Idiomatic Rust}}
 
+### Unsafe (Work in Progress)
+
+The [Unsafe](../unsafe-deep-dive/welcome.md) deep dive is a two-day class on the
+_unsafe_ Rust language. It covers the fundamentals of Rust's safety guarantees,
+the motivation for `unsafe`, review process for `unsafe` code, FFI basics, and
+building data structures that the borrow checker would normally reject.
+
+{{%course outline Unsafe}}
+
 ## Format
 
 The course is meant to be very interactive and we recommend letting the

src/unsafe-deep-dive/Cargo.toml

@@ -0,0 +1,5 @@
+# Foundations
+
+Some fundamental concepts and terms.
+
+{{%segment outline}}

src/unsafe-deep-dive/foundations.md

@@ -0,0 +1,19 @@
+---
+minutes: 2
+---
+
+# ... but actions on them might not be
+
+```rust
+fn main() {
+    let n: i64 = 12345;
+    let safe = &n as *const _;
+    println!("{safe:p}");
+}
+```
+
+<details>
+
+Modify the example to de-reference `safe` without an `unsafe` block.
+
+</details>

src/unsafe-deep-dive/foundations/actions-might-not-be.md

@@ -0,0 +1,25 @@
+---
+minutes: 2
+---
+
+# Data structures are safe ...
+
+Data structures are inert. They cannot do any harm by themselves.
+
+Safe Rust code can create raw pointers:
+
+```rust
+fn main() {
+    let n: i64 = 12345;
+    let safe = &raw const n;
+    println!("{safe:p}");
+}
+```
+
+<details>
+
+Consider a raw pointer to an integer, i.e., the value `safe` is the raw pointer
+type `*const i64`. Raw pointers can be out-of-bounds, misaligned, or be null.
+But the unsafe keyword is not required when creating them.
+
+</details>

src/unsafe-deep-dive/foundations/data-structures-are-safe.md

@@ -0,0 +1,52 @@
+---
+minutes: 10
+---
+
+# Less powerful than it seems
+
+The `unsafe` keyword does not allow you to break Rust.
+
+```rust,ignore
+use std::mem::transmute;
+
+let orig = b"RUST";
+let n: i32 = unsafe { transmute(orig) };
+
+println!("{n}")
+```
+
+<details>
+
+## Suggested outline
+
+- Request that someone explains what `std::mem::transmute` does
+- Discuss why it doesn't compile
+- Fix the code
+
+## Expected compiler output
+
+```ignore
+   Compiling playground v0.0.1 (/playground)
+error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
+ --> src/main.rs:5:27
+  |
+5 |     let n: i32 = unsafe { transmute(orig) };
+  |                           ^^^^^^^^^
+  |
+  = note: source type: `&[u8; 4]` (64 bits)
+  = note: target type: `i32` (32 bits)
+```
+
+## Suggested change
+
+```diff
+- let n: i32 = unsafe { transmute(orig) };
++ let n: i64 = unsafe { transmute(orig) };
+```
+
+## Notes on less familiar Rust
+
+- the `b` prefix on a string literal marks it as byte slice (`&[u8]`) rather
+  than a string slice (`&str`)
+
+</details>

src/unsafe-deep-dive/foundations/less-powerful.md

@@ -0,0 +1,98 @@
+---
+minutes: 6
+---
+
+# What is “unsafety”?
+
+Unsafe Rust is a superset of Safe Rust.
+
+Let's create a list of things that are enabled by the `unsafe` keyword.
+
+<details>
+
+## Definitions from authoritative docs:
+
+From the [unsafe keyword's documentation]():
+
+> Code or interfaces whose memory safety cannot be verified by the type system.
+>
+> ...
+>
+> Here are the abilities Unsafe Rust has in addition to Safe Rust:
+>
+> - Dereference raw pointers
+> - Implement unsafe traits
+> - Call unsafe functions
+> - Mutate statics (including external ones)
+> - Access fields of unions
+
+From the [reference](https://doc.rust-lang.org/reference/unsafety.html)
+
+> The following language level features cannot be used in the safe subset of
+> Rust:
+>
+> - Dereferencing a raw pointer.
+> - Reading or writing a mutable or external static variable.
+> - Accessing a field of a union, other than to assign to it.
+> - Calling an unsafe function (including an intrinsic or foreign function).
+> - Calling a safe function marked with a target_feature from a function that
+>   does not have a target_feature attribute enabling the same features (see
+>   attributes.codegen.target_feature.safety-restrictions).
+> - Implementing an unsafe trait.
+> - Declaring an extern block.
+> - Applying an unsafe attribute to an item.
+
+## Group exercise
+
+> You may have a group of learners who are not familiar with each other yet.
+> This is a way for you to gather some data about their confidence levels and
+> the psychological safety that they're feeling.
+
+### Part 1: Informal definition
+
+> Use this to gauge the confidence level of the group. If they are uncertain,
+> then tailor the next section to be more directed.
+
+Ask the class: **By raising your hand, indicate if you would feel comfortable
+defining unsafe?**
+
+If anyone's feeling confident, allow them to try to explain.
+
+### Part 2: Evidence gathering
+
+Ask the class to spend 3-5 minutes.
+
+- Find a use of the unsafe keyword. What contract/invariant/pre-condition is
+  being established or satisfied?
+- Write down terms that need to be defined (unsafe, memory safety, soundness,
+  undefined behavior)
+
+### Part 3: Write a working definition
+
+### Part 4: Remarks
+
+Mention that we'll be reviewing our definition at the end of the day.
+
+## Note: Avoid detailed discussion about precise semantics of memory safety
+
+It's possible that the group will slide into a discussion about the precise
+semantics of what memory safety actually is and how define pointer validity.
+This isn't a productive line of discussion. It can undermine confidence in less
+experienced learners.
+
+Perhaps refer people who wish to discuss this to the discussion within the
+official [documentation for pointer types] (excerpt below) as a place for
+further research.
+
+> Many functions in [this module] take raw pointers as arguments and read from
+> or write to them. For this to be safe, these pointers must be _valid_ for the
+> given access.
+>
+> ...
+>
+> The precise rules for validity are not determined yet.
+
+[this module]: https://doc.rust-lang.org/std/ptr/index.html
+[documentation for pointer types]: https://doc.rust-lang.org/std/ptr/index.html#safety
+
+</details>

src/unsafe-deep-dive/foundations/what-is-unsafe.md

@@ -0,0 +1,48 @@
+---
+minutes: 2
+---
+
+# When is unsafe used?
+
+The unsafe keyword indicates that the programmer is responsible for upholding
+Rust's safety guarantees.
+
+The keyword has two roles:
+
+- define pre-conditions that must be satisfied
+- assert to the compiler (= promise) that those defined pre-conditions are
+  satisfied
+
+## Further references
+
+- [The unsafe keyword chapter of the Rust Reference](https://doc.rust-lang.org/reference/unsafe-keyword.html)
+
+<details>
+
+Places where pre-conditions can be defined (Role 1)
+
+- [unsafe functions] (`unsafe fn foo() { ... }`). Example: `get_unchecked`
+  method on slices, which requires callers to verify that the index is
+  in-bounds.
+- unsafe traits (`unsafe trait`). Examples: [`Send`] and [`Sync`] marker traits
+  in the standard library.
+
+Places where pre-conditions must be satisfied (Role 2)
+
+- unsafe blocks (`unafe { ... }`)
+- implementing unsafe traits (`unsafe impl`)
+- access external items (`unsafe extern`)
+- adding
+  [unsafe attributes](https://doc.rust-lang.org/reference/attributes.html) o an
+  item. Examples: [`export_name`], [`link_section`] and [`no_mangle`]. Usage:
+  `#[unsafe(no_mangle)]`
+
+[unsafe functions]: https://doc.rust-lang.org/reference/unsafe-keyword.html#unsafe-functions-unsafe-fn
+[unsafe traits]: https://doc.rust-lang.org/reference/unsafe-keyword.html#unsafe-traits-unsafe-trait
+[`export_name`]: https://doc.rust-lang.org/reference/abi.html#the-export_name-attribute
+[`link_section`]: https://doc.rust-lang.org/reference/abi.html#the-link_section-attribute
+[`no_mangle`]: https://doc.rust-lang.org/reference/abi.html#the-no_mangle-attribute
+[`Send`]: https://doc.rust-lang.org/std/marker/trait.Send.html
+[`Sync`]: https://doc.rust-lang.org/std/marker/trait.Sync.html
+
+</details>

src/unsafe-deep-dive/foundations/when-is-unsafe-used.md

@@ -0,0 +1,24 @@
+---
+minutes: 1
+---
+
+# Motivations
+
+We know that writing code without the guarantees that Rust provides ...
+
+> “Use-after-free (UAF), integer overflows, and out of bounds (OOB) reads/writes
+> comprise 90% of vulnerabilities with OOB being the most common.”
+>
+> --— **Jeff Vander Stoep and Chong Zang**, Google.
+> "[Queue the Hardening Enhancements](https://security.googleblog.com/2019/05/queue-hardening-enhancements.html)"
+
+... so why is `unsafe` part of the language?
+
+{{%segment outline}}
+
+<details>
+
+The `unsafe` keyword exists because there is no compiler technology available
+today that makes it obsolete. Compilers cannot verify everything.
+
+</details>