Mutex deadlocks in production: the patterns I found in my codebase and how I diagnosed them

Three deadlocks in production, all with the same face: the service stopped responding — no error, no panic, no log. What I found while diagnosing them changed how I think about lock design in async Rust.