refactor ClientManager

DEVELOPING.md

@@ -80,3 +80,22 @@ To debug a hang, the best thing to do is to get a stack trace. Go's nice, and yo
     $ kill -ABRT <procid>
 
 This will kill Oragono and print out a stack trace for you to take a look at.
+
+## Concurrency design
+
+Oragono involves a fair amount of shared state. Here are some of the main points:
+
+1. Each client has a separate goroutine that listens for incoming messages and synchronously processes them.
+1. All sends to clients are asynchronous; `client.Send` appends the message to a queue, which is then processed on a separate goroutine. It is always safe to call `client.Send`.
+1. The server has a few of its own goroutines, for listening on sockets and handing off new client connections to their dedicated goroutines.
+1. A few tasks are done asynchronously in ad-hoc goroutines.
+
+In consequence, there is a lot of state (in particular, server and channel state) that can be read and written from multiple goroutines. This state is protected with mutexes. To avoid deadlocks, mutexes are arranged in "tiers"; while holding a mutex of one tier, you're only allowed to acquire mutexes of a strictly *higher* tier. The tiers are:
+
+1. Tier 1 mutexes: these are the "innermost" mutexes. They typically protect getters and setters on objects, or invariants that are local to the state of a single object. Example: `Channel.stateMutex`.
+1. Tier 2 mutexes: these protect some invariants of their own, but also need to access fields on other objects that themselves require synchronization. Example: `ChannelManager.RWMutex`.
+1. Tier 3 mutexes: these protect macroscopic operations, where it doesn't make sense for more than one to occur concurrently. Example; `Server.rehashMutex`, which prevents rehashes from overlapping.
+
+There are some mutexes that are "tier 0": anything in a subpackage of `irc` (e.g., `irc/logger` or `irc/connection_limits`) shouldn't acquire mutexes defined in `irc`.
+
+We are using `buntdb` for persistence; a `buntdb.DB` has an `RWMutex` inside it, with read-write transactions getting the `Lock()` and read-only transactions getting the `RLock()`. We haven't completely decided where this lock fits into the overall lock model. For now, it's probably better to err on the side of caution: if possible, don't acquire new locks inside the `buntdb` transaction, and be careful about what locks are held around the transaction as well.