Simplify the locking and remove some deadlocks