Fix set implementation to be truly threadsafe even with destroy() callbacks