When an object goes away, remove from queue of pending cache additions