Fix deadlock in g_object_remove_toggle_ref()