keymap: be more careful in xkb_keymap_unref