config: Use config file name and entry when deciding removal