aggregator: Avoid destroying sources we do not own