aggregator: Release pads' peeked buffer when removing the pad or finalizing it