basesrc: Don't reallocate buffers when flushing