audioaggregate: Don't hold object locks across calls to aggregate_one