aggregator: Use the sinkpads iterator directly to query upstream latencies