sched: Move up affinity check to mitigate useless redoing overhead