Fix overflow check in unroll cost calculation