perf arm-spe: Don't set data source if it's not a memory operation