perf symbols: Handle different endians properly during symbol load