buildman: Make the toolchain test more forgiving