x86: properly respect rex/{rex}

x86: properly respect rex/{rex}

This addresses two issues: For one, a user specified "rex" did not cause
the diagnostic to trigger when there was no other need for a REX prefix;
instead, another than the specified insn+operands was encoded. And then
(which is what this started from) .insn didn't respect {rex} (and was
otherwise similarly flawed as ordinary insns).

The latter requires splitting out code from md_assemble(), for it to
become re-usable. Besides the addition to address the first issue, that
code then also needs generalizing to account for immediate operands, as
with .insn we can't make assumptions anymore on all respective templates
having at most two operands (we still can build upon there being at most
two non-immediate operands, though). While moving the code also simplify
the first if(), by folding redundant checks.

In the new testcase also test a few more things which afaics weren't
tested till now.