x86: reject further invalid AVX-512 masking constructs

x86: reject further invalid AVX-512 masking constructs

For one the register type used for masking should be validated. And then
we shouldn't accept input producing encodings which will #UD when
executed, as is the case when EVEX.Z is set while EVEX.AAA is zero.