i386: Optimize away shift count masking of shifts/rotates some more [PR105778]

i386: Optimize away shift count masking of shifts/rotates some more [PR105778]

As the following testcase shows, our x86 backend support for optimizing
out useless masking of shift/rotate counts when using instructions
that naturally modulo the count themselves is insufficient.
The *_mask define_insn_and_split patterns use
(subreg:QI (and:SI (match_operand:SI) (match_operand "const_int_operand")))
for the masking, but that can catch only the case where the masking
is done in SImode, so typically in SImode in the source.
We then have another set of patterns, *_mask_1, which use
(and:QI (match_operand:QI) (match_operand "const_int_operand"))
If the masking is done in DImode or in theory in HImode, we don't match
it.
The following patch does 4 different things to improve this:
1) drops the mode from AND and MATCH_OPERAND inside of the subreg:QI
   and replaces that by checking that the register shift count has
   SWI48 mode - I think doing it this way is cheaper than adding
   another mode iterator to patterns which use already another mode
   iterator and sometimes a code iterator as well
2) the doubleword shift patterns were only handling the case where
   the shift count is masked with a constant that has the most significant
   bit clear, i.e. where we know the shift count is less than half the
   number of bits in double-word.  If the mask is equal to half the
   number of bits in double-word minus 1, the masking was optimized
   away, otherwise the AND was kept.
   But if the most significant bit isn't clear, e use a word-sized shift
   and SHRD instruction, where the former does the modulo and the latter
   modulo with 64 / 32 depending on what mode the CPU is in (so 64 for
   128-bit doubleword and 32 or 64-bit doubleword).  So we can also
   optimize away the masking when the mask has all the relevant bits set,
   masking with the most significant bit will remain for the cmove
   test.
3) as requested, this patch adds a bunch of force_reg calls before
   gen_lowpart
4) 1-3 above unfortunately regressed
   +FAIL: gcc.target/i386/bt-mask-2.c scan-assembler-not and[lq][ \\t]
   +FAIL: gcc.target/i386/pr57819.c scan-assembler-not and[lq][ \\t]
   where we during combine match the new pattern we didn't match
   before and in the end don't match the pattern we were testing for.
   These 2 tests are fixed by the *jcc_bt<mode>_mask_1 pattern
   addition and small tweak to target rtx_costs, because even with
   the pattern around we'd refuse to match it because it appeared to
   have higher instruction cost

2022-06-02  Jakub Jelinek  <jakub@redhat.com>

	PR target/105778
	* config/i386/i386.md (*ashl<dwi>3_doubleword_mask): Remove :SI
	from AND and its operands and just verify operands[2] has HImode,
	SImode or for TARGET_64BIT DImode.  Allow operands[3] to be a mask
	with all low 6 (64-bit) or 5 (32-bit) bits set and in that case
	just throw away the masking.  Use force_reg before calling
	gen_lowpart.
	(*ashl<dwi>3_doubleword_mask_1): Allow operands[3] to be a mask
	with all low 6 (64-bit) or 5 (32-bit) bits set and in that case
	just throw away the masking.
	(*ashl<mode>3_doubleword): Rename to ...
	(ashl<mode>3_doubleword): ... this.
	(*ashl<mode>3_mask): Remove :SI from AND and its operands and just
	verify operands[2] has HImode, SImode or for TARGET_64BIT DImode.
	Use force_reg before calling gen_lowpart.
	(*<insn><mode>3_mask): Likewise.
	(*<insn><dwi>3_doubleword_mask): Likewise.  Allow operands[3] to be
	a mask with all low 6 (64-bit) or 5 (32-bit) bits set and in that
	case just throw away the masking.  Use force_reg before calling
	gen_lowpart.
	(*<insn><dwi>3_doubleword_mask_1): Allow operands[3] to be a mask
	with all low 6 (64-bit) or 5 (32-bit) bits set and in that case just
	throw away the masking.
	(*<insn><mode>3_doubleword): Rename to ...
	(<insn><mode>3_doubleword): ... this.
	(*<insn><mode>3_mask): Remove :SI from AND and its operands and just
	verify operands[2] has HImode, SImode or for TARGET_64BIT DImode.
	Use force_reg before calling gen_lowpart.
	(splitter after it): Remove :SI from AND and its operands and just
	verify operands[2] has HImode, SImode or for TARGET_64BIT DImode.
	(*<btsc><mode>_mask, *<btsc><mode>_mask): Remove :SI from AND and its
	operands and just verify operands[1] has HImode, SImode or for
	TARGET_64BIT DImode.  Use force_reg before calling gen_lowpart.
	(*jcc_bt<mode>_mask_1): New define_insn_and_split pattern.
	* config/i386/i386.cc (ix86_rtx_costs): For ZERO_EXTRACT with
	ZERO_EXTEND QI->SI in last operand ignore the cost of the ZERO_EXTEND.

	* gcc.target/i386/pr105778.c: New test.