tree-optimization/104658 - avoid mixing mask & non-mask vector defs

tree-optimization/104658 - avoid mixing mask & non-mask vector defs

When pattern recognition fails to sanitize all defs of a mask
producing operation and the respective def is external or constant
we end up trying to produce a VECTOR_BOOLEAN_TYPE_P constructor
which in turn ends up exposing stmts like

  <signed-boolean:1> _135 = _49 ? -1 : 0;

which isn't handled well in followup SLP and generates awful code.

We do rely heavily on pattern recognition to sanitize mask vs.
data uses of bools but that fails here which means we also should
fail vectorization.  That avoids ICEing because of such stmts
and it also avoids generating weird code which makes the
vectorization not profitable.

The following patch simply disallows external VECTOR_BOOLEAN_TYPE_P
defs and arranges the promote to external code to instead promote
mask uses to extern (that's just a short-cut here).

I've also looked at aarch64 and with SVE and a fixed vector length
for the gcc.target/i386/pr101636.c testcase.  I see similar vectorization
(using <signed-boolean:4>) there but it's hard to decide whether the
old, the new or no vectorization is better for this.  The code
generated with traditional integer masks isn't as awkward but we
still get the != 0 promotion done for each scalar element which
doesn't look like intended - this operation should be visible upfront.

That also means some cases will now become a missed optimization
that needs to be fixed by bool pattern recognition.  But that can
possibly be delayed to GCC 13.

2022-02-22  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/104658
	* tree-vect-slp.cc (vect_slp_convert_to_external): Do not
	create VECTOR_BOOLEAN_TYPE_P extern defs.  Reset the vector
	type on nodes we promote.
	(vectorizable_bb_reduc_epilogue): Deal with externalized
	root.
	* tree-vect-stmts.cc (vect_maybe_update_slp_op_vectype): Do
	not allow VECTOR_BOOLEAN_TYPE_P extern defs.

	* gcc.target/i386/pr104658.c: New testcase.