* doc/c-tree.texi: Document new tree codes.
* doc/md.texi: Document new optabs.
* tree-pretty-print.c (dump_generic_node): Handle print of new tree
codes.
* optabs.c (optab_for_tree_code, init_optabs): Handle new optabs.
* optabs.h (optab_index): Add new.
(vec_extract_even_optab, vec_extract_odd_optab,
vec_interleave_high_optab, vec_interleave_low_optab): New optabs.
* genopinit.c (vec_extract_even_optab, vec_extract_odd_optab,
vec_interleave_high_optab, vec_interleave_low_optab): Initialize
new optabs.
* expr.c (expand_expr_real_1): Add implementation for new tree codes.
* tree-vectorizer.c (new_stmt_vec_info): Initialize new fields.
* tree-vectorizer.h (stmt_vec_info): Add new fields for interleaving
along with macros for their access.
* tree-data-ref.h (first_location_in_loop, data_reference): Update
comment.
* tree-vect-analyze.c (toplev.h): Include.
(vect_determine_vectorization_factor): Fix indentation.
(vect_insert_into_interleaving_chain,
vect_update_interleaving_chain, vect_equal_offsets): New functions.
(vect_analyze_data_ref_dependence): Add argument for interleaving
check. Check for interleaving if it's true.
(vect_check_dependences): New function.
(vect_analyze_data_ref_dependences): Call vect_check_dependences for
every ddr. Call vect_analyze_data_ref_dependence with new argument.
(vect_update_misalignment_for_peel): Update for interleaving.
(vect_verify_datarefs_alignment): Check only first data-ref for
interleaving.
(vect_enhance_data_refs_alignment): Update for interleaving. Check
only first data-ref for interleaving.
(vect_analyze_data_ref_access): Check interleaving, update
interleaving data.
(vect_analyze_data_refs): Call compute_data_dependences_for_loop
with different parameters.
* tree.def (VEC_EXTRACT_EVEN_EXPR, VEC_EXTRACT_ODD_EXPR,
VEC_INTERLEAVE_HIGH_EXPR, VEC_INTERLEAVE_LOW_EXPR): New tree codes.
* tree-inline.c (estimate_num_insns_1): Add cases for new codes.
* tree-vect-transform.c (vect_create_addr_base_for_vector_ref):
Update step in case of interleaving.
(vect_strided_store_supported, vect_permute_store_chain): New
functions.
(vectorizable_store): Handle strided stores.
(vect_strided_load_supported, vect_permute_load_chain,
vect_transform_strided_load): New functions.
(vectorizable_load): Handle strided loads.
(vect_transform_stmt): Add argument. Handle strided stores. Check
that vectorized stmt exists for patterns.
(vect_gen_niters_for_prolog_loop): Update calculation for
interleaving.
(vect_transform_loop): Remove stmt_vec_info for strided stores after
whole chain vectorization.
* config/rs6000/altivec.md (UNSPEC_EXTEVEN, UNSPEC_EXTODD,
UNSPEC_INTERHI, UNSPEC_INTERLO): New constants.
(vpkuhum_nomode, vpkuwum_nomode, vec_extract_even<mode>,
vec_extract_odd<mode>, altivec_vmrghsf, altivec_vmrglsf,
vec_interleave_high<mode>, vec_interleave_low<mode>): Implement.
From-SVN: r119088
+2006-11-22 Ira Rosen <irar@il.ibm.com>
+
+ * doc/c-tree.texi: Document new tree codes.
+ * doc/md.texi: Document new optabs.
+ * tree-pretty-print.c (dump_generic_node): Handle print of new tree
+ codes.
+ * optabs.c (optab_for_tree_code, init_optabs): Handle new optabs.
+ * optabs.h (optab_index): Add new.
+ (vec_extract_even_optab, vec_extract_odd_optab,
+ vec_interleave_high_optab, vec_interleave_low_optab): New optabs.
+ * genopinit.c (vec_extract_even_optab, vec_extract_odd_optab,
+ vec_interleave_high_optab, vec_interleave_low_optab): Initialize
+ new optabs.
+ * expr.c (expand_expr_real_1): Add implementation for new tree codes.
+ * tree-vectorizer.c (new_stmt_vec_info): Initialize new fields.
+ * tree-vectorizer.h (stmt_vec_info): Add new fields for interleaving
+ along with macros for their access.
+ * tree-data-ref.h (first_location_in_loop, data_reference): Update
+ comment.
+ * tree-vect-analyze.c (toplev.h): Include.
+ (vect_determine_vectorization_factor): Fix indentation.
+ (vect_insert_into_interleaving_chain,
+ vect_update_interleaving_chain, vect_equal_offsets): New functions.
+ (vect_analyze_data_ref_dependence): Add argument for interleaving
+ check. Check for interleaving if it's true.
+ (vect_check_dependences): New function.
+ (vect_analyze_data_ref_dependences): Call vect_check_dependences for
+ every ddr. Call vect_analyze_data_ref_dependence with new argument.
+ (vect_update_misalignment_for_peel): Update for interleaving.
+ (vect_verify_datarefs_alignment): Check only first data-ref for
+ interleaving.
+ (vect_enhance_data_refs_alignment): Update for interleaving. Check
+ only first data-ref for interleaving.
+ (vect_analyze_data_ref_access): Check interleaving, update
+ interleaving data.
+ (vect_analyze_data_refs): Call compute_data_dependences_for_loop
+ with different parameters.
+ * tree.def (VEC_EXTRACT_EVEN_EXPR, VEC_EXTRACT_ODD_EXPR,
+ VEC_INTERLEAVE_HIGH_EXPR, VEC_INTERLEAVE_LOW_EXPR): New tree codes.
+ * tree-inline.c (estimate_num_insns_1): Add cases for new codes.
+ * tree-vect-transform.c (vect_create_addr_base_for_vector_ref):
+ Update step in case of interleaving.
+ (vect_strided_store_supported, vect_permute_store_chain): New
+ functions.
+ (vectorizable_store): Handle strided stores.
+ (vect_strided_load_supported, vect_permute_load_chain,
+ vect_transform_strided_load): New functions.
+ (vectorizable_load): Handle strided loads.
+ (vect_transform_stmt): Add argument. Handle strided stores. Check
+ that vectorized stmt exists for patterns.
+ (vect_gen_niters_for_prolog_loop): Update calculation for
+ interleaving.
+ (vect_transform_loop): Remove stmt_vec_info for strided stores after
+ whole chain vectorization.
+ * config/rs6000/altivec.md (UNSPEC_EXTEVEN, UNSPEC_EXTODD,
+ UNSPEC_INTERHI, UNSPEC_INTERLO): New constants.
+ (vpkuhum_nomode, vpkuwum_nomode, vec_extract_even<mode>,
+ vec_extract_odd<mode>, altivec_vmrghsf, altivec_vmrglsf,
+ vec_interleave_high<mode>, vec_interleave_low<mode>): Implement.
+
2006-11-22 Steven Bosscher <steven@gcc.gnu.org>
* cse.c (enum taken): Remove PATH_AROUND.
(UNSPEC_REALIGN_LOAD 215)
(UNSPEC_REDUC_PLUS 217)
(UNSPEC_VECSH 219)
+ (UNSPEC_EXTEVEN_V4SI 220)
+ (UNSPEC_EXTEVEN_V8HI 221)
+ (UNSPEC_EXTEVEN_V16QI 222)
+ (UNSPEC_EXTEVEN_V4SF 223)
+ (UNSPEC_EXTODD_V4SI 224)
+ (UNSPEC_EXTODD_V8HI 225)
+ (UNSPEC_EXTODD_V16QI 226)
+ (UNSPEC_EXTODD_V4SF 227)
+ (UNSPEC_INTERHI_V4SI 228)
+ (UNSPEC_INTERHI_V8HI 229)
+ (UNSPEC_INTERHI_V16QI 230)
+ (UNSPEC_INTERHI_V4SF 231)
+ (UNSPEC_INTERLO_V4SI 232)
+ (UNSPEC_INTERLO_V8HI 233)
+ (UNSPEC_INTERLO_V16QI 234)
+ (UNSPEC_INTERLO_V4SF 235)
(UNSPEC_VCOND_V4SI 301)
(UNSPEC_VCOND_V4SF 302)
(UNSPEC_VCOND_V8HI 303)
(UNSPEC_VUPKLUH 319)
(UNSPEC_VPERMSI 320)
(UNSPEC_VPERMHI 321)
- ])
+ (UNSPEC_INTERHI 322)
+ (UNSPEC_INTERLO 323)
+])
(define_constants
[(UNSPECV_SET_VRSAVE 30)
"vmrghw %0,%1,%2"
[(set_attr "type" "vecperm")])
+(define_insn "altivec_vmrghsf"
+ [(set (match_operand:V4SF 0 "register_operand" "=v")
+ (vec_merge:V4SF (vec_select:V4SF (match_operand:V4SF 1 "register_operand" "v")
+ (parallel [(const_int 0)
+ (const_int 2)
+ (const_int 1)
+ (const_int 3)]))
+ (vec_select:V4SF (match_operand:V4SF 2 "register_operand" "v")
+ (parallel [(const_int 2)
+ (const_int 0)
+ (const_int 3)
+ (const_int 1)]))
+ (const_int 5)))]
+ "TARGET_ALTIVEC"
+ "vmrghw %0,%1,%2"
+ [(set_attr "type" "vecperm")])
+
(define_insn "altivec_vmrglb"
[(set (match_operand:V16QI 0 "register_operand" "=v")
(vec_merge:V16QI (vec_select:V16QI (match_operand:V16QI 1 "register_operand" "v")
"vmrglw %0,%1,%2"
[(set_attr "type" "vecperm")])
+(define_insn "altivec_vmrglsf"
+ [(set (match_operand:V4SF 0 "register_operand" "=v")
+ (vec_merge:V4SF (vec_select:V4SF (match_operand:V4SF 1 "register_operand" "v")
+ (parallel [(const_int 2)
+ (const_int 0)
+ (const_int 3)
+ (const_int 1)]))
+ (vec_select:V4SF (match_operand:V4SF 2 "register_operand" "v")
+ (parallel [(const_int 0)
+ (const_int 2)
+ (const_int 1)
+ (const_int 3)]))
+ (const_int 5)))]
+ "TARGET_ALTIVEC"
+ "vmrglw %0,%1,%2"
+ [(set_attr "type" "vecperm")])
+
(define_insn "altivec_vmuleub"
[(set (match_operand:V8HI 0 "register_operand" "=v")
(unspec:V8HI [(match_operand:V16QI 1 "register_operand" "v")
DONE;
}")
+
+(define_expand "vec_extract_evenv4si"
+ [(set (match_operand:V4SI 0 "register_operand" "")
+ (unspec:V8HI [(match_operand:V4SI 1 "register_operand" "")
+ (match_operand:V4SI 2 "register_operand" "")]
+ UNSPEC_EXTEVEN_V4SI))]
+ "TARGET_ALTIVEC"
+ "
+{
+ rtx mask = gen_reg_rtx (V16QImode);
+ rtvec v = rtvec_alloc (16);
+
+ RTVEC_ELT (v, 0) = gen_rtx_CONST_INT (QImode, 0);
+ RTVEC_ELT (v, 1) = gen_rtx_CONST_INT (QImode, 1);
+ RTVEC_ELT (v, 2) = gen_rtx_CONST_INT (QImode, 2);
+ RTVEC_ELT (v, 3) = gen_rtx_CONST_INT (QImode, 3);
+ RTVEC_ELT (v, 4) = gen_rtx_CONST_INT (QImode, 8);
+ RTVEC_ELT (v, 5) = gen_rtx_CONST_INT (QImode, 9);
+ RTVEC_ELT (v, 6) = gen_rtx_CONST_INT (QImode, 10);
+ RTVEC_ELT (v, 7) = gen_rtx_CONST_INT (QImode, 11);
+ RTVEC_ELT (v, 8) = gen_rtx_CONST_INT (QImode, 16);
+ RTVEC_ELT (v, 9) = gen_rtx_CONST_INT (QImode, 17);
+ RTVEC_ELT (v, 10) = gen_rtx_CONST_INT (QImode, 18);
+ RTVEC_ELT (v, 11) = gen_rtx_CONST_INT (QImode, 19);
+ RTVEC_ELT (v, 12) = gen_rtx_CONST_INT (QImode, 24);
+ RTVEC_ELT (v, 13) = gen_rtx_CONST_INT (QImode, 25);
+ RTVEC_ELT (v, 14) = gen_rtx_CONST_INT (QImode, 26);
+ RTVEC_ELT (v, 15) = gen_rtx_CONST_INT (QImode, 27);
+ emit_insn (gen_vec_initv16qi (mask, gen_rtx_PARALLEL (V16QImode, v)));
+ emit_insn (gen_altivec_vperm_v4si (operands[0], operands[1], operands[2], mask));
+
+ DONE;
+}")
+
+(define_expand "vec_extract_evenv4sf"
+ [(set (match_operand:V4SF 0 "register_operand" "")
+ (unspec:V8HI [(match_operand:V4SF 1 "register_operand" "")
+ (match_operand:V4SF 2 "register_operand" "")]
+ UNSPEC_EXTEVEN_V4SF))]
+ "TARGET_ALTIVEC"
+ "
+{
+ rtx mask = gen_reg_rtx (V16QImode);
+ rtvec v = rtvec_alloc (16);
+
+ RTVEC_ELT (v, 0) = gen_rtx_CONST_INT (QImode, 0);
+ RTVEC_ELT (v, 1) = gen_rtx_CONST_INT (QImode, 1);
+ RTVEC_ELT (v, 2) = gen_rtx_CONST_INT (QImode, 2);
+ RTVEC_ELT (v, 3) = gen_rtx_CONST_INT (QImode, 3);
+ RTVEC_ELT (v, 4) = gen_rtx_CONST_INT (QImode, 8);
+ RTVEC_ELT (v, 5) = gen_rtx_CONST_INT (QImode, 9);
+ RTVEC_ELT (v, 6) = gen_rtx_CONST_INT (QImode, 10);
+ RTVEC_ELT (v, 7) = gen_rtx_CONST_INT (QImode, 11);
+ RTVEC_ELT (v, 8) = gen_rtx_CONST_INT (QImode, 16);
+ RTVEC_ELT (v, 9) = gen_rtx_CONST_INT (QImode, 17);
+ RTVEC_ELT (v, 10) = gen_rtx_CONST_INT (QImode, 18);
+ RTVEC_ELT (v, 11) = gen_rtx_CONST_INT (QImode, 19);
+ RTVEC_ELT (v, 12) = gen_rtx_CONST_INT (QImode, 24);
+ RTVEC_ELT (v, 13) = gen_rtx_CONST_INT (QImode, 25);
+ RTVEC_ELT (v, 14) = gen_rtx_CONST_INT (QImode, 26);
+ RTVEC_ELT (v, 15) = gen_rtx_CONST_INT (QImode, 27);
+ emit_insn (gen_vec_initv16qi (mask, gen_rtx_PARALLEL (V16QImode, v)));
+ emit_insn (gen_altivec_vperm_v4sf (operands[0], operands[1], operands[2], mask));
+
+ DONE;
+}")
+
+(define_expand "vec_extract_evenv8hi"
+ [(set (match_operand:V4SI 0 "register_operand" "")
+ (unspec:V8HI [(match_operand:V8HI 1 "register_operand" "")
+ (match_operand:V8HI 2 "register_operand" "")]
+ UNSPEC_EXTEVEN_V8HI))]
+ "TARGET_ALTIVEC"
+ "
+{
+ rtx mask = gen_reg_rtx (V16QImode);
+ rtvec v = rtvec_alloc (16);
+
+ RTVEC_ELT (v, 0) = gen_rtx_CONST_INT (QImode, 0);
+ RTVEC_ELT (v, 1) = gen_rtx_CONST_INT (QImode, 1);
+ RTVEC_ELT (v, 2) = gen_rtx_CONST_INT (QImode, 4);
+ RTVEC_ELT (v, 3) = gen_rtx_CONST_INT (QImode, 5);
+ RTVEC_ELT (v, 4) = gen_rtx_CONST_INT (QImode, 8);
+ RTVEC_ELT (v, 5) = gen_rtx_CONST_INT (QImode, 9);
+ RTVEC_ELT (v, 6) = gen_rtx_CONST_INT (QImode, 12);
+ RTVEC_ELT (v, 7) = gen_rtx_CONST_INT (QImode, 13);
+ RTVEC_ELT (v, 8) = gen_rtx_CONST_INT (QImode, 16);
+ RTVEC_ELT (v, 9) = gen_rtx_CONST_INT (QImode, 17);
+ RTVEC_ELT (v, 10) = gen_rtx_CONST_INT (QImode, 20);
+ RTVEC_ELT (v, 11) = gen_rtx_CONST_INT (QImode, 21);
+ RTVEC_ELT (v, 12) = gen_rtx_CONST_INT (QImode, 24);
+ RTVEC_ELT (v, 13) = gen_rtx_CONST_INT (QImode, 25);
+ RTVEC_ELT (v, 14) = gen_rtx_CONST_INT (QImode, 28);
+ RTVEC_ELT (v, 15) = gen_rtx_CONST_INT (QImode, 29);
+ emit_insn (gen_vec_initv16qi (mask, gen_rtx_PARALLEL (V16QImode, v)));
+ emit_insn (gen_altivec_vperm_v8hi (operands[0], operands[1], operands[2], mask));
+
+ DONE;
+}")
+
+(define_expand "vec_extract_evenv16qi"
+ [(set (match_operand:V4SI 0 "register_operand" "")
+ (unspec:V8HI [(match_operand:V16QI 1 "register_operand" "")
+ (match_operand:V16QI 2 "register_operand" "")]
+ UNSPEC_EXTEVEN_V16QI))]
+ "TARGET_ALTIVEC"
+ "
+{
+ rtx mask = gen_reg_rtx (V16QImode);
+ rtvec v = rtvec_alloc (16);
+
+ RTVEC_ELT (v, 0) = gen_rtx_CONST_INT (QImode, 0);
+ RTVEC_ELT (v, 1) = gen_rtx_CONST_INT (QImode, 2);
+ RTVEC_ELT (v, 2) = gen_rtx_CONST_INT (QImode, 4);
+ RTVEC_ELT (v, 3) = gen_rtx_CONST_INT (QImode, 6);
+ RTVEC_ELT (v, 4) = gen_rtx_CONST_INT (QImode, 8);
+ RTVEC_ELT (v, 5) = gen_rtx_CONST_INT (QImode, 10);
+ RTVEC_ELT (v, 6) = gen_rtx_CONST_INT (QImode, 12);
+ RTVEC_ELT (v, 7) = gen_rtx_CONST_INT (QImode, 14);
+ RTVEC_ELT (v, 8) = gen_rtx_CONST_INT (QImode, 16);
+ RTVEC_ELT (v, 9) = gen_rtx_CONST_INT (QImode, 18);
+ RTVEC_ELT (v, 10) = gen_rtx_CONST_INT (QImode, 20);
+ RTVEC_ELT (v, 11) = gen_rtx_CONST_INT (QImode, 22);
+ RTVEC_ELT (v, 12) = gen_rtx_CONST_INT (QImode, 24);
+ RTVEC_ELT (v, 13) = gen_rtx_CONST_INT (QImode, 26);
+ RTVEC_ELT (v, 14) = gen_rtx_CONST_INT (QImode, 28);
+ RTVEC_ELT (v, 15) = gen_rtx_CONST_INT (QImode, 30);
+ emit_insn (gen_vec_initv16qi (mask, gen_rtx_PARALLEL (V16QImode, v)));
+ emit_insn (gen_altivec_vperm_v16qi (operands[0], operands[1], operands[2], mask));
+
+ DONE;
+}")
+
+(define_expand "vec_extract_oddv4si"
+ [(set (match_operand:V4SI 0 "register_operand" "")
+ (unspec:V8HI [(match_operand:V4SI 1 "register_operand" "")
+ (match_operand:V4SI 2 "register_operand" "")]
+ UNSPEC_EXTODD_V4SI))]
+ "TARGET_ALTIVEC"
+ "
+{
+ rtx mask = gen_reg_rtx (V16QImode);
+ rtvec v = rtvec_alloc (16);
+
+ RTVEC_ELT (v, 0) = gen_rtx_CONST_INT (QImode, 4);
+ RTVEC_ELT (v, 1) = gen_rtx_CONST_INT (QImode, 5);
+ RTVEC_ELT (v, 2) = gen_rtx_CONST_INT (QImode, 6);
+ RTVEC_ELT (v, 3) = gen_rtx_CONST_INT (QImode, 7);
+ RTVEC_ELT (v, 4) = gen_rtx_CONST_INT (QImode, 12);
+ RTVEC_ELT (v, 5) = gen_rtx_CONST_INT (QImode, 13);
+ RTVEC_ELT (v, 6) = gen_rtx_CONST_INT (QImode, 14);
+ RTVEC_ELT (v, 7) = gen_rtx_CONST_INT (QImode, 15);
+ RTVEC_ELT (v, 8) = gen_rtx_CONST_INT (QImode, 20);
+ RTVEC_ELT (v, 9) = gen_rtx_CONST_INT (QImode, 21);
+ RTVEC_ELT (v, 10) = gen_rtx_CONST_INT (QImode, 22);
+ RTVEC_ELT (v, 11) = gen_rtx_CONST_INT (QImode, 23);
+ RTVEC_ELT (v, 12) = gen_rtx_CONST_INT (QImode, 28);
+ RTVEC_ELT (v, 13) = gen_rtx_CONST_INT (QImode, 29);
+ RTVEC_ELT (v, 14) = gen_rtx_CONST_INT (QImode, 30);
+ RTVEC_ELT (v, 15) = gen_rtx_CONST_INT (QImode, 31);
+ emit_insn (gen_vec_initv16qi (mask, gen_rtx_PARALLEL (V16QImode, v)));
+ emit_insn (gen_altivec_vperm_v4si (operands[0], operands[1], operands[2], mask));
+
+ DONE;
+}")
+
+(define_expand "vec_extract_oddv4sf"
+ [(set (match_operand:V4SF 0 "register_operand" "")
+ (unspec:V8HI [(match_operand:V4SF 1 "register_operand" "")
+ (match_operand:V4SF 2 "register_operand" "")]
+ UNSPEC_EXTODD_V4SF))]
+ "TARGET_ALTIVEC"
+ "
+{
+ rtx mask = gen_reg_rtx (V16QImode);
+ rtvec v = rtvec_alloc (16);
+
+ RTVEC_ELT (v, 0) = gen_rtx_CONST_INT (QImode, 4);
+ RTVEC_ELT (v, 1) = gen_rtx_CONST_INT (QImode, 5);
+ RTVEC_ELT (v, 2) = gen_rtx_CONST_INT (QImode, 6);
+ RTVEC_ELT (v, 3) = gen_rtx_CONST_INT (QImode, 7);
+ RTVEC_ELT (v, 4) = gen_rtx_CONST_INT (QImode, 12);
+ RTVEC_ELT (v, 5) = gen_rtx_CONST_INT (QImode, 13);
+ RTVEC_ELT (v, 6) = gen_rtx_CONST_INT (QImode, 14);
+ RTVEC_ELT (v, 7) = gen_rtx_CONST_INT (QImode, 15);
+ RTVEC_ELT (v, 8) = gen_rtx_CONST_INT (QImode, 20);
+ RTVEC_ELT (v, 9) = gen_rtx_CONST_INT (QImode, 21);
+ RTVEC_ELT (v, 10) = gen_rtx_CONST_INT (QImode, 22);
+ RTVEC_ELT (v, 11) = gen_rtx_CONST_INT (QImode, 23);
+ RTVEC_ELT (v, 12) = gen_rtx_CONST_INT (QImode, 28);
+ RTVEC_ELT (v, 13) = gen_rtx_CONST_INT (QImode, 29);
+ RTVEC_ELT (v, 14) = gen_rtx_CONST_INT (QImode, 30);
+ RTVEC_ELT (v, 15) = gen_rtx_CONST_INT (QImode, 31);
+ emit_insn (gen_vec_initv16qi (mask, gen_rtx_PARALLEL (V16QImode, v)));
+ emit_insn (gen_altivec_vperm_v4sf (operands[0], operands[1], operands[2], mask));
+
+ DONE;
+}")
+
+(define_insn "vpkuhum_nomode"
+ [(set (match_operand:V16QI 0 "register_operand" "=v")
+ (unspec:V16QI [(match_operand 1 "register_operand" "v")
+ (match_operand 2 "register_operand" "v")]
+ UNSPEC_VPKUHUM))]
+ "TARGET_ALTIVEC"
+ "vpkuhum %0,%1,%2"
+ [(set_attr "type" "vecperm")])
+
+(define_insn "vpkuwum_nomode"
+ [(set (match_operand:V8HI 0 "register_operand" "=v")
+ (unspec:V8HI [(match_operand 1 "register_operand" "v")
+ (match_operand 2 "register_operand" "v")]
+ UNSPEC_VPKUWUM))]
+ "TARGET_ALTIVEC"
+ "vpkuwum %0,%1,%2"
+ [(set_attr "type" "vecperm")])
+
+(define_expand "vec_extract_oddv8hi"
+ [(set (match_operand:V8HI 0 "register_operand" "")
+ (unspec:V8HI [(match_operand:V8HI 1 "register_operand" "")
+ (match_operand:V8HI 2 "register_operand" "")]
+ UNSPEC_EXTODD_V8HI))]
+ "TARGET_ALTIVEC"
+ "
+{
+ emit_insn (gen_vpkuwum_nomode (operands[0], operands[1], operands[2]));
+ DONE;
+}")
+
+(define_expand "vec_extract_oddv16qi"
+ [(set (match_operand:V16QI 0 "register_operand" "")
+ (unspec:V16QI [(match_operand:V16QI 1 "register_operand" "")
+ (match_operand:V16QI 2 "register_operand" "")]
+ UNSPEC_EXTODD_V16QI))]
+ "TARGET_ALTIVEC"
+ "
+{
+ emit_insn (gen_vpkuhum_nomode (operands[0], operands[1], operands[2]));
+ DONE;
+}")
+(define_expand "vec_interleave_highv4sf"
+ [(set (match_operand:V4SF 0 "register_operand" "")
+ (unspec:V4SF [(match_operand:V4SF 1 "register_operand" "")
+ (match_operand:V4SF 2 "register_operand" "")]
+ UNSPEC_INTERHI_V4SF))]
+ "TARGET_ALTIVEC"
+ "
+{
+ emit_insn (gen_altivec_vmrghsf (operands[0], operands[1], operands[2]));
+ DONE;
+}")
+
+(define_expand "vec_interleave_lowv4sf"
+ [(set (match_operand:V4SF 0 "register_operand" "")
+ (unspec:V4SF [(match_operand:V4SF 1 "register_operand" "")
+ (match_operand:V4SF 2 "register_operand" "")]
+ UNSPEC_INTERLO_V4SF))]
+ "TARGET_ALTIVEC"
+ "
+{
+ emit_insn (gen_altivec_vmrglsf (operands[0], operands[1], operands[2]));
+ DONE;
+}")
+
+(define_expand "vec_interleave_high<mode>"
+ [(set (match_operand:VI 0 "register_operand" "")
+ (unspec:VI [(match_operand:VI 1 "register_operand" "")
+ (match_operand:VI 2 "register_operand" "")]
+ UNSPEC_INTERHI))]
+ "TARGET_ALTIVEC"
+ "
+{
+ emit_insn (gen_altivec_vmrgh<VI_char> (operands[0], operands[1], operands[2]));
+ DONE;
+}")
+
+(define_expand "vec_interleave_low<mode>"
+ [(set (match_operand:VI 0 "register_operand" "")
+ (unspec:VI [(match_operand:VI 1 "register_operand" "")
+ (match_operand:VI 2 "register_operand" "")]
+ UNSPEC_INTERLO))]
+ "TARGET_ALTIVEC"
+ "
+{
+ emit_insn (gen_altivec_vmrgl<VI_char> (operands[0], operands[1], operands[2]));
+ DONE;
+}")
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_PACK_MOD_EXPR
@tindex VEC_PACK_SAT_EXPR
+@tindex VEC_EXTRACT_EVEN_EXPR
+@tindex VEC_EXTRACT_ODD_EXPR
+@tindex VEC_INTERLEAVE_HIGH_EXPR
+@tindex VEC_INTERLEAVE_LOW_EXPR
The internal representation for expressions is for the most part quite
straightforward. However, there are a few facts that one must bear in
the elements of the two vectors are demoted and merged (concatenated) to form
the output vector.
+@item VEC_EXTRACT_EVEN_EXPR
+@item VEC_EXTRACT_ODD_EXPR
+These nodes represent extracting of the even/odd elements of the two input
+vectors, respectively. Their operands and result are vectors that contain the
+same number of elements of the same type.
+
+@item VEC_INTERLEAVE_HIGH_EXPR
+@item VEC_INTERLEAVE_LOW_EXPR
+These nodes represent merging and interleaving of the high/low elements of the
+two input vectors, respectively. The operands and the result are vectors that
+contain the same number of elements (@code{N}) of the same type.
+In the case of @code{VEC_INTERLEAVE_HIGH_EXPR}, the high @code{N/2} elements of
+the first input vector are interleaved with the high @code{N/2} elements of the
+second input vector. In the case of @code{VEC_INTERLEAVE_LOW_EXPR}, the low
+@code{N/2} elements of the first input vector are interleaved with the low
+@code{N/2} elements of the second input vector.
+
@end table
Extract given field from the vector value. Operand 1 is the vector, operand 2
specify field index and operand 0 place to store value into.
+@cindex @code{vec_extract_even@var{m}} instruction pattern
+@item @samp{vec_extract_even@var{m}}
+Extract even elements from the input vectors (operand 1 and operand 2).
+The even elements of operand 2 are concatenated to the even elements of operand
+1 in their original order. The result is stored in operand 0.
+The output and input vectors should have the same modes.
+
+@cindex @code{vec_extract_odd@var{m}} instruction pattern
+@item @samp{vec_extract_odd@var{m}}
+Extract odd elements from the input vectors (operand 1 and operand 2).
+The odd elements of operand 2 are concatenated to the odd elements of operand
+1 in their original order. The result is stored in operand 0.
+The output and input vectors should have the same modes.
+
+@cindex @code{vec_interleave_high@var{m}} instruction pattern
+@item @samp{vec_interleave_high@var{m}}
+Merge high elements of the two input vectors into the output vector. The output
+and input vectors should have the same modes (@code{N} elements). The high
+@code{N/2} elements of the first input vector are interleaved with the high
+@code{N/2} elements of the second input vector.
+
+@cindex @code{vec_interleave_low@var{m}} instruction pattern
+@item @samp{vec_interleave_low@var{m}}
+Merge low elements of the two input vectors into the output vector. The output
+and input vectors should have the same modes (@code{N} elements). The low
+@code{N/2} elements of the first input vector are interleaved with the low
+@code{N/2} elements of the second input vector.
+
@cindex @code{vec_init@var{m}} instruction pattern
@item @samp{vec_init@var{m}}
Initialize the vector to given values. Operand 0 is the vector to initialize
return temp;
}
+ case VEC_EXTRACT_EVEN_EXPR:
+ case VEC_EXTRACT_ODD_EXPR:
+ {
+ expand_operands (TREE_OPERAND (exp, 0), TREE_OPERAND (exp, 1),
+ NULL_RTX, &op0, &op1, 0);
+ this_optab = optab_for_tree_code (code, type);
+ temp = expand_binop (mode, this_optab, op0, op1, target, unsignedp,
+ OPTAB_WIDEN);
+ gcc_assert (temp);
+ return temp;
+ }
+
+ case VEC_INTERLEAVE_HIGH_EXPR:
+ case VEC_INTERLEAVE_LOW_EXPR:
+ {
+ expand_operands (TREE_OPERAND (exp, 0), TREE_OPERAND (exp, 1),
+ NULL_RTX, &op0, &op1, 0);
+ this_optab = optab_for_tree_code (code, type);
+ temp = expand_binop (mode, this_optab, op0, op1, target, unsignedp,
+ OPTAB_WIDEN);
+ gcc_assert (temp);
+ return temp;
+ }
+
case VEC_LSHIFT_EXPR:
case VEC_RSHIFT_EXPR:
{
"sync_lock_release[$A] = CODE_FOR_$(sync_lock_release$I$a$)",
"vec_set_optab->handlers[$A].insn_code = CODE_FOR_$(vec_set$a$)",
"vec_extract_optab->handlers[$A].insn_code = CODE_FOR_$(vec_extract$a$)",
+ "vec_extract_even_optab->handlers[$A].insn_code = CODE_FOR_$(vec_extract_even$a$)",
+ "vec_extract_odd_optab->handlers[$A].insn_code = CODE_FOR_$(vec_extract_odd$a$)",
+ "vec_interleave_high_optab->handlers[$A].insn_code = CODE_FOR_$(vec_interleave_high$a$)",
+ "vec_interleave_low_optab->handlers[$A].insn_code = CODE_FOR_$(vec_interleave_low$a$)",
"vec_init_optab->handlers[$A].insn_code = CODE_FOR_$(vec_init$a$)",
"vec_shl_optab->handlers[$A].insn_code = CODE_FOR_$(vec_shl_$a$)",
"vec_shr_optab->handlers[$A].insn_code = CODE_FOR_$(vec_shr_$a$)",
case ABS_EXPR:
return trapv ? absv_optab : abs_optab;
+ case VEC_EXTRACT_EVEN_EXPR:
+ return vec_extract_even_optab;
+
+ case VEC_EXTRACT_ODD_EXPR:
+ return vec_extract_odd_optab;
+
+ case VEC_INTERLEAVE_HIGH_EXPR:
+ return vec_interleave_high_optab;
+
+ case VEC_INTERLEAVE_LOW_EXPR:
+ return vec_interleave_low_optab;
+
default:
return NULL;
}
udot_prod_optab = init_optab (UNKNOWN);
vec_extract_optab = init_optab (UNKNOWN);
+ vec_extract_even_optab = init_optab (UNKNOWN);
+ vec_extract_odd_optab = init_optab (UNKNOWN);
+ vec_interleave_high_optab = init_optab (UNKNOWN);
+ vec_interleave_low_optab = init_optab (UNKNOWN);
vec_set_optab = init_optab (UNKNOWN);
vec_init_optab = init_optab (UNKNOWN);
vec_shl_optab = init_optab (UNKNOWN);
/* Optabs are tables saying how to generate insn bodies
for various machine modes and numbers of operands.
Each optab applies to one operation.
+
For example, add_optab applies to addition.
The insn_code slot is the enum insn_code that says how to
OTI_vec_set,
/* Extract specified field of vector operand. */
OTI_vec_extract,
+ /* Extract even/odd fields of vector operands. */
+ OTI_vec_extract_even,
+ OTI_vec_extract_odd,
+ /* Interleave fields of vector operands. */
+ OTI_vec_interleave_high,
+ OTI_vec_interleave_low,
/* Initialize vector operand. */
OTI_vec_init,
/* Whole vector shift. The shift amount is in bits. */
#define vec_set_optab (optab_table[OTI_vec_set])
#define vec_extract_optab (optab_table[OTI_vec_extract])
+#define vec_extract_even_optab (optab_table[OTI_vec_extract_even])
+#define vec_extract_odd_optab (optab_table[OTI_vec_extract_odd])
+#define vec_interleave_high_optab (optab_table[OTI_vec_interleave_high])
+#define vec_interleave_low_optab (optab_table[OTI_vec_interleave_low])
#define vec_init_optab (optab_table[OTI_vec_init])
#define vec_shl_optab (optab_table[OTI_vec_shl])
#define vec_shr_optab (optab_table[OTI_vec_shr])
+2006-11-22 Ira Rosen <irar@il.ibm.com>
+
+ * gcc.dg/vect/vect-1.c: Additional loop is now vectorizable on
+ platforms that have interleaving support.
+ * gcc.dg/vect/vect-107.c, gcc.dg/vect/vect-98.c: Likewise.
+ * gcc.dg/vect/vect-strided-a-u16-i2.c,
+ gcc.dg/vect/vect-strided-a-u16-i4.c, gcc.dg/vect/vect-strided-u16-i2.c,
+ gcc.dg/vect/vect-strided-u16-i4.c, gcc.dg/vect/vect-strided-u32-i4.c,
+ gcc.dg/vect/vect-strided-u32-i8.c, gcc.dg/vect/vect-strided-u8-i2.c,
+ gcc.dg/vect/vect-strided-u8-i2-gap.c,
+ gcc.dg/vect/vect-strided-u8-i8.c,
+ gcc.dg/vect/vect-strided-u8-i8-gap2.c,
+ gcc.dg/vect/vect-strided-u8-i8-gap4.c,
+ gcc.dg/vect/vect-strided-u8-i8-gap7.c,
+ gcc.dg/vect/vect-strided-float.c,
+ gcc.dg/vect/vect-strided-a-mult.c,
+ gcc.dg/vect/vect-strided-mult-char-ls.c,
+ gcc.dg/vect/vect-strided-a-u16-mult.c,
+ gcc.dg/vect/vect-strided-a-u32-mult.c,
+ gcc.dg/vect/vect-strided-a-u8-i2-gap.c,
+ gcc.dg/vect/vect-strided-a-u8-i8-gap2.c,
+ gcc.dg/vect/vect-strided-a-u8-i8-gap7.c,
+ gcc.dg/vect/vect-strided-mult.c,
+ gcc.dg/vect/vect-strided-u32-mult.c: New testcases.
+ * lib/target-supports.exp (vect_extract_even_odd, vect_interleave): New.
+
2006-11-22 Paul Thomas <pault@gcc.gnu.org>
PR fortran/25087
fbar (a);
- /* Not vectorizable yet (access pattern). */
+ /* Strided access. Vectorizable on platforms that support load of strided
+ accesses (extract of even/odd vector elements). */
for (i = 0; i < N/2; i++){
a[i] = b[2*i+1] * c[2*i+1] - b[2*i] * c[2*i];
d[i] = b[2*i] * c[2*i+1] + b[2*i+1] * c[2*i];
fbar (a);
}
-/* { dg-final { scan-tree-dump-times "vectorized 3 loops" 1 "vect" } } */
-/* { dg-final { scan-tree-dump-times "Vectorizing an unaligned access" 0 "vect" } } */
+/* { dg-final { scan-tree-dump-times "vectorized 4 loops" 1 "vect" { target vect_extract_even_odd } } } */
+/* { dg-final { scan-tree-dump-times "vectorized 3 loops" 1 "vect" { xfail vect_extract_even_odd } } } */
/* { dg-final { cleanup-tree-dump "vect" } } */
float c[N] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15};
float d[N] = {0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30};
- /* Strided access pattern. */
+ /* Strided access. Vectorizable on platforms that support load of strided
+ accesses (extract of even/odd vector elements). */
for (i = 0; i < N/2; i++)
{
a[i] = b[2*i+1] * c[2*i+1] - b[2*i] * c[2*i];
return main1 ();
}
-/* { dg-final { scan-tree-dump-times "vectorized 0 loops" 1 "vect" } } */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target vect_extract_even_odd } } } */
+/* { dg-final { scan-tree-dump-times "vectorized 0 loops" 1 "vect" { xfail vect_extract_even_odd } } } */
/* { dg-final { cleanup-tree-dump "vect" } } */
return main1 (ia);
}
-/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { xfail *-*-* } } } */
-/* { dg-final { scan-tree-dump-times "not vectorized: complicated access pattern" 1 "vect" } } */
+/* Needs interleaving support. */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 0 "vect" { xfail { vect_interleave && vect_extract_even_odd } } } } */
/* { dg-final { cleanup-tree-dump "vect" } } */
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned short a;
+ unsigned short b;
+} s;
+
+typedef struct {
+ unsigned int a;
+ unsigned int b;
+} ii;
+
+int
+main1 ()
+{
+ s arr[N];
+ s *ptr = arr;
+ ii iarr[N];
+ ii *iptr = iarr;
+ s res[N];
+ ii ires[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ iarr[i].a = i;
+ iarr[i].b = i * 3;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ for (i = 0; i < N; i++)
+ {
+ ires[i].a = iptr->b - iptr->a;
+ ires[i].b = iptr->b + iptr->a;
+ res[i].b = ptr->b - ptr->a;
+ res[i].a = ptr->b + ptr->a;
+ iptr++;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].b != arr[i].b - arr[i].a
+ || ires[i].a != iarr[i].b - iarr[i].a
+ || res[i].a != arr[i].b + arr[i].a
+ || ires[i].b != iarr[i].b + iarr[i].a
+)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+
+ check_vect ();
+
+ main1 ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned short a;
+ unsigned short b;
+} s;
+
+int
+main1 ()
+{
+ s arr[N];
+ s *ptr = arr;
+ s res[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].a = ptr->b - ptr->a;
+ res[i].b = ptr->b + ptr->a;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].a != arr[i].b - arr[i].a
+ || res[i].b != arr[i].a + arr[i].b)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+
+ check_vect ();
+
+ main1 ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned short a;
+ unsigned short b;
+ unsigned short c;
+ unsigned short d;
+} s;
+
+int
+main1 ()
+{
+ s arr[N];
+ s *ptr = arr;
+ s res[N];
+ int i;
+ unsigned short x, y, z, w;
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ for (i = 0; i < N; i++)
+ {
+ x = ptr->b - ptr->a;
+ y = ptr->d - ptr->c;
+ res[i].c = x + y;
+ z = ptr->a + ptr->c;
+ w = ptr->b + ptr->d;
+ res[i].a = z + w;
+ res[i].d = x + y;
+ res[i].b = x + y;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].a != arr[i].a + arr[i].c + arr[i].b + arr[i].d
+ || res[i].d != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].b != arr[i].b - arr[i].a + arr[i].d - arr[i].c)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+
+ check_vect ();
+
+ main1 ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned short a;
+ unsigned short b;
+} s;
+
+int
+main1 ()
+{
+ s arr[N];
+ s *ptr = arr;
+ unsigned int iarr[N];
+ unsigned int *iptr = iarr;
+ s res[N];
+ unsigned int ires[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ iarr[i] = i * 3;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ for (i = 0; i < N; i++)
+ {
+ ires[i] = *iptr;
+ res[i].b = ptr->b - ptr->a;
+ res[i].a = ptr->b + ptr->a;
+ iptr++;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].b != arr[i].b - arr[i].a
+ || ires[i] != iarr[i]
+ || res[i].a != arr[i].b + arr[i].a)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+
+ check_vect ();
+
+ main1 ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned int a;
+ unsigned int b;
+} ii;
+
+int
+main1 ()
+{
+ unsigned short arr[N];
+ unsigned short *ptr = arr;
+ ii iarr[N];
+ ii *iptr = iarr;
+ unsigned short res[N];
+ ii ires[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i] = i;
+ iarr[i].a = i;
+ iarr[i].b = i * 3;
+ if (arr[i] == 178)
+ abort();
+ }
+
+ for (i = 0; i < N; i++)
+ {
+ ires[i].a = iptr->b - iptr->a;
+ ires[i].b = iptr->b + iptr->a;
+ res[i] = *ptr;
+ iptr++;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i] != arr[i]
+ || ires[i].a != iarr[i].b - iarr[i].a
+ || ires[i].b != iarr[i].b + iarr[i].a)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+
+ check_vect ();
+
+ main1 ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 64
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+} s;
+
+int
+main1 ()
+{
+ s arr[N];
+ s *ptr = arr;
+ s res[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].a = ptr->a;
+ res[i].b = ptr->a;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].a != arr[i].a
+ || res[i].b != arr[i].a)
+ abort ();
+ }
+
+ ptr = arr;
+ /* Not vectorizable: gap in store. */
+ for (i = 0; i < N; i++)
+ {
+ res[i].a = ptr->b;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].a != arr[i].b)
+ abort ();
+ }
+
+
+ return 0;
+}
+
+int main (void)
+{
+ check_vect ();
+
+ main1 ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include <stdio.h>
+#include "tree-vect.h"
+
+#define N 16
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+ unsigned char c;
+ unsigned char d;
+ unsigned char e;
+ unsigned char f;
+ unsigned char g;
+ unsigned char h;
+} s;
+
+int
+main1 ()
+{
+ int i;
+ s arr[N];
+ s *ptr = arr;
+ s res[N];
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ arr[i].e = i + 5;
+ arr[i].f = i * 2 + 2;
+ arr[i].g = i - 3;
+ arr[i].h = 56;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].c = ptr->a;
+ res[i].a = ptr->f + ptr->a;
+ res[i].d = ptr->f - ptr->a;
+ res[i].b = ptr->f;
+ res[i].f = ptr->a;
+ res[i].e = ptr->f - ptr->a;
+ res[i].h = ptr->f;
+ res[i].g = ptr->f - ptr->a;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].a
+ || res[i].a != arr[i].f + arr[i].a
+ || res[i].d != arr[i].f - arr[i].a
+ || res[i].b != arr[i].f
+ || res[i].f != arr[i].a
+ || res[i].e != arr[i].f - arr[i].a
+ || res[i].h != arr[i].f
+ || res[i].g != arr[i].f - arr[i].a)
+ abort();
+ }
+}
+
+
+int main (void)
+{
+ check_vect ();
+
+ main1 ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 16
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+ unsigned char c;
+ unsigned char d;
+ unsigned char e;
+ unsigned char f;
+ unsigned char g;
+ unsigned char h;
+} s;
+
+int
+main1 ()
+{
+ int i;
+ s arr[N];
+ s *ptr = arr;
+ s res[N];
+ unsigned char u, t, s, x, y, z, w;
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ arr[i].e = i * 3 + 5;
+ arr[i].f = i * 5;
+ arr[i].g = i - 3;
+ arr[i].h = 67;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ for (i = 0; i < N; i++)
+ {
+ u = ptr->b - ptr->a;
+ t = ptr->d - ptr->c;
+ res[i].c = u + t;
+ x = ptr->b + ptr->d;
+ res[i].a = ptr->a + x;
+ res[i].d = u + t;
+ s = ptr->h - ptr->a;
+ res[i].b = s + t;
+ res[i].f = ptr->f + ptr->h;
+ res[i].e = ptr->b + ptr->e;
+ res[i].h = ptr->d;
+ res[i].g = u + t;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].a != arr[i].a + arr[i].b + arr[i].d
+ || res[i].d != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].b != arr[i].h - arr[i].a + arr[i].d - arr[i].c
+ || res[i].f != arr[i].f + arr[i].h
+ || res[i].e != arr[i].b + arr[i].e
+ || res[i].h != arr[i].d
+ || res[i].g != arr[i].b - arr[i].a + arr[i].d - arr[i].c)
+ abort();
+ }
+}
+
+
+int main (void)
+{
+ check_vect ();
+
+ main1 ();
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_float } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 16
+
+int
+main1 (void)
+{
+ int i;
+ float a[N*2];
+ float b[N*2] = {0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57,60,63,66,69,72,75,78,81,84,87,90,93};
+ float c[N*2] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31};
+
+ /* Strided access pattern. */
+ for (i = 0; i < N/2; i++)
+ {
+ a[i*2] = b[2*i+1] * c[2*i+1] - b[2*i] * c[2*i];
+ a[i*2+1] = b[2*i+8] * c[2*i+9] + b[2*i+9] * c[2*i+8];
+ }
+
+ /* Check results. */
+ for (i = 0; i < N/2; i++)
+ {
+ if (a[i*2] != b[2*i+1] * c[2*i+1] - b[2*i] * c[2*i]
+ || a[i*2+1] != b[2*i+8] * c[2*i+9] + b[2*i+9] * c[2*i+8])
+ abort();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ check_vect ();
+ return main1 ();
+}
+
+/* Needs interleaving support. */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { scan-tree-dump-times "vectorized 0 loops" 1 "vect" { xfail { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 32
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+} s;
+
+typedef struct {
+ unsigned int a;
+ unsigned int b;
+} ii;
+
+int
+main1 (s *arr, ii *iarr)
+{
+ s *ptr = arr;
+ ii *iptr = iarr;
+ s res[N];
+ ii ires[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ ires[i].a = iptr->b;
+ ires[i].b = iptr->a;
+ res[i].b = ptr->b - ptr->a;
+ res[i].a = ptr->b + ptr->a;
+ iptr++;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].b != arr[i].b - arr[i].a
+ || ires[i].a != iarr[i].b
+ || res[i].a != arr[i].b + arr[i].a
+ || ires[i].b != iarr[i].a
+)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+ ii iarr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ iarr[i].a = i;
+ iarr[i].b = i * 3;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr, iarr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned short a;
+ unsigned short b;
+} s;
+
+typedef struct {
+ unsigned int a;
+ unsigned int b;
+} ii;
+
+int
+main1 (s *arr, ii *iarr)
+{
+ s *ptr = arr;
+ ii *iptr = iarr;
+ s res[N];
+ ii ires[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ ires[i].a = iptr->b - iptr->a;
+ ires[i].b = iptr->b + iptr->a;
+ res[i].b = ptr->b - ptr->a;
+ res[i].a = ptr->b + ptr->a;
+ iptr++;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].b != arr[i].b - arr[i].a
+ || ires[i].a != iarr[i].b - iarr[i].a
+ || res[i].a != arr[i].b + arr[i].a
+ || ires[i].b != iarr[i].b + iarr[i].a
+)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+ ii iarr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ iarr[i].a = i;
+ iarr[i].b = i * 3;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr, iarr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned short a;
+ unsigned short b;
+} s;
+
+int
+main1 (s *arr)
+{
+ s *ptr = arr;
+ s res[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].a = ptr->b - ptr->a;
+ res[i].b = ptr->b + ptr->a;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].a != arr[i].b - arr[i].a
+ || res[i].b != arr[i].a + arr[i].b)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned short a;
+ unsigned short b;
+ unsigned short c;
+ unsigned short d;
+} s;
+
+int
+main1 (s *arr)
+{
+ int i;
+ s *ptr = arr;
+ s res[N];
+ unsigned short x, y, z, w;
+
+ for (i = 0; i < N; i++)
+ {
+ x = ptr->b - ptr->a;
+ y = ptr->d - ptr->c;
+ res[i].c = x + y;
+ z = ptr->a + ptr->c;
+ w = ptr->b + ptr->d;
+ res[i].a = z + w;
+ res[i].d = x + y;
+ res[i].b = x + y;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].a != arr[i].a + arr[i].c + arr[i].b + arr[i].d
+ || res[i].d != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].b != arr[i].b - arr[i].a + arr[i].d - arr[i].c)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ int a;
+ int b;
+ int c;
+ int d;
+} s;
+
+int
+main1 (s *arr)
+{
+ int i;
+ s *ptr = arr;
+ s res[N];
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].c = ptr->b - ptr->a + ptr->d - ptr->c;
+ res[i].a = ptr->a + ptr->c + ptr->b + ptr->d;
+ res[i].d = ptr->b - ptr->a + ptr->d - ptr->c;
+ res[i].b = ptr->b - ptr->a + ptr->d - ptr->c;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].a != arr[i].a + arr[i].c + arr[i].b + arr[i].d
+ || res[i].d != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].b != arr[i].b - arr[i].a + arr[i].d - arr[i].c)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ int a;
+ int b;
+ int c;
+ int d;
+ int e;
+ int f;
+ int g;
+ int h;
+} s;
+
+int
+main1 (s *arr)
+{
+ int i;
+ s *ptr = arr;
+ s res[N];
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].c = ptr->b - ptr->a + ptr->d - ptr->c;
+ res[i].a = ptr->a + ptr->g + ptr->b + ptr->d;
+ res[i].d = ptr->b - ptr->a + ptr->d - ptr->c;
+ res[i].b = ptr->h - ptr->a + ptr->d - ptr->c;
+ res[i].f = ptr->f + ptr->h;
+ res[i].e = ptr->b - ptr->e;
+ res[i].h = ptr->d - ptr->g;
+ res[i].g = ptr->b - ptr->a + ptr->d - ptr->c;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].a != arr[i].a + arr[i].g + arr[i].b + arr[i].d
+ || res[i].d != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].b != arr[i].h - arr[i].a + arr[i].d - arr[i].c
+ || res[i].f != arr[i].f + arr[i].h
+ || res[i].e != arr[i].b - arr[i].e
+ || res[i].h != arr[i].d - arr[i].g
+ || res[i].g != arr[i].b - arr[i].a + arr[i].d - arr[i].c)
+ abort();
+ }
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ arr[i].e = i * 3 + 5;
+ arr[i].f = i * 5;
+ arr[i].g = i - 3;
+ arr[i].h = 56;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 128
+
+typedef struct {
+ unsigned int a;
+ unsigned int b;
+} ii;
+
+int
+main1 (unsigned short *arr, ii *iarr)
+{
+ unsigned short *ptr = arr;
+ ii *iptr = iarr;
+ unsigned short res[N];
+ ii ires[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ ires[i].a = iptr->b - iptr->a;
+ ires[i].b = iptr->b + iptr->a;
+ res[i] = *ptr;
+ iptr++;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i] != arr[i]
+ || ires[i].a != iarr[i].b - iarr[i].a
+ || ires[i].b != iarr[i].b + iarr[i].a)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+ unsigned short arr[N];
+ ii iarr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i] = i;
+ iarr[i].a = i;
+ iarr[i].b = i * 3;
+ if (arr[i] == 178)
+ abort();
+ }
+ main1 (arr, iarr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 64
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+} s;
+
+int
+main1 (s *arr)
+{
+ s *ptr = arr;
+ s res[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].a = ptr->b;
+ res[i].b = ptr->b;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].a != arr[i].b
+ || res[i].b != arr[i].b)
+ abort ();
+ }
+
+ ptr = arr;
+ /* Not vectorizable: gap in store. */
+ for (i = 0; i < N; i++)
+ {
+ res[i].a = ptr->b;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].a != arr[i].b)
+ abort ();
+ }
+
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 64
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+} s;
+
+int
+main1 (s *arr)
+{
+ s *ptr = arr;
+ s res[N];
+ int i;
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].a = ptr->b - ptr->a;
+ res[i].b = ptr->b + ptr->a;
+ ptr++;
+ }
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].a != arr[i].b - arr[i].a
+ || res[i].b != arr[i].a + arr[i].b)
+ abort ();
+ }
+
+ return 0;
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include <stdio.h>
+#include "tree-vect.h"
+
+#define N 16
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+ unsigned char c;
+ unsigned char d;
+ unsigned char e;
+ unsigned char f;
+ unsigned char g;
+ unsigned char h;
+} s;
+
+int
+main1 (s *arr)
+{
+ int i;
+ s *ptr = arr;
+ s res[N];
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].c = ptr->b;
+ res[i].a = ptr->f + ptr->b;
+ res[i].d = ptr->f - ptr->b;
+ res[i].b = ptr->f;
+ res[i].f = ptr->b;
+ res[i].e = ptr->f - ptr->b;
+ res[i].h = ptr->f;
+ res[i].g = ptr->f - ptr->b;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b
+ || res[i].a != arr[i].f + arr[i].b
+ || res[i].d != arr[i].f - arr[i].b
+ || res[i].b != arr[i].f
+ || res[i].f != arr[i].b
+ || res[i].e != arr[i].f - arr[i].b
+ || res[i].h != arr[i].f
+ || res[i].g != arr[i].f - arr[i].b)
+ abort();
+ }
+}
+
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ arr[i].e = i + 5;
+ arr[i].f = i * 2 + 2;
+ arr[i].g = i - 3;
+ arr[i].h = 56;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 16
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+ unsigned char c;
+ unsigned char d;
+ unsigned char e;
+ unsigned char f;
+ unsigned char g;
+ unsigned char h;
+} s;
+
+int
+main1 (s *arr)
+{
+ int i;
+ s *ptr = arr;
+ s res[N];
+ unsigned char x;
+
+ for (i = 0; i < N; i++)
+ {
+ res[i].c = ptr->b + ptr->c;
+ x = ptr->c + ptr->f;
+ res[i].a = x + ptr->b;
+ res[i].d = ptr->b + ptr->c;
+ res[i].b = ptr->c;
+ res[i].f = ptr->f + ptr->e;
+ res[i].e = ptr->b + ptr->e;
+ res[i].h = ptr->c;
+ res[i].g = ptr->b + ptr->c;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b + arr[i].c
+ || res[i].a != arr[i].c + arr[i].f + arr[i].b
+ || res[i].d != arr[i].b + arr[i].c
+ || res[i].b != arr[i].c
+ || res[i].f != arr[i].f + arr[i].e
+ || res[i].e != arr[i].b + arr[i].e
+ || res[i].h != arr[i].c
+ || res[i].g != arr[i].b + arr[i].c)
+ abort();
+ }
+}
+
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ arr[i].e = i * 3 + 5;
+ arr[i].f = i * 5;
+ arr[i].g = i - 3;
+ arr[i].h = 56;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include "tree-vect.h"
+
+#define N 16
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+ unsigned char c;
+ unsigned char d;
+ unsigned char e;
+ unsigned char f;
+ unsigned char g;
+ unsigned char h;
+} s;
+
+int
+main1 (s *arr)
+{
+ int i;
+ s *ptr = arr;
+ s res[N];
+ unsigned char u, t, s, x, y, z, w;
+
+ for (i = 0; i < N; i++)
+ {
+ u = ptr->b - ptr->a;
+ t = ptr->d - ptr->c;
+ res[i].c = u + t;
+ x = ptr->b + ptr->d;
+ res[i].a = ptr->a + x;
+ res[i].d = u + t;
+ s = ptr->h - ptr->a;
+ res[i].b = s + t;
+ res[i].f = ptr->f + ptr->h;
+ res[i].e = ptr->b + ptr->e;
+ res[i].h = ptr->d;
+ res[i].g = u + t;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].a != arr[i].a + arr[i].b + arr[i].d
+ || res[i].d != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].b != arr[i].h - arr[i].a + arr[i].d - arr[i].c
+ || res[i].f != arr[i].f + arr[i].h
+ || res[i].e != arr[i].b + arr[i].e
+ || res[i].h != arr[i].d
+ || res[i].g != arr[i].b - arr[i].a + arr[i].d - arr[i].c)
+ abort();
+ }
+}
+
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ arr[i].e = i * 3 + 5;
+ arr[i].f = i * 5;
+ arr[i].g = i - 3;
+ arr[i].h = 67;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
--- /dev/null
+/* { dg-require-effective-target vect_int } */
+
+#include <stdarg.h>
+#include <stdio.h>
+#include "tree-vect.h"
+
+#define N 32
+
+typedef struct {
+ unsigned char a;
+ unsigned char b;
+ unsigned char c;
+ unsigned char d;
+ unsigned char e;
+ unsigned char f;
+ unsigned char g;
+ unsigned char h;
+} s;
+
+int
+main1 (s *arr)
+{
+ int i;
+ s *ptr = arr;
+ s res[N];
+ unsigned char u, t, s, x, y, z, w;
+
+ for (i = 0; i < N; i++)
+ {
+ u = ptr->b - ptr->a;
+ t = ptr->d - ptr->c;
+ res[i].c = u + t;
+ s = ptr->a + ptr->g;
+ x = ptr->b + ptr->d;
+ res[i].a = s + x;
+ res[i].d = u + t;
+ s = ptr->h - ptr->a;
+ x = ptr->d - ptr->c;
+ res[i].b = s + x;
+ res[i].f = ptr->f + ptr->h;
+ res[i].e = ptr->b + ptr->e;
+ res[i].h = ptr->d - ptr->g;
+ res[i].g = u + t;
+ ptr++;
+ }
+
+ /* check results: */
+ for (i = 0; i < N; i++)
+ {
+ if (res[i].c != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].a != arr[i].a + arr[i].g + arr[i].b + arr[i].d
+ || res[i].d != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ || res[i].b != arr[i].h - arr[i].a + arr[i].d - arr[i].c
+ || res[i].f != arr[i].f + arr[i].h
+ || res[i].e != arr[i].b + arr[i].e
+ || res[i].h != arr[i].d - arr[i].g
+ || res[i].g != arr[i].b - arr[i].a + arr[i].d - arr[i].c
+ )
+ abort();
+ }
+}
+
+int main (void)
+{
+ int i;
+ s arr[N];
+
+ check_vect ();
+
+ for (i = 0; i < N; i++)
+ {
+ arr[i].a = i;
+ arr[i].b = i * 2;
+ arr[i].c = 17;
+ arr[i].d = i+34;
+ arr[i].e = i;
+ arr[i].f = i + 5;
+ arr[i].g = i + 3;
+ arr[i].h = 67;
+ if (arr[i].a == 178)
+ abort();
+ }
+
+ main1 (arr);
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { vect_interleave && vect_extract_even_odd } } } } */
+/* { dg-final { cleanup-tree-dump "vect" } } */
+
return $et_vect_int_mult_saved
}
+# Return 1 if the target supports vector even/odd elements extraction, 0 otherwise.
+
+proc check_effective_target_vect_extract_even_odd { } {
+ global et_vect_extract_even_odd_saved
+
+ if [info exists et_vect_extract_even_odd_saved] {
+ verbose "check_effective_target_vect_extract_even_odd: using cached result" 2
+ } else {
+ set et_vect_extract_even_odd_saved 0
+ if { [istarget powerpc*-*-*] } {
+ set et_vect_extract_even_odd_saved 1
+ }
+ }
+
+ verbose "check_effective_target_vect_extract_even_odd: returning $et_vect_extract_even_odd_saved" 2
+ return $et_vect_extract_even_odd_saved
+}
+
+# Return 1 if the target supports vector interleaving, 0 otherwise.
+
+proc check_effective_target_vect_interleave { } {
+ global et_vect_interleave_saved
+
+ if [info exists et_vect_interleave_saved] {
+ verbose "check_effective_target_vect_interleave: using cached result" 2
+ } else {
+ set et_vect_interleave_saved 0
+ if { [istarget powerpc*-*-*]
+ || [istarget i?86-*-*]
+ || [istarget x86_64-*-*] } {
+ set et_vect_interleave_saved 1
+ }
+ }
+
+ verbose "check_effective_target_vect_interleave: returning $et_vect_interleave_saved" 2
+ return $et_vect_interleave_saved
+}
+
# Return 1 if the target supports section-anchors
proc check_effective_target_section_anchors { } {
#include "lambda.h"
-/** {base_address + offset + init} is the first location accessed by data-ref
- in the loop, and step is the stride of data-ref in the loop in bytes;
- e.g.:
-
+/*
+ The first location accessed by data-ref in the loop is the address of data-ref's
+ base (BASE_ADDRESS) plus the initial offset from the base. We divide the initial offset
+ into two parts: loop invariant offset (OFFSET) and constant offset (INIT).
+ STEP is the stride of data-ref in the loop in bytes.
+
Example 1 Example 2
data-ref a[j].b[i][j] a + x + 16B (a is int*)
-First location info:
+ First location info:
base_address &a a
- offset j_0*D_j + i_0*D_i + C_a x
- init C_b 16
+ offset j_0*D_j + i_0*D_i x
+ init C_b + C_a 16
step D_j 4
access_fn NULL {16, +, 1}
-Base object info:
+ Base object info:
base_object a NULL
access_fn <access_fns of indexes of b> NULL
- **/
+ */
struct first_location_in_loop
{
tree base_address;
tree step;
/* Access function related to first location in the loop. */
VEC(tree,heap) *access_fns;
-
};
struct base_object_info
struct ptr_info_def *ptr_info;
subvar_t subvars;
- /* Alignment information. */
- /* The offset of the data-reference from its base in bytes. */
+ /* Alignment information.
+ MISALIGNMENT is the offset of the data-reference from its base in bytes.
+ ALIGNED_TO is the maximum data-ref's alignment.
+
+ Example 1,
+ for i
+ for (j = 3; j < N; j++)
+ a[j].b[i][j] = 0;
+
+ For a[j].b[i][j], the offset from base (calculated in get_inner_reference()
+ will be 'i * C_i + j * C_j + C'.
+ We try to substitute the variables of the offset expression
+ with initial_condition of the corresponding access_fn in the loop.
+ 'i' cannot be substituted, since its access_fn in the inner loop is i. 'j'
+ will be substituted with 3.
+
+ Example 2
+ for (j = 3; j < N; j++)
+ a[j].b[5][j] = 0;
+
+ Here the offset expression (j * C_j + C) will not contain variables after
+ subsitution of j=3 (3*C_j + C).
+
+ Misalignment can be calculated only if all the variables can be
+ substituted with constants, otherwise, we record maximum possible alignment
+ in ALIGNED_TO. In Example 1, since 'i' cannot be substituted,
+ MISALIGNMENT will be NULL_TREE, and the biggest divider of C_i (a power of
+ 2) will be recorded in ALIGNED_TO.
+
+ In Example 2, MISALIGNMENT will be the value of 3*C_j + C in bytes, and
+ ALIGNED_TO will be NULL_TREE.
+ */
tree misalignment;
- /* The maximum data-ref's alignment. */
tree aligned_to;
/* The type of the data-ref. */
case WIDEN_MULT_EXPR:
+ case VEC_EXTRACT_EVEN_EXPR:
+ case VEC_EXTRACT_ODD_EXPR:
+ case VEC_INTERLEAVE_HIGH_EXPR:
+ case VEC_INTERLEAVE_LOW_EXPR:
+
case RESX_EXPR:
*count += 1;
break;
}
break;
+ case VEC_EXTRACT_EVEN_EXPR:
+ pp_string (buffer, " VEC_EXTRACT_EVEN_EXPR < ");
+ dump_generic_node (buffer, TREE_OPERAND (node, 0), spc, flags, false);
+ pp_string (buffer, ", ");
+ dump_generic_node (buffer, TREE_OPERAND (node, 1), spc, flags, false);
+ pp_string (buffer, " > ");
+ break;
+
+ case VEC_EXTRACT_ODD_EXPR:
+ pp_string (buffer, " VEC_EXTRACT_ODD_EXPR < ");
+ dump_generic_node (buffer, TREE_OPERAND (node, 0), spc, flags, false);
+ pp_string (buffer, ", ");
+ dump_generic_node (buffer, TREE_OPERAND (node, 1), spc, flags, false);
+ pp_string (buffer, " > ");
+ break;
+
+ case VEC_INTERLEAVE_HIGH_EXPR:
+ pp_string (buffer, " VEC_INTERLEAVE_HIGH_EXPR < ");
+ dump_generic_node (buffer, TREE_OPERAND (node, 0), spc, flags, false);
+ pp_string (buffer, ", ");
+ dump_generic_node (buffer, TREE_OPERAND (node, 1), spc, flags, false);
+ pp_string (buffer, " > ");
+ break;
+
+ case VEC_INTERLEAVE_LOW_EXPR:
+ pp_string (buffer, " VEC_INTERLEAVE_LOW_EXPR < ");
+ dump_generic_node (buffer, TREE_OPERAND (node, 0), spc, flags, false);
+ pp_string (buffer, ", ");
+ dump_generic_node (buffer, TREE_OPERAND (node, 1), spc, flags, false);
+ pp_string (buffer, " > ");
+ break;
+
default:
NIY;
}
#include "tree-data-ref.h"
#include "tree-scalar-evolution.h"
#include "tree-vectorizer.h"
+#include "toplev.h"
/* Main analysis functions. */
static loop_vec_info vect_analyze_loop_form (struct loop *);
static bool exist_non_indexing_operands_for_use_p (tree, tree);
static tree vect_get_loop_niters (struct loop *, tree *);
static bool vect_analyze_data_ref_dependence
- (struct data_dependence_relation *, loop_vec_info);
+ (struct data_dependence_relation *, loop_vec_info, bool);
static bool vect_compute_data_ref_alignment (struct data_reference *);
static bool vect_analyze_data_ref_access (struct data_reference *);
static bool vect_can_advance_ivs_p (loop_vec_info);
static void vect_update_misalignment_for_peel
(struct data_reference *, struct data_reference *, int npeel);
-
/* Function vect_determine_vectorization_factor
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "nunits = %d", nunits);
- if (!vectorization_factor
- || (nunits > vectorization_factor))
- vectorization_factor = nunits;
+ if (!vectorization_factor
+ || (nunits > vectorization_factor))
+ vectorization_factor = nunits;
+
}
}
}
+/* Function vect_insert_into_interleaving_chain.
+
+ Insert DRA into the interleaving chain of DRB according to DRA's INIT. */
+
+static void
+vect_insert_into_interleaving_chain (struct data_reference *dra,
+ struct data_reference *drb)
+{
+ tree prev, next, next_init;
+ stmt_vec_info stmtinfo_a = vinfo_for_stmt (DR_STMT (dra));
+ stmt_vec_info stmtinfo_b = vinfo_for_stmt (DR_STMT (drb));
+
+ prev = DR_GROUP_FIRST_DR (stmtinfo_b);
+ next = DR_GROUP_NEXT_DR (vinfo_for_stmt (prev));
+ while (next)
+ {
+ next_init = DR_INIT (STMT_VINFO_DATA_REF (vinfo_for_stmt (next)));
+ if (tree_int_cst_compare (next_init, DR_INIT (dra)) > 0)
+ {
+ /* Insert here. */
+ DR_GROUP_NEXT_DR (vinfo_for_stmt (prev)) = DR_STMT (dra);
+ DR_GROUP_NEXT_DR (stmtinfo_a) = next;
+ return;
+ }
+ prev = next;
+ next = DR_GROUP_NEXT_DR (vinfo_for_stmt (prev));
+ }
+
+ /* We got to the end of the list. Insert here. */
+ DR_GROUP_NEXT_DR (vinfo_for_stmt (prev)) = DR_STMT (dra);
+ DR_GROUP_NEXT_DR (stmtinfo_a) = NULL_TREE;
+}
+
+
+/* Function vect_update_interleaving_chain.
+
+ For two data-refs DRA and DRB that are a part of a chain interleaved data
+ accesses, update the interleaving chain. DRB's INIT is smaller than DRA's.
+
+ There are four possible cases:
+ 1. New stmts - both DRA and DRB are not a part of any chain:
+ FIRST_DR = DRB
+ NEXT_DR (DRB) = DRA
+ 2. DRB is a part of a chain and DRA is not:
+ no need to update FIRST_DR
+ no need to insert DRB
+ insert DRA according to init
+ 3. DRA is a part of a chain and DRB is not:
+ if (init of FIRST_DR > init of DRB)
+ FIRST_DR = DRB
+ NEXT(FIRST_DR) = previous FIRST_DR
+ else
+ insert DRB according to its init
+ 4. both DRA and DRB are in some interleaving chains:
+ choose the chain with the smallest init of FIRST_DR
+ insert the nodes of the second chain into the first one. */
+
+static void
+vect_update_interleaving_chain (struct data_reference *drb,
+ struct data_reference *dra)
+{
+ stmt_vec_info stmtinfo_a = vinfo_for_stmt (DR_STMT (dra));
+ stmt_vec_info stmtinfo_b = vinfo_for_stmt (DR_STMT (drb));
+ tree next_init, init_dra_chain, init_drb_chain, first_a, first_b;
+ tree node, prev, next, node_init, first_stmt;
+
+ /* 1. New stmts - both DRA and DRB are not a part of any chain. */
+ if (!DR_GROUP_FIRST_DR (stmtinfo_a) && !DR_GROUP_FIRST_DR (stmtinfo_b))
+ {
+ DR_GROUP_FIRST_DR (stmtinfo_a) = DR_STMT (drb);
+ DR_GROUP_FIRST_DR (stmtinfo_b) = DR_STMT (drb);
+ DR_GROUP_NEXT_DR (stmtinfo_b) = DR_STMT (dra);
+ return;
+ }
+
+ /* 2. DRB is a part of a chain and DRA is not. */
+ if (!DR_GROUP_FIRST_DR (stmtinfo_a) && DR_GROUP_FIRST_DR (stmtinfo_b))
+ {
+ DR_GROUP_FIRST_DR (stmtinfo_a) = DR_GROUP_FIRST_DR (stmtinfo_b);
+ /* Insert DRA into the chain of DRB. */
+ vect_insert_into_interleaving_chain (dra, drb);
+ return;
+ }
+
+ /* 3. DRA is a part of a chain and DRB is not. */
+ if (DR_GROUP_FIRST_DR (stmtinfo_a) && !DR_GROUP_FIRST_DR (stmtinfo_b))
+ {
+ tree old_first_stmt = DR_GROUP_FIRST_DR (stmtinfo_a);
+ tree init_old = DR_INIT (STMT_VINFO_DATA_REF (vinfo_for_stmt (
+ old_first_stmt)));
+ tree tmp;
+
+ if (tree_int_cst_compare (init_old, DR_INIT (drb)) > 0)
+ {
+ /* DRB's init is smaller than the init of the stmt previously marked
+ as the first stmt of the interleaving chain of DRA. Therefore, we
+ update FIRST_STMT and put DRB in the head of the list. */
+ DR_GROUP_FIRST_DR (stmtinfo_b) = DR_STMT (drb);
+ DR_GROUP_NEXT_DR (stmtinfo_b) = old_first_stmt;
+
+ /* Update all the stmts in the list to point to the new FIRST_STMT. */
+ tmp = old_first_stmt;
+ while (tmp)
+ {
+ DR_GROUP_FIRST_DR (vinfo_for_stmt (tmp)) = DR_STMT (drb);
+ tmp = DR_GROUP_NEXT_DR (vinfo_for_stmt (tmp));
+ }
+ }
+ else
+ {
+ /* Insert DRB in the list of DRA. */
+ vect_insert_into_interleaving_chain (drb, dra);
+ DR_GROUP_FIRST_DR (stmtinfo_b) = DR_GROUP_FIRST_DR (stmtinfo_a);
+ }
+ return;
+ }
+
+ /* 4. both DRA and DRB are in some interleaving chains. */
+ first_a = DR_GROUP_FIRST_DR (stmtinfo_a);
+ first_b = DR_GROUP_FIRST_DR (stmtinfo_b);
+ if (first_a == first_b)
+ return;
+ init_dra_chain = DR_INIT (STMT_VINFO_DATA_REF (vinfo_for_stmt (first_a)));
+ init_drb_chain = DR_INIT (STMT_VINFO_DATA_REF (vinfo_for_stmt (first_b)));
+
+ if (tree_int_cst_compare (init_dra_chain, init_drb_chain) > 0)
+ {
+ /* Insert the nodes of DRA chain into the DRB chain.
+ After inserting a node, continue from this node of the DRB chain (don't
+ start from the beginning. */
+ node = DR_GROUP_FIRST_DR (stmtinfo_a);
+ prev = DR_GROUP_FIRST_DR (stmtinfo_b);
+ first_stmt = first_b;
+ }
+ else
+ {
+ /* Insert the nodes of DRB chain into the DRA chain.
+ After inserting a node, continue from this node of the DRA chain (don't
+ start from the beginning. */
+ node = DR_GROUP_FIRST_DR (stmtinfo_b);
+ prev = DR_GROUP_FIRST_DR (stmtinfo_a);
+ first_stmt = first_a;
+ }
+
+ while (node)
+ {
+ node_init = DR_INIT (STMT_VINFO_DATA_REF (vinfo_for_stmt (node)));
+ next = DR_GROUP_NEXT_DR (vinfo_for_stmt (prev));
+ while (next)
+ {
+ next_init = DR_INIT (STMT_VINFO_DATA_REF (vinfo_for_stmt (next)));
+ if (tree_int_cst_compare (next_init, node_init) > 0)
+ {
+ /* Insert here. */
+ DR_GROUP_NEXT_DR (vinfo_for_stmt (prev)) = node;
+ DR_GROUP_NEXT_DR (vinfo_for_stmt (node)) = next;
+ prev = node;
+ break;
+ }
+ prev = next;
+ next = DR_GROUP_NEXT_DR (vinfo_for_stmt (prev));
+ }
+ if (!next)
+ {
+ /* We got to the end of the list. Insert here. */
+ DR_GROUP_NEXT_DR (vinfo_for_stmt (prev)) = node;
+ DR_GROUP_NEXT_DR (vinfo_for_stmt (node)) = NULL_TREE;
+ prev = node;
+ }
+ DR_GROUP_FIRST_DR (vinfo_for_stmt (node)) = first_stmt;
+ node = DR_GROUP_NEXT_DR (vinfo_for_stmt (node));
+ }
+}
+
+
+/* Function vect_equal_offsets.
+
+ Check if OFFSET1 and OFFSET2 are identical expressions. */
+
+static bool
+vect_equal_offsets (tree offset1, tree offset2)
+{
+ bool res0, res1;
+
+ STRIP_NOPS (offset1);
+ STRIP_NOPS (offset2);
+
+ if (offset1 == offset2)
+ return true;
+
+ if (TREE_CODE (offset1) != TREE_CODE (offset2)
+ || !BINARY_CLASS_P (offset1)
+ || !BINARY_CLASS_P (offset2))
+ return false;
+
+ res0 = vect_equal_offsets (TREE_OPERAND (offset1, 0),
+ TREE_OPERAND (offset2, 0));
+ res1 = vect_equal_offsets (TREE_OPERAND (offset1, 1),
+ TREE_OPERAND (offset2, 1));
+
+ return (res0 && res1);
+}
+
+
+/* Function vect_check_interleaving.
+
+ Check if DRA and DRB are a part of interleaving. In case they are, insert
+ DRA and DRB in an interleaving chain. */
+
+static void
+vect_check_interleaving (struct data_reference *dra,
+ struct data_reference *drb)
+{
+ HOST_WIDE_INT type_size_a, type_size_b, diff_mod_size, step, init_a, init_b;
+
+ /* Check that the data-refs have same first location (except init) and they
+ are both either store or load (not load and store). */
+ if ((DR_BASE_ADDRESS (dra) != DR_BASE_ADDRESS (drb)
+ && (TREE_CODE (DR_BASE_ADDRESS (dra)) != ADDR_EXPR
+ || TREE_CODE (DR_BASE_ADDRESS (drb)) != ADDR_EXPR
+ || TREE_OPERAND (DR_BASE_ADDRESS (dra), 0)
+ != TREE_OPERAND (DR_BASE_ADDRESS (drb),0)))
+ || !vect_equal_offsets (DR_OFFSET (dra), DR_OFFSET (drb))
+ || !tree_int_cst_compare (DR_INIT (dra), DR_INIT (drb))
+ || DR_IS_READ (dra) != DR_IS_READ (drb))
+ return;
+
+ /* Check:
+ 1. data-refs are of the same type
+ 2. their steps are equal
+ 3. the step is greater than the difference between data-refs' inits */
+ type_size_a = TREE_INT_CST_LOW (TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dra))));
+ type_size_b = TREE_INT_CST_LOW (TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (drb))));
+
+ if (type_size_a != type_size_b
+ || tree_int_cst_compare (DR_STEP (dra), DR_STEP (drb)))
+ return;
+
+ init_a = TREE_INT_CST_LOW (DR_INIT (dra));
+ init_b = TREE_INT_CST_LOW (DR_INIT (drb));
+ step = TREE_INT_CST_LOW (DR_STEP (dra));
+
+ if (init_a > init_b)
+ {
+ /* If init_a == init_b + the size of the type * k, we have an interleaving,
+ and DRB is accessed before DRA. */
+ diff_mod_size = (init_a - init_b) % type_size_a;
+
+ if ((init_a - init_b) > step)
+ return;
+
+ if (diff_mod_size == 0)
+ {
+ vect_update_interleaving_chain (drb, dra);
+ if (vect_print_dump_info (REPORT_DR_DETAILS))
+ {
+ fprintf (vect_dump, "Detected interleaving ");
+ print_generic_expr (vect_dump, DR_REF (dra), TDF_SLIM);
+ fprintf (vect_dump, " and ");
+ print_generic_expr (vect_dump, DR_REF (drb), TDF_SLIM);
+ }
+ return;
+ }
+ }
+ else
+ {
+ /* If init_b == init_a + the size of the type * k, we have an
+ interleaving, and DRA is accessed before DRB. */
+ diff_mod_size = (init_b - init_a) % type_size_a;
+
+ if ((init_b - init_a) > step)
+ return;
+
+ if (diff_mod_size == 0)
+ {
+ vect_update_interleaving_chain (dra, drb);
+ if (vect_print_dump_info (REPORT_DR_DETAILS))
+ {
+ fprintf (vect_dump, "Detected interleaving ");
+ print_generic_expr (vect_dump, DR_REF (dra), TDF_SLIM);
+ fprintf (vect_dump, " and ");
+ print_generic_expr (vect_dump, DR_REF (drb), TDF_SLIM);
+ }
+ return;
+ }
+ }
+}
+
+
/* Function vect_analyze_data_ref_dependence.
Return TRUE if there (might) exist a dependence between a memory-reference
static bool
vect_analyze_data_ref_dependence (struct data_dependence_relation *ddr,
- loop_vec_info loop_vinfo)
+ loop_vec_info loop_vinfo,
+ bool check_interleaving)
{
unsigned int i;
struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
unsigned int loop_depth;
if (DDR_ARE_DEPENDENT (ddr) == chrec_known)
+ {
+ /* Independent data accesses. */
+ if (check_interleaving)
+ vect_check_interleaving (dra, drb);
+ return false;
+ }
+
+ if ((DR_IS_READ (dra) && DR_IS_READ (drb)) || dra == drb)
return false;
if (DDR_ARE_DEPENDENT (ddr) == chrec_dont_know)
}
+/* Function vect_check_dependences.
+
+ Return TRUE if there is a store-store or load-store dependence between
+ data-refs in DDR, otherwise return FALSE. */
+
+static bool
+vect_check_dependences (struct data_dependence_relation *ddr)
+{
+ struct data_reference *dra = DDR_A (ddr);
+ struct data_reference *drb = DDR_B (ddr);
+
+ if (DDR_ARE_DEPENDENT (ddr) == chrec_known || dra == drb)
+ /* Independent or same data accesses. */
+ return false;
+
+ if (DR_IS_READ (dra) == DR_IS_READ (drb) && DR_IS_READ (dra))
+ /* Two loads. */
+ return false;
+
+ if (vect_print_dump_info (REPORT_DR_DETAILS))
+ {
+ fprintf (vect_dump, "possible store or store/load dependence between ");
+ print_generic_expr (vect_dump, DR_REF (dra), TDF_SLIM);
+ fprintf (vect_dump, " and ");
+ print_generic_expr (vect_dump, DR_REF (drb), TDF_SLIM);
+ }
+ return true;
+}
+
+
/* Function vect_analyze_data_ref_dependences.
Examine all the data references in the loop, and make sure there do not
unsigned int i;
VEC (ddr_p, heap) *ddrs = LOOP_VINFO_DDRS (loop_vinfo);
struct data_dependence_relation *ddr;
+ bool check_interleaving = true;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "=== vect_analyze_dependences ===");
+ /* We allow interleaving only if there are no store-store and load-store
+ dependencies in the loop. */
for (i = 0; VEC_iterate (ddr_p, ddrs, i, ddr); i++)
- if (vect_analyze_data_ref_dependence (ddr, loop_vinfo))
+ {
+ if (vect_check_dependences (ddr))
+ {
+ check_interleaving = false;
+ break;
+ }
+ }
+
+ for (i = 0; VEC_iterate (ddr_p, ddrs, i, ddr); i++)
+ if (vect_analyze_data_ref_dependence (ddr, loop_vinfo, check_interleaving))
return false;
return true;
struct data_reference *current_dr;
int dr_size = GET_MODE_SIZE (TYPE_MODE (TREE_TYPE (DR_REF (dr))));
int dr_peel_size = GET_MODE_SIZE (TYPE_MODE (TREE_TYPE (DR_REF (dr_peel))));
+ stmt_vec_info stmt_info = vinfo_for_stmt (DR_STMT (dr));
+ stmt_vec_info peel_stmt_info = vinfo_for_stmt (DR_STMT (dr_peel));
+
+ /* For interleaved data accesses the step in the loop must be multiplied by
+ the size of the interleaving group. */
+ if (DR_GROUP_FIRST_DR (stmt_info))
+ dr_size *= DR_GROUP_SIZE (vinfo_for_stmt (DR_GROUP_FIRST_DR (stmt_info)));
+ if (DR_GROUP_FIRST_DR (peel_stmt_info))
+ dr_peel_size *= DR_GROUP_SIZE (peel_stmt_info);
if (known_alignment_for_access_p (dr)
&& known_alignment_for_access_p (dr_peel)
- && (DR_MISALIGNMENT (dr)/dr_size ==
- DR_MISALIGNMENT (dr_peel)/dr_peel_size))
+ && (DR_MISALIGNMENT (dr) / dr_size ==
+ DR_MISALIGNMENT (dr_peel) / dr_peel_size))
{
DR_MISALIGNMENT (dr) = 0;
return;
{
if (current_dr != dr)
continue;
- gcc_assert (DR_MISALIGNMENT (dr)/dr_size ==
- DR_MISALIGNMENT (dr_peel)/dr_peel_size);
+ gcc_assert (DR_MISALIGNMENT (dr) / dr_size ==
+ DR_MISALIGNMENT (dr_peel) / dr_peel_size);
DR_MISALIGNMENT (dr) = 0;
return;
}
if (known_alignment_for_access_p (dr)
&& known_alignment_for_access_p (dr_peel))
- {
+ {
DR_MISALIGNMENT (dr) += npeel * dr_size;
DR_MISALIGNMENT (dr) %= UNITS_PER_SIMD_WORD;
return;
for (i = 0; VEC_iterate (data_reference_p, datarefs, i, dr); i++)
{
+ tree stmt = DR_STMT (dr);
+ stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
+
+ /* For interleaving, only the alignment of the first access matters. */
+ if (DR_GROUP_FIRST_DR (stmt_info)
+ && DR_GROUP_FIRST_DR (stmt_info) != stmt)
+ continue;
+
supportable_dr_alignment = vect_supportable_dr_alignment (dr);
if (!supportable_dr_alignment)
{
bool do_peeling = false;
bool do_versioning = false;
bool stat;
+ tree stmt;
+ stmt_vec_info stmt_info;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "=== vect_enhance_data_refs_alignment ===");
TODO: Use a cost model. */
for (i = 0; VEC_iterate (data_reference_p, datarefs, i, dr); i++)
- if (!DR_IS_READ (dr) && !aligned_access_p (dr))
- {
- dr0 = dr;
- do_peeling = true;
- break;
- }
+ {
+ stmt = DR_STMT (dr);
+ stmt_info = vinfo_for_stmt (stmt);
+
+ /* For interleaving, only the alignment of the first access
+ matters. */
+ if (DR_GROUP_FIRST_DR (stmt_info)
+ && DR_GROUP_FIRST_DR (stmt_info) != stmt)
+ continue;
+
+ if (!DR_IS_READ (dr) && !aligned_access_p (dr))
+ {
+ if (DR_GROUP_FIRST_DR (stmt_info))
+ {
+ /* For interleaved access we peel only if number of iterations in
+ the prolog loop ({VF - misalignment}), is a multiple of the
+ number of the interelaved accesses. */
+ int elem_size, mis_in_elements;
+ int vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
+
+ /* FORNOW: handle only known alignment. */
+ if (!known_alignment_for_access_p (dr))
+ {
+ do_peeling = false;
+ break;
+ }
+
+ elem_size = UNITS_PER_SIMD_WORD / vf;
+ mis_in_elements = DR_MISALIGNMENT (dr) / elem_size;
+
+ if ((vf - mis_in_elements) % DR_GROUP_SIZE (stmt_info))
+ {
+ do_peeling = false;
+ break;
+ }
+ }
+ dr0 = dr;
+ do_peeling = true;
+ break;
+ }
+ }
/* Often peeling for alignment will require peeling for loop-bound, which in
turn requires that we know how to adjust the loop ivs after the loop. */
mis = DR_MISALIGNMENT (dr0);
mis /= GET_MODE_SIZE (TYPE_MODE (TREE_TYPE (DR_REF (dr0))));
npeel = LOOP_VINFO_VECT_FACTOR (loop_vinfo) - mis;
+
+ /* For interleaved data access every iteration accesses all the
+ members of the group, therefore we divide the number of iterations
+ by the group size. */
+ stmt_info = vinfo_for_stmt (DR_STMT (dr0));
+ if (DR_GROUP_FIRST_DR (stmt_info))
+ npeel /= DR_GROUP_SIZE (stmt_info);
+
if (vect_print_dump_info (REPORT_DETAILS))
- fprintf (vect_dump, "Try peeling by %d",npeel);
+ fprintf (vect_dump, "Try peeling by %d", npeel);
}
/* Ensure that all data refs can be vectorized after the peel. */
if (dr == dr0)
continue;
+ stmt = DR_STMT (dr);
+ stmt_info = vinfo_for_stmt (stmt);
+ /* For interleaving, only the alignment of the first access
+ matters. */
+ if (DR_GROUP_FIRST_DR (stmt_info)
+ && DR_GROUP_FIRST_DR (stmt_info) != stmt)
+ continue;
+
save_misalignment = DR_MISALIGNMENT (dr);
vect_update_misalignment_for_peel (dr, dr0, npeel);
supportable_dr_alignment = vect_supportable_dr_alignment (dr);
{
for (i = 0; VEC_iterate (data_reference_p, datarefs, i, dr); i++)
{
- if (aligned_access_p (dr))
- continue;
+ stmt = DR_STMT (dr);
+ stmt_info = vinfo_for_stmt (stmt);
+
+ /* For interleaving, only the alignment of the first access
+ matters. */
+ if (aligned_access_p (dr)
+ || (DR_GROUP_FIRST_DR (stmt_info)
+ && DR_GROUP_FIRST_DR (stmt_info) != stmt))
+ continue;
- supportable_dr_alignment = vect_supportable_dr_alignment (dr);
+ supportable_dr_alignment = vect_supportable_dr_alignment (dr);
if (!supportable_dr_alignment)
{
vect_analyze_data_ref_access (struct data_reference *dr)
{
tree step = DR_STEP (dr);
+ HOST_WIDE_INT dr_step = TREE_INT_CST_LOW (step);
tree scalar_type = TREE_TYPE (DR_REF (dr));
+ HOST_WIDE_INT type_size = TREE_INT_CST_LOW (TYPE_SIZE_UNIT (scalar_type));
+ tree stmt = DR_STMT (dr);
+ /* For interleaving, STRIDE is STEP counted in elements, i.e., the size of the
+ interleaving group (including gaps). */
+ HOST_WIDE_INT stride = dr_step / type_size;
+
+ if (!step)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "bad data-ref access");
+ return false;
+ }
- if (!step || tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type)))
+ /* Consecutive? */
+ if (!tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type)))
{
+ /* Mark that it is not interleaving. */
+ DR_GROUP_FIRST_DR (vinfo_for_stmt (stmt)) = NULL_TREE;
+ return true;
+ }
+
+ /* Not consecutive access is possible only if it is a part of interleaving. */
+ if (!DR_GROUP_FIRST_DR (vinfo_for_stmt (stmt)))
+ {
+ /* Check if it this DR is a part of interleaving, and is a single
+ element of the group that is accessed in the loop. */
+
+ /* Gaps are supported only for loads. STEP must be a multiple of the type
+ size. The size of the group must be a power of 2. */
+ if (DR_IS_READ (dr)
+ && (dr_step % type_size) == 0
+ && stride > 0
+ && exact_log2 (stride) != -1)
+ {
+ DR_GROUP_FIRST_DR (vinfo_for_stmt (stmt)) = stmt;
+ DR_GROUP_SIZE (vinfo_for_stmt (stmt)) = stride;
+ if (vect_print_dump_info (REPORT_DR_DETAILS))
+ {
+ fprintf (vect_dump, "Detected single element interleaving %d ",
+ DR_GROUP_SIZE (vinfo_for_stmt (stmt)));
+ print_generic_expr (vect_dump, DR_REF (dr), TDF_SLIM);
+ fprintf (vect_dump, " step ");
+ print_generic_expr (vect_dump, step, TDF_SLIM);
+ }
+ return true;
+ }
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "not consecutive access");
return false;
}
+
+ if (DR_GROUP_FIRST_DR (vinfo_for_stmt (stmt)) == stmt)
+ {
+ /* First stmt in the interleaving chain. Check the chain. */
+ tree next = DR_GROUP_NEXT_DR (vinfo_for_stmt (stmt));
+ struct data_reference *data_ref = dr;
+ unsigned int count = 1;
+ tree next_step;
+ tree prev_init = DR_INIT (data_ref);
+ tree prev = stmt;
+ HOST_WIDE_INT diff, count_in_bytes;
+
+ while (next)
+ {
+ /* Skip same data-refs. In case that two or more stmts share data-ref
+ (supported only for loads), we vectorize only the first stmt, and
+ the rest get their vectorized loads from the the first one. */
+ if (!tree_int_cst_compare (DR_INIT (data_ref),
+ DR_INIT (STMT_VINFO_DATA_REF (
+ vinfo_for_stmt (next)))))
+ {
+ /* For load use the same data-ref load. (We check in
+ vect_check_dependences() that there are no two stores to the
+ same location). */
+ DR_GROUP_SAME_DR_STMT (vinfo_for_stmt (next)) = prev;
+
+ prev = next;
+ next = DR_GROUP_NEXT_DR (vinfo_for_stmt (next));
+ continue;
+ }
+ prev = next;
+
+ /* Check that all the accesses have the same STEP. */
+ next_step = DR_STEP (STMT_VINFO_DATA_REF (vinfo_for_stmt (next)));
+ if (tree_int_cst_compare (step, next_step))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "not consecutive access in interleaving");
+ return false;
+ }
+
+ data_ref = STMT_VINFO_DATA_REF (vinfo_for_stmt (next));
+ /* Check that the distance between two accesses is equal to the type
+ size. Otherwise, we have gaps. */
+ diff = (TREE_INT_CST_LOW (DR_INIT (data_ref))
+ - TREE_INT_CST_LOW (prev_init)) / type_size;
+ if (!DR_IS_READ (data_ref) && diff != 1)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "interleaved store with gaps");
+ return false;
+ }
+ /* Store the gap from the previous member of the group. If there is no
+ gap in the access, DR_GROUP_GAP is always 1. */
+ DR_GROUP_GAP (vinfo_for_stmt (next)) = diff;
+
+ prev_init = DR_INIT (data_ref);
+ next = DR_GROUP_NEXT_DR (vinfo_for_stmt (next));
+ /* Count the number of data-refs in the chain. */
+ count++;
+ }
+
+ /* COUNT is the number of accesses found, we multiply it by the size of
+ the type to get COUNT_IN_BYTES. */
+ count_in_bytes = type_size * count;
+ /* Check the size of the interleaving is not greater than STEP. */
+ if (dr_step < count_in_bytes)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ {
+ fprintf (vect_dump, "interleaving size is greater than step for ");
+ print_generic_expr (vect_dump, DR_REF (dr), TDF_SLIM);
+ }
+ return false;
+ }
+
+ /* Check that STEP is a multiple of type size. */
+ if ((dr_step % type_size) != 0)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ {
+ fprintf (vect_dump, "step is not a multiple of type size: step ");
+ print_generic_expr (vect_dump, step, TDF_SLIM);
+ fprintf (vect_dump, " size ");
+ print_generic_expr (vect_dump, TYPE_SIZE_UNIT (scalar_type),
+ TDF_SLIM);
+ }
+ return false;
+ }
+
+ /* FORNOW: we handle only interleaving that is a power of 2. */
+ if (exact_log2 (stride) == -1)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "interleaving is not a power of 2");
+ return false;
+ }
+ DR_GROUP_SIZE (vinfo_for_stmt (stmt)) = stride;
+ }
return true;
}
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "=== vect_analyze_data_refs ===");
- compute_data_dependences_for_loop (loop, false,
+ compute_data_dependences_for_loop (loop, true,
&LOOP_VINFO_DATAREFS (loop_vinfo),
&LOOP_VINFO_DDRS (loop_vinfo));
#include "real.h"
/* Utility functions for the code transformation. */
-static bool vect_transform_stmt (tree, block_stmt_iterator *);
+static bool vect_transform_stmt (tree, block_stmt_iterator *, bool *);
static tree vect_create_destination_var (tree, tree);
static tree vect_create_data_ref_ptr
(tree, block_stmt_iterator *, tree, tree *, tree *, bool);
if (offset)
{
tree tmp = create_tmp_var (TREE_TYPE (base_offset), "offset");
+ tree step;
+
+ /* For interleaved access step we divide STEP by the size of the
+ interleaving group. */
+ if (DR_GROUP_SIZE (stmt_info))
+ step = fold_build2 (TRUNC_DIV_EXPR, TREE_TYPE (offset), DR_STEP (dr),
+ build_int_cst (TREE_TYPE (offset),
+ DR_GROUP_SIZE (stmt_info)));
+ else
+ step = DR_STEP (dr);
+
add_referenced_var (tmp);
- offset = fold_build2 (MULT_EXPR, TREE_TYPE (offset), offset,
- DR_STEP (dr));
+ offset = fold_build2 (MULT_EXPR, TREE_TYPE (offset), offset, step);
base_offset = fold_build2 (PLUS_EXPR, TREE_TYPE (base_offset),
base_offset, offset);
base_offset = force_gimple_operand (base_offset, &new_stmt, false, tmp);
}
+/* Function vect_strided_store_supported.
+
+ Returns TRUE is INTERLEAVE_HIGH and INTERLEAVE_LOW operations are supported,
+ and FALSE otherwise. */
+
+static bool
+vect_strided_store_supported (tree vectype)
+{
+ optab interleave_high_optab, interleave_low_optab;
+ int mode;
+
+ mode = (int) TYPE_MODE (vectype);
+
+ /* Check that the operation is supported. */
+ interleave_high_optab = optab_for_tree_code (VEC_INTERLEAVE_HIGH_EXPR,
+ vectype);
+ interleave_low_optab = optab_for_tree_code (VEC_INTERLEAVE_LOW_EXPR,
+ vectype);
+ if (!interleave_high_optab || !interleave_low_optab)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "no optab for interleave.");
+ return false;
+ }
+
+ if (interleave_high_optab->handlers[(int) mode].insn_code
+ == CODE_FOR_nothing
+ || interleave_low_optab->handlers[(int) mode].insn_code
+ == CODE_FOR_nothing)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "interleave op not supported by target.");
+ return false;
+ }
+ return true;
+}
+
+
+/* Function vect_permute_store_chain.
+
+ Given a chain of interleaved strores in DR_CHAIN of LENGTH that must be
+ a power of 2, generate interleave_high/low stmts to reorder the data
+ correctly for the stores. Return the final references for stores in
+ RESULT_CHAIN.
+
+ E.g., LENGTH is 4 and the scalar type is short, i.e., VF is 8.
+ The input is 4 vectors each containg 8 elements. We assign a number to each
+ element, the input sequence is:
+
+ 1st vec: 0 1 2 3 4 5 6 7
+ 2nd vec: 8 9 10 11 12 13 14 15
+ 3rd vec: 16 17 18 19 20 21 22 23
+ 4th vec: 24 25 26 27 28 29 30 31
+
+ The output sequence should be:
+
+ 1st vec: 0 8 16 24 1 9 17 25
+ 2nd vec: 2 10 18 26 3 11 19 27
+ 3rd vec: 4 12 20 28 5 13 21 30
+ 4th vec: 6 14 22 30 7 15 23 31
+
+ i.e., we interleave the contents of the four vectors in their order.
+
+ We use interleave_high/low instructions to create such output. The input of
+ each interleave_high/low operation is two vectors:
+ 1st vec 2nd vec
+ 0 1 2 3 4 5 6 7
+ the even elements of the result vector are obtained left-to-right from the
+ high/low elements of the first vector. The odd elements of the result are
+ obtained left-to-right from the high/low elements of the second vector.
+ The output of interleave_high will be: 0 4 1 5
+ and of interleave_low: 2 6 3 7
+
+
+ The permutaion is done in log LENGTH stages. In each stage interleave_high
+ and interleave_low stmts are created for each pair of vectors in DR_CHAIN,
+ where the first argument is taken from the first half of DR_CHAIN and the
+ second argument from it's second half.
+ In our example,
+
+ I1: interleave_high (1st vec, 3rd vec)
+ I2: interleave_low (1st vec, 3rd vec)
+ I3: interleave_high (2nd vec, 4th vec)
+ I4: interleave_low (2nd vec, 4th vec)
+
+ The output for the first stage is:
+
+ I1: 0 16 1 17 2 18 3 19
+ I2: 4 20 5 21 6 22 7 23
+ I3: 8 24 9 25 10 26 11 27
+ I4: 12 28 13 29 14 30 15 31
+
+ The output of the second stage, i.e. the final result is:
+
+ I1: 0 8 16 24 1 9 17 25
+ I2: 2 10 18 26 3 11 19 27
+ I3: 4 12 20 28 5 13 21 30
+ I4: 6 14 22 30 7 15 23 31. */
+
+static bool
+vect_permute_store_chain (VEC(tree,heap) *dr_chain,
+ unsigned int length,
+ tree stmt,
+ block_stmt_iterator *bsi,
+ VEC(tree,heap) **result_chain)
+{
+ tree perm_dest, perm_stmt, vect1, vect2, high, low;
+ tree vectype = STMT_VINFO_VECTYPE (vinfo_for_stmt (stmt));
+ tree scalar_dest;
+ int i;
+ unsigned int j;
+ VEC(tree,heap) *first, *second;
+
+ scalar_dest = TREE_OPERAND (stmt, 0);
+ first = VEC_alloc (tree, heap, length/2);
+ second = VEC_alloc (tree, heap, length/2);
+
+ /* Check that the operation is supported. */
+ if (!vect_strided_store_supported (vectype))
+ return false;
+
+ *result_chain = VEC_copy (tree, heap, dr_chain);
+
+ for (i = 0; i < exact_log2 (length); i++)
+ {
+ for (j = 0; j < length/2; j++)
+ {
+ vect1 = VEC_index (tree, dr_chain, j);
+ vect2 = VEC_index (tree, dr_chain, j+length/2);
+
+ /* high = interleave_high (vect1, vect2); */
+ perm_dest = create_tmp_var (vectype, "vect_inter_high");
+ add_referenced_var (perm_dest);
+ perm_stmt = build2 (MODIFY_EXPR, vectype, perm_dest,
+ build2 (VEC_INTERLEAVE_HIGH_EXPR, vectype, vect1,
+ vect2));
+ high = make_ssa_name (perm_dest, perm_stmt);
+ TREE_OPERAND (perm_stmt, 0) = high;
+ vect_finish_stmt_generation (stmt, perm_stmt, bsi);
+ VEC_replace (tree, *result_chain, 2*j, high);
+
+ /* low = interleave_low (vect1, vect2); */
+ perm_dest = create_tmp_var (vectype, "vect_inter_low");
+ add_referenced_var (perm_dest);
+ perm_stmt = build2 (MODIFY_EXPR, vectype, perm_dest,
+ build2 (VEC_INTERLEAVE_LOW_EXPR, vectype, vect1,
+ vect2));
+ low = make_ssa_name (perm_dest, perm_stmt);
+ TREE_OPERAND (perm_stmt, 0) = low;
+ vect_finish_stmt_generation (stmt, perm_stmt, bsi);
+ VEC_replace (tree, *result_chain, 2*j+1, low);
+ }
+ dr_chain = VEC_copy (tree, heap, *result_chain);
+ }
+ return true;
+}
+
+
/* Function vectorizable_store.
Check if STMT defines a non scalar data-ref (array/pointer/structure) that
tree op;
tree vec_oprnd = NULL_TREE;
stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
- struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info);
+ struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info), *first_dr = NULL;
tree vectype = STMT_VINFO_VECTYPE (stmt_info);
loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
enum machine_mode vec_mode;
def_operand_p def_p;
tree def, def_stmt;
enum vect_def_type dt;
- stmt_vec_info prev_stmt_info;
+ stmt_vec_info prev_stmt_info = NULL;
tree dataref_ptr = NULL_TREE;
int nunits = TYPE_VECTOR_SUBPARTS (vectype);
int ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits;
int j;
-
+ tree next_stmt, first_stmt;
+ bool strided_store = false;
+ unsigned int group_size, i;
+ VEC(tree,heap) *dr_chain = NULL, *oprnds = NULL, *result_chain = NULL;
gcc_assert (ncopies >= 1);
/* Is vectorizable store? */
scalar_dest = TREE_OPERAND (stmt, 0);
if (TREE_CODE (scalar_dest) != ARRAY_REF
- && TREE_CODE (scalar_dest) != INDIRECT_REF)
+ && TREE_CODE (scalar_dest) != INDIRECT_REF
+ && !DR_GROUP_FIRST_DR (stmt_info))
return false;
op = TREE_OPERAND (stmt, 1);
if (!STMT_VINFO_DATA_REF (stmt_info))
return false;
+ if (DR_GROUP_FIRST_DR (stmt_info))
+ {
+ strided_store = true;
+ if (!vect_strided_store_supported (vectype))
+ return false;
+ }
if (!vec_stmt) /* transformation not required. */
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "transform store. ncopies = %d",ncopies);
- alignment_support_cheme = vect_supportable_dr_alignment (dr);
+ if (strided_store)
+ {
+ first_stmt = DR_GROUP_FIRST_DR (stmt_info);
+ first_dr = STMT_VINFO_DATA_REF (vinfo_for_stmt (first_stmt));
+ group_size = DR_GROUP_SIZE (vinfo_for_stmt (first_stmt));
+
+ DR_GROUP_STORE_COUNT (vinfo_for_stmt (first_stmt))++;
+
+ /* We vectorize all the stmts of the interleaving group when we
+ reach the last stmt in the group. */
+ if (DR_GROUP_STORE_COUNT (vinfo_for_stmt (first_stmt))
+ < DR_GROUP_SIZE (vinfo_for_stmt (first_stmt)))
+ {
+ *vec_stmt = NULL_TREE;
+ return true;
+ }
+ }
+ else
+ {
+ first_stmt = stmt;
+ first_dr = dr;
+ group_size = 1;
+ }
+
+ dr_chain = VEC_alloc (tree, heap, group_size);
+ oprnds = VEC_alloc (tree, heap, group_size);
+
+ alignment_support_cheme = vect_supportable_dr_alignment (first_dr);
gcc_assert (alignment_support_cheme);
gcc_assert (alignment_support_cheme == dr_aligned); /* FORNOW */
vector stmt by a factor VF/nunits. For more details see documentation in
vect_get_vec_def_for_copy_stmt. */
+ /* In case of interleaving (non-unit strided access):
+
+ S1: &base + 2 = x2
+ S2: &base = x0
+ S3: &base + 1 = x1
+ S4: &base + 3 = x3
+
+ We create vectorized storess starting from base address (the access of the
+ first stmt in the chain (S2 in the above example), when the last store stmt
+ of the chain (S4) is reached:
+
+ VS1: &base = vx2
+ VS2: &base + vec_size*1 = vx0
+ VS3: &base + vec_size*2 = vx1
+ VS4: &base + vec_size*3 = vx3
+
+ Then permutation statements are generated:
+
+ VS5: vx5 = VEC_INTERLEAVE_HIGH_EXPR < vx0, vx3 >
+ VS6: vx6 = VEC_INTERLEAVE_LOW_EXPR < vx0, vx3 >
+ ...
+
+ And they are put in STMT_VINFO_VEC_STMT of the corresponding scalar stmts
+ (the order of the data-refs in the output of vect_permute_store_chain
+ corresponds to the order of scalar stmts in the interleaving chain - see
+ the documentaion of vect_permute_store_chain()).
+
+ In case of both multiple types and interleaving, above vector stores and
+ permutation stmts are created for every copy. The result vector stmts are
+ put in STMT_VINFO_VEC_STMT for the first copy and in the corresponding
+ STMT_VINFO_RELATED_STMT for the next copies.
+ */
+
prev_stmt_info = NULL;
for (j = 0; j < ncopies; j++)
{
if (j == 0)
{
- vec_oprnd = vect_get_vec_def_for_operand (op, stmt, NULL);
- dataref_ptr = vect_create_data_ref_ptr (stmt, bsi, NULL_TREE, &dummy,
- &ptr_incr, false);
+ /* For interleaved stores we collect vectorized defs for all the
+ stores in the group in DR_CHAIN and OPRNDS. DR_CHAIN is then used
+ as an input to vect_permute_store_chain(), and OPRNDS as an input
+ to vect_get_vec_def_for_stmt_copy() for the next copy.
+ If the store is not strided, GROUP_SIZE is 1, and DR_CHAIN and
+ OPRNDS are of size 1.
+ */
+ next_stmt = first_stmt;
+ for (i = 0; i < group_size; i++)
+ {
+ /* Since gaps are not supported for interleaved stores, GROUP_SIZE
+ is the exact number of stmts in the chain. Therefore, NEXT_STMT
+ can't be NULL_TREE. In case that there is no interleaving,
+ GROUP_SIZE is 1, and only one iteration of the loop will be
+ executed.
+ */
+ gcc_assert (next_stmt);
+ op = TREE_OPERAND (next_stmt, 1);
+ vec_oprnd = vect_get_vec_def_for_operand (op, next_stmt, NULL);
+ VEC_quick_push(tree, dr_chain, vec_oprnd);
+ VEC_quick_push(tree, oprnds, vec_oprnd);
+ next_stmt = DR_GROUP_NEXT_DR (vinfo_for_stmt (next_stmt));
+ }
+ dataref_ptr = vect_create_data_ref_ptr (first_stmt, bsi, NULL_TREE,
+ &dummy, &ptr_incr, false);
}
else
{
- vec_oprnd = vect_get_vec_def_for_stmt_copy (dt, vec_oprnd);
+ /* For interleaved stores we created vectorized defs for all the
+ defs stored in OPRNDS in the previous iteration (previous copy).
+ DR_CHAIN is then used as an input to vect_permute_store_chain(),
+ and OPRNDS as an input to vect_get_vec_def_for_stmt_copy() for the
+ next copy.
+ If the store is not strided, GROUP_SIZE is 1, and DR_CHAIN and
+ OPRNDS are of size 1.
+ */
+ for (i = 0; i < group_size; i++)
+ {
+ vec_oprnd = vect_get_vec_def_for_stmt_copy (dt,
+ VEC_index (tree, oprnds, i));
+ VEC_replace(tree, dr_chain, i, vec_oprnd);
+ VEC_replace(tree, oprnds, i, vec_oprnd);
+ }
dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, bsi, stmt);
}
- /* Arguments are ready. create the new vector stmt. */
- data_ref = build_fold_indirect_ref (dataref_ptr);
- new_stmt = build2 (MODIFY_EXPR, vectype, data_ref, vec_oprnd);
- vect_finish_stmt_generation (stmt, new_stmt, bsi);
+ if (strided_store)
+ {
+ result_chain = VEC_alloc (tree, heap, group_size);
+ /* Permute. */
+ if (!vect_permute_store_chain (dr_chain, group_size, stmt, bsi,
+ &result_chain))
+ return false;
+ }
- /* Set the V_MAY_DEFS for the vector pointer. If this virtual def has a
- use outside the loop and a loop peel is performed then the def may be
- renamed by the peel. Mark it for renaming so the later use will also
- be renamed. */
- copy_virtual_operands (new_stmt, stmt);
- if (j == 0)
+ next_stmt = first_stmt;
+ for (i = 0; i < group_size; i++)
{
- /* The original store is deleted so the same SSA_NAMEs can be used.
- */
- FOR_EACH_SSA_TREE_OPERAND (def, stmt, iter, SSA_OP_VMAYDEF)
+ /* For strided stores vectorized defs are interleaved in
+ vect_permute_store_chain(). */
+ if (strided_store)
+ vec_oprnd = VEC_index(tree, result_chain, i);
+
+ data_ref = build_fold_indirect_ref (dataref_ptr);
+ /* Arguments are ready. Create the new vector stmt. */
+ new_stmt = build2 (MODIFY_EXPR, vectype, data_ref, vec_oprnd);
+ vect_finish_stmt_generation (stmt, new_stmt, bsi);
+
+ /* Set the V_MAY_DEFS for the vector pointer. If this virtual def has a
+ use outside the loop and a loop peel is performed then the def may be
+ renamed by the peel. Mark it for renaming so the later use will also
+ be renamed. */
+ copy_virtual_operands (new_stmt, next_stmt);
+ if (j == 0)
{
- SSA_NAME_DEF_STMT (def) = new_stmt;
- mark_sym_for_renaming (SSA_NAME_VAR (def));
+ /* The original store is deleted so the same SSA_NAMEs can be used.
+ */
+ FOR_EACH_SSA_TREE_OPERAND (def, next_stmt, iter, SSA_OP_VMAYDEF)
+ {
+ SSA_NAME_DEF_STMT (def) = new_stmt;
+ mark_sym_for_renaming (SSA_NAME_VAR (def));
+ }
+
+ STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt;
}
-
- STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt;
- }
- else
- {
- /* Create new names for all the definitions created by COPY and
- add replacement mappings for each new name. */
- FOR_EACH_SSA_DEF_OPERAND (def_p, new_stmt, iter, SSA_OP_VMAYDEF)
+ else
{
- create_new_def_for (DEF_FROM_PTR (def_p), new_stmt, def_p);
- mark_sym_for_renaming (SSA_NAME_VAR (DEF_FROM_PTR (def_p)));
+ /* Create new names for all the definitions created by COPY and
+ add replacement mappings for each new name. */
+ FOR_EACH_SSA_DEF_OPERAND (def_p, new_stmt, iter, SSA_OP_VMAYDEF)
+ {
+ create_new_def_for (DEF_FROM_PTR (def_p), new_stmt, def_p);
+ mark_sym_for_renaming (SSA_NAME_VAR (DEF_FROM_PTR (def_p)));
+ }
+
+ STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt;
}
- STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt;
+ prev_stmt_info = vinfo_for_stmt (new_stmt);
+ next_stmt = DR_GROUP_NEXT_DR (vinfo_for_stmt (next_stmt));
+ if (!next_stmt)
+ break;
+ /* Bump the vector pointer. */
+ dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, bsi, stmt);
}
-
- prev_stmt_info = vinfo_for_stmt (new_stmt);
}
return true;
Output:
REALIGNMENT_TOKEN - the result of a call to the builtin_mask_for_load
target hook, if defined.
- Return value - the result of the loop-header phi node.
-*/
+ Return value - the result of the loop-header phi node. */
static tree
vect_setup_realignment (tree stmt, block_stmt_iterator *bsi,
}
+/* Function vect_strided_load_supported.
+
+ Returns TRUE is EXTRACT_EVEN and EXTRACT_ODD operations are supported,
+ and FALSE otherwise. */
+
+static bool
+vect_strided_load_supported (tree vectype)
+{
+ optab perm_even_optab, perm_odd_optab;
+ int mode;
+
+ mode = (int) TYPE_MODE (vectype);
+
+ perm_even_optab = optab_for_tree_code (VEC_EXTRACT_EVEN_EXPR, vectype);
+ if (!perm_even_optab)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "no optab for perm_even.");
+ return false;
+ }
+
+ if (perm_even_optab->handlers[mode].insn_code == CODE_FOR_nothing)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "perm_even op not supported by target.");
+ return false;
+ }
+
+ perm_odd_optab = optab_for_tree_code (VEC_EXTRACT_ODD_EXPR, vectype);
+ if (!perm_odd_optab)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "no optab for perm_odd.");
+ return false;
+ }
+
+ if (perm_odd_optab->handlers[mode].insn_code == CODE_FOR_nothing)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "perm_odd op not supported by target.");
+ return false;
+ }
+ return true;
+}
+
+
+/* Function vect_permute_load_chain.
+
+ Given a chain of interleaved loads in DR_CHAIN of LENGTH that must be
+ a power of 2, generate extract_even/odd stmts to reorder the input data
+ correctly. Return the final references for loads in RESULT_CHAIN.
+
+ E.g., LENGTH is 4 and the scalar type is short, i.e., VF is 8.
+ The input is 4 vectors each containg 8 elements. We assign a number to each
+ element, the input sequence is:
+
+ 1st vec: 0 1 2 3 4 5 6 7
+ 2nd vec: 8 9 10 11 12 13 14 15
+ 3rd vec: 16 17 18 19 20 21 22 23
+ 4th vec: 24 25 26 27 28 29 30 31
+
+ The output sequence should be:
+
+ 1st vec: 0 4 8 12 16 20 24 28
+ 2nd vec: 1 5 9 13 17 21 25 29
+ 3rd vec: 2 6 10 14 18 22 26 30
+ 4th vec: 3 7 11 15 19 23 27 31
+
+ i.e., the first output vector should contain the first elements of each
+ interleaving group, etc.
+
+ We use extract_even/odd instructions to create such output. The input of each
+ extract_even/odd operation is two vectors
+ 1st vec 2nd vec
+ 0 1 2 3 4 5 6 7
+
+ and the output is the vector of extracted even/odd elements. The output of
+ extract_even will be: 0 2 4 6
+ and of extract_odd: 1 3 5 7
+
+
+ The permutaion is done in log LENGTH stages. In each stage extract_even and
+ extract_odd stmts are created for each pair of vectors in DR_CHAIN in their
+ order. In our example,
+
+ E1: extract_even (1st vec, 2nd vec)
+ E2: extract_odd (1st vec, 2nd vec)
+ E3: extract_even (3rd vec, 4th vec)
+ E4: extract_odd (3rd vec, 4th vec)
+
+ The output for the first stage will be:
+
+ E1: 0 2 4 6 8 10 12 14
+ E2: 1 3 5 7 9 11 13 15
+ E3: 16 18 20 22 24 26 28 30
+ E4: 17 19 21 23 25 27 29 31
+
+ In order to proceed and create the correct sequence for the next stage (or
+ for the correct output, if the second stage is the last one, as in our
+ example), we first put the output of extract_even operation and then the
+ output of extract_odd in RESULT_CHAIN (which is then copied to DR_CHAIN).
+ The input for the second stage is:
+
+ 1st vec (E1): 0 2 4 6 8 10 12 14
+ 2nd vec (E3): 16 18 20 22 24 26 28 30
+ 3rd vec (E2): 1 3 5 7 9 11 13 15
+ 4th vec (E4): 17 19 21 23 25 27 29 31
+
+ The output of the second stage:
+
+ E1: 0 4 8 12 16 20 24 28
+ E2: 2 6 10 14 18 22 26 30
+ E3: 1 5 9 13 17 21 25 29
+ E4: 3 7 11 15 19 23 27 31
+
+ And RESULT_CHAIN after reordering:
+
+ 1st vec (E1): 0 4 8 12 16 20 24 28
+ 2nd vec (E3): 1 5 9 13 17 21 25 29
+ 3rd vec (E2): 2 6 10 14 18 22 26 30
+ 4th vec (E4): 3 7 11 15 19 23 27 31. */
+
+static bool
+vect_permute_load_chain (VEC(tree,heap) *dr_chain,
+ unsigned int length,
+ tree stmt,
+ block_stmt_iterator *bsi,
+ VEC(tree,heap) **result_chain)
+{
+ tree perm_dest, perm_stmt, data_ref, first_vect, second_vect;
+ tree vectype = STMT_VINFO_VECTYPE (vinfo_for_stmt (stmt));
+ int i;
+ unsigned int j;
+
+ /* Check that the operation is supported. */
+ if (!vect_strided_load_supported (vectype))
+ return false;
+
+ *result_chain = VEC_copy (tree, heap, dr_chain);
+ for (i = 0; i < exact_log2 (length); i++)
+ {
+ for (j = 0; j < length; j +=2)
+ {
+ first_vect = VEC_index (tree, dr_chain, j);
+ second_vect = VEC_index (tree, dr_chain, j+1);
+
+ /* data_ref = permute_even (first_data_ref, second_data_ref); */
+ perm_dest = create_tmp_var (vectype, "vect_perm_even");
+ add_referenced_var (perm_dest);
+
+ perm_stmt = build2 (MODIFY_EXPR, vectype, perm_dest,
+ build2 (VEC_EXTRACT_EVEN_EXPR, vectype,
+ first_vect, second_vect));
+
+ data_ref = make_ssa_name (perm_dest, perm_stmt);
+ TREE_OPERAND (perm_stmt, 0) = data_ref;
+ vect_finish_stmt_generation (stmt, perm_stmt, bsi);
+ mark_new_vars_to_rename (perm_stmt);
+
+ VEC_replace (tree, *result_chain, j/2, data_ref);
+
+ /* data_ref = permute_odd (first_data_ref, second_data_ref); */
+ perm_dest = create_tmp_var (vectype, "vect_perm_odd");
+ add_referenced_var (perm_dest);
+
+ perm_stmt = build2 (MODIFY_EXPR, vectype, perm_dest,
+ build2 (VEC_EXTRACT_ODD_EXPR, vectype,
+ first_vect, second_vect));
+ data_ref = make_ssa_name (perm_dest, perm_stmt);
+ TREE_OPERAND (perm_stmt, 0) = data_ref;
+ vect_finish_stmt_generation (stmt, perm_stmt, bsi);
+ mark_new_vars_to_rename (perm_stmt);
+
+ VEC_replace (tree, *result_chain, j/2+length/2, data_ref);
+ }
+ dr_chain = VEC_copy (tree, heap, *result_chain);
+ }
+ return true;
+}
+
+
+/* Function vect_transform_strided_load.
+
+ Given a chain of input interleaved data-refs (in DR_CHAIN), build statements
+ to perform their permutation and ascribe the result vectorized statements to
+ the scalar statements.
+*/
+
+static bool
+vect_transform_strided_load (tree stmt, VEC(tree,heap) *dr_chain, int size,
+ block_stmt_iterator *bsi)
+{
+ stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
+ tree first_stmt = DR_GROUP_FIRST_DR (stmt_info);
+ tree next_stmt, new_stmt;
+ VEC(tree,heap) *result_chain = NULL;
+ unsigned int i, gap_count;
+ tree tmp_data_ref;
+
+ /* DR_CHAIN contains input data-refs that are a part of the interleaving.
+ RESULT_CHAIN is the output of vect_permute_load_chain, it contains permuted
+ vectors, that are ready for vector computation. */
+ result_chain = VEC_alloc (tree, heap, size);
+ /* Permute. */
+ if (!vect_permute_load_chain (dr_chain, size, stmt, bsi, &result_chain))
+ return false;
+
+ /* Put a permuted data-ref in the VECTORIZED_STMT field.
+ Since we scan the chain starting from it's first node, their order
+ corresponds the order of data-refs in RESULT_CHAIN. */
+ next_stmt = first_stmt;
+ gap_count = 1;
+ for (i = 0; VEC_iterate(tree, result_chain, i, tmp_data_ref); i++)
+ {
+ if (!next_stmt)
+ break;
+
+ /* Skip the gaps. Loads created for the gaps will be removed by dead
+ code elimination pass later.
+ DR_GROUP_GAP is the number of steps in elements from the previous
+ access (if there is no gap DR_GROUP_GAP is 1). We skip loads that
+ correspond to the gaps.
+ */
+ if (gap_count < DR_GROUP_GAP (vinfo_for_stmt (next_stmt)))
+ {
+ gap_count++;
+ continue;
+ }
+
+ while (next_stmt)
+ {
+ new_stmt = SSA_NAME_DEF_STMT (tmp_data_ref);
+ /* We assume that if VEC_STMT is not NULL, this is a case of multiple
+ copies, and we put the new vector statement in the first available
+ RELATED_STMT. */
+ if (!STMT_VINFO_VEC_STMT (vinfo_for_stmt (next_stmt)))
+ STMT_VINFO_VEC_STMT (vinfo_for_stmt (next_stmt)) = new_stmt;
+ else
+ {
+ tree prev_stmt = STMT_VINFO_VEC_STMT (vinfo_for_stmt (next_stmt));
+ tree rel_stmt = STMT_VINFO_RELATED_STMT (
+ vinfo_for_stmt (prev_stmt));
+ while (rel_stmt)
+ {
+ prev_stmt = rel_stmt;
+ rel_stmt = STMT_VINFO_RELATED_STMT (vinfo_for_stmt (rel_stmt));
+ }
+ STMT_VINFO_RELATED_STMT (vinfo_for_stmt (prev_stmt)) = new_stmt;
+ }
+ next_stmt = DR_GROUP_NEXT_DR (vinfo_for_stmt (next_stmt));
+ gap_count = 1;
+ /* If NEXT_STMT accesses the same DR as the previous statement,
+ put the same TMP_DATA_REF as its vectorized statement; otherwise
+ get the next data-ref from RESULT_CHAIN. */
+ if (!next_stmt || !DR_GROUP_SAME_DR_STMT (vinfo_for_stmt (next_stmt)))
+ break;
+ }
+ }
+ return true;
+}
+
+
/* vectorizable_load.
Check if STMT reads a non scalar data-ref (array/pointer/structure) that
stmt_vec_info prev_stmt_info;
loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
- struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info);
+ struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info), *first_dr;
tree vectype = STMT_VINFO_VECTYPE (stmt_info);
tree new_temp;
int mode;
- tree new_stmt;
+ tree new_stmt = NULL_TREE;
tree dummy;
enum dr_alignment_support alignment_support_cheme;
tree dataref_ptr = NULL_TREE;
tree ptr_incr;
int nunits = TYPE_VECTOR_SUBPARTS (vectype);
int ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits;
- int j;
+ int i, j, group_size;
tree msq = NULL_TREE, lsq;
tree offset = NULL_TREE;
tree realignment_token = NULL_TREE;
tree phi_stmt = NULL_TREE;
+ VEC(tree,heap) *dr_chain = NULL;
+ bool strided_load = false;
+ tree first_stmt;
/* Is vectorizable load? */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
op = TREE_OPERAND (stmt, 1);
- if (TREE_CODE (op) != ARRAY_REF && TREE_CODE (op) != INDIRECT_REF)
+ if (TREE_CODE (op) != ARRAY_REF
+ && TREE_CODE (op) != INDIRECT_REF
+ && !DR_GROUP_FIRST_DR (stmt_info))
return false;
if (!STMT_VINFO_DATA_REF (stmt_info))
return false;
}
+ /* Check if the load is a part of an interleaving chain. */
+ if (DR_GROUP_FIRST_DR (stmt_info))
+ {
+ strided_load = true;
+
+ /* Check if interleaving is supported. */
+ if (!vect_strided_load_supported (vectype))
+ return false;
+ }
+
if (!vec_stmt) /* transformation not required. */
{
STMT_VINFO_TYPE (stmt_info) = load_vec_info_type;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "transform load.");
- alignment_support_cheme = vect_supportable_dr_alignment (dr);
+ if (strided_load)
+ {
+ first_stmt = DR_GROUP_FIRST_DR (stmt_info);
+ /* Check if the chain of loads is already vectorized. */
+ if (STMT_VINFO_VEC_STMT (vinfo_for_stmt (first_stmt)))
+ {
+ *vec_stmt = STMT_VINFO_VEC_STMT (stmt_info);
+ return true;
+ }
+ first_dr = STMT_VINFO_DATA_REF (vinfo_for_stmt (first_stmt));
+ group_size = DR_GROUP_SIZE (vinfo_for_stmt (first_stmt));
+ dr_chain = VEC_alloc (tree, heap, group_size);
+ }
+ else
+ {
+ first_stmt = stmt;
+ first_dr = dr;
+ group_size = 1;
+ }
+
+ alignment_support_cheme = vect_supportable_dr_alignment (first_dr);
gcc_assert (alignment_support_cheme);
+
/* In case the vectorization factor (VF) is bigger than the number
of elements that we can fit in a vectype (nunits), we have to generate
more than one vector stmt - i.e - we need to "unroll" the
information we recorded in RELATED_STMT field is used to vectorize
stmt S2. */
+ /* In case of interleaving (non-unit strided access):
+
+ S1: x2 = &base + 2
+ S2: x0 = &base
+ S3: x1 = &base + 1
+ S4: x3 = &base + 3
+
+ Vectorized loads are created in the order of memory accesses
+ starting from the access of the first stmt of the chain:
+
+ VS1: vx0 = &base
+ VS2: vx1 = &base + vec_size*1
+ VS3: vx3 = &base + vec_size*2
+ VS4: vx4 = &base + vec_size*3
+
+ Then permutation statements are generated:
+
+ VS5: vx5 = VEC_EXTRACT_EVEN_EXPR < vx0, vx1 >
+ VS6: vx6 = VEC_EXTRACT_ODD_EXPR < vx0, vx1 >
+ ...
+
+ And they are put in STMT_VINFO_VEC_STMT of the corresponding scalar stmts
+ (the order of the data-refs in the output of vect_permute_load_chain
+ corresponds to the order of scalar stmts in the interleaving chain - see
+ the documentaion of vect_permute_load_chain()).
+ The generation of permutation stmts and recording them in
+ STMT_VINFO_VEC_STMT is done in vect_transform_strided_load().
+
+ In case of both multiple types and interleaving, the vector loads and
+ permutation stmts above are created for every copy. The result vector stmts
+ are put in STMT_VINFO_VEC_STMT for the first copy and in the corresponding
+ STMT_VINFO_RELATED_STMT for the next copies. */
+
/* If the data reference is aligned (dr_aligned) or potentially unaligned
on a target that supports unaligned accesses (dr_unaligned_supported)
we generate the following code:
vec_dest = realign_load (msq, lsq, realignment_token)
indx = indx + 1;
msq = lsq;
- }
- */
+ } */
if (alignment_support_cheme == dr_unaligned_software_pipeline)
{
- msq = vect_setup_realignment (stmt, bsi, &realignment_token);
+ msq = vect_setup_realignment (first_stmt, bsi, &realignment_token);
phi_stmt = SSA_NAME_DEF_STMT (msq);
offset = size_int (TYPE_VECTOR_SUBPARTS (vectype) - 1);
}
{
/* 1. Create the vector pointer update chain. */
if (j == 0)
- dataref_ptr = vect_create_data_ref_ptr (stmt, bsi, offset,
+ dataref_ptr = vect_create_data_ref_ptr (first_stmt, bsi, offset,
&dummy, &ptr_incr, false);
else
dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, bsi, stmt);
- /* 2. Create the vector-load in the loop. */
- switch (alignment_support_cheme)
- {
- case dr_aligned:
- gcc_assert (aligned_access_p (dr));
- data_ref = build_fold_indirect_ref (dataref_ptr);
- break;
- case dr_unaligned_supported:
- {
- int mis = DR_MISALIGNMENT (dr);
- tree tmis = (mis == -1 ? size_zero_node : size_int (mis));
-
- gcc_assert (!aligned_access_p (dr));
- tmis = size_binop (MULT_EXPR, tmis, size_int(BITS_PER_UNIT));
- data_ref =
- build2 (MISALIGNED_INDIRECT_REF, vectype, dataref_ptr, tmis);
- break;
- }
- case dr_unaligned_software_pipeline:
- gcc_assert (!aligned_access_p (dr));
- data_ref = build1 (ALIGN_INDIRECT_REF, vectype, dataref_ptr);
- break;
- default:
- gcc_unreachable ();
- }
- vec_dest = vect_create_destination_var (scalar_dest, vectype);
- new_stmt = build2 (MODIFY_EXPR, vectype, vec_dest, data_ref);
- new_temp = make_ssa_name (vec_dest, new_stmt);
- TREE_OPERAND (new_stmt, 0) = new_temp;
- vect_finish_stmt_generation (stmt, new_stmt, bsi);
- copy_virtual_operands (new_stmt, stmt);
- mark_new_vars_to_rename (new_stmt);
-
- /* 3. Handle explicit realignment if necessary/supported. */
- if (alignment_support_cheme == dr_unaligned_software_pipeline)
- {
- /* Create in loop:
- <vec_dest = realign_load (msq, lsq, realignment_token)> */
- lsq = TREE_OPERAND (new_stmt, 0);
- if (!realignment_token)
- realignment_token = dataref_ptr;
- vec_dest = vect_create_destination_var (scalar_dest, vectype);
- new_stmt =
- build3 (REALIGN_LOAD_EXPR, vectype, msq, lsq, realignment_token);
- new_stmt = build2 (MODIFY_EXPR, vectype, vec_dest, new_stmt);
- new_temp = make_ssa_name (vec_dest, new_stmt);
- TREE_OPERAND (new_stmt, 0) = new_temp;
- vect_finish_stmt_generation (stmt, new_stmt, bsi);
- if (j == ncopies - 1)
- add_phi_arg (phi_stmt, lsq, loop_latch_edge (loop));
- msq = lsq;
- }
+ for (i = 0; i < group_size; i++)
+ {
+ /* 2. Create the vector-load in the loop. */
+ switch (alignment_support_cheme)
+ {
+ case dr_aligned:
+ gcc_assert (aligned_access_p (first_dr));
+ data_ref = build_fold_indirect_ref (dataref_ptr);
+ break;
+ case dr_unaligned_supported:
+ {
+ int mis = DR_MISALIGNMENT (first_dr);
+ tree tmis = (mis == -1 ? size_zero_node : size_int (mis));
+
+ gcc_assert (!aligned_access_p (first_dr));
+ tmis = size_binop (MULT_EXPR, tmis, size_int(BITS_PER_UNIT));
+ data_ref =
+ build2 (MISALIGNED_INDIRECT_REF, vectype, dataref_ptr, tmis);
+ break;
+ }
+ case dr_unaligned_software_pipeline:
+ gcc_assert (!aligned_access_p (first_dr));
+ data_ref = build1 (ALIGN_INDIRECT_REF, vectype, dataref_ptr);
+ break;
+ default:
+ gcc_unreachable ();
+ }
+ vec_dest = vect_create_destination_var (scalar_dest, vectype);
+ new_stmt = build2 (MODIFY_EXPR, vectype, vec_dest, data_ref);
+ new_temp = make_ssa_name (vec_dest, new_stmt);
+ TREE_OPERAND (new_stmt, 0) = new_temp;
+ vect_finish_stmt_generation (stmt, new_stmt, bsi);
+ copy_virtual_operands (new_stmt, stmt);
+ mark_new_vars_to_rename (new_stmt);
+
+ /* 3. Handle explicit realignment if necessary/supported. */
+ if (alignment_support_cheme == dr_unaligned_software_pipeline)
+ {
+ /* Create in loop:
+ <vec_dest = realign_load (msq, lsq, realignment_token)> */
+ lsq = TREE_OPERAND (new_stmt, 0);
+ if (!realignment_token)
+ realignment_token = dataref_ptr;
+ vec_dest = vect_create_destination_var (scalar_dest, vectype);
+ new_stmt =
+ build3 (REALIGN_LOAD_EXPR, vectype, msq, lsq, realignment_token);
+ new_stmt = build2 (MODIFY_EXPR, vectype, vec_dest, new_stmt);
+ new_temp = make_ssa_name (vec_dest, new_stmt);
+ TREE_OPERAND (new_stmt, 0) = new_temp;
+ vect_finish_stmt_generation (stmt, new_stmt, bsi);
+ if (i == group_size - 1 && j == ncopies - 1)
+ add_phi_arg (phi_stmt, lsq, loop_latch_edge (loop));
+ msq = lsq;
+ }
+ if (strided_load)
+ VEC_quick_push (tree, dr_chain, new_temp);
+ if (i < group_size - 1)
+ dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, bsi, stmt);
+ }
- if (j == 0)
- STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt;
+ if (strided_load)
+ {
+ if (!vect_transform_strided_load (stmt, dr_chain, group_size, bsi))
+ return false;
+ *vec_stmt = STMT_VINFO_VEC_STMT (stmt_info);
+ dr_chain = VEC_alloc (tree, heap, group_size);
+ }
else
- STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt;
- prev_stmt_info = vinfo_for_stmt (new_stmt);
+ {
+ if (j == 0)
+ STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt;
+ else
+ STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt;
+ prev_stmt_info = vinfo_for_stmt (new_stmt);
+ }
}
return true;
Create a vectorized stmt to replace STMT, and insert it at BSI. */
bool
-vect_transform_stmt (tree stmt, block_stmt_iterator *bsi)
+vect_transform_stmt (tree stmt, block_stmt_iterator *bsi, bool *strided_store)
{
bool is_store = false;
tree vec_stmt = NULL_TREE;
case store_vec_info_type:
done = vectorizable_store (stmt, bsi, &vec_stmt);
gcc_assert (done);
- is_store = true;
+ if (DR_GROUP_FIRST_DR (stmt_info))
+ {
+ /* In case of interleaving, the whole chain is vectorized when the
+ last store in the chain is reached. Store stmts before the last
+ one are skipped, and there vec_stmt_info shoudn't be freed
+ meanwhile. */
+ *strided_store = true;
+ if (STMT_VINFO_VEC_STMT (stmt_info))
+ is_store = true;
+ }
+ else
+ is_store = true;
break;
case condition_vec_info_type:
gcc_unreachable ();
}
- gcc_assert (vec_stmt);
- STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
- orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
- if (orig_stmt_in_pattern)
- {
- stmt_vec_info stmt_vinfo = vinfo_for_stmt (orig_stmt_in_pattern);
- if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
- {
- gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
-
- /* STMT was inserted by the vectorizer to replace a computation
- idiom. ORIG_STMT_IN_PATTERN is a stmt in the original
- sequence that computed this idiom. We need to record a pointer
- to VEC_STMT in the stmt_info of ORIG_STMT_IN_PATTERN. See more
- detail in the documentation of vect_pattern_recog. */
-
- STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
- }
- }
+ gcc_assert (vec_stmt || *strided_store);
+ if (vec_stmt)
+ {
+ STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
+ orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
+ if (orig_stmt_in_pattern)
+ {
+ stmt_vec_info stmt_vinfo = vinfo_for_stmt (orig_stmt_in_pattern);
+ if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
+ {
+ gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
+
+ /* STMT was inserted by the vectorizer to replace a
+ computation idiom. ORIG_STMT_IN_PATTERN is a stmt in the
+ original sequence that computed this idiom. We need to
+ record a pointer to VEC_STMT in the stmt_info of
+ ORIG_STMT_IN_PATTERN. See more details in the
+ documentation of vect_pattern_recog. */
+
+ STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
+ }
+ }
+ }
}
if (STMT_VINFO_LIVE_P (stmt_info))
prolog_niters = min ( LOOP_NITERS , (VF - addr_mis/elem_size)&(VF-1) )
(elem_size = element type size; an element is the scalar element
- whose type is the inner type of the vectype) */
+ whose type is the inner type of the vectype)
+
+ For interleaving,
+
+ prolog_niters = min ( LOOP_NITERS ,
+ (VF/group_size - addr_mis/elem_size)&(VF/group_size-1) )
+ where group_size is the size of the interleaved group.
+*/
static tree
vect_gen_niters_for_prolog_loop (loop_vec_info loop_vinfo, tree loop_niters)
tree vectype = STMT_VINFO_VECTYPE (stmt_info);
int vectype_align = TYPE_ALIGN (vectype) / BITS_PER_UNIT;
tree niters_type = TREE_TYPE (loop_niters);
+ int group_size = 1;
+ int element_size = GET_MODE_SIZE (TYPE_MODE (TREE_TYPE (DR_REF (dr))));
+
+ if (DR_GROUP_FIRST_DR (stmt_info))
+ {
+ /* For interleaved access element size must be multipled by the size of
+ the interleaved group. */
+ group_size = DR_GROUP_SIZE (vinfo_for_stmt (
+ DR_GROUP_FIRST_DR (stmt_info)));
+ element_size *= group_size;
+ }
pe = loop_preheader_edge (loop);
if (LOOP_PEELING_FOR_ALIGNMENT (loop_vinfo) > 0)
{
int byte_misalign = LOOP_PEELING_FOR_ALIGNMENT (loop_vinfo);
- int element_size = GET_MODE_SIZE (TYPE_MODE (TREE_TYPE (DR_REF (dr))));
int elem_misalign = byte_misalign / element_size;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "known alignment = %d.", byte_misalign);
- iters = build_int_cst (niters_type, (vf - elem_misalign)&(vf-1));
+ iters = build_int_cst (niters_type,
+ (vf - elem_misalign)&(vf/group_size-1));
}
else
{
int vectorization_factor = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
bitmap_iterator bi;
unsigned int j;
+ bool strided_store;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "=== vec_transform_loop ===");
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "transform statement.");
- is_store = vect_transform_stmt (stmt, &si);
- if (is_store)
- {
- /* Free the attached stmt_vec_info and remove the stmt. */
- stmt_ann_t ann = stmt_ann (stmt);
- free (stmt_info);
- set_stmt_info (ann, NULL);
- bsi_remove (&si, true);
- continue;
+ strided_store = false;
+ is_store = vect_transform_stmt (stmt, &si, &strided_store);
+ if (is_store)
+ {
+ stmt_ann_t ann;
+ if (DR_GROUP_FIRST_DR (stmt_info))
+ {
+ /* Interleaving. If IS_STORE is TRUE, the vectorization of the
+ interleaving chain was completed - free all the stores in
+ the chain. */
+ tree next = DR_GROUP_FIRST_DR (stmt_info);
+ tree tmp;
+ stmt_vec_info next_stmt_info;
+
+ while (next)
+ {
+ next_stmt_info = vinfo_for_stmt (next);
+ /* Free the attached stmt_vec_info and remove the stmt. */
+ ann = stmt_ann (next);
+ tmp = DR_GROUP_NEXT_DR (next_stmt_info);
+ free (next_stmt_info);
+ set_stmt_info (ann, NULL);
+ next = tmp;
+ }
+ bsi_remove (&si, true);
+ continue;
+ }
+ else
+ {
+ /* Free the attached stmt_vec_info and remove the stmt. */
+ ann = stmt_ann (stmt);
+ free (stmt_info);
+ set_stmt_info (ann, NULL);
+ bsi_remove (&si, true);
+ continue;
+ }
}
-
+ else
+ {
+ if (strided_store)
+ {
+ /* This is case of skipped interleaved store. We don't free
+ its stmt_vec_info. */
+ bsi_remove (&si, true);
+ continue;
+ }
+ }
bsi_next (&si);
} /* stmts in BB */
} /* BBs in loop */
else
STMT_VINFO_DEF_TYPE (res) = vect_loop_def;
STMT_VINFO_SAME_ALIGN_REFS (res) = VEC_alloc (dr_p, heap, 5);
+ DR_GROUP_FIRST_DR (res) = NULL_TREE;
+ DR_GROUP_NEXT_DR (res) = NULL_TREE;
+ DR_GROUP_SIZE (res) = 0;
+ DR_GROUP_STORE_COUNT (res) = 0;
+ DR_GROUP_GAP (res) = 0;
+ DR_GROUP_SAME_DR_STMT (res) = NULL_TREE;
return res;
}
/* Classify the def of this stmt. */
enum vect_def_type def_type;
+ /* Interleaving info. */
+ /* First data-ref in the interleaving group. */
+ tree first_dr;
+ /* Pointer to the next data-ref in the group. */
+ tree next_dr;
+ /* The size of the interleaving group. */
+ unsigned int size;
+ /* For stores, number of stores from this group seen. We vectorize the last
+ one. */
+ unsigned int store_count;
+ /* For loads only, the gap from the previous load. For consecutive loads, GAP
+ is 1. */
+ unsigned int gap;
+ /* In case that two or more stmts share data-ref, this is the pointer to the
+ previously detected stmt with the same dr. */
+ tree same_dr_stmt;
} *stmt_vec_info;
/* Access Functions. */
-#define STMT_VINFO_TYPE(S) (S)->type
-#define STMT_VINFO_STMT(S) (S)->stmt
-#define STMT_VINFO_LOOP_VINFO(S) (S)->loop_vinfo
-#define STMT_VINFO_RELEVANT(S) (S)->relevant
-#define STMT_VINFO_LIVE_P(S) (S)->live
-#define STMT_VINFO_VECTYPE(S) (S)->vectype
-#define STMT_VINFO_VEC_STMT(S) (S)->vectorized_stmt
-#define STMT_VINFO_DATA_REF(S) (S)->data_ref_info
-#define STMT_VINFO_IN_PATTERN_P(S) (S)->in_pattern_p
-#define STMT_VINFO_RELATED_STMT(S) (S)->related_stmt
-#define STMT_VINFO_SAME_ALIGN_REFS(S) (S)->same_align_refs
-#define STMT_VINFO_DEF_TYPE(S) (S)->def_type
+#define STMT_VINFO_TYPE(S) (S)->type
+#define STMT_VINFO_STMT(S) (S)->stmt
+#define STMT_VINFO_LOOP_VINFO(S) (S)->loop_vinfo
+#define STMT_VINFO_RELEVANT(S) (S)->relevant
+#define STMT_VINFO_LIVE_P(S) (S)->live
+#define STMT_VINFO_VECTYPE(S) (S)->vectype
+#define STMT_VINFO_VEC_STMT(S) (S)->vectorized_stmt
+#define STMT_VINFO_DATA_REF(S) (S)->data_ref_info
+#define STMT_VINFO_IN_PATTERN_P(S) (S)->in_pattern_p
+#define STMT_VINFO_RELATED_STMT(S) (S)->related_stmt
+#define STMT_VINFO_SAME_ALIGN_REFS(S) (S)->same_align_refs
+#define STMT_VINFO_DEF_TYPE(S) (S)->def_type
+#define STMT_VINFO_DR_GROUP_FIRST_DR(S) (S)->first_dr
+#define STMT_VINFO_DR_GROUP_NEXT_DR(S) (S)->next_dr
+#define STMT_VINFO_DR_GROUP_SIZE(S) (S)->size
+#define STMT_VINFO_DR_GROUP_STORE_COUNT(S) (S)->store_count
+#define STMT_VINFO_DR_GROUP_GAP(S) (S)->gap
+#define STMT_VINFO_DR_GROUP_SAME_DR_STMT(S)(S)->same_dr_stmt
+
+#define DR_GROUP_FIRST_DR(S) (S)->first_dr
+#define DR_GROUP_NEXT_DR(S) (S)->next_dr
+#define DR_GROUP_SIZE(S) (S)->size
+#define DR_GROUP_STORE_COUNT(S) (S)->store_count
+#define DR_GROUP_GAP(S) (S)->gap
+#define DR_GROUP_SAME_DR_STMT(S) (S)->same_dr_stmt
#define STMT_VINFO_RELEVANT_P(S) ((S)->relevant != vect_unused_in_loop)
DEFTREECODE (VEC_PACK_MOD_EXPR, "vec_pack_mod_expr", tcc_binary, 2)
DEFTREECODE (VEC_PACK_SAT_EXPR, "vec_pack_sat_expr", tcc_binary, 2)
+/* Extract even/odd fields from vectors. */
+DEFTREECODE (VEC_EXTRACT_EVEN_EXPR, "vec_extracteven_expr", tcc_binary, 2)
+DEFTREECODE (VEC_EXTRACT_ODD_EXPR, "vec_extractodd_expr", tcc_binary, 2)
+
+/* Merge input vectors interleaving their fields. */
+DEFTREECODE (VEC_INTERLEAVE_HIGH_EXPR, "vec_interleavehigh_expr", tcc_binary, 2)
+DEFTREECODE (VEC_INTERLEAVE_LOW_EXPR, "vec_interleavelow_expr", tcc_binary, 2)
+
/*
Local variables:
mode:c
projects / platform / upstream / gcc.git / commitdiff