sse.md (FMAMODE_NOVF512): New mode iterator.

	* config/i386/sse.md (FMAMODE_NOVF512): New mode iterator.
	(<sd_mask_codefor>fma_fmadd_<mode><sd_maskz_name><round_name>):
	Split out
	<sd_mask_codefor>fma_fmadd_<VF_512:mode><sd_maskz_name><round_name>.
	Use FMAMODE_NOVF512 mode iterator.
	(<sd_mask_codefor>fma_fmsub_<mode><sd_maskz_name><round_name>): Ditto.
	(<sd_mask_codefor>fma_fnmadd_<mode><sd_maskz_name><round_name>): Ditto.
	(<sd_mask_codefor>fma_fnmsub_<mode><sd_maskz_name><round_name>): Ditto.
	(<sd_mask_codefor>fma_fmaddsub_<mode><sd_maskz_name><round_name>):
	Split out
	<sd_mask_codefor>fma_fmaddsub_<VF_512:mode><sd_maskz_name><round_name>.
	Use VF_128_256 mode iterator.
	(<sd_mask_codefor>fma_fmsubadd_<mode><sd_maskz_name><round_name>):
	Ditto.

From-SVN: r208948

diff --git a/gcc/ChangeLog b/gcc/ChangeLog
index ded35eb..b7b6bca 100644 (file)
--- a/gcc/ChangeLog
+++ b/gcc/ChangeLog
@@ -1,3 +1,20 @@
+2014-03-30  Uros Bizjak  <ubizjak@gmail.com>
+
+       * config/i386/sse.md (FMAMODE_NOVF512): New mode iterator.
+       (<sd_mask_codefor>fma_fmadd_<mode><sd_maskz_name><round_name>):
+       Split out
+       <sd_mask_codefor>fma_fmadd_<VF_512:mode><sd_maskz_name><round_name>.
+       Use FMAMODE_NOVF512 mode iterator.
+       (<sd_mask_codefor>fma_fmsub_<mode><sd_maskz_name><round_name>): Ditto.
+       (<sd_mask_codefor>fma_fnmadd_<mode><sd_maskz_name><round_name>): Ditto.
+       (<sd_mask_codefor>fma_fnmsub_<mode><sd_maskz_name><round_name>): Ditto.
+       (<sd_mask_codefor>fma_fmaddsub_<mode><sd_maskz_name><round_name>):
+       Split out
+       <sd_mask_codefor>fma_fmaddsub_<VF_512:mode><sd_maskz_name><round_name>.
+       Use VF_128_256 mode iterator.
+       (<sd_mask_codefor>fma_fmsubadd_<mode><sd_maskz_name><round_name>):
+       Ditto.
+
 2014-03-28  Jan Hubicka  <hubicka@ucw.cz>
 
        * cgraph.c (cgraph_redirect_edge_call_stmt_to_callee): Clear
@@ -67,7 +84,8 @@
 2014-03-28  Jan Hubicka  <hubicka@ucw.cz>
 
        * cgraph.c: Include expr.h and tree-dfa.h.
-       (cgraph_redirect_edge_call_stmt_to_callee): If call in noreturn; remove LHS.
+       (cgraph_redirect_edge_call_stmt_to_callee): If call in noreturn;
+       remove LHS.
 
 2014-03-28  Vladimir Makarov  <vmakarov@redhat.com>
 
@@ -111,8 +129,7 @@
        Optimize vec_extract of 64-bit values, where the value being
        extracted is in the top word, where we can use scalar
        instructions.  Add direct move and store support.  Combine the big
-       endian/little endian vector select load support into a single
-       insn.
+       endian/little endian vector select load support into a single insn.
        (vsx_extract_<mode>_internal1): Likewise.
        (vsx_extract_<mode>_internal2): Likewise.
        (vsx_extract_<mode>_load): Likewise.
@@ -257,12 +274,13 @@
 
        PR ipa/60315
        * cif-code.def (UNREACHABLE) New code.
-       * ipa-inline.c (inline_small_functions): Skip edges to __builtlin_unreachable.
+       * ipa-inline.c (inline_small_functions): Skip edges to
+       __builtlin_unreachable.
        (estimate_edge_growth): Allow edges to __builtlin_unreachable.
        * ipa-inline-analysis.c (edge_set_predicate): Redirect edges with false
        predicate to __bulitin_unreachable.
-       (set_cond_stmt_execution_predicate): Fix issue when invert_tree_comparison
-       returns ERROR_MARK.
+       (set_cond_stmt_execution_predicate): Fix issue when
+       invert_tree_comparison returns ERROR_MARK.
        * ipa-pure-const.c (propagate_pure_const, propagate_nothrow): Do not
        propagate to inline clones.
        * cgraph.c (verify_edge_corresponds_to_fndecl): Allow redirection
@@ -284,8 +302,8 @@
        (addv<mode>4, subv<mode>4, mulv<mode>4): If operands[2] is CONST_INT,
        don't generate (sign_extend (const_int)).
        (*addv<mode>4, *subv<mode>4, *mulv<mode>4): Disallow CONST_INT_P
-       operands[2].  Use We constraint instead of <i> and <general_sext_operand>
-       predicate instead of <general_operand>.
+       operands[2].  Use We constraint instead of <i> and
+       <general_sext_operand> predicate instead of <general_operand>.
        (*addv<mode>4_1, *subv<mode>4_1, *mulv<mode>4_1): New insns.
        * config/i386/constraints.md (We): New constraint.
        * config/i386/predicates.md (x86_64_sext_operand,
@@ -323,8 +341,7 @@
 
 2014-03-25  Richard Biener  <rguenther@suse.de>
 
-       * lto-wrapper.c (merge_and_complain): Handle OPT_fPIE like
-       OPT_fpie.
+       * lto-wrapper.c (merge_and_complain): Handle OPT_fPIE like OPT_fpie.
        (run_gcc): Likewise.
 
 2014-03-25  Jakub Jelinek  <jakub@redhat.com>
@@ -550,13 +567,10 @@
 2014-03-18  Basile Starynkevitch  <basile@starynkevitch.net>
 
        * plugin.def: Improve comment for PLUGIN_INCLUDE_FILE.
-       * doc/plugins.texi (Plugin callbacks): Mention
-       PLUGIN_INCLUDE_FILE.
+       * doc/plugins.texi (Plugin callbacks): Mention PLUGIN_INCLUDE_FILE.
        Italicize plugin event names in description.  Explain that
-       PLUGIN_PRAGMAS has no sense for lto1. Explain
-       PLUGIN_INCLUDE_FILE.
-       Remind that no GCC functions should be called after
-       PLUGIN_FINISH.
+       PLUGIN_PRAGMAS has no sense for lto1.  Explain PLUGIN_INCLUDE_FILE.
+       Remind that no GCC functions should be called after PLUGIN_FINISH.
        Explain what pragmas with expansion are.
 
 2014-03-18  Martin Liska  <mliska@suse.cz>
@@ -814,8 +828,7 @@
 
 2014-03-13  Cesar Philippidis  <cesar@codesourcery.com>
 
-       * lto-wrapper.c (maybe_unlink_file): Suppress diagnostic
-       messages.
+       * lto-wrapper.c (maybe_unlink_file): Suppress diagnostic messages.
 
 2014-03-13  Jakub Jelinek  <jakub@redhat.com>
 

diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index f30b27e..72a4d6d 100644 (file)
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -2712,50 +2712,46 @@
        (fma:FMAMODEM
          (match_operand:FMAMODEM 1 "nonimmediate_operand")
          (match_operand:FMAMODEM 2 "nonimmediate_operand")
-         (match_operand:FMAMODEM 3 "nonimmediate_operand")))]
-  "")
+         (match_operand:FMAMODEM 3 "nonimmediate_operand")))])
 
 (define_expand "fms<mode>4"
   [(set (match_operand:FMAMODEM 0 "register_operand")
        (fma:FMAMODEM
          (match_operand:FMAMODEM 1 "nonimmediate_operand")
          (match_operand:FMAMODEM 2 "nonimmediate_operand")
-         (neg:FMAMODEM (match_operand:FMAMODEM 3 "nonimmediate_operand"))))]
-  "")
+         (neg:FMAMODEM (match_operand:FMAMODEM 3 "nonimmediate_operand"))))])
 
 (define_expand "fnma<mode>4"
   [(set (match_operand:FMAMODEM 0 "register_operand")
        (fma:FMAMODEM
          (neg:FMAMODEM (match_operand:FMAMODEM 1 "nonimmediate_operand"))
          (match_operand:FMAMODEM 2 "nonimmediate_operand")
-         (match_operand:FMAMODEM 3 "nonimmediate_operand")))]
-  "")
+         (match_operand:FMAMODEM 3 "nonimmediate_operand")))])
 
 (define_expand "fnms<mode>4"
   [(set (match_operand:FMAMODEM 0 "register_operand")
        (fma:FMAMODEM
          (neg:FMAMODEM (match_operand:FMAMODEM 1 "nonimmediate_operand"))
          (match_operand:FMAMODEM 2 "nonimmediate_operand")
-         (neg:FMAMODEM (match_operand:FMAMODEM 3 "nonimmediate_operand"))))]
-  "")
+         (neg:FMAMODEM (match_operand:FMAMODEM 3 "nonimmediate_operand"))))])
 
 ;; The builtins for intrinsics are not constrained by SSE math enabled.
-(define_mode_iterator FMAMODE [(SF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F")
-                              (DF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F")
-                              (V4SF "TARGET_FMA || TARGET_FMA4")
-                              (V2DF "TARGET_FMA || TARGET_FMA4")
-                              (V8SF "TARGET_FMA || TARGET_FMA4")
-                              (V4DF "TARGET_FMA || TARGET_FMA4")
-                              (V16SF "TARGET_AVX512F")
-                              (V8DF "TARGET_AVX512F")])
+(define_mode_iterator FMAMODE
+  [(SF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F")
+   (DF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F")
+   (V4SF "TARGET_FMA || TARGET_FMA4")
+   (V2DF "TARGET_FMA || TARGET_FMA4")
+   (V8SF "TARGET_FMA || TARGET_FMA4")
+   (V4DF "TARGET_FMA || TARGET_FMA4")
+   (V16SF "TARGET_AVX512F")
+   (V8DF "TARGET_AVX512F")])
 
 (define_expand "fma4i_fmadd_<mode>"
   [(set (match_operand:FMAMODE 0 "register_operand")
        (fma:FMAMODE
          (match_operand:FMAMODE 1 "nonimmediate_operand")
          (match_operand:FMAMODE 2 "nonimmediate_operand")
-         (match_operand:FMAMODE 3 "nonimmediate_operand")))]
-  "")
+         (match_operand:FMAMODE 3 "nonimmediate_operand")))])
 
 (define_expand "avx512f_fmadd_<mode>_maskz<round_expand_name>"
   [(match_operand:VF_512 0 "register_operand")
@@ -2771,12 +2767,20 @@
   DONE;
 })
 
+(define_mode_iterator FMAMODE_NOVF512
+  [(SF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F")
+   (DF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F")
+   (V4SF "TARGET_FMA || TARGET_FMA4")
+   (V2DF "TARGET_FMA || TARGET_FMA4")
+   (V8SF "TARGET_FMA || TARGET_FMA4")
+   (V4DF "TARGET_FMA || TARGET_FMA4")])
+
 (define_insn "<sd_mask_codefor>fma_fmadd_<mode><sd_maskz_name><round_name>"
-  [(set (match_operand:FMAMODE 0 "register_operand" "=v,v,v,x,x")
-       (fma:FMAMODE
-         (match_operand:FMAMODE 1 "<round_nimm_predicate>" "%0,0,v,x,x")
-         (match_operand:FMAMODE 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
-         (match_operand:FMAMODE 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")))]
+  [(set (match_operand:FMAMODE_NOVF512 0 "register_operand" "=v,v,v,x,x")
+       (fma:FMAMODE_NOVF512
+         (match_operand:FMAMODE_NOVF512 1 "<round_nimm_predicate>" "%0,0,v,x,x")
+         (match_operand:FMAMODE_NOVF512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
+         (match_operand:FMAMODE_NOVF512 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")))]
   "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
   "@
    vfmadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
@@ -2788,6 +2792,21 @@
    (set_attr "type" "ssemuladd")
    (set_attr "mode" "<MODE>")])
 
+(define_insn "<sd_mask_codefor>fma_fmadd_<mode><sd_maskz_name><round_name>"
+  [(set (match_operand:VF_512 0 "register_operand" "=v,v,v")
+       (fma:VF_512
+         (match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v")
+         (match_operand:VF_512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>")
+         (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0")))]
+  "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
+  "@
+   vfmadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
+   vfmadd213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>}
+   vfmadd231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}"
+  [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f")
+   (set_attr "type" "ssemuladd")
+   (set_attr "mode" "<MODE>")])
+
 (define_insn "avx512f_fmadd_<mode>_mask<round_name>"
   [(set (match_operand:VF_512 0 "register_operand" "=v,v")
        (vec_merge:VF_512
@@ -2821,12 +2840,12 @@
    (set_attr "mode" "<MODE>")])
 
 (define_insn "<sd_mask_codefor>fma_fmsub_<mode><sd_maskz_name><round_name>"
-  [(set (match_operand:FMAMODE 0 "register_operand" "=v,v,v,x,x")
-       (fma:FMAMODE
-         (match_operand:FMAMODE   1 "<round_nimm_predicate>" "%0, 0, v, x,x")
-         (match_operand:FMAMODE   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
-         (neg:FMAMODE
-           (match_operand:FMAMODE 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))))]
+  [(set (match_operand:FMAMODE_NOVF512 0 "register_operand" "=v,v,v,x,x")
+       (fma:FMAMODE_NOVF512
+         (match_operand:FMAMODE_NOVF512   1 "<round_nimm_predicate>" "%0,0,v,x,x")
+         (match_operand:FMAMODE_NOVF512   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
+         (neg:FMAMODE_NOVF512
+           (match_operand:FMAMODE_NOVF512 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))))]
   "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
   "@
    vfmsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
@@ -2838,6 +2857,22 @@
    (set_attr "type" "ssemuladd")
    (set_attr "mode" "<MODE>")])
 
+(define_insn "<sd_mask_codefor>fma_fmsub_<mode><sd_maskz_name><round_name>"
+  [(set (match_operand:VF_512 0 "register_operand" "=v,v,v")
+       (fma:VF_512
+         (match_operand:VF_512   1 "<round_nimm_predicate>" "%0,0,v")
+         (match_operand:VF_512   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>")
+         (neg:VF_512
+           (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0"))))]
+  "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
+  "@
+   vfmsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
+   vfmsub213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>}
+   vfmsub231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}"
+  [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f")
+   (set_attr "type" "ssemuladd")
+   (set_attr "mode" "<MODE>")])
+
 (define_insn "avx512f_fmsub_<mode>_mask<round_name>"
   [(set (match_operand:VF_512 0 "register_operand" "=v,v")
        (vec_merge:VF_512
@@ -2873,12 +2908,12 @@
    (set_attr "mode" "<MODE>")])
 
 (define_insn "<sd_mask_codefor>fma_fnmadd_<mode><sd_maskz_name><round_name>"
-  [(set (match_operand:FMAMODE 0 "register_operand" "=v,v,v,x,x")
-       (fma:FMAMODE
-         (neg:FMAMODE
-           (match_operand:FMAMODE 1 "<round_nimm_predicate>" "%0,0,v,x,x"))
-         (match_operand:FMAMODE   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
-         (match_operand:FMAMODE   3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")))]
+  [(set (match_operand:FMAMODE_NOVF512 0 "register_operand" "=v,v,v,x,x")
+       (fma:FMAMODE_NOVF512
+         (neg:FMAMODE_NOVF512
+           (match_operand:FMAMODE_NOVF512 1 "<round_nimm_predicate>" "%0,0,v,x,x"))
+         (match_operand:FMAMODE_NOVF512   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
+         (match_operand:FMAMODE_NOVF512   3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")))]
   "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
   "@
    vfnmadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
@@ -2890,6 +2925,22 @@
    (set_attr "type" "ssemuladd")
    (set_attr "mode" "<MODE>")])
 
+(define_insn "<sd_mask_codefor>fma_fnmadd_<mode><sd_maskz_name><round_name>"
+  [(set (match_operand:VF_512 0 "register_operand" "=v,v,v")
+       (fma:VF_512
+         (neg:VF_512
+           (match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v"))
+         (match_operand:VF_512   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>")
+         (match_operand:VF_512   3 "<round_nimm_predicate>" "v,<round_constraint>,0")))]
+  "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
+  "@
+   vfnmadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
+   vfnmadd213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>}
+   vfnmadd231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}"
+  [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f")
+   (set_attr "type" "ssemuladd")
+   (set_attr "mode" "<MODE>")])
+
 (define_insn "avx512f_fnmadd_<mode>_mask<round_name>"
   [(set (match_operand:VF_512 0 "register_operand" "=v,v")
        (vec_merge:VF_512
@@ -2925,13 +2976,13 @@
    (set_attr "mode" "<MODE>")])
 
 (define_insn "<sd_mask_codefor>fma_fnmsub_<mode><sd_maskz_name><round_name>"
-  [(set (match_operand:FMAMODE 0 "register_operand" "=v,v,v,x,x")
-       (fma:FMAMODE
-         (neg:FMAMODE
-           (match_operand:FMAMODE 1 "<round_nimm_predicate>" "%0,0,v,x,x"))
-         (match_operand:FMAMODE   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
-         (neg:FMAMODE
-           (match_operand:FMAMODE 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))))]
+  [(set (match_operand:FMAMODE_NOVF512 0 "register_operand" "=v,v,v,x,x")
+       (fma:FMAMODE_NOVF512
+         (neg:FMAMODE_NOVF512
+           (match_operand:FMAMODE_NOVF512 1 "<round_nimm_predicate>" "%0,0,v,x,x"))
+         (match_operand:FMAMODE_NOVF512   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
+         (neg:FMAMODE_NOVF512
+           (match_operand:FMAMODE_NOVF512 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))))]
   "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
   "@
    vfnmsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
@@ -2943,6 +2994,23 @@
    (set_attr "type" "ssemuladd")
    (set_attr "mode" "<MODE>")])
 
+(define_insn "<sd_mask_codefor>fma_fnmsub_<mode><sd_maskz_name><round_name>"
+  [(set (match_operand:VF_512 0 "register_operand" "=v,v,v")
+       (fma:VF_512
+         (neg:VF_512
+           (match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v"))
+         (match_operand:VF_512   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>")
+         (neg:VF_512
+           (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0"))))]
+  "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
+  "@
+   vfnmsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
+   vfnmsub213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>}
+   vfnmsub231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}"
+  [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f")
+   (set_attr "type" "ssemuladd")
+   (set_attr "mode" "<MODE>")])
+
 (define_insn "avx512f_fnmsub_<mode>_mask<round_name>"
   [(set (match_operand:VF_512 0 "register_operand" "=v,v")
        (vec_merge:VF_512
@@ -3014,11 +3082,11 @@
 })
 
 (define_insn "<sd_mask_codefor>fma_fmaddsub_<mode><sd_maskz_name><round_name>"
-  [(set (match_operand:VF 0 "register_operand" "=v,v,v,x,x")
-       (unspec:VF
-         [(match_operand:VF 1 "<round_nimm_predicate>" "%0,0,v,x,x")
-          (match_operand:VF 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
-          (match_operand:VF 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")]
+  [(set (match_operand:VF_128_256 0 "register_operand" "=v,v,v,x,x")
+       (unspec:VF_128_256
+         [(match_operand:VF_128_256 1 "<round_nimm_predicate>" "%0,0,v,x,x")
+          (match_operand:VF_128_256 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
+          (match_operand:VF_128_256 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")]
          UNSPEC_FMADDSUB))]
   "(TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F) && <sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
   "@
@@ -3031,6 +3099,22 @@
    (set_attr "type" "ssemuladd")
    (set_attr "mode" "<MODE>")])
 
+(define_insn "<sd_mask_codefor>fma_fmaddsub_<mode><sd_maskz_name><round_name>"
+  [(set (match_operand:VF_512 0 "register_operand" "=v,v,v")
+       (unspec:VF_512
+         [(match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v")
+          (match_operand:VF_512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>")
+          (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0")]
+         UNSPEC_FMADDSUB))]
+  "TARGET_AVX512F && <sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
+  "@
+   vfmaddsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
+   vfmaddsub213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>}
+   vfmaddsub231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}"
+  [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f")
+   (set_attr "type" "ssemuladd")
+   (set_attr "mode" "<MODE>")])
+
 (define_insn "avx512f_fmaddsub_<mode>_mask<round_name>"
   [(set (match_operand:VF_512 0 "register_operand" "=v,v")
        (vec_merge:VF_512
@@ -3066,12 +3150,12 @@
    (set_attr "mode" "<MODE>")])
 
 (define_insn "<sd_mask_codefor>fma_fmsubadd_<mode><sd_maskz_name><round_name>"
-  [(set (match_operand:VF 0 "register_operand" "=v,v,v,x,x")
-       (unspec:VF
-         [(match_operand:VF   1 "<round_nimm_predicate>" "%0,0,v,x,x")
-          (match_operand:VF   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
-          (neg:VF
-            (match_operand:VF 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))]
+  [(set (match_operand:VF_128_256 0 "register_operand" "=v,v,v,x,x")
+       (unspec:VF_128_256
+         [(match_operand:VF_128_256   1 "<round_nimm_predicate>" "%0,0,v,x,x")
+          (match_operand:VF_128_256   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m")
+          (neg:VF_128_256
+            (match_operand:VF_128_256 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))]
          UNSPEC_FMADDSUB))]
   "(TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F) && <sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
   "@
@@ -3084,6 +3168,23 @@
    (set_attr "type" "ssemuladd")
    (set_attr "mode" "<MODE>")])
 
+(define_insn "<sd_mask_codefor>fma_fmsubadd_<mode><sd_maskz_name><round_name>"
+  [(set (match_operand:VF_512 0 "register_operand" "=v,v,v")
+       (unspec:VF_512
+         [(match_operand:VF_512   1 "<round_nimm_predicate>" "%0,0,v")
+          (match_operand:VF_512   2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>")
+          (neg:VF_512
+            (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0"))]
+         UNSPEC_FMADDSUB))]
+  "TARGET_AVX512F && <sd_mask_mode512bit_condition> && <round_mode512bit_condition>"
+  "@
+   vfmsubadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>}
+   vfmsubadd213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>}
+   vfmsubadd231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}"
+  [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f")
+   (set_attr "type" "ssemuladd")
+   (set_attr "mode" "<MODE>")])
+
 (define_insn "avx512f_fmsubadd_<mode>_mask<round_name>"
   [(set (match_operand:VF_512 0 "register_operand" "=v,v")
        (vec_merge:VF_512