The body region must contain exactly one block that terminates with
"loop.yield". Calling ForOp::build will create such a region and insert
- the terminator implicitly if none is defined, so will the parsing even
+ the terminator implicitly if none is defined, so will the parsing even
in cases when it is absent from the custom format. For example:
```mlir
```
"loop.for" can also operate on loop-carried variables and returns the final values
- after loop termination. The initial values of the variables are passed as additional SSA
+ after loop termination. The initial values of the variables are passed as additional SSA
operands to the "loop.for" following the 3 loop control SSA values mentioned above
- (lower bound, upper bound and step). The operation region has equivalent arguments
+ (lower bound, upper bound and step). The operation region has equivalent arguments
for each variable representing the value of the variable at the current iteration.
-
- The region must terminate with a "loop.yield" that passes all the current iteration
+
+ The region must terminate with a "loop.yield" that passes all the current iteration
variables to the next iteration, or to the "loop.for" result, if at the last iteration.
- "loop.for" results hold the final values after the last iteration.
-
+ "loop.for" results hold the final values after the last iteration.
+
For example, to sum-reduce a memref:
-
+
```mlir
func @reduce(%buffer: memref<1024xf32>, %lb: index, %ub: index, %step: index) -> (f32) {
// Initial sum set to 0.
loop.yield %sum_next : f32
}
return %sum : f32
- }
+ }
```
- If the "loop.for" defines any values, a yield must be explicitly present.
- The number and types of the "loop.for" results must match the initial values
+ If the "loop.for" defines any values, a yield must be explicitly present.
+ The number and types of the "loop.for" results must match the initial values
in the "iter_args" binding and the yield operands.
-
- Another example with a nested "loop.if" (see "loop.if" for details)
+
+ Another example with a nested "loop.if" (see "loop.if" for details)
to perform conditional reduction:
```mlir
Index:$upperBound,
Index:$step,
Variadic<AnyType>:$initArgs);
- let results = (outs Variadic<AnyType>:$results);
+ let results = (outs Variadic<AnyType>:$results);
let regions = (region SizedRegion<1>:$region);
let skipDefaultBuilders = 1;
void setLowerBound(Value bound) { getOperation()->setOperand(0, bound); }
void setUpperBound(Value bound) { getOperation()->setOperand(1, bound); }
void setStep(Value step) { getOperation()->setOperand(2, step); }
-
+
/// Number of region arguments for loop-carried values
unsigned getNumRegionIterArgs() {
- return getBody()->getNumArguments() - 1;
+ return getBody()->getNumArguments() - 1;
}
/// Number of operands controlling the loop: lb, ub, step
unsigned getNumControlOperands() { return 3; }
/// Does the operation hold operands for loop-carried values
- bool hasIterOperands() {
+ bool hasIterOperands() {
return getOperation()->getNumOperands() > getNumControlOperands();
}
/// Get Number of loop-carried values
```
"loop.if" may also return results that are defined in its regions. The values
- defined are determined by which execution path is taken.
+ defined are determined by which execution path is taken.
For example:
```mlir
%x, %y = loop.if %b -> (f32, f32) {
%y_true = ...
loop.yield %x_true, %y_true : f32, f32
} else {
- %x_false = ...
- %y_false = ...
+ %x_false = ...
+ %y_false = ...
loop.yield %x_false, %y_false : f32, f32
}
```
defines no values, the "loop.yield" can be left out, and will be
inserted implicitly. Otherwise, it must be explicit.
Also, if "loop.if" defines one or more values, the 'else' block cannot
- be omitted.
+ be omitted.
For example:
```mlir
}
def ParallelOp : Loop_Op<"parallel",
- [SameVariadicOperandSize, SingleBlockImplicitTerminator<"YieldOp">]> {
+ [AttrSizedOperandSegments, SingleBlockImplicitTerminator<"YieldOp">]> {
let summary = "parallel for operation";
let description = [{
- The "loop.parallel" operation represents a loop nest taking 3 groups of SSA
- values as operands that represent the lower bounds, upper bounds and steps,
- respectively. The operation defines a variadic number of SSA values for its
- induction variables. It has one region capturing the loop body. The
- induction variables are represented as an argument of this region. These SSA
- values always have type index, which is the size of the machine word. The
- steps are values of type index, required to be positive.
- The lower and upper bounds specify a half-open range: the range includes the
- lower bound but does not include the upper bound.
+ The "loop.parallel" operation represents a loop nest taking 4 groups of SSA
+ values as operands that represent the lower bounds, upper bounds, steps and
+ initial values, respectively. The operation defines a variadic number of
+ SSA values for its induction variables. It has one region capturing the
+ loop body. The induction variables are represented as an argument of this
+ region. These SSA values always have type index, which is the size of the
+ machine word. The steps are values of type index, required to be positive.
+ The lower and upper bounds specify a half-open range: the range includes
+ the lower bound but does not include the upper bound. The initial values
+ have the same types as results of "loop.parallel". If there are no results,
+ the keyword `init` can be omitted.
Semantically we require that the iteration space can be iterated in any
order, and the loop body can be executed in parallel. If there are data
The parallel loop operation supports reduction of values produced by
individual iterations into a single result. This is modeled using the
loop.reduce operation (see loop.reduce for details). Each result of a
- loop.parallel operation is associated with a reduce operation that is an
- immediate child. Reduces are matched to result values in order of their
- appearance in the body. Consequently, we require that the body region has
- the same number of results as it has reduce operations.
+ loop.parallel operation is associated with an initial value operand and
+ reduce operation that is an immediate child. Reductions are matched to result
+ and initial values in order of their appearance in the body. Consequently,
+ we require that the body region has the same number of results and initial
+ values as it has reduce operations.
The body region must contain exactly one block that terminates with
"loop.yield" without operands. Parsing ParallelOp will create such a region
let arguments = (ins Variadic<Index>:$lowerBound,
Variadic<Index>:$upperBound,
- Variadic<Index>:$step);
+ Variadic<Index>:$step,
+ Variadic<AnyType>:$initVals);
let results = (outs Variadic<AnyType>:$results);
let regions = (region SizedRegion<1>:$region);
let builders = [
OpBuilder<"Builder *builder, OperationState &result, "
"ValueRange lowerBounds, ValueRange upperBounds, "
- "ValueRange steps">,
- OpBuilder<"Builder *builder, OperationState &result, "
- "ValueRange lowerBounds, ValueRange upperBounds, "
- "ValueRange steps, ArrayRef<Type> resultTypes">
+ "ValueRange steps, ValueRange initVals = {}">,
];
let extraClassDeclaration = [{
return getBody()->getNumArguments();
}
iterator_range<Block::args_iterator> getInductionVars() {
- return {getBody()->args_begin(), getBody()->args_end()};
+ return getBody()->getArguments();
}
unsigned getNumLoops() { return step().size(); }
+ unsigned getNumReductions() { return initVals().size(); }
}];
}
let description = [{
"loop.yield" yields an SSA value from a loop dialect op region and
terminates the regions. The semantics of how the values are yielded
- is defined by the parent operation.
+ is defined by the parent operation.
If "loop.yield" has any operands, the operands must match the parent
- operation's results.
- If the parent operation defines no values, then the "loop.yield" may be
+ operation's results.
+ If the parent operation defines no values, then the "loop.yield" may be
left out in the custom syntax and the builders will insert one implicitly.
Otherwise, it has to be present in the syntax to indicate which values
- are yielded.
+ are yielded.
}];
let arguments = (ins Variadic<AnyType>:$results);
//===----------------------------------------------------------------------===//
void ParallelOp::build(Builder *builder, OperationState &result, ValueRange lbs,
- ValueRange ubs, ValueRange steps) {
+ ValueRange ubs, ValueRange steps, ValueRange initVals) {
result.addOperands(lbs);
result.addOperands(ubs);
result.addOperands(steps);
+ result.addOperands(initVals);
+ result.addAttribute(
+ ParallelOp::getOperandSegmentSizeAttr(),
+ builder->getI32VectorAttr({static_cast<int32_t>(lbs.size()),
+ static_cast<int32_t>(ubs.size()),
+ static_cast<int32_t>(steps.size()),
+ static_cast<int32_t>(initVals.size())}));
Region *bodyRegion = result.addRegion();
ParallelOp::ensureTerminator(*bodyRegion, *builder, result.location);
for (size_t i = 0, e = steps.size(); i < e; ++i)
bodyRegion->front().addArgument(builder->getIndexType());
-}
-
-void ParallelOp::build(Builder *builder, OperationState &result, ValueRange lbs,
- ValueRange ubs, ValueRange steps,
- ArrayRef<Type> resultTypes) {
- result.addTypes(resultTypes);
- build(builder, result, lbs, ubs, steps);
+ for (Value init : initVals)
+ result.addTypes(init.getType());
}
static LogicalResult verify(ParallelOp op) {
// number of tuple elements in step.
Block *body = op.getBody();
if (body->getNumArguments() != stepValues.size())
- return op.emitOpError(
- "expects the same number of induction variables as bound and step "
- "values");
+ return op.emitOpError()
+ << "expects the same number of induction variables: "
+ << body->getNumArguments()
+ << " as bound and step values: " << stepValues.size();
for (auto arg : body->getArguments())
if (!arg.getType().isIndex())
return op.emitOpError(
// Check that the number of results is the same as the number of ReduceOps.
SmallVector<ReduceOp, 4> reductions(body->getOps<ReduceOp>());
- if (op.results().size() != reductions.size())
- return op.emitOpError(
- "expects number of results to be the same as number of reductions");
+ auto resultsSize = op.results().size();
+ auto reductionsSize = reductions.size();
+ auto initValsSize = op.initVals().size();
+ if (resultsSize != reductionsSize)
+ return op.emitOpError()
+ << "expects number of results: " << resultsSize
+ << " to be the same as number of reductions: " << reductionsSize;
+ if (resultsSize != initValsSize)
+ return op.emitOpError()
+ << "expects number of results: " << resultsSize
+ << " to be the same as number of initial values: " << initValsSize;
// Check that the types of the results and reductions are the same.
for (auto resultAndReduce : llvm::zip(op.results(), reductions)) {
auto reduceType = reduceOp.operand().getType();
if (resultType != reduceType)
return reduceOp.emitOpError()
- << "expects type of reduce to be the same as result type: "
- << resultType;
+ << "expects type of reduce: " << reduceType
+ << " to be the same as result type: " << resultType;
}
return success();
}
parser.resolveOperands(steps, builder.getIndexType(), result.operands))
return failure();
+ // Parse step value.
+ SmallVector<OpAsmParser::OperandType, 4> initVals;
+ if (succeeded(parser.parseOptionalKeyword("init"))) {
+ if (parser.parseOperandList(initVals, -1, OpAsmParser::Delimiter::Paren))
+ return failure();
+ }
+
// Now parse the body.
Region *body = result.addRegion();
SmallVector<Type, 4> types(ivs.size(), builder.getIndexType());
if (parser.parseRegion(*body, ivs, types))
return failure();
+ // Set `operand_segment_sizes` attribute.
+ result.addAttribute(
+ ParallelOp::getOperandSegmentSizeAttr(),
+ builder.getI32VectorAttr({static_cast<int32_t>(lower.size()),
+ static_cast<int32_t>(upper.size()),
+ static_cast<int32_t>(steps.size()),
+ static_cast<int32_t>(initVals.size())}));
+
// Parse attributes and optional results (in case there is a reduce).
if (parser.parseOptionalAttrDict(result.attributes) ||
parser.parseOptionalColonTypeList(result.types))
return failure();
+ if (!initVals.empty())
+ parser.resolveOperands(initVals, result.types, parser.getNameLoc(),
+ result.operands);
// Add a terminator if none was parsed.
ForOp::ensureTerminator(*body, builder, result.location);
p << op.getOperationName() << " (" << op.getBody()->getArguments() << ") = ("
<< op.lowerBound() << ") to (" << op.upperBound() << ") step (" << op.step()
<< ")";
+ if (!op.initVals().empty())
+ p << " init (" << op.initVals() << ")";
p.printRegion(op.region(), /*printEntryBlockArgs=*/false);
- p.printOptionalAttrDict(op.getAttrs());
+ p.printOptionalAttrDict(
+ op.getAttrs(), /*elidedAttrs=*/ParallelOp::getOperandSegmentSizeAttr());
if (!op.results().empty())
p << " : " << op.getResultTypes();
}
"loop.parallel"(%arg0, %arg1, %arg2) ({
^bb0(%i0: f32):
loop.yield
- }): (index, index, index) -> ()
+ }) {operand_segment_sizes = dense<[1, 1, 1, 0]>: vector<4xi32>}: (index, index, index) -> ()
return
}
func @parallel_body_wrong_number_of_arguments(
%arg0: index, %arg1: index, %arg2: index) {
- // expected-error@+1 {{'loop.parallel' op expects the same number of induction variables as bound and step values}}
+ // expected-error@+1 {{'loop.parallel' op expects the same number of induction variables: 2 as bound and step values: 1}}
"loop.parallel"(%arg0, %arg1, %arg2) ({
^bb0(%i0: index, %i1: index):
loop.yield
- }): (index, index, index) -> ()
+ }) {operand_segment_sizes = dense<[1, 1, 1, 0]>: vector<4xi32>}: (index, index, index) -> ()
return
}
func @parallel_fewer_results_than_reduces(
%arg0 : index, %arg1: index, %arg2: index) {
- // expected-error@+1 {{expects number of results to be the same as number of reductions}}
+ // expected-error@+1 {{expects number of results: 0 to be the same as number of reductions: 1}}
loop.parallel (%i0) = (%arg0) to (%arg1) step (%arg2) {
%c0 = constant 1.0 : f32
loop.reduce(%c0) {
func @parallel_more_results_than_reduces(
%arg0 : index, %arg1 : index, %arg2 : index) {
- // expected-error@+1 {{expects number of results to be the same as number of reductions}}
- %res = loop.parallel (%i0) = (%arg0) to (%arg1) step (%arg2) {
+ // expected-error@+2 {{expects number of results: 1 to be the same as number of reductions: 0}}
+ %zero = constant 1.0 : f32
+ %res = loop.parallel (%i0) = (%arg0) to (%arg1) step (%arg2) init (%zero) {
} : f32
return
// -----
-func @parallel_different_types_of_results_and_reduces(
+func @parallel_more_results_than_initial_values(
%arg0 : index, %arg1: index, %arg2: index) {
+ // expected-error@+1 {{expects number of results: 1 to be the same as number of initial values: 0}}
%res = loop.parallel (%i0) = (%arg0) to (%arg1) step (%arg2) {
- // expected-error@+1 {{expects type of reduce to be the same as result type: 'f32'}}
+ loop.reduce(%arg0) {
+ ^bb0(%lhs: index, %rhs: index):
+ loop.reduce.return %lhs : index
+ } : index
+ } : f32
+ return
+}
+
+// -----
+
+func @parallel_different_types_of_results_and_reduces(
+ %arg0 : index, %arg1: index, %arg2: index) {
+ %zero = constant 0.0 : f32
+ %res = loop.parallel (%i0) = (%arg0) to (%arg1) step (%arg2) init (%zero) {
+ // expected-error@+1 {{expects type of reduce: 'index' to be the same as result type: 'f32'}}
loop.reduce(%arg0) {
^bb0(%lhs: index, %rhs: index):
loop.reduce.return %lhs : index
// -----
func @reduce_empty_block(%arg0 : index, %arg1 : f32) {
- %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+ %zero = constant 0.0 : f32
+ %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) init (%zero) {
// expected-error@+1 {{the block inside reduce should not be empty}}
loop.reduce(%arg1) {
^bb0(%lhs : f32, %rhs : f32):
// -----
func @reduce_too_many_args(%arg0 : index, %arg1 : f32) {
- %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+ %zero = constant 0.0 : f32
+ %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) init (%zero) {
// expected-error@+1 {{expects two arguments to reduce block of type 'f32'}}
loop.reduce(%arg1) {
^bb0(%lhs : f32, %rhs : f32, %other : f32):
// -----
func @reduce_wrong_args(%arg0 : index, %arg1 : f32) {
- %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+ %zero = constant 0.0 : f32
+ %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) init (%zero) {
// expected-error@+1 {{expects two arguments to reduce block of type 'f32'}}
loop.reduce(%arg1) {
^bb0(%lhs : f32, %rhs : i32):
// -----
func @reduce_wrong_terminator(%arg0 : index, %arg1 : f32) {
- %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+ %zero = constant 0.0 : f32
+ %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) init (%zero) {
// expected-error@+1 {{the block inside reduce should be terminated with a 'loop.reduce.return' op}}
loop.reduce(%arg1) {
^bb0(%lhs : f32, %rhs : f32):
// -----
func @reduceReturn_wrong_type(%arg0 : index, %arg1: f32) {
- %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+ %zero = constant 0.0 : f32
+ %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) init (%zero) {
loop.reduce(%arg1) {
^bb0(%lhs : f32, %rhs : f32):
%c0 = constant 1 : index
loop.yield %c0 : f32
}
return
-}
\ No newline at end of file
+}
-// RUN: mlir-opt %s | FileCheck %s
+// RUN: mlir-opt %s | FileCheck %s --dump-input-on-failure
// Verify the printed output can be parsed.
-// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s --dump-input-on-failure
// Verify the generic form can be parsed.
-// RUN: mlir-opt -mlir-print-op-generic %s | mlir-opt | FileCheck %s
+// RUN: mlir-opt -mlir-print-op-generic %s | mlir-opt | FileCheck %s --dump-input-on-failure
func @std_for(%arg0 : index, %arg1 : index, %arg2 : index) {
loop.for %i0 = %arg0 to %arg1 step %arg2 {
%min = select %min_cmp, %i0, %i1 : index
%max_cmp = cmpi "sge", %i0, %i1 : index
%max = select %max_cmp, %i0, %i1 : index
- %red = loop.parallel (%i2) = (%min) to (%max) step (%i1) {
- %zero = constant 0.0 : f32
- loop.reduce(%zero) {
+ %zero = constant 0.0 : f32
+ %red = loop.parallel (%i2) = (%min) to (%max) step (%i1) init (%zero) {
+ %one = constant 1.0 : f32
+ loop.reduce(%one) {
^bb0(%lhs : f32, %rhs: f32):
%res = addf %lhs, %rhs : f32
loop.reduce.return %res : f32
// CHECK-NEXT: %[[MIN:.*]] = select %[[MIN_CMP]], %[[I0]], %[[I1]] : index
// CHECK-NEXT: %[[MAX_CMP:.*]] = cmpi "sge", %[[I0]], %[[I1]] : index
// CHECK-NEXT: %[[MAX:.*]] = select %[[MAX_CMP]], %[[I0]], %[[I1]] : index
-// CHECK-NEXT: loop.parallel (%{{.*}}) = (%[[MIN]]) to (%[[MAX]]) step (%[[I1]]) {
-// CHECK-NEXT: %[[ZERO:.*]] = constant 0.000000e+00 : f32
-// CHECK-NEXT: loop.reduce(%[[ZERO]]) {
+// CHECK-NEXT: %[[ZERO:.*]] = constant 0.000000e+00 : f32
+// CHECK-NEXT: loop.parallel (%{{.*}}) = (%[[MIN]]) to (%[[MAX]])
+// CHECK-SAME: step (%[[I1]]) init (%[[ZERO]]) {
+// CHECK-NEXT: %[[ONE:.*]] = constant 1.000000e+00 : f32
+// CHECK-NEXT: loop.reduce(%[[ONE]]) {
// CHECK-NEXT: ^bb0(%[[LHS:.*]]: f32, %[[RHS:.*]]: f32):
// CHECK-NEXT: %[[RES:.*]] = addf %[[LHS]], %[[RHS]] : f32
// CHECK-NEXT: loop.reduce.return %[[RES]] : f32
projects / platform / upstream / llvm.git / commitdiff