The documentation you're currently reading is for version 2.7.2. Click here to view documentation for the latest stable version.
Mistral is an OpenStack project that manages and executes workflows as a service. Mistral is automatically installed as a separate service named “mistral” along with EWC. A Mistral workflow can be defined as a EWC action in a Mistral workbook using the v2 Workflow Language.
Expression languages such as YAQL are used for formatting variables and condition evaluations. Starting with EWC v2.2, Jinja2 is also supported where YAQL expressions are accepted. Both workbook and workflow definitions are supported.
On action execution, EWC writes the definition to Mistral and executes the workflow. A workflow can invoke other EWC actions natively as subtasks. EWC handles the translations and calls transparently in Mistral and actively polls Mistral for execution results. EWC actions in the workflow can be traced back to the original parent action that invoked the workflow.
Essential Mistral Links:
- Mistral workflow definition language, aka v2 WorkFlow Language
- YAQL documentation and YAQL online evaluator
- Jinja2 template engine documentation and Jinja2 online evaluator
Workflow examples in this documentation use YAQL expressions unless otherwise stated.
Similarly to ActionChains, Mistral workflows have an action metadata file in
/opt/stackstorm/packs/<mypack>/actions, and the workflow definition itself in
Let’s start with a very basic workflow that calls a EWC action and notifies EWC when the workflow is done. The files used in this example are also located under /usr/share/doc/st2/examples if EWC is already installed (see also deploy examples).
The first task is named
run-cmd. It executes a shell command on the server where EWC is
installed. A task can reference any registered EWC action directly. In this example, the
run-cmd task calls
core.local and passing the cmd as input.
core.local is an action
that comes installed with EWC. When the workflow is invoked, EWC will translate the workflow
definition appropriately before sending it to Mistral. Let’s save this as
/opt/stackstorm/packs/examples/actions/workflows/mistral-basic.yaml on our EWC server.
version: '2.0' examples.mistral-basic: description: A basic workflow that runs an arbitrary linux command. type: direct input: - cmd - timeout output: stdout: <% $.stdout %> tasks: task1: action: core.local cmd=<% $.cmd %> timeout=<% $.timeout %> publish: stdout: <% task(task1).result.stdout %> stderr: <% task(task1).result.stderr %>
This is the corresponding EWC action metadata for the example above. The EWC pack for this
workflow action is named “examples”. Note that the workflow is named fully qualified as
<pack>.<action in the definition above. The EWC action runner is
mistral-v2. The entry
point for the EWC action refers to the YAML file of the workflow definition. Let’s save this
--- description: Run a local linux command enabled: true runner_type: mistral-v2 entry_point: workflows/mistral-basic.yaml name: mistral-basic pack: examples parameters: cmd: required: true type: string timeout: type: integer default: 60
The following table list optional parameters that can be defined in the workflow action. In the example, these optional parameters are set to immutable. It is good practice to set them to immutable even if they are empty since these are Mistral-specific parameters for the workflow author.
|workflow||If definition is a workbook containing many workflows, this specifies the main workflow to execute.|
|task||If the type of workflow is “reverse”, this specifies the task to invoke.|
|context||A dictionary containing additional workflow start up parameters.|
st2 action create /opt/stackstorm/packs/examples/actions/mistral-basic.yaml to
create this workflow action. This will register the workflow as
EWC. To execute the workflow, run
st2 run examples.mistral-basic cmd=date -a where
tells the command to return and not wait for the workflow to complete.
If the workflow completed successfully, both the workflow
examples.mistral-basic and the
core.local should have a
succeeded status in the EWC action execution list. By
st2 execution list only returns top level executions. This means subtasks are not
+--------------------------+--------------+--------------+-----------+-----------------+---------------+ | id | action.ref | context.user | status | start_timestamp | end_timestamp | +--------------------------+--------------+--------------+-----------+-----------------+---------------+ | 54ee54c61e2e24152b769a47 | examples | stanley | succeeded | Wed, 25 Feb | Wed, 25 Feb | | | .mistral- | | | 2015 23:03:34 | 2015 23:03:34 | | | basic | | | UTC | UTC | +--------------------------+--------------+--------------+-----------+-----------------+---------------+
To display subtasks, run
st2 execution get <execution-id> --show-tasks:
+--------------------------+------------+--------------+-----------+------------------------------+------------------------------+ | id | action.ref | context.user | status | start_timestamp | end_timestamp | +--------------------------+------------+--------------+-----------+------------------------------+------------------------------+ | 54ee54c91e2e24152b769a49 | core.local | stanley | succeeded | Wed, 25 Feb 2015 23:03:37 | Wed, 25 Feb 2015 23:03:37 | | | | | | UTC | UTC | +--------------------------+------------+--------------+-----------+------------------------------+------------------------------+
The following is a simple extension of the previous workflow definition. In this example, we have
a second task named “task2”. It might be natural to think that “task2” will be executed after
“task1”, i.e, in sequential order. However, when no tasks attributes like
on-error are defined, tasks are run in parallel. This is possible with
Mistral because it provides a join flow control which allows us to synchronize multiple parallel
workflow branches and aggregate their data.
version: '2.0' examples.mistral-basic-two-tasks-with-notifications: description: A basic workflow that runs two Linux commands (one in each task). type: direct output: stdout: <% $.stdout %> tasks: task1: action: core.local cmd="echo task1" publish: stdout: <% task(task1).result.stdout %> stderr: <% task(task1).result.stderr %> task2: action: core.local cmd="echo task2" publish: stdout: <% task(task2).result.stdout %> stderr: <% task(task2).result.stderr %>
Pausing and Resuming Workflow Execution¶
An execution of a Mistral workflow can be paused by running
st2 execution pause <execution-id>.
An execution must be in a running state in order for pause to be successful. The execution will
initially go into a
pausing state, and will go into a
paused state when no more tasks are
in an active state such as
canceling. When a workflow execution
is paused, it can be resumed by running
st2 execution resume <execution-id>.
resume operation will cascade down to subworkflows, whether it’s another
workflow defined in a workbook or it’s another EWC action that is a Mistral workflow or Action
Chain. If the
pause operation is performed from a subworkflow or subchain, then the
will cascade up to the parent workflow or parent chain. However, if the
resume operation is
performed from a subworkflow or subchain, the
resume will not cascade up to the parent workflow
or parent chain. This allows users to resume and troubleshoot branches individually.
Canceling Workflow Execution¶
An execution of a Mistral workflow can be cancelled by running
st2 execution cancel <execution-id>. Workflow tasks that are still running will not be
canceled and will run to completion. No new tasks for the workflow will be scheduled.
Re-running Workflow Execution¶
An execution of a Mistral workflow can be re-run on error. The execution either can be re-run from
the beginning or from the task(s) that failed. The latter is useful for long running workflows with
temporary service or network outages. Re-running the workflow execution from the beginning is
exactly like re-running any EWC execution with the command
st2 execution re-run <execution-id>.
The re-run is a completely separate execution with a new execution ID in both EWC and Mistral.
Re-running the workflow from where it errored is slightly different. To retain context, the
original workflow execution is reused in Mistral but a new EWC execution will be created to stay
consistent in EWC. The re-run command has a new
--tasks option that takes a list of task
names to re-run.
For example, given a workflow that fails at task3 and task4 on separate parallel branches, the
st2 execution re-run <execution-id> --tasks task3 task4 will resume the Mistral
workflow execution and re-run both task3 and task4 using original inputs. Both the workflow and
task execution in Mistral have to be in an
errored state for re-run.
If using a Mistral workbook, tasks of subworkflows can also be re-run. For example, if the main
workflow has a task1 that calls subflow1, then to re-run subtask1 of subflow1, the syntax for the
st2 execution re-run command would be
st2 execution re-run <execution-id> --tasks task1.subtask1.
If the task to re-run is a “with-items” task, there is an option to re-run only failed iterations.
For example, task1 is a with-items task with 5 items. Let’s say 2 of the items failed. By
st2 execution re-run --tasks task1 task2 --no-reset task1 option, task1 will
only re-run the 2 items that failed. If the
--no-reset option is not provided, then all 5
items will be re-run.
Re-running workflow execution from the task(s) that failed is currently an experimental feature and subject to bug(s) and change(s). Please also note that re-running a subtask nested in another EWC action is not currently supported.
Publishing Variables in Mistral Workflows¶
A Mistral task can publish results from a task as variables that can be consumed in other tasks:
tasks: get_hostname: action: core.local input: cmd: "hostname" publish: hostname: <% task(get_hostname).result.stdout %>
In the above example,
get_hostname is a
core.local action which runs the command
core.local action produces output consisting of the fields
We just want to publish the variable
stdout from it, for the rest of tasks to consume. To
reference the result of the task, use the
task function, which returns a dictionary containing
attributes for the task such as id, state, result, and additional info.
Another example is shown below:
tasks: create_new_node: action: rackspace.create_vm input: name: <% $.hostname %> flavor_id: <% $.vm_size_id %> image_id: <% $.vm_image_id %> key_material: <% $.ssh_pub_key %> metadata: asg: <% $.asg %> publish: ipv4_address: '<% task(create_new_node).result.result.public_ips %>' ipv6_address: '<% task(create_new_node).result.result.public_ips %>'
In the above example, the action
rackspace.create_vm is a Python action that produces a result
object. We just want to publish the IP addresses from the
public_ips list field from the
Please note that
result.result is not a typo. The Python action posts output to a key named
result for the st2 action execution and the Mistral task function puts the result of the Python
result of its output dictionary.
Such published variables are accessible as input parameters to other tasks in the workflow. An
example of using
ipv4_address from the above example in another task is shown below:
tasks: # ... <snap> setup_ipv4_dns: action: rackspace.create_dns_record wait-before: 1 # delay, in seconds input: name: '<% $.hostname %>.<% $.asg %>.<% $.domain %>' zone_id: <% $.dns_zone_id %> type: 'A' data: <% $.ipv4_address %> # .... </snap>
Stitching Together a More Complex Workflow¶
The following is a mock up of a more complex workflow. In this mock up running simple
sleep commands, the workflow demonstrates nested workflows, fork, and join:
version: "2.0" name: examples.mistral-workbook-complex description: A sample workflow that demonstrates nested workflows, forks, and join. workflows: main: type: direct input: - vm_name - cpu_cores - memory_mb output: vm_id: <% $.vm_id %> ip: <% $.ip %> tasks: register_dns: action: core.local input: cmd: "sleep 1; printf 'Registering <% $.vm_name %>...'" publish: ip: "10.1.23.99" status_message: "DNS for <% $.vm_name %> is registered." on-success: - configure_vm - notify create_vm: wait-before: 1 workflow: create_vm input: name: <% $.vm_name %> cpu_cores: <% $.cpu_cores %> memory_mb: <% $.memory_mb %> publish: vm_id: <% task(create_vm).result.vm_id %> status_message: "VM <% $.vm_name %> is created." on-success: - configure_vm - notify configure_vm: join: all workflow: configure_vm input: vm_id: <% $.vm_id %> ip: <% $.ip %> publish: status_message: "VM <% $.vm_name %> is reconfigured." on-success: - close_request - notify close_request: action: std.noop publish: status_message: "VM request is fulfilled." on-success: - notify notify: action: core.local input: cmd: "printf '<% $.status_message %>'" create_vm: type: direct input: - name - cpu_cores - memory_mb output: vm_id: <% $.vm_id %> tasks: create: action: core.local input: cmd: "printf 'vm1234'; sleep 5" publish: vm_id: <% task(create).result.stdout %> configure_vm: type: direct input: - vm_id - ip tasks: add_disks: action: core.local input: cmd: "sleep 1; printf 'disks created'" add_nics: action: core.local input: cmd: "sleep 1; printf 'nics created'" install_apps: action: core.local input: cmd: "sleep 1; printf 'apps installed'"
Since there are multiple workflows defined in this workbook, the workflow author has to specify which workflow to execute in the metadata:
--- description: Run a series of simulated actions. enabled: true entry_point: workflows/mistral-workbook-complex.yaml name: mistral-workbook-complex pack: examples parameters: cpu_cores: default: 1 type: integer memory_mb: default: 1024 type: integer vm_name: required: true type: string workflow: default: examples.mistral-workbook-complex.main immutable: true type: string runner_type: mistral-v2
To test out this workflow, save the metadata file to
and the workflow file to
st2 action create /opt/stackstorm/packs/examples/actions/mistral-workbook-complex.yaml to
create the action and run
st2 run examples.mistral-workbook-complex vm_name="vmtest1" -a to
The Mistral CLI includes tools for performing high-level sanity checks of Mistral workflow YAML files:
# Validate a workflow mistral workflow-validate /path/to/workflow.yaml # Validate a workbook mistral workbook-validate /path/to/workbook.yaml
These sanity checks simply provide a test against the Mistral DSL schema. They do NOT test YAQL or Jinja2 expressions.
There are more workflow examples under /usr/share/doc/st2/examples. These include error handling, repeat, and retries.
Check out this step-by-step tutorial on building a workflow in EWC https://stackstorm.com/2015/07/08/automating-with-mistral-workflow/
More details about Mistral can be found at https://docs.openstack.org/mistral/latest/.
Mistral Workflows Completion, Latency, and Performance¶
During the workflow runtime, EWC and Mistral handshake multiple times over HTTP. This applies when launching the workflow execution, completing a task, or completing the workflow.
Prior to v2.7, EWC queries Mistral to check on workflow execution status and the status of individual tasks via st2resultstracker. This mechanism has a number of configuration settings. See Troubleshooting Mistral Workflow Completion Latency section about how to fine-tune the Mistral workflows completion time opposed to CPU usage.
Since v2.7, the results tracking mechanism is replaced with a callback mechanism from Mistral. Instead of EWC querying Mistral at regular interval, Mistral is configured to callback EWC on task and workflow completion. With the callback mechanism, it is possible to trace the events sent to EWC.
# Identify the Mistral workflow execution ID which is different st2 execution get <st2-action-execution-id> -dj | grep workflow_execution_id # Grep the log entries from the Mistral log, typically at /var/log/mistral/mistral-server.log sudo tail -n 1000 /var/log/mistral/mistral-server.log | grep stackstorm_notifier | grep <mistral-wf-ex-id>
The returned list of log entries would look something similar to the following.
2018-03-28 22:40:05,811 140124959618704 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The workflow event WORKFLOW_LAUNCHED for 839925d9-02c7-47be-ad8e-ce0943749a7b will be published to st2. 2018-03-28 22:40:05,844 140124959618704 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The workflow event WORKFLOW_LAUNCHED for 839925d9-02c7-47be-ad8e-ce0943749a7b is published to st2. 2018-03-28 22:40:06,492 140124958584912 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The task event TASK_SUCCEEDED for c8731e6a-2464-4a59-bf46-501a80215298 will be processed for st2. 2018-03-28 22:40:06,492 140124958584912 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The task event TASK_SUCCEEDED for c8731e6a-2464-4a59-bf46-501a80215298 is processed for st2. 2018-03-28 22:40:07,195 140124956804432 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The workflow event WORKFLOW_SUCCEEDED for 839925d9-02c7-47be-ad8e-ce0943749a7b will be published to st2. 2018-03-28 22:40:07,371 140124956804432 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The workflow event WORKFLOW_SUCCEEDED for 839925d9-02c7-47be-ad8e-ce0943749a7b is published to st2.
The results tracking mechanism is still available for manual intervention and can be enabled on an individual workflow basis in case EWC or Mistral services is offline during a callback operation.
# Enable the results tracking for an individual workflow execution st2-track-result <st2-action-execution-id> --config-dir /etc/st2 # Disable the results tracking for an individual workflow execution st2-track-result <st2-action-execution-id> --config-dir /etc/st2 --delete