Skip to main content

​
Overview

The Orchestration engine is a horizontally scalable service. Multiple engine worker processes can run on a single controller node or across multiple controller nodes, sharing work through a message queue. In convergence mode, the engine distributes individual resource operations across all available workers, enabling parallel provisioning of independent resources within a single stack.
Administrator Access Required — This operation requires the admin role. Contact your Xloud administrator if you do not have sufficient permissions.

​
Engine Worker Architecture

​
Worker Configuration

​
Engine Workers Per Node

The heat_engine_workers setting controls how many engine worker processes run on each controller node. Each worker is an independent process that picks up tasks from the message queue.
In XDeploy, navigate to Configuration → Services → Orchestration and adjust the Engine Workers slider. A common starting point is 2–4 workers per CPU core available on the controller node.

​
Worker Sizing Guidelines

Controller vCPU CountRecommended Engine WorkersRecommended API Workers
4 vCPUs22
8 vCPUs44
16 vCPUs84
32+ vCPUs168
Engine workers are I/O bound (waiting on cloud service API calls), not CPU bound. You can safely set heat_engine_workers higher than the physical CPU count. Monitor the message queue depth in XIMP to identify bottlenecks.

​
Convergence Mode

Convergence mode enables the engine to process independent resources in a stack concurrently across all available workers. This significantly reduces total stack creation time for large templates.
ModeBehaviorBest For
Convergence (default)Resources are processed in parallel by multiple workersLarge stacks (50+ resources), independent resource graphs
Non-convergenceResources processed sequentially by a single workerSimple stacks, environments where strict ordering is required
Enable convergence in globals
heat_convergence_engine: true   # Default — recommended for production
Convergence mode requires the database to be accessible from all engine workers simultaneously. Ensure your database connection pool is sized appropriately: heat_db_max_pool_size should be at least heat_engine_workers * 2.

​
Performance Tuning

SettingDefaultDescription
heat_engine_workers4Engine worker processes per controller node
heat_api_workers4API worker processes per controller node
heat_db_max_pool_size10Max database connections per worker
heat_db_max_overflow20Max overflow connections above the pool
heat_rpc_response_timeout120Seconds before an RPC call is considered failed
heat_max_stacks_per_tenant100Per-project stack limit
heat_max_resources_per_stack1000Per-stack resource limit

​
Monitoring Engine Health

List active engine services
openstack orchestration service list
Expected output — all services show status: up: 
+----------+-----------+------+--------+-------------------+-----+
| Hostname | Binary    | Port | Status | Updated At        | ... |
+----------+-----------+------+--------+-------------------+-----+
| ctrl-01  | heat-eng  | 0    | up     | 2026-03-18T10:00Z | ... |
| ctrl-01  | heat-api  | 8004 | up     | 2026-03-18T10:00Z | ... |
+----------+-----------+------+--------+-------------------+-----+
Set up an alert in XIMP for orchestration_engine_up == 0 to detect engine worker failures before they affect user deployments.

​
Next Steps

Configuration

Full configuration reference for the Orchestration service

Security

Trust-based authorization and policy configuration

Architecture

Engine internals and dependency resolution design

Admin Troubleshooting

Resolve engine worker failures and performance degradation