Operations

Operations is the command center for your cloud deployment. Every infrastructure action — bootstrap, prechecks, deploy, reconfigure, upgrade, stop, and destroy — runs through this tool. It wraps all deployment playbooks into a visual wizard with a live terminal, real-time progress tracking, and searchable task history.

Prerequisites

Bootstrap completed with all dependency groups passing
Hosts configured with SSH connectivity to all target nodes
Configuration saved with networking, storage, and service settings
Images loaded and available in the local registry

Deployment Wizard

The Operations wizard guides you through four steps to execute any deployment action. Each step validates input before allowing you to proceed.

Select Action

Select the deployment action to execute from the available categories: Deploy, Manage, Backend, or Advanced. Each action is described with its purpose and expected duration.

Select Target Hosts

Select which servers to target:

All Hosts — execute across the entire cluster
Specific Hosts — select individual servers via a checklist
Host Patterns — target groups such as all compute nodes or all storage nodes

For initial deployment, select All Hosts. For targeted maintenance or upgrades, select specific hosts to minimize disruption.

Select Services (Optional)

Narrow the scope to specific cloud services or deploy all enabled services. Selecting individual services is useful for targeted reconfiguration or troubleshooting without affecting the rest of the cluster.

Run Deployment

Review the action summary and click Run to execute. Destructive actions (Stop, Destroy) require explicit confirmation before execution begins.

Once a deployment action starts, interrupting it mid-execution can leave services in an inconsistent state. Allow the action to complete fully before taking further steps.

Available Actions

Deploy
Manage
Backend
Advanced

Actions for initial deployment and readiness validation.

Action	Purpose	Typical Duration
Bootstrap Servers	Prepare target servers — install Docker, configure networking, set up prerequisites	5-15 min
Prechecks	Validate that everything is ready — ports, services, configurations, and connectivity	2-5 min
Deploy	Deploy all enabled cloud services as Docker containers across the cluster	30-90 min
Deploy Bifrost	Deploy the bare-metal provisioning service for automated server enrollment	10-20 min
Deploy Servers	Provision bare-metal servers through the Bifrost enrollment pipeline	15-30 min
Post-Deploy	Create the admin user, default networks, base flavors, and initial project configuration	5-10 min

Always run Prechecks before Deploy. It catches configuration issues, port conflicts, and missing dependencies that would otherwise cause failures 30+ minutes into deployment.

Actions for ongoing lifecycle management after initial deployment.

Action	Purpose
Pull Images	Pre-pull container images to all nodes before an upgrade or redeployment
Reconfigure	Apply configuration changes to running services without a full redeployment
Upgrade	Rolling upgrade to a new release version with minimal downtime
Upgrade Bifrost	Upgrade the bare-metal provisioning service to a new release version
Nova Libvirt Cleanup	Clean up stale libvirt resources — orphaned domains, volumes, and network filters on compute nodes
Stop	Stop all containers across the cluster (causes full downtime, requires confirmation)
Destroy	Remove all containers, configurations, and data completely (requires confirmation)

Destroy is irreversible. It removes all containers, configuration files, and data from every target node. This action cannot be undone. Ensure you have verified backups before executing.

Database and message queue maintenance operations.

Action	Purpose
MariaDB Backup	Create a full database backup of the Galera cluster for disaster recovery
MariaDB Recovery	Recover a broken or corrupted database from a previous backup
RabbitMQ Reset	Reset message queue state when queues are stuck or consumers are stalled
RabbitMQ Upgrade	Upgrade the RabbitMQ version to match a new release

MariaDB Recovery overwrites the current database state with backup data. Any changes made after the backup was taken are lost.

Infrastructure-level and diagnostic operations.

Action	Purpose
Deploy Containers	Deploy only the container infrastructure layer without configuring services
Octavia Certificates	Generate or renew TLS certificates for the Xloud Load Balancer service
Generate Config	Generate all service configuration files without deploying — useful for pre-review
Validate Config	Check configuration files for syntax errors and missing required values
Gather Facts	Collect system information (hardware, network, OS) from all target hosts

Generate Config and Validate Config are non-destructive — they do not modify any running services. Use them freely for pre-deployment review.

Command Modifiers

Actions in the Operations wizard support optional modifiers that control execution behavior.

Modifier	Description	Applicable Actions
Dry-run mode	Simulate the action without making changes. Shows what tasks would execute and which hosts would be affected.	All deployment and management actions
Backup mode	Create a backup before executing. Supports full (complete state snapshot) and incremental (changes since last backup) modes.	Deploy, Reconfigure, Upgrade
Offline mode	Execute without internet access, using pre-staged packages and images.	Bootstrap Servers
Verbose logging	Increase Ansible output verbosity from level 0 (default) through level 4 (maximum debug output). Higher levels include connection debugging, module arguments, and full task execution details.	All actions

Use dry-run mode before any production change. It validates the action plan without modifying the cluster, letting you catch configuration issues before they affect running services.

Live Terminal

Every action execution streams full terminal output in real time, providing complete visibility into each playbook task as it runs.

Real-Time Metrics

Live counters for task success, warnings, failures, and overall completion percentage update as each task finishes.

Searchable History

All task output is archived and searchable. Filter by task name, host, or status to quickly locate specific events across long deployment runs.

Understanding Terminal Output

Each line in the terminal represents an Ansible task. Tasks display their target host, task name, and result status:

ok — Task completed successfully, no changes needed
changed — Task completed successfully and modified the system
failed — Task encountered an error (deployment may continue or stop depending on the task)
skipped — Task was not applicable to the current configuration

Quick Guide

The Operations tool includes a Quick Guide button that opens a recommended workflow tutorial. This guide walks through the standard deployment sequence and explains when to use each action.

Recommended Workflow

Follow this sequence for a successful first deployment. Each step must complete without errors before proceeding to the next.

Run Prechecks

Execute Prechecks against all target hosts. This validates port availability, service connectivity, configuration consistency, and Docker readiness across the cluster.

Fix Reported Issues

Review any failures or warnings from the Prechecks output. Common issues include missing Docker configuration, incorrect NIC assignments, and port conflicts. Resolve every failure before continuing.

Run Deploy

Execute the Deploy action against all hosts. This provisions every enabled cloud service as Docker containers, configures HAProxy load balancing, and starts all services.

Deployment duration scales with cluster size and network speed. A 3-node cluster typically completes in 30-45 minutes. Larger clusters with 10+ nodes may take up to 90 minutes.

Run Post-Deploy

Execute Post-Deploy to create the admin user account, default tenant networks, base compute flavors, and initial project structure. This step finalizes the environment for production use.

Verify via Dashboard and CLI

Log in to the Xloud Dashboard (https://connect.<your-domain>) with the admin credentials created during Post-Deploy. Verify that all services are listed and operational.

Verify service endpoints

source admin-openrc.sh
openstack service list
openstack compute service list
openstack network agent list

All services report status up — the deployment is complete and operational.

For configuration changes after initial deployment, use Reconfigure instead of a full redeployment. Reconfigure applies changes incrementally to running services without downtime, completing in minutes instead of 30-90 minutes.

Troubleshooting

Prechecks report port conflicts

Cause: Another service or previous deployment remnant is using a port required by a cloud service.Resolution: Identify the conflicting process using ss -tlnp | grep <port> on the affected host. Stop or remove the conflicting service. If a previous deployment was not fully destroyed, run Destroy before redeploying.

Deploy fails with image pull errors

Cause: Container images are not available in the local registry, or Docker cannot reach the registry.Resolution: Return to the Images tool and verify that all required images are loaded and pushed to the local registry. Confirm that docker-registry:4000 is listed as an insecure registry in the Docker daemon configuration on every node.

Post-Deploy fails to create admin user

Cause: The Xloud Identity service is not yet fully initialized when Post-Deploy runs.Resolution: Wait 30-60 seconds after Deploy completes before running Post-Deploy. The identity service needs time to complete database migrations and start accepting requests. Re-run Post-Deploy after the brief delay.

Core Services

Other Services

Deployment Wizard

Select Action

Select Target Hosts

Select Services (Optional)

Run Deployment

Available Actions

Command Modifiers

Live Terminal

Real-Time Metrics

Searchable History

Recommended Workflow

Run Prechecks

Fix Reported Issues

Run Deploy

Run Post-Deploy

Verify via Dashboard and CLI

Troubleshooting

Next Steps

Management

Cloud Fleet

Core Services

Other Services

​Deployment Wizard

Select Action

Select Target Hosts

Select Services (Optional)

Run Deployment

​Available Actions

​Command Modifiers

​Live Terminal

Real-Time Metrics

Searchable History

​Recommended Workflow

Run Prechecks

Fix Reported Issues

Run Deploy

Run Post-Deploy

Verify via Dashboard and CLI

​Troubleshooting

​Next Steps

Management

Cloud Fleet

Deployment Wizard

Available Actions

Command Modifiers

Live Terminal

Recommended Workflow

Troubleshooting

Next Steps