# API Authentication
Source: https://docs.xloud.tech/api-reference/authentication

Authenticate with the Xloud API using password tokens, application credentials, and scoped tokens. Includes token lifecycle management and CLI examples.

## Overview

All Xloud API requests require a valid bearer token in the `X-Auth-Token` request header. The Identity service (Keystone) issues tokens upon successful authentication. Tokens are time-limited, project-scoped, and carry the role assignments of the authenticated user.

<Note>
  **Prerequisites**

  * An active Xloud account
  * Project name, username, and password (or application credential ID and secret)
  * Identity endpoint: `https://api.<your-domain>/identity/v3`
</Note>

***

## Token-Based Authentication

### Password Authentication

Authenticate with a username and password to receive a project-scoped token:

<CodeGroup>
  ```bash title="cURL" theme={null}
  curl -i -X POST https://api.<your-domain>/identity/v3/auth/tokens \
    -H "Content-Type: application/json" \
    -d '{
      "auth": {
        "identity": {
          "methods": ["password"],
          "password": {
            "user": {
              "name": "myuser",
              "domain": { "name": "Default" },
              "password": "mypassword"
            }
          }
        },
        "scope": {
          "project": {
            "name": "myproject",
            "domain": { "name": "Default" }
          }
        }
      }
    }'
  ```

  ```python title="Python (requests)" theme={null}
  import requests

  auth_payload = {
      "auth": {
          "identity": {
              "methods": ["password"],
              "password": {
                  "user": {
                      "name": "myuser",
                      "domain": {"name": "Default"},
                      "password": "mypassword"
                  }
              }
          },
          "scope": {
              "project": {
                  "name": "myproject",
                  "domain": {"name": "Default"}
              }
          }
      }
  }

  resp = requests.post(
      "https://api.<your-domain>/identity/v3/auth/tokens",
      json=auth_payload
  )
  token = resp.headers["X-Subject-Token"]
  print(f"Token: {token}")
  ```
</CodeGroup>

The response contains:

* `X-Subject-Token` header — the token string to use in subsequent calls
* JSON body with token metadata including expiry time and service catalog

```json title="Token metadata response (excerpt)" theme={null}
{
  "token": {
    "expires_at": "2026-03-19T10:00:00.000000Z",
    "issued_at": "2026-03-18T10:00:00.000000Z",
    "methods": ["password"],
    "project": {
      "id": "abc123",
      "name": "myproject",
      "domain": { "id": "default", "name": "Default" }
    },
    "roles": [
      { "id": "def456", "name": "member" }
    ],
    "catalog": [...]
  }
}
```

***

## Application Credentials

Application credentials allow services and automation scripts to authenticate without exposing user passwords. They are scoped to a specific project and role set at creation time.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Application Credentials" icon="compass">
        Log in to the **Xloud Dashboard** (`https://connect.<your-domain>`) and navigate to
        **Project → Identity → Application Credentials**. Click **Create Application Credential**.
      </Step>

      <Step title="Configure the credential" icon="settings">
        | Field            | Value                    | Description                                                 |
        | ---------------- | ------------------------ | ----------------------------------------------------------- |
        | **Name**         | `ci-deploy-prod`         | Descriptive identifier                                      |
        | **Secret**       | Auto-generated or custom | Store this immediately — it is shown only once              |
        | **Expiration**   | Optional date            | Leave blank for non-expiring credentials                    |
        | **Roles**        | `member`                 | Restrict to minimum required roles                          |
        | **Unrestricted** | Disabled                 | Only enable if the credential must manage other credentials |

        <Warning>
          Copy the **Secret** value immediately after creation. It cannot be retrieved again. If lost, delete and recreate the credential.
        </Warning>
      </Step>

      <Step title="Download the clouds.yaml" icon="download">
        Click **Download clouds.yaml** to get a ready-to-use configuration file for the
        Xloud CLI and Python SDK.

        <Check>The `clouds.yaml` file is saved to your local machine and can be placed at `~/.config/xloud/clouds.yaml`.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create an application credential" theme={null}
    openstack application credential create ci-deploy-prod \
      --role member \
      --description "CI/CD deployment credential for production project"
    ```

    Store the `secret` value from the output immediately.

    ```bash title="Authenticate using an application credential" theme={null}
    curl -i -X POST https://api.<your-domain>/identity/v3/auth/tokens \
      -H "Content-Type: application/json" \
      -d '{
        "auth": {
          "identity": {
            "methods": ["application_credential"],
            "application_credential": {
              "id": "<credential-id>",
              "secret": "<credential-secret>"
            }
          }
        }
      }'
    ```
  </Tab>
</Tabs>

***

## Scoped Tokens

Tokens can be scoped to different resources depending on the operation required:

| Scope              | Use Case                                      | Request `scope` field                |
| ------------------ | --------------------------------------------- | ------------------------------------ |
| **Project-scoped** | Manage resources within a project             | `"project": { "name": "myproject" }` |
| **Domain-scoped**  | Manage users and projects within a domain     | `"domain": { "name": "Default" }`    |
| **System-scoped**  | Administrative operations across all projects | `"system": { "all": true }`          |
| **Unscoped**       | Discover available projects before scoping    | Omit the `scope` field               |

### Exchange an Unscoped Token for a Project-Scoped Token

```bash title="Step 1: Obtain unscoped token" theme={null}
curl -i -X POST https://api.<your-domain>/identity/v3/auth/tokens \
  -H "Content-Type: application/json" \
  -d '{
    "auth": {
      "identity": {
        "methods": ["password"],
        "password": {
          "user": {
            "name": "myuser",
            "domain": { "name": "Default" },
            "password": "mypassword"
          }
        }
      }
    }
  }'
```

```bash title="Step 2: List available projects" theme={null}
curl -X GET https://api.<your-domain>/identity/v3/auth/projects \
  -H "X-Auth-Token: $UNSCOPED_TOKEN"
```

```bash title="Step 3: Exchange for project-scoped token" theme={null}
curl -i -X POST https://api.<your-domain>/identity/v3/auth/tokens \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: $UNSCOPED_TOKEN" \
  -d '{
    "auth": {
      "identity": {
        "methods": ["token"],
        "token": { "id": "'"$UNSCOPED_TOKEN"'" }
      },
      "scope": {
        "project": { "id": "<project-id>" }
      }
    }
  }'
```

***

## Token Lifecycle

| Operation           | Endpoint                      | Method                          |
| ------------------- | ----------------------------- | ------------------------------- |
| Create token        | `/identity/v3/auth/tokens`    | POST                            |
| Validate token      | `/identity/v3/auth/tokens`    | GET (with `X-Subject-Token`)    |
| Revoke token        | `/identity/v3/auth/tokens`    | DELETE (with `X-Subject-Token`) |
| List tokens (admin) | `/identity/v3/OS-PKI/revoked` | GET                             |

```bash title="Validate a token" theme={null}
curl -i -X GET https://api.<your-domain>/identity/v3/auth/tokens \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "X-Subject-Token: $TOKEN_TO_VALIDATE"
```

```bash title="Revoke a token" theme={null}
curl -i -X DELETE https://api.<your-domain>/identity/v3/auth/tokens \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "X-Subject-Token: $TOKEN_TO_REVOKE"
```

<Info>
  Tokens expire after the configured token TTL (default: 1 hour). Long-running automation scripts should implement token refresh logic — detect `401 Unauthorized` responses and re-authenticate automatically.
</Info>

***

## Using the OpenRC File

For CLI and script use, source an OpenRC credentials file instead of manually passing credentials:

```bash title="Source OpenRC file" theme={null}
source admin-openrc.sh
```

```bash title="Sample OpenRC file content" theme={null}
export OS_AUTH_URL=https://api.<your-domain>/identity/v3
export OS_PROJECT_NAME=myproject
export OS_USERNAME=myuser
export OS_PASSWORD=mypassword
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
```

Download the OpenRC file from **Xloud Dashboard → Project → API Access → Download OpenStack RC File**.

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="401 Unauthorized on every request" icon="lock">
    **Cause**: Token is expired, invalid, or not included in the request header.

    **Resolution**:

    1. Re-authenticate to obtain a fresh token
    2. Confirm the token is passed in `X-Auth-Token` (not `Authorization: Bearer`)
    3. Verify the Identity endpoint URL is correct
  </Accordion>

  <Accordion title="403 Forbidden despite valid token" icon="shield">
    **Cause**: The token is valid but the user's role does not permit this operation.

    **Resolution**: Verify your role assignment in the project:

    ```bash title="Check role assignments" theme={null}
    openstack role assignment list --user $OS_USERNAME --project $OS_PROJECT_NAME
    ```

    Contact your administrator to assign the appropriate role.
  </Accordion>

  <Accordion title="Token scope mismatch" icon="key">
    **Cause**: The token is scoped to a different project than the resource being accessed.

    **Resolution**: Re-authenticate with the correct project scope. Confirm the `OS_PROJECT_NAME` environment variable matches the project containing the resource.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Compute API" href="/api-reference/compute-api">
    Launch and manage instances using the Compute API
  </Card>

  <Card title="Identity API" href="/services/identity/index">
    Manage users, projects, and roles via the Identity API
  </Card>

  <Card title="Automation Guide" href="/integrations/index">
    Scripting patterns and token refresh strategies for automation
  </Card>

  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Create and manage application credentials for non-interactive access
  </Card>
</CardGroup>


# Compute API Reference
Source: https://docs.xloud.tech/api-reference/compute-api

Xloud Compute API (Nova) overview, key endpoints, request/response examples, and common operations for managing virtual machine instances programmatically.

## Overview

The Xloud Compute API provides programmatic control over virtual machine instances, flavors, keypairs, availability zones, server groups, and hypervisor resources. The API follows RESTful conventions with JSON request and response bodies.

<Note>
  **Prerequisites**

  * A valid project-scoped token from the [Identity API](/api-reference/authentication)
  * Base URL: `https://api.<your-domain>/compute/v2.1`
  * Minimum API microversion: `2.1`
</Note>

***

## API Versioning (Microversions)

The Compute API uses microversions to add new capabilities without breaking existing clients. Specify the desired microversion in the `X-OpenStack-Nova-Microversion` header:

```bash title="Request with specific microversion" theme={null}
curl -X GET https://api.<your-domain>/compute/v2.1/servers \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "X-OpenStack-Nova-Microversion: 2.79"
```

```bash title="Query supported microversions" theme={null}
curl -X GET https://api.<your-domain>/compute \
  -H "X-Auth-Token: $OS_TOKEN"
```

***

## Key Endpoints

| Resource        | Method | Endpoint                      | Description                                 |
| --------------- | ------ | ----------------------------- | ------------------------------------------- |
| List instances  | GET    | `/servers`                    | List all instances in the project           |
| Instance detail | GET    | `/servers/detail`             | List instances with full details            |
| Get instance    | GET    | `/servers/{id}`               | Get a specific instance                     |
| Create instance | POST   | `/servers`                    | Launch a new instance                       |
| Delete instance | DELETE | `/servers/{id}`               | Delete an instance                          |
| Instance action | POST   | `/servers/{id}/action`        | Perform an action (reboot, resize, etc.)    |
| List flavors    | GET    | `/flavors/detail`             | List all available instance sizes           |
| List images     | GET    | `/images`                     | List available images                       |
| List keypairs   | GET    | `/os-keypairs`                | List SSH keypairs                           |
| Create keypair  | POST   | `/os-keypairs`                | Register or generate an SSH keypair         |
| List AZs        | GET    | `/os-availability-zone`       | List availability zones                     |
| Server groups   | GET    | `/os-server-groups`           | List server groups (affinity/anti-affinity) |
| Hypervisors     | GET    | `/os-hypervisors/detail`      | List hypervisors (admin)                    |
| Quotas          | GET    | `/os-quota-sets/{project_id}` | Show project compute quotas                 |

***

## Create an Instance

<ParamField type="string">
  Display name for the instance.
</ParamField>

<ParamField type="string">
  Flavor ID or URL. Determines vCPU, RAM, and disk allocation.
</ParamField>

<ParamField type="string">
  Image ID or URL to boot from. Omit when booting from volume.
</ParamField>

<ParamField type="array">
  List of network objects. Each must include `uuid` (network ID) or `port` (port ID).
</ParamField>

<ParamField type="string">
  SSH keypair name to inject into the instance for key-based authentication.
</ParamField>

<ParamField type="array">
  List of security group name objects. Defaults to the `default` group.
</ParamField>

<ParamField type="string">
  Base64-encoded cloud-init user data script executed at first boot.
</ParamField>

<ParamField type="string">
  Target availability zone. Omit to let the scheduler choose.
</ParamField>

<RequestExample>
  ```bash title="cURL" theme={null}
  curl -X POST https://api.<your-domain>/compute/v2.1/servers \
    -H "X-Auth-Token: $OS_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "server": {
        "name": "web-server-01",
        "flavorRef": "m1.medium",
        "imageRef": "ubuntu-22.04-amd64",
        "networks": [{ "uuid": "a1b2c3d4-e5f6-..." }],
        "key_name": "my-keypair",
        "security_groups": [{ "name": "web-servers" }]
      }
    }'
  ```

  ```python title="Python" theme={null}
  import requests

  token = "your-token"
  server = {
      "server": {
          "name": "web-server-01",
          "flavorRef": "m1.medium",
          "imageRef": "ubuntu-22.04-amd64",
          "networks": [{"uuid": "a1b2c3d4-e5f6-..."}],
          "key_name": "my-keypair",
          "security_groups": [{"name": "web-servers"}]
      }
  }
  resp = requests.post(
      "https://api.<your-domain>/compute/v2.1/servers",
      json=server,
      headers={"X-Auth-Token": token, "Content-Type": "application/json"}
  )
  print(resp.json()["server"]["id"])
  ```
</RequestExample>

<ResponseExample>
  ```json title="202 Accepted" theme={null}
  {
    "server": {
      "id": "b1c2d3e4-f5a6-7890-b1c2-d3e4f5a67890",
      "name": "web-server-01",
      "status": "BUILD",
      "links": [
        { "href": "https://api.<your-domain>/compute/v2.1/servers/b1c2d3e4...", "rel": "self" }
      ],
      "adminPass": "auto-generated-password",
      "OS-DCF:diskConfig": "MANUAL"
    }
  }
  ```
</ResponseExample>

***

## Instance Actions

Perform operations on a running instance via the `/action` endpoint:

<Tabs>
  <Tab title="Reboot" icon="refresh-cw">
    ```bash title="Soft reboot (graceful OS restart)" theme={null}
    curl -X POST https://api.<your-domain>/compute/v2.1/servers/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "reboot": { "type": "SOFT" } }'
    ```

    ```bash title="Hard reboot (power cycle)" theme={null}
    curl -X POST https://api.<your-domain>/compute/v2.1/servers/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "reboot": { "type": "HARD" } }'
    ```
  </Tab>

  <Tab title="Stop / Start" icon="power">
    ```bash title="Stop instance (power off)" theme={null}
    curl -X POST https://api.<your-domain>/compute/v2.1/servers/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "os-stop": null }'
    ```

    ```bash title="Start stopped instance" theme={null}
    curl -X POST https://api.<your-domain>/compute/v2.1/servers/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "os-start": null }'
    ```
  </Tab>

  <Tab title="Resize" icon="expand">
    ```bash title="Resize to a larger flavor" theme={null}
    curl -X POST https://api.<your-domain>/compute/v2.1/servers/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "resize": { "flavorRef": "m1.large" } }'
    ```

    ```bash title="Confirm the resize" theme={null}
    curl -X POST https://api.<your-domain>/compute/v2.1/servers/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "confirmResize": null }'
    ```
  </Tab>

  <Tab title="Snapshot" icon="camera">
    ```bash title="Create instance snapshot (image)" theme={null}
    curl -X POST https://api.<your-domain>/compute/v2.1/servers/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "createImage": { "name": "web-server-01-snapshot", "metadata": {} } }'
    ```
  </Tab>
</Tabs>

***

## List Instances with Filtering

```bash title="List all running instances" theme={null}
curl -X GET "https://api.<your-domain>/compute/v2.1/servers?status=ACTIVE" \
  -H "X-Auth-Token: $OS_TOKEN"
```

```bash title="List instances with full details" theme={null}
curl -X GET https://api.<your-domain>/compute/v2.1/servers/detail \
  -H "X-Auth-Token: $OS_TOKEN"
```

```bash title="List instances across all projects (admin)" theme={null}
curl -X GET "https://api.<your-domain>/compute/v2.1/servers/detail?all_tenants=1" \
  -H "X-Auth-Token: $OS_TOKEN"
```

Common filter parameters:

| Parameter     | Description                                                      |
| ------------- | ---------------------------------------------------------------- |
| `status`      | Filter by instance status: `ACTIVE`, `SHUTOFF`, `BUILD`, `ERROR` |
| `name`        | Filter by instance name (supports regex)                         |
| `flavor`      | Filter by flavor ID                                              |
| `image`       | Filter by image ID                                               |
| `all_tenants` | `1` to list across all projects (admin only)                     |
| `limit`       | Maximum number of results per page                               |
| `marker`      | Pagination cursor — last instance ID from previous page          |

***

## Error Reference

| HTTP Status | Error Code           | Description                                   |
| ----------- | -------------------- | --------------------------------------------- |
| `400`       | `badRequest`         | Invalid request body or parameter             |
| `401`       | `unauthorized`       | Token missing, invalid, or expired            |
| `403`       | `forbidden`          | Insufficient permissions for this operation   |
| `404`       | `itemNotFound`       | Instance or resource not found                |
| `409`       | `conflictingRequest` | Operation not valid in current instance state |
| `413`       | `entityTooLarge`     | Request exceeds quota limits                  |
| `429`       | `overLimit`          | Rate limit exceeded                           |

***

## Next Steps

<CardGroup>
  <Card title="Storage API" href="/api-reference/storage-api">
    Attach volumes and manage snapshots via the Storage API
  </Card>

  <Card title="Networking API" href="/api-reference/networking-api">
    Create networks and assign floating IPs for your instances
  </Card>

  <Card title="Authentication" href="/api-reference/authentication">
    Manage tokens and application credentials
  </Card>

  <Card title="Automation" href="/integrations/index">
    Automate instance lifecycle with scripts and SDKs
  </Card>
</CardGroup>


# API Reference
Source: https://docs.xloud.tech/api-reference/index

REST API documentation for the Xloud Cloud Platform. Programmatically control compute, storage, networking, and identity services.

## Overview

The Xloud Cloud Platform exposes a comprehensive set of RESTful APIs. These APIs give you programmatic control over every infrastructure resource — compute instances, block storage volumes, networks, identity tokens, and more. All APIs use token-based authentication and return JSON-formatted responses.

<Note>
  **Prerequisites**

  * An active Xloud account with project membership
  * API access enabled (contact your administrator if not available)
  * API base URL: `https://api.<your-domain>` or the internal endpoint provided during deployment
</Note>

***

## API Services

<CardGroup>
  <Card title="Authentication" icon="key" href="/api-reference/authentication">
    Token-based auth, application credentials, and scoped tokens
  </Card>

  <Card title="Compute API" icon="server" href="/api-reference/compute-api">
    Instances, flavors, keypairs, and console access
  </Card>

  <Card title="Storage API" icon="hard-drive" href="/api-reference/storage-api">
    Volumes, snapshots, backups, and volume types
  </Card>

  <Card title="Networking API" icon="network" href="/api-reference/networking-api">
    Networks, subnets, routers, security groups, and floating IPs
  </Card>

  <Card title="Identity API" icon="fingerprint" href="/services/identity/index">
    Projects, users, roles, domains, and service catalog
  </Card>

  <Card title="Automation" icon="workflow" href="/integrations/index">
    Scripting patterns, SDK usage, and webhook integration
  </Card>
</CardGroup>

***

## Quick Reference

| Service                | Base Path          | Current Version |
| ---------------------- | ------------------ | --------------- |
| Identity (Keystone)    | `/identity/v3`     | v3              |
| Compute (Nova)         | `/compute/v2.1`    | v2.1            |
| Block Storage (Cinder) | `/volume/v3`       | v3              |
| Networking (Neutron)   | `/networking/v2.0` | v2.0            |
| Image Service (Glance) | `/image/v2`        | v2              |
| Object Storage (Swift) | `/object-store/v1` | v1              |
| Object Storage (S3)    | `/s3`              | S3 v4           |

***

## Authentication Overview

All API calls require a valid token in the `X-Auth-Token` header. Obtain a token by authenticating against the Identity service:

```bash title="Obtain authentication token" theme={null}
curl -i -X POST https://api.<your-domain>/identity/v3/auth/tokens \
  -H "Content-Type: application/json" \
  -d '{
    "auth": {
      "identity": {
        "methods": ["password"],
        "password": {
          "user": {
            "name": "admin",
            "domain": { "name": "Default" },
            "password": "your_password"
          }
        }
      },
      "scope": {
        "project": {
          "name": "admin",
          "domain": { "name": "Default" }
        }
      }
    }
  }'
```

The `X-Subject-Token` response header contains the token value. Use it in all subsequent API calls:

```bash title="Use token in API calls" theme={null}
export OS_TOKEN="<X-Subject-Token value>"
curl -X GET https://api.<your-domain>/compute/v2.1/servers \
  -H "X-Auth-Token: $OS_TOKEN"
```

See the [Authentication reference](/api-reference/authentication) for application credentials, token scoping, and token renewal.

***

## Response Format

All API responses use JSON. Standard HTTP status codes indicate success or failure:

| Status                  | Meaning                                                       |
| ----------------------- | ------------------------------------------------------------- |
| `200 OK`                | Request succeeded. Response body contains the resource.       |
| `201 Created`           | Resource created. Response body contains the new resource.    |
| `202 Accepted`          | Asynchronous operation started. Poll the resource for status. |
| `204 No Content`        | Request succeeded. No response body (common for DELETE).      |
| `400 Bad Request`       | Invalid request body or parameters. Check the error message.  |
| `401 Unauthorized`      | Invalid or expired token. Re-authenticate.                    |
| `403 Forbidden`         | Valid token, but insufficient permissions for this operation. |
| `404 Not Found`         | Resource does not exist or is not visible to your project.    |
| `409 Conflict`          | Operation conflicts with current resource state.              |
| `429 Too Many Requests` | Rate limit exceeded. Back off and retry.                      |

***

## Next Steps

<CardGroup>
  <Card title="Authentication Guide" href="/api-reference/authentication">
    Start here — learn how to obtain tokens and use application credentials
  </Card>

  <Card title="Automation Patterns" href="/integrations/index">
    Scripting, SDK usage, and CI/CD pipeline integration examples
  </Card>
</CardGroup>


# Networking API Reference
Source: https://docs.xloud.tech/api-reference/networking-api

Xloud Networking API (Neutron) overview, key endpoints, and examples for managing networks, subnets, routers, security groups, and floating IPs.

## Overview

The Xloud Networking API provides full programmatic control over virtual network resources — networks, subnets, routers, ports, security groups, floating IPs, and QoS policies. The API uses a RESTful design with JSON bodies and project-scoped resource isolation.

<Note>
  **Prerequisites**

  * A valid project-scoped token from the [Identity API](/api-reference/authentication)
  * Base URL: `https://api.<your-domain>/networking/v2.0`
  * Network resources are project-isolated by default — shared resources are admin-managed
</Note>

***

## Key Endpoints

| Resource              | Method | Endpoint                                | Description                          |
| --------------------- | ------ | --------------------------------------- | ------------------------------------ |
| List networks         | GET    | `/networks`                             | List networks visible to the project |
| Create network        | POST   | `/networks`                             | Create a new tenant network          |
| Get network           | GET    | `/networks/{id}`                        | Get network details                  |
| Delete network        | DELETE | `/networks/{id}`                        | Delete a network                     |
| List subnets          | GET    | `/subnets`                              | List subnets in the project          |
| Create subnet         | POST   | `/subnets`                              | Create a subnet on a network         |
| List routers          | GET    | `/routers`                              | List routers in the project          |
| Create router         | POST   | `/routers`                              | Create a new router                  |
| Add interface         | PUT    | `/routers/{id}/add_router_interface`    | Attach subnet to router              |
| Remove interface      | PUT    | `/routers/{id}/remove_router_interface` | Detach subnet from router            |
| List ports            | GET    | `/ports`                                | List all ports                       |
| Create port           | POST   | `/ports`                                | Create a network port                |
| List security groups  | GET    | `/security-groups`                      | List security groups                 |
| Create security group | POST   | `/security-groups`                      | Create a security group              |
| Add SG rule           | POST   | `/security-group-rules`                 | Add a rule to a security group       |
| List floating IPs     | GET    | `/floatingips`                          | List floating IP allocations         |
| Create floating IP    | POST   | `/floatingips`                          | Allocate a floating IP               |
| Associate floating IP | PUT    | `/floatingips/{id}`                     | Associate with a port                |
| List QoS policies     | GET    | `/qos/policies`                         | List QoS policies (admin)            |
| Create QoS policy     | POST   | `/qos/policies`                         | Create a QoS policy (admin)          |

***

## Create a Network and Subnet

<RequestExample>
  ```bash title="Create network" theme={null}
  curl -X POST https://api.<your-domain>/networking/v2.0/networks \
    -H "X-Auth-Token: $OS_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "network": {
        "name": "app-network",
        "admin_state_up": true
      }
    }'
  ```

  ```bash title="Create subnet on the network" theme={null}
  curl -X POST https://api.<your-domain>/networking/v2.0/subnets \
    -H "X-Auth-Token: $OS_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "subnet": {
        "name": "app-subnet",
        "network_id": "<network-id>",
        "ip_version": 4,
        "cidr": "192.168.10.0/24",
        "enable_dhcp": true,
        "dns_nameservers": ["8.8.8.8", "8.8.4.4"],
        "allocation_pools": [
          { "start": "192.168.10.10", "end": "192.168.10.200" }
        ]
      }
    }'
  ```
</RequestExample>

<ResponseExample>
  ```json title="201 Created — Network" theme={null}
  {
    "network": {
      "id": "d4e5f6a7-b8c9-0123-d4e5-f6a7b8c90123",
      "name": "app-network",
      "status": "ACTIVE",
      "admin_state_up": true,
      "subnets": [],
      "shared": false,
      "tenant_id": "abc123"
    }
  }
  ```
</ResponseExample>

***

## Routers

<Tabs>
  <Tab title="Create Router" icon="route">
    ```bash title="Create a router with external gateway" theme={null}
    curl -X POST https://api.<your-domain>/networking/v2.0/routers \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "router": {
          "name": "app-router",
          "admin_state_up": true,
          "external_gateway_info": {
            "network_id": "<external-network-id>"
          }
        }
      }'
    ```
  </Tab>

  <Tab title="Attach Subnet" icon="link">
    ```bash title="Attach subnet to router" theme={null}
    curl -X PUT \
      "https://api.<your-domain>/networking/v2.0/routers/{router_id}/add_router_interface" \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "subnet_id": "<subnet-id>" }'
    ```
  </Tab>

  <Tab title="Remove Subnet" icon="unlink">
    ```bash title="Remove subnet from router" theme={null}
    curl -X PUT \
      "https://api.<your-domain>/networking/v2.0/routers/{router_id}/remove_router_interface" \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "subnet_id": "<subnet-id>" }'
    ```
  </Tab>
</Tabs>

***

## Security Groups

```bash title="Create a security group" theme={null}
curl -X POST https://api.<your-domain>/networking/v2.0/security-groups \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "security_group": {
      "name": "web-servers",
      "description": "Allow HTTP/HTTPS and SSH inbound"
    }
  }'
```

```bash title="Allow SSH inbound (TCP 22)" theme={null}
curl -X POST https://api.<your-domain>/networking/v2.0/security-group-rules \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "security_group_rule": {
      "security_group_id": "<sg-id>",
      "direction": "ingress",
      "protocol": "tcp",
      "port_range_min": 22,
      "port_range_max": 22,
      "remote_ip_prefix": "0.0.0.0/0"
    }
  }'
```

```bash title="Allow HTTPS inbound (TCP 443)" theme={null}
curl -X POST https://api.<your-domain>/networking/v2.0/security-group-rules \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "security_group_rule": {
      "security_group_id": "<sg-id>",
      "direction": "ingress",
      "protocol": "tcp",
      "port_range_min": 443,
      "port_range_max": 443,
      "remote_ip_prefix": "0.0.0.0/0"
    }
  }'
```

Security group rule parameters:

| Parameter          | Values                       | Description                      |
| ------------------ | ---------------------------- | -------------------------------- |
| `direction`        | `ingress`, `egress`          | Traffic direction                |
| `protocol`         | `tcp`, `udp`, `icmp`, `null` | IP protocol                      |
| `port_range_min`   | `1`–`65535`                  | Start of port range              |
| `port_range_max`   | `1`–`65535`                  | End of port range                |
| `remote_ip_prefix` | CIDR e.g. `10.0.0.0/8`       | Source or destination IP range   |
| `remote_group_id`  | Security group ID            | Allow traffic from another group |

***

## Floating IPs

```bash title="Allocate a floating IP from external network" theme={null}
curl -X POST https://api.<your-domain>/networking/v2.0/floatingips \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "floatingip": {
      "floating_network_id": "<external-network-id>"
    }
  }'
```

```bash title="Associate floating IP with a port" theme={null}
curl -X PUT https://api.<your-domain>/networking/v2.0/floatingips/{floatingip_id} \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "floatingip": {
      "port_id": "<instance-port-id>"
    }
  }'
```

```bash title="Disassociate floating IP (keep allocated)" theme={null}
curl -X PUT https://api.<your-domain>/networking/v2.0/floatingips/{floatingip_id} \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "floatingip": { "port_id": null } }'
```

```bash title="Release floating IP back to the pool" theme={null}
curl -X DELETE https://api.<your-domain>/networking/v2.0/floatingips/{floatingip_id} \
  -H "X-Auth-Token: $OS_TOKEN"
```

***

## Ports

<ParamField type="string">
  Network ID to create the port on.
</ParamField>

<ParamField type="string">
  Display name for the port.
</ParamField>

<ParamField type="array">
  List of objects specifying `subnet_id` and optionally a specific `ip_address`.
</ParamField>

<ParamField type="array">
  List of security group IDs to apply to this port.
</ParamField>

```bash title="Create a port with specific IP" theme={null}
curl -X POST https://api.<your-domain>/networking/v2.0/ports \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "port": {
      "network_id": "<network-id>",
      "name": "db-port",
      "fixed_ips": [
        { "subnet_id": "<subnet-id>", "ip_address": "192.168.10.50" }
      ],
      "security_groups": ["<sg-id>"]
    }
  }'
```

***

## Network Resource Status

| Status   | Meaning                                    |
| -------- | ------------------------------------------ |
| `ACTIVE` | Resource is operational                    |
| `DOWN`   | Resource exists but is not passing traffic |
| `BUILD`  | Resource is being provisioned              |
| `ERROR`  | Provisioning or operation failed           |

***

## Next Steps

<CardGroup>
  <Card title="Compute API" href="/api-reference/compute-api">
    Launch instances connected to the networks you create
  </Card>

  <Card title="Network QoS" href="/services/networking/qos">
    Apply bandwidth limits and DSCP marking via the QoS API
  </Card>

  <Card title="Identity API" href="/services/identity/index">
    Manage project membership and resource access
  </Card>

  <Card title="Automation" href="/integrations/index">
    Script full network topology provisioning end-to-end
  </Card>
</CardGroup>


# Storage API Reference
Source: https://docs.xloud.tech/api-reference/storage-api

Xloud Block Storage API (Cinder) overview, key endpoints, and examples for managing volumes, snapshots, backups, and volume types programmatically.

## Overview

The Xloud Block Storage API provides programmatic management of persistent storage volumes, volume snapshots, volume backups, volume types, and QoS specifications. Volumes attach to compute instances as block devices — independent of instance lifecycle.

<Note>
  **Prerequisites**

  * A valid project-scoped token from the [Identity API](/api-reference/authentication)
  * Base URL: `https://api.<your-domain>/volume/v3`
  * All operations are project-scoped unless using admin credentials
</Note>

***

## Key Endpoints

| Resource        | Method | Endpoint                      | Description                         |
| --------------- | ------ | ----------------------------- | ----------------------------------- |
| List volumes    | GET    | `/volumes`                    | List volumes in the current project |
| Volume detail   | GET    | `/volumes/detail`             | List volumes with full metadata     |
| Get volume      | GET    | `/volumes/{id}`               | Get a specific volume               |
| Create volume   | POST   | `/volumes`                    | Create a new volume                 |
| Extend volume   | POST   | `/volumes/{id}/action`        | Extend volume size                  |
| Delete volume   | DELETE | `/volumes/{id}`               | Delete a volume                     |
| Attach volume   | POST   | `/volumes/{id}/action`        | Reserve for attachment              |
| List snapshots  | GET    | `/snapshots`                  | List volume snapshots               |
| Create snapshot | POST   | `/snapshots`                  | Create a volume snapshot            |
| Delete snapshot | DELETE | `/snapshots/{id}`             | Delete a snapshot                   |
| List backups    | GET    | `/backups`                    | List volume backups                 |
| Create backup   | POST   | `/backups`                    | Create a volume backup              |
| Restore backup  | POST   | `/backups/{id}/restore`       | Restore backup to a volume          |
| Volume types    | GET    | `/types`                      | List available volume types         |
| QoS specs       | GET    | `/qos-specs`                  | List QoS specifications (admin)     |
| Quotas          | GET    | `/os-quota-sets/{project_id}` | Show project storage quotas         |

***

## Create a Volume

<ParamField type="string">
  Display name for the volume.
</ParamField>

<ParamField type="integer">
  Volume size in gigabytes.
</ParamField>

<ParamField type="string">
  Volume type name or ID. Determines the storage backend and QoS policy.
</ParamField>

<ParamField type="string">
  Target availability zone. Must match the compute availability zone to attach.
</ParamField>

<ParamField type="string">
  Clone from an existing volume by providing its ID.
</ParamField>

<ParamField type="string">
  Create from a snapshot by providing the snapshot ID.
</ParamField>

<ParamField type="object">
  Key-value metadata pairs for custom tagging.
</ParamField>

<RequestExample>
  ```bash title="cURL" theme={null}
  curl -X POST https://api.<your-domain>/volume/v3/volumes \
    -H "X-Auth-Token: $OS_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "volume": {
        "name": "db-data-01",
        "size": 200,
        "volume_type": "ssd-standard",
        "availability_zone": "nova",
        "metadata": {
          "environment": "production",
          "service": "mysql"
        }
      }
    }'
  ```

  ```python title="Python" theme={null}
  import requests

  volume = {
      "volume": {
          "name": "db-data-01",
          "size": 200,
          "volume_type": "ssd-standard",
          "availability_zone": "nova",
          "metadata": {"environment": "production"}
      }
  }
  resp = requests.post(
      "https://api.<your-domain>/volume/v3/volumes",
      json=volume,
      headers={"X-Auth-Token": token}
  )
  print(resp.json()["volume"]["id"])
  ```
</RequestExample>

<ResponseExample>
  ```json title="202 Accepted" theme={null}
  {
    "volume": {
      "id": "c3d4e5f6-a7b8-9012-c3d4-e5f6a7b89012",
      "name": "db-data-01",
      "status": "creating",
      "size": 200,
      "volume_type": "ssd-standard",
      "availability_zone": "nova",
      "metadata": { "environment": "production" },
      "created_at": "2026-03-18T10:00:00.000000",
      "bootable": "false",
      "encrypted": false
    }
  }
  ```
</ResponseExample>

***

## Volume Operations

<Tabs>
  <Tab title="Attach / Detach" icon="link">
    Volumes are attached to instances via the Compute API. The Storage API provides the `os-reserve` and `os-attach` actions for low-level attachment management.

    ```bash title="Attach volume via Compute API (recommended)" theme={null}
    curl -X POST https://api.<your-domain>/compute/v2.1/servers/{server_id}/os-volume_attachments \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "volumeAttachment": {
          "volumeId": "<volume-id>",
          "device": "/dev/vdb"
        }
      }'
    ```

    ```bash title="Detach volume from instance" theme={null}
    curl -X DELETE \
      "https://api.<your-domain>/compute/v2.1/servers/{server_id}/os-volume_attachments/{volume_id}" \
      -H "X-Auth-Token: $OS_TOKEN"
    ```

    <Info>
      Always detach a volume from the guest OS before issuing the API detach call to prevent filesystem corruption.
    </Info>
  </Tab>

  <Tab title="Extend" icon="expand">
    ```bash title="Extend volume to 400 GB" theme={null}
    curl -X POST https://api.<your-domain>/volume/v3/volumes/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "os-extend": { "new_size": 400 } }'
    ```

    <Warning>
      Extending a volume only expands the block device. After extension, resize the filesystem from inside the guest OS (e.g., `resize2fs` for ext4, `xfs_growfs` for XFS).
    </Warning>
  </Tab>

  <Tab title="Retype" icon="shuffle">
    ```bash title="Change volume type (live migration)" theme={null}
    curl -X POST https://api.<your-domain>/volume/v3/volumes/{id}/action \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "os-retype": {
          "new_type": "archive-hdd",
          "migration_policy": "on-demand"
        }
      }'
    ```

    Set `migration_policy` to `on-demand` to migrate data immediately, or `never` to only change the type metadata without data migration.
  </Tab>

  <Tab title="Clone" icon="copy">
    ```bash title="Clone an existing volume" theme={null}
    curl -X POST https://api.<your-domain>/volume/v3/volumes \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "volume": {
          "name": "db-data-01-clone",
          "size": 200,
          "source_volid": "<source-volume-id>"
        }
      }'
    ```
  </Tab>
</Tabs>

***

## Snapshots

```bash title="Create a volume snapshot" theme={null}
curl -X POST https://api.<your-domain>/volume/v3/snapshots \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "snapshot": {
      "name": "db-data-01-snap-20260318",
      "volume_id": "<volume-id>",
      "force": false,
      "metadata": { "backup_type": "daily" }
    }
  }'
```

```bash title="Create volume from snapshot" theme={null}
curl -X POST https://api.<your-domain>/volume/v3/volumes \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "volume": {
      "name": "db-data-restored",
      "snapshot_id": "<snapshot-id>",
      "size": 200
    }
  }'
```

<Tip>
  Set `force: true` in the snapshot request to snapshot a volume that is currently attached to a running instance. Ensure the filesystem is quiesced (via `sync` or application-level flush) before forcing a snapshot.
</Tip>

***

## Backups

Backups are stored in Xloud Object Storage and are independent of volume availability.

```bash title="Create a full backup" theme={null}
curl -X POST https://api.<your-domain>/volume/v3/backups \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "backup": {
      "name": "db-data-01-backup-20260318",
      "volume_id": "<volume-id>",
      "incremental": false,
      "container": "volume-backups"
    }
  }'
```

```bash title="Create incremental backup (faster, less storage)" theme={null}
curl -X POST https://api.<your-domain>/volume/v3/backups \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "backup": {
      "name": "db-data-01-incr-20260318",
      "volume_id": "<volume-id>",
      "incremental": true
    }
  }'
```

```bash title="Restore backup to a new volume" theme={null}
curl -X POST https://api.<your-domain>/volume/v3/backups/{backup_id}/restore \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "restore": {
      "name": "db-data-01-restored"
    }
  }'
```

***

## Volume Status Reference

| Status             | Description                             |
| ------------------ | --------------------------------------- |
| `creating`         | Volume is being provisioned             |
| `available`        | Ready to attach                         |
| `in-use`           | Currently attached to an instance       |
| `deleting`         | Deletion in progress                    |
| `extending`        | Size extension in progress              |
| `retyping`         | Volume type change in progress          |
| `backing-up`       | Backup in progress                      |
| `restoring-backup` | Backup restore in progress              |
| `error`            | An error occurred — check volume events |
| `error_deleting`   | Deletion failed                         |

***

## Next Steps

<CardGroup>
  <Card title="Compute API" href="/api-reference/compute-api">
    Attach volumes to compute instances via the Compute API
  </Card>

  <Card title="Networking API" href="/api-reference/networking-api">
    Configure networks for your storage-backed instances
  </Card>

  <Card title="Storage QoS" href="/services/storage/qos">
    Configure IOPS and throughput limits for volume types
  </Card>

  <Card title="Automation" href="/integrations/index">
    Automate backup workflows with scripts and scheduling
  </Card>
</CardGroup>


# CLI Setup
Source: https://docs.xloud.tech/cli-setup

Install and configure the command-line interface for managing Xloud Cloud Platform resources.

## Overview

The Xloud command-line interface gives you full control over every platform service directly from your terminal. With a single authenticated session you can provision instances, manage volumes, configure networks, and automate infrastructure workflows — without opening a browser. The CLI is compatible with all major operating systems and integrates cleanly into CI/CD pipelines and shell scripts.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * Python 3.8 or later installed on your workstation
  * Network access to your Xloud API endpoint
</Note>

***

## Installation

Install the Xloud CLI using the package manager for your operating system.

<Tabs>
  <Tab title="Ubuntu / Debian" icon="server">
    ```bash title="Install on Ubuntu / Debian" theme={null}
    sudo apt update
    sudo apt install -y python3-pip python3-venv
    pip3 install python-openstackclient
    ```

    <Tip>
      Install into a virtual environment to avoid conflicts with system Python packages:
      `python3 -m venv ~/.xloud-cli && source ~/.xloud-cli/bin/activate && pip install python-openstackclient`
    </Tip>
  </Tab>

  <Tab title="CentOS / RHEL" icon="server">
    ```bash title="Install on CentOS / RHEL" theme={null}
    sudo dnf install -y python3-pip
    pip3 install --user python-openstackclient
    ```

    Add the user bin directory to your PATH if it is not already present:

    ```bash title="Update PATH" theme={null}
    echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
    source ~/.bashrc
    ```
  </Tab>

  <Tab title="macOS" icon="monitor">
    ```bash title="Install on macOS (Homebrew)" theme={null}
    brew install python@3
    pip3 install python-openstackclient
    ```

    Alternatively, use `pipx` for an isolated installation:

    ```bash title="Install with pipx" theme={null}
    brew install pipx
    pipx install python-openstackclient
    ```
  </Tab>

  <Tab title="pip (any OS)" icon="package">
    ```bash title="Install via pip" theme={null}
    pip install python-openstackclient
    ```

    <Warning>
      Python 3.8 or later is required. Verify with `python3 --version` before installing.
    </Warning>
  </Tab>
</Tabs>

Confirm the installation completed successfully:

```bash title="Verify installation" theme={null}
openstack --version
```

<Check>The command prints the installed version, e.g., `openstack 6.x.x`.</Check>

***

## Authentication

The CLI authenticates using an RC (credentials) file that sets environment variables for your session. Your administrator provides this file, or you can create it manually.

<Steps>
  <Step title="Obtain your RC file">
    Your administrator provides the RC (credentials) file for your project. This file
    contains the authentication endpoint, project name, and your username.

    Alternatively, create the file manually with the following content:

    ```bash title="openrc.sh" theme={null}
    export OS_AUTH_URL=https://api.<your-domain>:5000/v3
    export OS_PROJECT_NAME=<your-project>
    export OS_PROJECT_DOMAIN_NAME=Default
    export OS_USERNAME=<your-username>
    export OS_USER_DOMAIN_NAME=Default
    export OS_REGION_NAME=RegionOne
    export OS_IDENTITY_API_VERSION=3
    echo "Please enter your Xloud password for project $OS_PROJECT_NAME as user $OS_USERNAME:"
    read -sr OS_PASSWORD_INPUT
    export OS_PASSWORD=$OS_PASSWORD_INPUT
    ```

    <Tip>
      Each project has its own RC file. Ask your administrator for the correct
      `OS_AUTH_URL` and `OS_PROJECT_NAME` values for your environment.
    </Tip>
  </Step>

  <Step title="Source the credentials file">
    Open a terminal and source the file to load your credentials into the
    current shell session:

    ```bash title="Load credentials" theme={null}
    source openrc.sh
    ```

    Enter your Xloud account password when prompted.
  </Step>

  <Step title="Verify authentication" icon="circle-check">
    Confirm the CLI can authenticate successfully:

    ```bash title="Verify token" theme={null}
    openstack token issue
    ```

    A valid response displays a token ID, expiry, and your project details.

    <Check>Authentication is working. Your session is valid for 1 hour by default.</Check>
  </Step>
</Steps>

<Info>
  The RC file exports the following environment variables to configure the CLI client.
  You can also set these manually or store them in a `clouds.yaml` file for multi-cloud management.

  | Variable                  | Purpose                              |
  | ------------------------- | ------------------------------------ |
  | `OS_AUTH_URL`             | Xloud Identity service endpoint      |
  | `OS_PROJECT_NAME`         | Target project for all operations    |
  | `OS_PROJECT_DOMAIN_NAME`  | Project domain (typically `Default`) |
  | `OS_USERNAME`             | Your Xloud account username          |
  | `OS_USER_DOMAIN_NAME`     | User domain (typically `Default`)    |
  | `OS_REGION_NAME`          | Target region for API calls          |
  | `OS_IDENTITY_API_VERSION` | Identity API version (`3`)           |
</Info>

***

## Command Structure

Every CLI command follows the same pattern: resource type, action, and optional flags.

```
openstack <resource> <action> [options] [arguments]
```

<CodeGroup>
  ```bash title="List resources" theme={null}
  # Pattern: openstack <resource> list [--filter value]
  openstack server list --status ACTIVE
  openstack volume list --status available
  openstack network list --internal
  ```

  ```bash title="Create resources" theme={null}
  # Pattern: openstack <resource> create --option value <name>
  openstack server create \
    --flavor m1.medium \
    --image ubuntu-24.04 \
    --network private \
    --key-name my-keypair \
    web-server-01
  ```

  ```bash title="Show / delete resources" theme={null}
  # Pattern: openstack <resource> show <name-or-id>
  openstack server show web-server-01
  openstack server show --format json web-server-01

  # Pattern: openstack <resource> delete <name-or-id>
  openstack server delete web-server-01
  ```
</CodeGroup>

<Tip>
  Most commands accept either the resource **name** or **UUID** as the final argument.
  Use UUIDs in scripts to avoid ambiguity when multiple resources share the same name.
</Tip>

***

## Output Formats

The CLI supports multiple output formats controlled by the `--format` flag.

| Flag             | Output Type                    | Best Used For                               |
| ---------------- | ------------------------------ | ------------------------------------------- |
| `--format table` | Human-readable table (default) | Interactive use, quick inspection           |
| `--format json`  | JSON object or array           | Scripts, parsing, API-like output           |
| `--format yaml`  | YAML document                  | Configuration files, readable serialization |
| `--format csv`   | Comma-separated values         | Spreadsheet import, bulk reporting          |
| `--format value` | Plain values, one per line     | Shell variable assignment, `grep` pipelines |

<CodeGroup>
  ```bash title="JSON output" theme={null}
  openstack server show my-vm --format json
  ```

  ```bash title="Extract a single field" theme={null}
  # Use --format value with -c to extract one column
  openstack server show my-vm --format value -c status
  ```

  ```bash title="CSV for reporting" theme={null}
  openstack server list --format csv --quote all
  ```
</CodeGroup>

***

## Tips and Tricks

<AccordionGroup>
  <Accordion title="Set up shell aliases for common commands" icon="zap">
    Add aliases to your shell profile (`~/.bashrc` or `~/.zshrc`) to speed up repetitive tasks:

    ```bash title="Recommended aliases (~/.bashrc)" theme={null}
    alias osl='openstack server list'
    alias ovl='openstack volume list'
    alias onl='openstack network list'
    alias oil='openstack image list'
    alias oss='openstack server show'
    alias osrc='source ~/openrc.sh'
    ```

    Reload your shell after adding aliases:

    ```bash title="Reload shell config" theme={null}
    source ~/.bashrc
    ```
  </Accordion>

  <Accordion title="Enable bash completion" icon="check-check">
    The CLI ships with a completion script that enables tab-completion for commands,
    resource names, and flags.

    ```bash title="Enable bash completion" theme={null}
    # Add to ~/.bashrc
    eval "$(openstack complete)"
    ```

    ```zsh title="Enable zsh completion" theme={null}
    # Add to ~/.zshrc
    eval "$(openstack complete)"
    ```

    After reloading your shell, press **Tab** after any partial command to see available
    completions.
  </Accordion>

  <Accordion title="Enable debug mode for troubleshooting" icon="bug">
    Pass `--debug` to any command to print the full HTTP request and response, including
    headers and token details. This is useful for diagnosing authentication errors or
    unexpected API responses.

    ```bash title="Debug a failing command" theme={null}
    openstack server list --debug 2>&1 | less
    ```

    <Warning>
      Debug output includes your authentication token. Avoid sharing or storing this
      output in logs or tickets.
    </Warning>
  </Accordion>

  <Accordion title="Manage multiple clouds with clouds.yaml" icon="cloud">
    Instead of sourcing separate RC files, define all your environments in a single
    `clouds.yaml` file stored at `~/.config/openstack/clouds.yaml`:

    ```yaml title="~/.config/openstack/clouds.yaml" theme={null}
    clouds:
      xloud-prod:
        auth:
          auth_url: https://api.<your-domain>:5000/v3
          project_name: production
          username: your-username
          password: your-password
          user_domain_name: Default
          project_domain_name: Default
        region_name: RegionOne
        identity_api_version: 3
      xloud-dev:
        auth:
          auth_url: https://api-dev.<your-domain>:5000/v3
          project_name: development
          username: your-username
          password: your-password
          user_domain_name: Default
          project_domain_name: Default
        region_name: RegionOne
        identity_api_version: 3
    ```

    Switch between environments using `--os-cloud`:

    ```bash title="Use a named cloud profile" theme={null}
    openstack --os-cloud xloud-prod server list
    openstack --os-cloud xloud-dev server list
    ```

    <Tip>
      Set `OS_CLOUD=xloud-prod` in your shell to make a profile the default for all
      commands in that session.
    </Tip>
  </Accordion>
</AccordionGroup>

***

## Service CLI References

Each service has a dedicated CLI reference with complete command syntax, options, and examples.

<CardGroup>
  <Card title="Compute CLI" icon="server" href="/services/compute/cli-reference">
    Instances, flavors, keypairs, server groups, console, and live migration commands.
  </Card>

  <Card title="Block Storage CLI" icon="hard-drive" href="/services/storage/cli-reference">
    Volumes, snapshots, backups, volume types, and cross-project transfers.
  </Card>

  <Card title="Networking CLI" icon="network" href="/services/networking/cli-reference">
    Networks, subnets, routers, floating IPs, security groups, and ports.
  </Card>

  <Card title="Images CLI" icon="image" href="/services/images/cli-reference">
    Upload, download, share, and manage virtual machine images.
  </Card>

  <Card title="Orchestration CLI" icon="layers" href="/services/orchestration/cli-reference">
    Heat stacks, resources, events, outputs, and template validation.
  </Card>

  <Card title="Identity CLI" icon="fingerprint" href="/services/identity/cli-reference">
    Projects, users, roles, groups, and application credentials.
  </Card>

  <Card title="Load Balancer CLI" icon="combine" href="/services/load-balancer/cli-reference">
    Load balancers, listeners, pools, members, and health monitors.
  </Card>

  <Card title="DNS CLI" icon="globe" href="/services/dns/cli-reference">
    Zones, record sets, PTR records, and zone transfers.
  </Card>

  <Card title="Key Manager CLI" icon="key" href="/services/key-manager/cli-reference">
    Secrets, containers, orders, and access control lists.
  </Card>

  <Card title="Object Storage CLI" icon="box" href="/services/object-storage/cli-reference">
    Containers, objects, large object uploads, and temporary URLs.
  </Card>

  <Card title="Kubernetes CLI" icon="container" href="/services/kubernetes/cli-reference">
    Cluster templates, clusters, and node group management.
  </Card>

  <Card title="SDS CLI" icon="database" href="/services/sds/cli-reference">
    Ceph cluster health, pools, OSDs, RBD images, and RGW buckets.
  </Card>
</CardGroup>


# Advanced Configuration
Source: https://docs.xloud.tech/deployment/advanced-config

Per-service config file editor with syntax validation and Git version control

## Overview

When the [Configuration](/deployment/configuration) forms do not expose a specific setting, Advanced Configuration provides a full code editor for individual service configuration files. It supports YAML, INI, and Jinja2 templates with syntax validation and auto-formatting. Integrated Git version control tracks every change.

<Note>
  **Prerequisites**

  * [Configuration](/deployment/configuration) completed with base settings saved
  * Services selected for deployment
  * Understanding of the specific service configuration options being modified
</Note>

***

## Editor Layout

Log in to **XDeploy** (`https://xdeploy.<your-domain>`) on the deployment node using administrative access.

Navigate to **Advanced Configuration** in the left sidebar. The editor interface is organized into three panels.

<CardGroup>
  <Card title="Service Tree" icon="folder">
    Searchable tree of all services organized by category: compute, network, storage, identity, and orchestration. Click a service to browse its configuration files.
  </Card>

  <Card title="Code Editor" icon="terminal">
    Full text editor for YAML, INI, and Jinja2 files. Includes syntax validation, auto-formatting, file preview, multi-tab support, and a cursor position status bar.
  </Card>

  <Card title="File Browser" icon="file-text">
    Lists configuration files for the selected service. Toolbar actions include New File, Upload, Download, and Delete.
  </Card>
</CardGroup>

***

## Editing Configuration Files

<Steps>
  <Step title="Select a service" icon="search">
    Use the Service Tree on the left to browse or search for the service you want to configure. Services are organized by category:

    | Category          | Services                                                    |
    | ----------------- | ----------------------------------------------------------- |
    | **Compute**       | Xloud Compute (nova), Libvirt, Scheduler                    |
    | **Network**       | Xloud Networking (neutron), L3 Agent, DHCP Agent, OVS Agent |
    | **Storage**       | Xloud Block Storage (cinder), Backup                        |
    | **Identity**      | Xloud Identity (keystone)                                   |
    | **Orchestration** | Xloud Orchestration (heat)                                  |
    | **Dashboard**     | Xloud Dashboard                                             |
    | **Monitoring**    | Prometheus, Grafana                                         |

    Click a service name to load its configuration files in the File Browser panel.
  </Step>

  <Step title="Open a configuration file" icon="file-text">
    Select a file from the File Browser to open it in the Code Editor. The editor provides:

    * **Syntax highlighting** for YAML, INI, and Jinja2 formats
    * **Validation** that flags syntax errors before saving
    * **Auto-formatting** to normalize indentation and spacing
    * **Multi-tab editing** to work on multiple files simultaneously
  </Step>

  <Step title="Modify and save" icon="save">
    Make your changes in the editor. The validation engine checks for syntax errors in real time. Click **Save** to persist the changes to the configuration directory.

    <Check>The file is saved to the configuration override directory and will be applied during the next deployment or reconfiguration.</Check>
  </Step>
</Steps>

<Tip>
  Use the **Preview** button to view the final rendered configuration before saving. This is especially useful for Jinja2 templates where the rendered output may differ from the source template.
</Tip>

***

## Git Integration

Advanced Configuration includes built-in version control for tracking configuration changes. Every modification is trackable, reversible, and auditable.

<Steps>
  <Step title="Initialize the repository" icon="folder">
    Click **Initialize Git** to create a version control repository in the configuration directory. This is a one-time operation.
  </Step>

  <Step title="Track changes" icon="eye">
    After initialization, the editor tracks all file modifications. Use **View Diff** to see pending changes before committing them.
  </Step>

  <Step title="Commit changes" icon="save">
    Commit your changes with a descriptive message. The commit history provides a complete audit trail of who changed what and when.
  </Step>

  <Step title="Branch for experiments" icon="copy">
    Create a branch before making experimental changes (e.g., `test-nova-tuning`). If something breaks during testing, switch back to the main branch to restore the working configuration instantly.
  </Step>
</Steps>

Additional Git operations available in the toolbar:

| Operation          | Description                                          |
| ------------------ | ---------------------------------------------------- |
| **Pull**           | Fetch and merge changes from a remote repository     |
| **Push**           | Push local commits to a remote repository            |
| **Branch**         | Create, switch, or delete branches                   |
| **Resolve Issues** | Handle merge conflicts when branch histories diverge |

<Tip>
  Create a branch before making experimental changes. If something breaks, switch back to the main branch to restore working configuration immediately.
</Tip>

***

## How Config Overrides Work

The deployment automation supports per-service configuration overrides. Dropping a configuration file into the service-specific override directory merges your custom settings into the deployed configuration on the next **Reconfigure** operation.

<Tree>
  <Tree.Folder name="etc">
    <Tree.Folder name="xavs">
      <Tree.Folder name="config">
        <Tree.Folder name="nova">
          <Tree.File name="nova.conf" />
        </Tree.Folder>

        <Tree.Folder name="neutron">
          <Tree.File name="neutron.conf" />

          <Tree.File name="ml2_conf.ini" />
        </Tree.Folder>

        <Tree.Folder name="horizon">
          <Tree.File name="local_settings.py" />
        </Tree.Folder>

        <Tree.Folder name="cinder">
          <Tree.File name="cinder.conf" />
        </Tree.Folder>
      </Tree.Folder>
    </Tree.Folder>
  </Tree.Folder>
</Tree>

<Info>
  Override files do not replace the entire service configuration. They are **merged** with the base configuration generated by the deployment automation. Only the specific keys you define in your override file are changed --- all other settings retain their default values.
</Info>

<Warning>
  After modifying configuration overrides, you must run a **Reconfigure** operation from the [Operations](/deployment/operations) tool for changes to take effect. Editing files here does not automatically apply changes to running services.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Operations" icon="play" href="/deployment/operations">
    Deploy or reconfigure services to apply your configuration changes
  </Card>

  <Card title="Configuration" icon="settings" href="/deployment/configuration">
    Return to the guided configuration forms for common deployment settings
  </Card>
</CardGroup>


# Bootstrap
Source: https://docs.xloud.tech/deployment/bootstrap

Server readiness validation and dependency installation for cloud deployment

Bootstrap is the preflight checklist for your server. Before deploying cloud services, every target server must meet hardware requirements and have all software dependencies installed. Bootstrap automates detection, validation, and installation — ensuring a clean foundation for deployment.

<Note>
  **Prerequisites**

  * Physical or virtual server with root or sudo access
  * Internet connection (required for package installation)
  * Ubuntu 24.04 LTS (XOS) recommended as the base operating system
</Note>

***

## Bootstrap Tabs

The Bootstrap module is organized into four tabs. Navigate between tabs to inspect specific subsystems or install dependencies.

<Tabs>
  <Tab title="Overview" icon="monitor">
    Quick snapshot of server health without modifying anything. Four information cards are displayed:

    <CardGroup>
      <Card title="Operating System" icon="monitor">
        Linux distribution, kernel version, package manager, and XDeploy mode (AIO or Multi-Node)
      </Card>

      <Card title="System Status" icon="activity">
        Uptime, CPU load average and usage percentage, memory used vs total
      </Card>

      <Card title="Storage Overview" icon="hard-drive">
        Root filesystem type and free space, total disk count, storage type (SSD/HDD)
      </Card>

      <Card title="Network Overview" icon="globe">
        Interface count, internet connectivity, primary IP address, DNS resolution status
      </Card>
    </CardGroup>

    Quick Actions at the bottom: **Run Hardware Checks** and **Install All Information**.
  </Tab>

  <Tab title="Hardware" icon="cpu">
    Runs automated hardware requirement checks and shows detailed specifications. Each check shows a green checkmark (pass) or red indicator (fail).

    **5 Requirement Checks:**

    | Check                           | Minimum                    | Why                                                   |
    | ------------------------------- | -------------------------- | ----------------------------------------------------- |
    | Root Filesystem                 | 100 GB                     | Services, Docker images, and logs                     |
    | Network Interfaces              | 2                          | Management NIC + provider/external NIC                |
    | Additional Disk or Volume Group | Optional                   | Required for persistent block storage volumes         |
    | CPU Cores                       | 4 (16+ production)         | Control plane needs \~2 cores, compute needs the rest |
    | RAM                             | 8 GB (32-64 GB production) | Each service container uses 200-500 MB                |

    <Info>
      Servers below minimums are flagged but not blocked — you can proceed for development and testing.
    </Info>

    **Hardware Details** (expandable sections):

    <AccordionGroup>
      <Accordion title="CPU Information" icon="cpu">
        Model name, physical cores, logical threads, architecture (x86\_64/ARM), and hardware virtualization support (VT-x/AMD-V). Virtualization is required for running VMs.
      </Accordion>

      <Accordion title="Memory Information" icon="bar-chart">
        Total installed, currently available, in use, and swap size.
      </Accordion>

      <Accordion title="Storage Information" icon="hard-drive">
        Root filesystem type (ext4, xfs), all block devices (sda, nvme0n1, etc.), LVM volume groups, and mount points. Identifies disks available for block storage.
      </Accordion>

      <Accordion title="Network Information" icon="globe">
        Active interfaces, link types (ethernet, bond, bridge), all IP addresses, and default gateway.
      </Accordion>
    </AccordionGroup>
  </Tab>

  <Tab title="Dependencies" icon="package">
    The Dependencies tab validates and installs 7 software groups required for deployment. Each group addresses a specific layer of the deployment stack.

    <Warning>
      ALL 7 dependency groups must pass before proceeding to the next stage. If even one group fails, deployment will break — sometimes 45 minutes in, when a missing library causes a playbook task to fail.
    </Warning>

    | Group                    | What It Installs                                                  | Why                                                                                               |
    | ------------------------ | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
    | **Python Dependencies**  | git, python3-dev, libffi-dev, gcc, libssl-dev, python3-venv       | Ansible modules and Python libraries require compilation of C extensions                          |
    | **System Tools**         | Monitoring, networking, storage, and virtualization CLI utilities | Ansible playbooks detect interfaces, disks, and containers using these tools                      |
    | **Python Environment**   | Isolated Python virtual environment                               | Prevents version conflicts between system Python packages and deployment tooling                  |
    | **XDeploy Dependencies** | Ansible + XAVS Deployment Automation                              | Core deployment engine and service playbooks that provision every cloud component                 |
    | **CLI Clients**          | nova, neutron, cinder, keystone, swift, octavia, magnum, barbican | Command-line tools for post-deployment management and troubleshooting                             |
    | **Configuration Setup**  | `/etc/xavs` directory with correct ownership and permissions      | Scaffolds the configuration directory structure: globals, inventory, and password files           |
    | **Password Generation**  | Auto-generated secure random passwords                            | Produces 50+ internal passwords for databases, service accounts, and message queue authentication |

    <Tip>
      Use **Install All** to install every dependency group in sequence with a single action. Once a group passes validation, its individual Install button is disabled to prevent accidental re-installation.
    </Tip>
  </Tab>

  <Tab title="Logs" icon="terminal">
    Every Bootstrap action is recorded in a live scrolling terminal log with timestamps. Use this log to diagnose failures — it captures the full output of every package installation, validation check, and error message. The log persists across page refreshes, so you can return to it at any time during the session.
  </Tab>
</Tabs>

***

## Validation

After all dependency groups are installed, verify readiness:

<Steps>
  <Step title="Check All Groups Pass" icon="circle-check">
    Every dependency group in the Bootstrap interface displays a green status indicator. No group should show a warning or error state.
  </Step>

  <Step title="Review the Overview Tab" icon="eye">
    Return to the Overview tab and confirm that hardware values (CPU, RAM, disk, network interfaces) meet or exceed the minimums listed above.
  </Step>

  <Step title="Verify Internet Connectivity" icon="globe">
    The Network tab must show successful internet connectivity and DNS resolution. Offline deployments require pre-staged package archives.
  </Step>
</Steps>

<Check>
  All 7 dependency groups installed and passing — the server is ready for the next stage.
</Check>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Python Dependencies group fails with compilation errors" icon="alert-triangle">
    **Cause**: Missing build tools or development headers on the base operating system.

    **Resolution**: Verify that `build-essential`, `python3-dev`, and `libffi-dev` are available in the system package repository. On XOS, these packages are pre-installed. On other Ubuntu 24.04 installations, run:

    ```bash title="Install build prerequisites" theme={null}
    sudo apt update && sudo apt install -y build-essential python3-dev libffi-dev libssl-dev
    ```

    Retry the Python Dependencies installation after installing the missing packages.
  </Accordion>

  <Accordion title="Network Overview shows no internet connectivity" icon="alert-triangle">
    **Cause**: DNS resolution failure or firewall blocking outbound connections.

    **Resolution**: Verify that `/etc/resolv.conf` contains valid nameservers and that outbound HTTPS (port 443) is not blocked by a firewall or proxy. Bootstrap requires internet access to download packages from Ubuntu and Python package repositories.

    ```bash title="Test DNS and connectivity" theme={null}
    nslookup archive.ubuntu.com
    curl -s https://pypi.org/simple/ | head -1
    ```
  </Accordion>

  <Accordion title="Dependency installation hangs or times out" icon="clock">
    **Cause**: Slow or unreliable network connection to package repositories.

    **Resolution**: Check network throughput between the server and the internet. If using a corporate proxy, ensure the proxy is configured in the system environment variables (`http_proxy`, `https_proxy`). Consider using a local package mirror for air-gapped or bandwidth-constrained environments.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Hosts" icon="server" href="/deployment/hosts">
    Define your cluster inventory and establish SSH connectivity to all nodes
  </Card>

  <Card title="Configuration" icon="settings" href="/deployment/configuration">
    Set up networking, storage backends, and enable optional cloud services
  </Card>
</CardGroup>


# Cloud Fleet
Source: https://docs.xloud.tech/deployment/cloud-fleet

Interactive topology map for visualizing your entire cloud infrastructure

Cloud Fleet provides a visual bird's-eye map of your entire cluster. Every host appears as a node in an interactive force-directed graph, colored by health status and connected by network planes. Click any host to inspect live metadata. Use Cloud Fleet to understand relationships between servers, validate high availability placement, and explain architecture to stakeholders.

<Note>
  **Prerequisites**

  * All cluster hosts configured in [Hosts](/deployment/hosts)
  * At least one successful deployment via [Operations](/deployment/operations)
  * SSH connectivity established to all nodes
</Note>

***

## Fleet Dashboard

The top statistics bar provides an immediate summary of your entire infrastructure. These values update in real time as Cloud Fleet queries the cluster.

| Metric             | Description                                                              |
| ------------------ | ------------------------------------------------------------------------ |
| **Hosts Online**   | Total number of reachable hosts across all roles                         |
| **Role Groups**    | Count of distinct role groups (controller, compute, storage, monitoring) |
| **Active Links**   | Number of network pathways between hosts currently carrying traffic      |
| **Overall Health** | Aggregate cluster status: Healthy, Degraded, or Critical                 |
| **Cluster Name**   | The configured cluster identifier from your deployment                   |
| **Region**         | Geographic or logical region label assigned during configuration         |
| **Version**        | Currently deployed XAVS platform version                                 |

<Info>
  A cluster is **Healthy** when all hosts are reachable and all services are running. **Degraded** indicates at least one host or service is in a warning state. **Critical** means one or more hosts are unreachable or core services have failed.
</Info>

***

## Topology View

The topology view is a three-panel layout that displays infrastructure relationships, the interactive map, and detailed metadata simultaneously.

<CardGroup>
  <Card title="Context Sidebar" icon="list">
    Displays role distribution with host counts per role, available network planes (management, storage, external, octavia), and a searchable, filterable host list. Use the search bar to locate specific hosts by name or IP address.
  </Card>

  <Card title="Interactive Map" icon="globe">
    SVG force-directed topology graph with drag-to-pan, scroll-to-zoom, and dedicated controls for zoom in, zoom out, fit-to-screen, and reset. Hosts are rendered as color-coded nodes based on health status. Network connections between hosts appear as color-coded lines representing their respective network planes.
  </Card>

  <Card title="Detail Inspector" icon="eye">
    Click any host node or network connection to populate the inspector panel. Displays: resource type, hostname, metadata (IP addresses, role assignments, installed services), health signals, dependency chain, and related network paths.
  </Card>
</CardGroup>

### Health Status Colors

Each host node in the topology graph is colored by its current health state:

| Color  | Status   | Meaning                                                              |
| ------ | -------- | -------------------------------------------------------------------- |
| Green  | Healthy  | All services running, host reachable, no active alerts               |
| Yellow | Warning  | One or more services degraded, resource usage approaching thresholds |
| Red    | Critical | Host unreachable, core service failure, or active critical alert     |
| Gray   | Standby  | Host configured but not yet deployed or intentionally powered off    |

***

## Network Planes

Cloud Fleet visualizes the four distinct network planes that connect your cluster nodes. Each plane is rendered as a separate set of color-coded connections on the topology map.

| Plane      | Color  | Purpose                                                               |
| ---------- | ------ | --------------------------------------------------------------------- |
| Management | Blue   | API calls, SSH access, Ansible communication, inter-service messaging |
| Storage    | Green  | Xloud Distributed Storage replication traffic and client I/O          |
| External   | Orange | Virtual machine internet access, floating IP routing                  |
| Octavia    | Purple | Load balancer management traffic and health monitor probes            |

<Tip>
  Export the topology as SVG or PNG for architecture documentation and stakeholder presentations. Use the export controls in the top-right corner of the interactive map.
</Tip>

<Warning>
  If a network plane appears disconnected between two hosts, verify the physical cabling, VLAN tagging, and interface assignment in [Hosts](/deployment/hosts). A missing storage plane connection degrades replication performance and can lead to data unavailability.
</Warning>

***

## Connection Inventory

Below the topology map, a tabular inventory lists every link between hosts in the cluster. Use this table to audit connectivity at a glance.

| Column          | Description                                                                       |
| --------------- | --------------------------------------------------------------------------------- |
| **Source**      | Originating host name and IP address                                              |
| **Destination** | Target host name and IP address                                                   |
| **Plane**       | Network plane this connection belongs to (management, storage, external, octavia) |
| **Status**      | Connection health: Active, Degraded, or Down                                      |
| **Latency**     | Measured round-trip time between the two hosts                                    |

<Info>
  The connection inventory updates automatically as the topology map refreshes. Filter by plane type or status to focus on specific segments of your infrastructure.
</Info>

***

## Next Steps

<CardGroup>
  <Card title="Operations" icon="play" href="/deployment/operations">
    Deploy, upgrade, reconfigure, and manage cloud services across the cluster
  </Card>

  <Card title="XSDS Storage" icon="database" href="/deployment/xsds">
    Configure distributed storage tiers and manage the storage cluster
  </Card>
</CardGroup>


# Cluster License
Source: https://docs.xloud.tech/deployment/cluster-license

License your deployed cloud cluster with hardware-bound activation keys

The Cluster License activates your deployed cloud infrastructure. It collects hardware fingerprints from all nodes, generates encrypted license requests, and distributes signed licenses across the cluster.

<Note>
  **Prerequisites**

  * Cloud infrastructure deployed and operational via [Operations](/deployment/operations)
  * All cluster nodes accessible via SSH from the management node
  * Access to [license.xloud.tech](https://license.xloud.tech) for license file submission
</Note>

<Warning>
  This licenses the **cloud cluster itself** — the deployed infrastructure running your workloads. To license the XDeploy management tool, use [XDeploy Key](/deployment/xdeploy-key) instead.
</Warning>

***

## License Dashboard

The License Dashboard presents two panels: an overview of the current license state and a per-node status table.

<CardGroup>
  <Card title="License Overview" icon="shield-check">
    Four summary indicators at the top of the dashboard:

    | Indicator          | Values                                 |
    | ------------------ | -------------------------------------- |
    | **License Status** | Active, Expired, or None               |
    | **Days Remaining** | Countdown to license expiration        |
    | **Licensed Nodes** | Number of nodes covered by the license |
    | **License Type**   | Trial or Enterprise                    |

    Below the indicators, a details section displays the customer name, issue and expiry dates, and enabled features. Use the **Verify** button to re-validate license integrity against the current hardware state.
  </Card>

  <Card title="Cluster Nodes" icon="server">
    A per-node table showing which hosts in the cluster have the license installed and verified. Each row displays:

    | Column                | Description                                                  |
    | --------------------- | ------------------------------------------------------------ |
    | **Hostname**          | The node's configured hostname                               |
    | **IP Address**        | Management network IP                                        |
    | **License Installed** | Whether the signed license file is present on this node      |
    | **Verified**          | Whether the license passes integrity validation on this node |
  </Card>
</CardGroup>

<Info>
  The **Verify** button performs a live integrity check. It re-reads the hardware fingerprint and compares it against the signed license. If hardware has changed (e.g., a node was replaced), the verification will fail and a new license request is required.
</Info>

***

## Activation Process

<Steps>
  <Step title="Request License" icon="download">
    Click **Request License** to open the license generation modal. The tool scans all cluster nodes over SSH and collects hardware fingerprints including server UUID and Machine ID.

    A preview displays before generating the request:

    | Field           | Description                                 |
    | --------------- | ------------------------------------------- |
    | **Total Nodes** | Number of hosts discovered in the cluster   |
    | **CPU Sockets** | Total physical CPU sockets across all nodes |
    | **Total Cores** | Aggregate CPU core count across the cluster |

    The tool generates an encrypted `.xlic` request file. Download this file to your local machine.

    <Tip>
      The `.xlic` request file contains only hardware fingerprints — no sensitive data such as passwords, IP addresses, or configuration details are included.
    </Tip>
  </Step>

  <Step title="Submit Request" icon="upload">
    Upload the `.xlic` request file to [license.xloud.tech](https://license.xloud.tech). After validation, you receive a signed license file in return.

    <Info>
      License processing is typically immediate for existing customers with active support agreements. New customers may require manual approval.
    </Info>
  </Step>

  <Step title="Upload License" icon="file-text">
    Return to the Cluster License page in XDeploy. Drag and drop the signed `.xlic` file into the upload area. The tool validates the digital signature and confirms the license details before installation.
  </Step>

  <Step title="Activate All Nodes" icon="circle-check">
    Click **Activate** to distribute the license to all Dashboard nodes via automation. The tool connects to each node over SSH, installs the license file, and verifies integrity.

    <Check>
      License installed and verified on all cluster nodes. The cluster is fully activated.
    </Check>
  </Step>
</Steps>

***

## License Features

The license controls which features are available in the Xloud Dashboard and enforces node count limits. Without a valid license, the cloud infrastructure runs with restricted functionality.

| Aspect                 | Licensed                             | Unlicensed                  |
| ---------------------- | ------------------------------------ | --------------------------- |
| **Node Count**         | Up to the licensed maximum           | Limited to evaluation count |
| **Dashboard Features** | All enabled features accessible      | Core features only          |
| **Support Access**     | Full support based on agreement tier | Community support only      |
| **Updates**            | Access to platform updates           | Updates restricted          |

<Warning>
  An expired license does not shut down running workloads, but it prevents new deployments, configuration changes, and Dashboard access to premium features. Renew before expiration to avoid operational disruptions.
</Warning>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Verification fails after hardware replacement" icon="alert-triangle">
    **Cause**: The hardware fingerprint of the replacement node does not match the original fingerprint in the signed license.

    **Resolution**: Generate a new license request that includes the updated hardware fingerprint. Upload the new request to [license.xloud.tech](https://license.xloud.tech) and install the new signed license file.
  </Accordion>

  <Accordion title="License upload rejected with signature error" icon="alert-triangle">
    **Cause**: The `.xlic` file was corrupted during download or transfer, or the file was modified after signing.

    **Resolution**: Re-download the signed license file from the license portal. Ensure no text editors or transfer tools modify the file content (binary-safe transfer required). Upload the file again.
  </Accordion>

  <Accordion title="Activation fails on specific nodes" icon="alert-triangle">
    **Cause**: SSH connectivity issues between the management node and the target node, or insufficient permissions on the target node.

    **Resolution**: Verify SSH connectivity from the management node to the failing node. Confirm that the deployment user has write access to the license directory. Check the [Hosts](/deployment/hosts) configuration for SSH credential issues.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="XDeploy Key" icon="fingerprint" href="/deployment/xdeploy-key">
    Activate the XDeploy management platform with a hardware-bound license
  </Card>

  <Card title="Management" icon="users" href="/deployment/management">
    Create users, projects, and quotas for your activated cluster
  </Card>
</CardGroup>


# Configuration
Source: https://docs.xloud.tech/deployment/configuration

Configure networking, storage, monitoring, and security settings for your cloud deployment

## Overview

Configuration is the main settings interface for your entire cloud. All settings are saved to `globals.yml` --- the master deployment configuration file that controls every aspect of your environment. Instead of editing YAML manually, XDeploy provides forms, dropdowns, toggles, and validation across eight configuration tabs.

<Note>
  **Prerequisites**

  * Bootstrap completed on the deployment server
  * Hosts configured with verified SSH access
  * Network interface names identified on all target servers
  * Virtual IP (VIP) addresses planned for your management subnet
</Note>

***

## Configuration Tabs

Log in to **XDeploy** (`https://xdeploy.<your-domain>`) on the deployment node using administrative access.

Navigate to **Configuration** in the left sidebar. The configuration interface is organized into eight tabs, each handling a distinct area of your deployment settings.

<Tabs>
  <Tab title="XAVS Settings" icon="settings">
    The primary settings tab covering networking, domains, and TLS configuration. These settings form the foundation of your entire deployment.

    ***

    ### Network Configuration

    | Field                                | Required | Description                                                                                                           |
    | ------------------------------------ | -------- | --------------------------------------------------------------------------------------------------------------------- |
    | **API/Management Network Interface** | Yes      | Select the NIC used for management traffic. Auto-fills the Internal VIP field based on the selected interface subnet. |
    | **Internal VIP Address**             | Yes      | Virtual IP for internal API calls. Floats between controllers via HAProxy for high availability.                      |
    | **External VIP Address**             | No       | Public-facing VIP for external access and the Xloud Dashboard. Required only if external access is needed.            |

    <Warning>
      The Internal VIP is the single most important setting in your deployment. If it is wrong, nothing works. Ensure it is an **unused** IP address on the management subnet that does not conflict with any existing host.
    </Warning>

    ***

    ### Domain Configuration

    | Field                   | Description                                                                                           |
    | ----------------------- | ----------------------------------------------------------------------------------------------------- |
    | **Enable Domain Setup** | Toggle to configure FQDNs instead of raw IP addresses for service endpoints                           |
    | **Internal FQDN**       | Hostname for internal API endpoints (e.g., `xavs-int.example.com`). Must resolve to the Internal VIP. |
    | **External FQDN**       | Hostname for external access (e.g., `xavs.example.com`). Must resolve to the External VIP.            |

    ***

    ### TLS/SSL Configuration

    Visible when Domain Setup is enabled. Configure certificate-based encryption for API endpoints.

    | Field                | Description                                                                                           |
    | -------------------- | ----------------------------------------------------------------------------------------------------- |
    | **Internal API TLS** | Toggle to enable TLS for internal service communication. Upload the certificate and private key.      |
    | **External API TLS** | Toggle to enable TLS for the public-facing Dashboard and API. Upload the certificate and private key. |

    Certificates are saved to `/etc/xavs/certificates/` on the deployment node and distributed to all controllers during deployment.

    <Tip>
      For production deployments, always enable TLS on external endpoints. Internal TLS is recommended when the management network is shared or untrusted.
    </Tip>
  </Tab>

  <Tab title="Network" icon="network">
    Configures the Xloud Networking service --- interfaces, provider networks, and optional networking features.

    ***

    ### External Interface

    | Field                          | Description                                                                          |
    | ------------------------------ | ------------------------------------------------------------------------------------ |
    | **External Interface Mode**    | **Single** (one NIC for all external traffic) or **Multiple** (per-host NIC mapping) |
    | **Neutron External Interface** | The NIC used for provider networks and external VM connectivity                      |

    <Warning>
      The selected external interface **loses its current IP address** after deployment. It is enslaved to a network bridge for provider traffic. Ensure this NIC name exists on **all hosts** that require external connectivity.
    </Warning>

    ***

    ### Feature Toggles

    | Feature               | Default | Purpose                                                            |
    | --------------------- | ------- | ------------------------------------------------------------------ |
    | **Provider Networks** | ON      | Direct VM connection to physical networks, floating IP support     |
    | **VPNaaS**            | OFF     | Site-to-site VPN tunnels between tenant networks                   |
    | **QoS**               | OFF     | Bandwidth limits, DSCP marking, minimum rate guarantees            |
    | **Trunk**             | OFF     | 802.1Q VLAN trunking on VM ports                                   |
    | **Agent HA**          | OFF     | High availability for L3 and DHCP agents across multiple nodes     |
    | **SR-IOV**            | OFF     | Hardware-accelerated networking via Single Root I/O Virtualization |

    <Info>
      Feature toggles marked OFF are premium or specialized capabilities. Enable them based on your workload requirements. Each can be activated at any time via reconfiguration.
    </Info>
  </Tab>

  <Tab title="Storage" icon="hard-drive">
    Configures Xloud Block Storage --- the persistent volume service for VM disks.

    ***

    ### Service Toggles

    | Field                           | Description                                                                        |
    | ------------------------------- | ---------------------------------------------------------------------------------- |
    | **Enable Block Storage**        | Master switch for the volume service. Disabling removes all storage functionality. |
    | **Enable Block Storage Backup** | Enables volume backup and restore capabilities                                     |
    | **Storage Backend Type**        | **Single** (one backend) or **Multiple** (tiered storage with multiple backends)   |

    ***

    ### Backend Options

    | Backend          | Description                                     | Best For                                    |
    | ---------------- | ----------------------------------------------- | ------------------------------------------- |
    | **LVM**          | Local disk volumes using Logical Volume Manager | Single-node deployments, testing            |
    | **Ceph RBD**     | Distributed replicated block storage            | Production, high availability (recommended) |
    | **NFS**          | Network file share as a volume backend          | NAS appliance integration                   |
    | **iSCSI**        | Block storage over the network                  | Enterprise SAN arrays                       |
    | **VMware VMDK**  | VMware datastore volumes                        | VMware-integrated environments              |
    | **Pure Storage** | FlashArray via iSCSI, Fibre Channel, or NVMe    | High-performance enterprise storage         |

    <Tip>
      For production multi-node deployments, Xloud recommends **Ceph RBD**. It provides data replication across nodes, eliminating single points of failure for VM disks.
    </Tip>
  </Tab>

  <Tab title="Load Balancer" icon="network">
    Configures the Xloud Load Balancer service for distributing network traffic across backend instances.

    ***

    ### Enable Load Balancer

    | Field                    | Description                                                             |
    | ------------------------ | ----------------------------------------------------------------------- |
    | **Enable Load Balancer** | Master switch to enable the load balancer service across the deployment |

    <Warning>
      Enabling the load balancer service deploys additional infrastructure components including controller agents and amphora VMs. Ensure sufficient compute capacity before enabling.
    </Warning>

    ***

    ### Controller Interface

    | Field                               | Description                                                                                                         |
    | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
    | **LB Management Network Interface** | Select the network interface used for load balancer management traffic between the controller and amphora instances |

    ***

    ### Amphora Network Configuration

    Configure the dedicated management network used for communication between the load balancer controller and amphora instances.

    | Field                | Description                                                     |
    | -------------------- | --------------------------------------------------------------- |
    | **Network Name**     | Name for the load balancer management network                   |
    | **Network CIDR**     | Subnet range for the management network (e.g., `172.24.0.0/24`) |
    | **Physical Network** | The underlying physical network for the management VLAN         |
    | **VLAN ID**          | VLAN tag for isolating load balancer management traffic         |
    | **IP Pool Start**    | First IP address in the allocation pool for amphora instances   |
    | **IP Pool End**      | Last IP address in the allocation pool for amphora instances    |

    ***

    ### Amphora Flavor Configuration

    Define the compute resources allocated to each amphora instance. Amphora instances are lightweight VMs that perform the actual load balancing.

    | Field     | Default | Description                                 |
    | --------- | ------- | ------------------------------------------- |
    | **vCPUs** | 2       | Number of virtual CPUs per amphora instance |
    | **RAM**   | 4096 MB | Memory allocation per amphora instance      |
    | **Disk**  | 20 GB   | Root disk size per amphora instance         |

    ***

    ### Topology

    | Option              | Description                                                                                  |
    | ------------------- | -------------------------------------------------------------------------------------------- |
    | **SINGLE**          | One amphora instance per load balancer. Suitable for non-critical workloads.                 |
    | **ACTIVE\_STANDBY** | Two amphora instances per load balancer with automatic failover. Recommended for production. |

    ***

    ### Certificate Configuration

    Configure the certificate authority parameters used for secure communication between the load balancer controller and amphora instances.

    | Field            | Description                                                 |
    | ---------------- | ----------------------------------------------------------- |
    | **Country**      | Two-letter country code for the CA certificate (e.g., `US`) |
    | **State**        | State or province for the CA certificate                    |
    | **Organization** | Organization name for the CA certificate                    |
    | **Org Unit**     | Organizational unit for the CA certificate                  |

    <Tip>
      The ACTIVE\_STANDBY topology provides high availability for load balancers at the cost of double the amphora resources. For production deployments, always use ACTIVE\_STANDBY.
    </Tip>
  </Tab>

  <Tab title="Monitoring" icon="bar-chart">
    Configure observability, logging, and security monitoring for your cloud environment.

    | Feature                    | Default | Description                                                                                                                     |
    | -------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------- |
    | **Enable Prometheus**      | OFF     | Metrics collection from all cloud services and infrastructure                                                                   |
    | **Enable Grafana**         | OFF     | Visual dashboards for metrics visualization and analysis                                                                        |
    | **Enable Central Logging** | OFF     | Aggregated log search and analysis across all nodes                                                                             |
    | **Enable Security Suite**  | OFF     | Integrated SIEM, vulnerability scanning, OS hardening, and security posture dashboard (visible when Central Logging is enabled) |

    ***

    ### CIS Compliance Level

    Visible when Security Suite is enabled. Select the compliance benchmark level:

    | Level       | Description                                                                    |
    | ----------- | ------------------------------------------------------------------------------ |
    | **Level 1** | Essential security controls with minimal performance impact                    |
    | **Level 2** | Extended controls for high-security environments                               |
    | **STIG**    | Defense-grade hardening based on DISA Security Technical Implementation Guides |
    | **All**     | Apply all available compliance benchmarks                                      |

    ***

    ### Alert Configuration

    Visible when Prometheus is enabled. Configure alert delivery channels.

    | Field           | Description                                    |
    | --------------- | ---------------------------------------------- |
    | **SMTP Server** | Mail server for email alert delivery           |
    | **Alert Email** | Recipient address for alert notifications      |
    | **Webhook URL** | HTTP endpoint for webhook-based alert delivery |

    Each channel includes a **Test** button to verify connectivity before deployment.

    <Tip>
      Configure at least one alert channel before deploying to production. Without alerts, critical infrastructure issues may go undetected.
    </Tip>
  </Tab>

  <Tab title="Advance Features" icon="layers">
    Enable or disable premium and specialized services. Each toggle activates the corresponding service and its dependencies during deployment.

    | Feature                                 | Default | Description                                                                                                                                          |
    | --------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
    | **Enable KMS**                          | OFF     | When enabled, the Xloud Key Management service (Barbican) is deployed for secret storage, encryption keys, and certificate management                |
    | **Enable Host HA**                      | OFF     | Enable high availability for hosts. Automatically detects and recovers from compute node failures by evacuating affected instances to healthy nodes. |
    | **Enable Dynamic Cluster Optimization** | OFF     | Enable the dynamic cluster optimization service for automated workload balancing, resource consolidation, and thermal management across the cluster  |
    | **Enable DB Backup Utility**            | OFF     | Enable the database backup utility for scheduled and on-demand backups of cloud service databases                                                    |
    | **Enable ProxySQL**                     | OFF     | Enable ProxySQL for database load balancing and query routing across multiple database replicas                                                      |
    | **Enable Disk Encryption**              | OFF     | Enable disk encryption using key management for volume-level encryption at rest. Requires KMS to be enabled first.                                   |
    | **Enable Insecure Registry**            | OFF     | Enable custom Docker registry configuration (`docker-registry:4000`) for internal container image distribution                                       |

    <Warning>
      **Enable Disk Encryption** depends on **Enable KMS**. If KMS is not enabled, the disk encryption toggle has no effect. Enable KMS first, then enable disk encryption.
    </Warning>

    <Info>
      These are premium or specialized capabilities that are disabled by default. Enable them based on your workload requirements. Each feature can be activated at any time via reconfiguration and redeployment.
    </Info>
  </Tab>

  <Tab title="Custom Config" icon="terminal">
    A raw YAML editor for deployment settings not covered by the other tabs. Content entered here is appended directly to `globals.yml` during deployment.

    | Field                    | Description                                                       |
    | ------------------------ | ----------------------------------------------------------------- |
    | **Custom Configuration** | Raw YAML key-value pairs appended to the deployment configuration |
    | **Documentation**        | Free-text area for comments and notes about custom settings       |

    <Tip>
      Use Custom Configuration as an escape hatch. There are hundreds of deployment variables available in the underlying automation --- the other tabs cover the most common ones. Consult the [xavs-ansible variable reference](https://docs.xloud.tech) for the full list.
    </Tip>

    ```yaml title="Example custom configuration" theme={null}
    nova_compute_virt_type: "kvm"
    neutron_plugin_agent: "openvswitch"
    haproxy_max_connections: 40000
    ```
  </Tab>

  <Tab title="Logs" icon="clock">
    View the history of configuration changes --- what was saved, when it was saved, and what changed between versions.

    XDeploy automatically backs up the last **5 versions** of your configuration. If a configuration change causes issues, you can review previous versions to identify what changed.

    | Column        | Description                               |
    | ------------- | ----------------------------------------- |
    | **Timestamp** | Date and time the configuration was saved |
    | **Changes**   | Summary of fields that were modified      |
    | **Version**   | Sequential version number for tracking    |
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Container Images" icon="package" href="/deployment/images">
    Pull or extract container images required for deploying cloud services
  </Card>

  <Card title="Operations" icon="play" href="/deployment/operations">
    Deploy, upgrade, reconfigure, and manage cloud services across your cluster
  </Card>
</CardGroup>


# Hosts
Source: https://docs.xloud.tech/deployment/hosts

Build your cluster inventory, configure SSH access, and manage target nodes

## Overview

Hosts is where you build your cluster inventory --- the list of servers that form your cloud. Define which servers exist, their IP addresses, assign roles (controller, compute, storage), and set up SSH for remote configuration. Every server that participates in your cloud must be registered here before deployment can proceed.

<Note>
  **Prerequisites**

  * Bootstrap completed on the deployment server
  * IP addresses of all target servers
  * SSH credentials (username and password) for all target servers
  * Network connectivity between the deployment node and every target server
</Note>

***

## Deployment Modes

<CardGroup>
  <Card title="All-in-One (AIO)" icon="server">
    Everything on **one server**. For labs, demos, and testing environments. XDeploy auto-detects the local hostname and IP address, then assigns all roles to localhost. No SSH configuration is required --- the deployment node is also the target.
  </Card>

  <Card title="Multi-Node Cluster" icon="network">
    Production setup with **separate servers** for different roles. Supports high availability, horizontal scaling, and workload isolation. Each server is assigned one or more roles and connected via SSH for remote configuration management.
  </Card>
</CardGroup>

***

## Node Roles

Every server in your cluster is assigned one or more roles that determine which services run on it. The following table describes each role and the services it hosts.

| Role                | Purpose               | Services                                                                                                |
| ------------------- | --------------------- | ------------------------------------------------------------------------------------------------------- |
| **Controller**      | Cloud control plane   | Xloud Identity, Xloud Dashboard, Xloud Compute API, Xloud Networking Server, MariaDB, RabbitMQ, HAProxy |
| **Network**         | Network agents        | L3 Router Agent, DHCP Agent (often combined with Controller)                                            |
| **Compute**         | Virtual machine hosts | Xloud Compute, Libvirt --- where VMs run                                                                |
| **Storage**         | Persistent volumes    | Xloud Block Storage volumes --- where VM disk data lives                                                |
| **Monitoring**      | Observability stack   | Prometheus, Grafana, logging services                                                                   |
| **XAVS-Deployment** | Ansible controller    | Runs deployment playbooks (usually the same server as Controller)                                       |
| **XSDS-Bootstrap**  | First storage node    | Bootstraps the Xloud Distributed Storage cluster                                                        |
| **XSDS**            | Storage nodes         | Additional Xloud Distributed Storage nodes                                                              |

<Info>
  A single server can hold multiple roles. In smaller deployments, it is common to combine Controller, Network, and XAVS-Deployment on the same server.
</Info>

***

## Hosts Tabs

The Hosts module is organized into four tabs. Each tab handles a distinct part of the cluster inventory and connectivity workflow.

<Tabs>
  <Tab title="Hosts" icon="server">
    The Hosts tab is where you register servers into the cluster inventory. Choose your deployment mode first (AIO or Multi-Node), then add hosts using one of three methods.

    Three ways to add hosts:

    <AccordionGroup>
      <Accordion title="Single Add" icon="plus">
        Add one host at a time through the form. Enter hostname (e.g., xd1.example), IP address, SSH port (default 22), then select roles using the toggle buttons: Controller, Network, Compute, Storage, Monitoring, XAVS-Deployment, XSDS-Bootstrap, XSDS.

        Click **Ping** to verify connectivity before adding.
      </Accordion>

      <Accordion title="Bulk Paste" icon="copy">
        Paste multiple hosts, one per line: `hostname,ip,roles` (roles separated by `|` or `;`).

        ```text title="Example" theme={null}
        ctrl-01,192.168.1.10,controller|network|xavs-deployment
        compute-01,192.168.1.11,compute
        compute-02,192.168.1.12,compute|storage
        ```

        Click **Preview** to verify, **Ping All** to check connectivity, then **Commit** to add.
      </Accordion>

      <Accordion title="Import File" icon="upload">
        Upload a JSON or CSV file with host definitions. Download sample files for the correct format. Useful for replicating host configs across environments.
      </Accordion>
    </AccordionGroup>

    **Network Scanner**: Enter a subnet (e.g., `192.168.1.0/24`) and click **Scan Network** to discover responsive hosts automatically.

    **Node Inventory Table**: Shows all added hosts with columns for Hostname, IP, Port, Roles (as colored badges), Ping status, SSH status. Searchable and paginated with **Ping All** and **SSH Check All** buttons.

    <Tip>The inventory table generates the Ansible inventory file automatically — you never need to manually edit the nodes file.</Tip>
  </Tab>

  <Tab title="SSH Access" icon="key">
    SSH connectivity is required for XDeploy to remotely configure and deploy services on every node. The SSH Access tab handles key generation, distribution, and connectivity verification.

    <Steps>
      <Step title="Select hosts" icon="users">
        Select which hosts to configure SSH for. Quick-select buttons: **ALL**, **CTRL**, **NET**, **COMP**, **STOR**. Individual checkboxes also available.
      </Step>

      <Step title="Configure credentials" icon="key">
        Enter the SSH username and password. Used **once** to copy the SSH public key — after that, password-based auth is no longer needed.
      </Step>

      <Step title="Initialize SSH" icon="lock">
        Generates an SSH key pair (if needed), distributes the public key to every selected host via `ssh-copy-id`.
      </Step>

      <Step title="Registry aliases" icon="globe">
        Adds `docker-registry` hostname to `/etc/hosts` on each node so all servers can pull container images from the local registry.
      </Step>

      <Step title="Verify" icon="circle-check">
        Tests SSH connectivity to every host. Green = ready for deployment, red = check credentials or firewall.
      </Step>
    </Steps>

    <Warning>
      After SSH keys are working, use **Disable Password Auth** to prevent brute-force attacks on production deployments.
    </Warning>
  </Tab>

  <Tab title="DNS" icon="globe">
    Manages hostname resolution across the cluster. Many services communicate using hostnames, not just IP addresses. This tab distributes hostname-to-IP mappings to `/etc/hosts` on every node.

    <Warning>If a compute node cannot resolve the controller hostname, internal API calls fail silently. Always verify DNS resolution after setup.</Warning>
  </Tab>

  <Tab title="Logs" icon="terminal">
    The Logs tab records all host management operations with timestamps --- SSH key distribution, host additions, connectivity tests, and DNS distribution results. Use this log to diagnose SSH failures or trace when hosts were added or modified.
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Configuration" icon="settings" href="/deployment/configuration">
    Configure networking, storage, monitoring, and security settings for your deployment
  </Card>

  <Card title="Operations" icon="play" href="/deployment/operations">
    Deploy, upgrade, reconfigure, and manage cloud services across your cluster
  </Card>
</CardGroup>


# Images
Source: https://docs.xloud.tech/deployment/images

Manage Docker container images and local registry for cloud service deployment

Every cloud service in your Xloud environment runs inside a Docker container. Before deployment, container images must be available on every target server. The Images tool helps you pull, extract, store, and distribute container images. This is a critical step for both online and air-gapped (offline) deployments.

<Note>
  **Prerequisites**

  * [Bootstrap](/deployment/bootstrap) completed with all dependency groups passing
  * Docker installed and running on all target nodes
  * Network connectivity to the internet (online mode) or a `.tar.gz` image archive file (offline mode)
</Note>

***

## Overview Tab

The Overview tab provides a real-time status dashboard for Docker and the local registry without modifying anything on the system.

<CardGroup>
  <Card title="Docker Engine" icon="box">
    Running status, installed Docker version, and total number of locally cached images on the current server.
  </Card>

  <Card title="Local Registry" icon="database">
    Registry container (`docker-registry:4000`) operational status and total stored image count. The registry serves images to all cluster nodes over the internal network.
  </Card>

  <Card title="Registry Controls" icon="settings">
    Action buttons to Deploy, Start, and Stop the local registry container. Deploy creates the container for the first time; Start and Stop manage its lifecycle afterward.
  </Card>
</CardGroup>

<Info>
  The local registry runs as a Docker container listening on port 4000. It is accessible to all cluster nodes at `docker-registry:4000` — no TLS required for internal communication.
</Info>

***

## Getting Images

Container images can be loaded from two sources depending on your network environment. After loading images into Docker, push them to the local registry so all cluster nodes can access them.

<Tabs>
  <Tab title="Extract / Pull" icon="download">
    Use this tab to load container images from offline archives or pull them from an online registry. The tab provides two sub-tabs: **Extract** for offline deployments and **Pull** for online deployments.

    ***

    ### Extract (Offline)

    Use Extract when your servers have no internet access or when deploying in air-gapped environments. Images can be loaded from two sources:

    <Tabs>
      <Tab title="USB Device" icon="plug">
        Load images directly from a USB storage device connected to the deployment server. This is the primary method for air-gapped data centers where images are delivered on physical media.

        <Steps>
          <Step title="Connect USB Storage" icon="plug">
            Connect the USB storage device containing the image archive to the deployment server. The tool automatically detects connected USB storage devices and displays them in the **USB Storage Detection** section.
          </Step>

          <Step title="Select Archive" icon="folder">
            Browse the detected USB device and select the `.tar.gz` image archive file. The tool displays the archive filename and size for confirmation.
          </Step>

          <Step title="Copy to Server" icon="download">
            Set the **Destination** path (default: `/var/lib/xavs/images`) and click **Copy to Server** to transfer the archive from the USB device to local server storage. This ensures extraction runs from local disk for maximum performance.

            <Info>Copying from USB to local storage first avoids slow extraction speeds caused by USB transfer rates. The default destination has sufficient space for full release archives.</Info>
          </Step>

          <Step title="Extract and Load Images" icon="play">
            After the copy completes, click **Extract & Load Images** to decompress the archive and load all container images into the local Docker daemon.
          </Step>
        </Steps>
      </Tab>

      <Tab title="Server Storage" icon="hard-drive">
        Load images from an archive file already present on the server's local storage. Use this when the archive was transferred via SCP, SFTP, or other file transfer methods.

        <Steps>
          <Step title="Browse for Archive" icon="folder">
            Browse the server filesystem and select the `.tar.gz` image archive file. The tool validates the archive format before proceeding.
          </Step>

          <Step title="Extract Images" icon="play">
            Click **Extract** to load all images from the archive into the local Docker daemon. This process reads each image layer from the archive and registers it with Docker.

            <Info>Extraction duration depends on archive size and disk speed. A full release archive typically takes 5-15 minutes to extract.</Info>
          </Step>
        </Steps>
      </Tab>
    </Tabs>

    After extraction completes from either source, click **Push to Local Registry** to distribute all loaded images to `docker-registry:4000`. This makes images available to every node in the cluster.

    ***

    ### Pull (Online)

    Use Pull when the deployment node has internet access and can reach the Xloud cloud registry.

    <Steps>
      <Step title="Verify Connectivity" icon="globe">
        Confirm that the deployment node can reach the Xloud container registry over HTTPS. The tool tests connectivity automatically and displays the result.
      </Step>

      <Step title="Pull Images" icon="download">
        Click **Pull** to download all required container images from the configured image catalog. Each image is downloaded with its full layer chain and stored in the local Docker daemon.
      </Step>

      <Step title="Push to Local Registry" icon="upload">
        After pulling completes, push all images to the local registry so cluster nodes pull from the internal network instead of the internet during deployment.
      </Step>
    </Steps>
  </Tab>
</Tabs>

<Tip>
  A local registry makes image pulls instant and deterministic across the cluster. Without it, multiple nodes pulling simultaneously from the internet is slow and unreliable — especially during initial deployment when 50+ images are needed.
</Tip>

***

## Registry Management

The local Docker registry is the central image distribution point for your cluster. All nodes pull container images from this registry during deployment and scaling operations.

<AccordionGroup>
  <Accordion title="Deploy the Registry" icon="plus">
    Creates and starts the `docker-registry` container for the first time. The registry listens on port 4000 and stores images in a persistent Docker volume.
  </Accordion>

  <Accordion title="Start / Stop / Restart" icon="power">
    Lifecycle controls for the registry container. Stop halts the registry; Start resumes it. Stored images persist across restarts in the backing volume.
  </Accordion>

  <Accordion title="Browse Catalog" icon="list">
    Displays the complete catalog of images stored in the registry, including all available tags per image. Use this view to verify that all required service images are present before running deployment.
  </Accordion>

  <Accordion title="Monitor Storage" icon="hard-drive">
    Shows disk space consumed by the registry volume. Large environments with multiple image versions can consume significant storage — monitor usage and clean up unused tags periodically.
  </Accordion>

  <Accordion title="Delete Unused Images" icon="trash">
    Removes old or unused image tags from the registry to reclaim disk space. Exercise caution — deleting an image that a running container depends on prevents that container from restarting.
  </Accordion>
</AccordionGroup>

<Warning>
  The local registry serves images to your entire cluster. If it goes down, new containers cannot start during maintenance, scaling operations, or node recovery. Keep the registry running at all times in production.
</Warning>

***

## Docker Configuration

The Images tool provides an editor for `/etc/docker/daemon.json` — the Docker daemon configuration file. Changes here affect how Docker pulls, stores, and logs container images.

| Setting                   | Purpose                                                              | Default                 |
| ------------------------- | -------------------------------------------------------------------- | ----------------------- |
| **Insecure Registries**   | Allow Docker to pull from `docker-registry:4000` over HTTP (non-TLS) | `docker-registry:4000`  |
| **Registry Mirrors**      | Configure Docker Hub mirror URLs for faster or cached pulls          | None                    |
| **Storage Driver**        | Set the filesystem driver for image layer storage                    | `overlay2`              |
| **Logging Configuration** | Configure container log rotation to prevent disk exhaustion          | JSON file with rotation |

<Tip>
  The most common deployment issue is Docker refusing to pull from the local registry because `docker-registry:4000` is not listed as an insecure registry. Verify this setting on every node before running deployment.
</Tip>

<Warning>
  Changes to the Docker daemon configuration require a Docker service restart to take effect. Restarting Docker stops all running containers briefly — schedule configuration changes during maintenance windows on production clusters.
</Warning>

***

## Validation

After loading images and configuring the registry, verify readiness before proceeding to deployment.

<Steps>
  <Step title="Verify Registry Status" icon="circle-check">
    The Overview tab shows the local registry with a green running status. Confirm the stored image count matches the expected number for your release version.
  </Step>

  <Step title="Confirm Image Availability" icon="eye">
    Browse the registry catalog and verify that all core service images are present — including compute, networking, identity, storage, and dashboard images.
  </Step>

  <Step title="Test Pull from Another Node" icon="server">
    From a compute or storage node, verify that Docker can pull an image from the local registry:

    ```bash title="Test registry pull from a cluster node" theme={null}
    docker pull docker-registry:4000/xloud/nova-compute:2025.1
    ```

    A successful pull confirms that the registry is accessible from across the cluster.
  </Step>
</Steps>

<Check>
  Registry running, all images loaded, and cross-node pulls verified — images are ready for deployment.
</Check>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Docker refuses to pull from the local registry" icon="alert-triangle">
    **Cause**: The local registry is not listed as an insecure registry in the Docker daemon configuration.

    **Resolution**: Open the Docker Configuration section in the Images tool and add `docker-registry:4000` to the insecure registries list. Restart the Docker daemon after saving the change. Repeat on every node in the cluster.
  </Accordion>

  <Accordion title="Archive extraction fails with checksum errors" icon="alert-triangle">
    **Cause**: The `.tar.gz` archive was corrupted during transfer.

    **Resolution**: Re-download or re-transfer the archive file and verify its integrity using the checksum provided with the release. Retry extraction after replacing the corrupted file.
  </Accordion>

  <Accordion title="Registry container fails to start" icon="alert-triangle">
    **Cause**: Port 4000 is already in use by another process, or the Docker volume is corrupted.

    **Resolution**: Check for port conflicts with `ss -tlnp | grep 4000`. If no conflicts exist, remove and redeploy the registry container to recreate the volume from scratch.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Operations" icon="play" href="/deployment/operations">
    Deploy cloud services across the cluster using the loaded container images
  </Card>

  <Card title="Configuration" icon="settings" href="/deployment/configuration">
    Review and adjust cloud service configuration before deployment
  </Card>
</CardGroup>


# XDeploy Platform
Source: https://docs.xloud.tech/deployment/index

Complete deployment and lifecycle management for your Xloud cloud infrastructure

XDeploy is the deployment and management platform that transforms bare servers into a fully operational cloud infrastructure. It provides a guided wizard-style interface for every stage. This covers server preparation, hardware validation, production deployment, and ongoing lifecycle management.

<Note>
  **Prerequisites** — Before starting any deployment workflow, ensure you have:

  * Physical or virtual servers running [XOS](/products/xos)
  * Root or sudo access on all target nodes
  * Network connectivity between all cluster nodes
  * A valid XDeploy activation key from [license.xloud.tech](https://license.xloud.tech)
</Note>

***

## Full Deployment Walkthrough

<div>
  <div>
    End-to-end XDEPLOY walkthrough — activation, hosts, XSDS storage, cluster configuration, load balancer, monitoring, advanced features, cluster licensing, and the Operations workflow that brings the cluster live.

    Watch this before you begin — the 20-minute tour covers every module and decision point you will hit during a production deployment.
  </div>

  <iframe title="How to Deploy Xloud with XDEPLOY — Full Walkthrough" />
</div>

***

## Deployment Workflow

Follow this sequence for a successful deployment. Each stage builds on the previous one.

<Steps>
  <Step title="Activate XDeploy" icon="fingerprint">
    Enter your activation key to unlock the platform. Without activation, all tools remain locked.
  </Step>

  <Step title="Bootstrap" icon="check">
    Validate hardware, install dependencies (Python, Ansible, CLI clients).
  </Step>

  <Step title="Hosts" icon="server">
    Build cluster inventory, establish SSH, assign node roles.
  </Step>

  <Step title="XSDS Bootstrap" icon="database">
    Deploy the distributed storage cluster. **Skip this step if not using XSDS as your storage backend.**
  </Step>

  <Step title="Configuration" icon="settings">
    Set networking, storage, monitoring, and feature toggles.
  </Step>

  <Step title="Images" icon="package">
    Pull or extract container images for all services.
  </Step>

  <Step title="Deploy" icon="play">
    Run the deployment playbook (30-90 minutes).
  </Step>

  <Step title="License" icon="key">
    Activate the deployed cluster with your license file.
  </Step>

  <Step title="Management" icon="users">
    Create users, projects, quotas, and credentials.
  </Step>
</Steps>

<Tip>
  For production deployments, always run **Prechecks** before **Deploy** in the Operations tool. Prechecks validate ports, services, and configurations before committing to a full deployment.
</Tip>

***

## Pre-Deployment Checklist

Complete every step below before initiating your first deployment.

<Steps>
  <Step title="Install XOS on All Nodes" icon="monitor">
    All target servers must be running **XOS**. XOS includes all required kernel modules, drivers, and base packages pre-configured for cloud deployment.

    <Info>XOS is available as an ISO from your Xloud representative or the [license portal](https://license.xloud.tech). See the [XOS product page](/products/xos) for details.</Info>
  </Step>

  <Step title="Verify Hardware Requirements" icon="cpu">
    Confirm every target server meets or exceeds these minimums:

    | Resource           | Minimum | Production                 |
    | ------------------ | ------- | -------------------------- |
    | CPU Cores          | 4       | 16+                        |
    | RAM                | 8 GB    | 32-64 GB                   |
    | Root Filesystem    | 100 GB  | 200+ GB                    |
    | Network Interfaces | 2       | 4+ (bonded for redundancy) |

    <Info>Each cloud service container consumes 200-500 MB of RAM. A production control plane with all services enabled requires 32 GB minimum.</Info>
  </Step>

  <Step title="Plan Your Network" icon="network">
    Separate traffic types onto dedicated VLANs and subnets for security and performance:

    | Network                    | Purpose                                        | VLAN Required          |
    | -------------------------- | ---------------------------------------------- | ---------------------- |
    | API / Management           | API endpoints, SSH, Ansible, internal services | Yes                    |
    | VM Traffic (Tenant)        | Instance-to-instance communication             | Yes                    |
    | Provider / External        | Floating IPs, VM internet access               | Yes                    |
    | Load Balancer (Octavia)    | Amphora management traffic                     | Yes (if LBaaS enabled) |
    | Storage Client (XSDS)      | Client-to-Ceph I/O                             | Recommended            |
    | Storage Replication (XSDS) | Ceph OSD-to-OSD replication                    | Recommended            |
    | Additional Provider VLANs  | Customer-specific external networks            | As needed              |

    <Warning>Never mix management and provider traffic on a single NIC. If tenant instance traffic saturates the interface, the control plane becomes unreachable.</Warning>
  </Step>

  <Step title="Prepare DNS and SSL" icon="globe">
    For production deployments with HTTPS API endpoints:

    * Configure DNS records pointing to your Internal and External VIP addresses
    * Obtain TLS/SSL certificates for your domain (wildcard or per-service)
    * XDeploy supports both internal and external API TLS independently

    <Tip>Domain and SSL configuration is optional for lab/testing environments. You can use raw IP addresses without TLS for initial deployments.</Tip>
  </Step>

  <Step title="Gather Credentials" icon="key">
    Collect the following before starting:

    * **SSH credentials** from the deployment node to all target servers (password or key-based)
    * **XDeploy activation key** (`.xlic`) from [license.xloud.tech](https://license.xloud.tech)
    * **Cluster license key** (`.xlic`) from [license.xloud.tech](https://license.xloud.tech)
  </Step>

  <Step title="Choose Architecture and Node Roles" icon="layout">
    Select the deployment architecture that matches your requirements:

    <CardGroup>
      <Card title="All-in-One (AIO)" icon="square">
        Single server runs all roles. For labs, demos, and development. Minimum 1 node.
      </Card>

      <Card title="Hyper-Converged (HCI)" icon="layers">
        Each node runs compute + storage together. Reduces hardware count. Minimum 3 nodes for HA.
      </Card>

      <Card title="Distributed" icon="grid">
        Dedicated nodes per role — separate controller, compute, and storage. Maximum performance and isolation. Minimum 5+ nodes.
      </Card>
    </CardGroup>

    <Tip>For production environments, Xloud recommends HCI with a minimum of 3 nodes or Distributed with 5+ nodes for full high availability.</Tip>
  </Step>
</Steps>

<Warning>
  Deploying on servers that do not meet minimum requirements leads to failures 30-45 minutes into deployment. Always run [Bootstrap](/deployment/bootstrap) hardware checks first.
</Warning>

***

## Platform Tools

XDeploy provides 11 specialized tools, each handling a distinct stage of the deployment and management lifecycle.

<CardGroup>
  <Card title="Bootstrap" icon="circle-check" href="/deployment/bootstrap">
    Server readiness validation and dependency installation
  </Card>

  <Card title="Hosts" icon="server" href="/deployment/hosts">
    Cluster inventory management and SSH connectivity setup
  </Card>

  <Card title="Configuration" icon="settings" href="/deployment/configuration">
    Cloud settings via guided forms — networking, storage, features
  </Card>

  <Card title="Advanced Config" icon="file-text" href="/deployment/advanced-config">
    Per-service configuration file editor with version tracking
  </Card>

  <Card title="Images" icon="package" href="/deployment/images">
    Container image management, registry, and extraction
  </Card>

  <Card title="Operations" icon="play" href="/deployment/operations">
    Deploy, upgrade, reconfigure, and destroy cloud services
  </Card>

  <Card title="Management" icon="users" href="/deployment/management">
    Users, projects, quotas, and credential management
  </Card>

  <Card title="Cloud Fleet" icon="globe" href="/deployment/cloud-fleet">
    Interactive infrastructure topology map
  </Card>

  <Card title="XSDS Storage" icon="database" href="/deployment/xsds">
    Distributed storage deployment and tier management
  </Card>

  <Card title="Cluster License" icon="key" href="/deployment/cluster-license">
    Cluster licensing and activation
  </Card>

  <Card title="XDeploy Key" icon="fingerprint" href="/deployment/xdeploy-key">
    XDeploy tool activation and license management
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="XDeploy Key" icon="fingerprint" href="/deployment/xdeploy-key">
    Start here — activate XDeploy before using any other tools
  </Card>

  <Card title="Bootstrap" icon="circle-check" href="/deployment/bootstrap">
    Validate server hardware and install all required dependencies
  </Card>
</CardGroup>


# Management
Source: https://docs.xloud.tech/deployment/management

Create administrator accounts and scale compute and API workers for your deployed cloud

Management handles post-deployment administration tasks. After your cloud is deployed, you need administrator accounts for accessing the Dashboard and APIs, and you may need to scale compute or API workers to match demand. The Management tool provides two tabs for these operations.

<Note>
  **Prerequisites**

  * Cloud deployment completed via [Operations](/deployment/operations) (Deploy + Post-Deploy)
  * Access to XDeploy on the deployment node
</Note>

***

## Management Tabs

<Tabs>
  <Tab title="User Management" icon="users">
    Create administrator accounts for accessing the Xloud Dashboard and APIs. Each user account receives admin and heat\_stack\_owner roles, granting full control over the cloud environment.

    ***

    ### Create Administrator Account

    | Field                | Required | Description                                                                                                               |
    | -------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------- |
    | **Username**         | Yes      | Unique login identifier. Must be 3-32 characters, alphanumeric only. The username `admin` is reserved and cannot be used. |
    | **Password**         | Yes      | Account password. Minimum 12 characters with at least one uppercase letter, one number, and one special character.        |
    | **Confirm Password** | Yes      | Re-enter the password to confirm accuracy                                                                                 |
    | **Email**            | No       | Optional email address associated with the account                                                                        |

    Click **Create Administrator Account** to provision the user.

    <Warning>
      Usernames cannot be changed after creation. Choose usernames carefully following your organization's naming conventions. The reserved username `admin` cannot be used --- it is reserved for the built-in system administrator account created during deployment.
    </Warning>

    <Info>
      Every account created through this interface receives the **admin** and **heat\_stack\_owner** roles automatically. These roles grant full access to all projects, users, and orchestration capabilities. For restricted user accounts with limited roles, use the [Xloud Dashboard](/services/dashboard/user-guide) identity management panel after deployment.
    </Info>
  </Tab>

  <Tab title="Scaling" icon="trending-up">
    Scale compute workers and API workers to match your cluster's workload demands. Scaling adjusts the number of service processes running across your deployment.

    ***

    ### Compute Workers

    Compute workers handle virtual machine lifecycle operations --- launching, stopping, migrating, and monitoring instances. Increase compute workers to handle higher instance concurrency.

    | Setting                  | Description                                         |
    | ------------------------ | --------------------------------------------------- |
    | **Compute Worker Count** | Number of compute worker processes per compute node |

    ***

    ### API Workers

    API workers handle incoming requests to the cloud APIs. Increase API workers to handle higher request throughput from the Dashboard, CLI, and automation tools.

    | Setting              | Description                                        |
    | -------------------- | -------------------------------------------------- |
    | **API Worker Count** | Number of API worker processes per controller node |

    <Tip>
      A general guideline is one API worker per CPU core on the controller node. Over-provisioning API workers beyond available CPU cores can increase latency due to context switching overhead.
    </Tip>

    <Warning>
      Changing worker counts requires a service restart. Plan scaling operations during maintenance windows to avoid brief API interruptions.
    </Warning>
  </Tab>
</Tabs>

***

## Validation

After creating accounts and adjusting scaling settings, verify the configuration is correct.

<Steps>
  <Step title="Test User Login" icon="circle-check">
    Log in to the **Xloud Dashboard** (`https://connect.<your-domain>`) using a newly created administrator account. Verify that the user has full administrative access and can view all projects.
  </Step>

  <Step title="Verify Account Roles" icon="shield-check">
    Confirm that the created account has the expected roles:

    ```bash title="Check role assignments" theme={null}
    source openrc.sh
    openstack role assignment list --user <username> --names
    ```

    The output should show both **admin** and **heat\_stack\_owner** roles assigned to the user.
  </Step>

  <Step title="Verify Worker Scaling" icon="activity">
    After adjusting worker counts, confirm the new worker processes are running:

    ```bash title="Check API worker count" theme={null}
    docker exec nova_api ps aux | grep nova-api | wc -l
    ```

    The count should match the configured API worker setting.

    <Check>Administrator accounts created and worker scaling verified.</Check>
  </Step>
</Steps>

***

## Best Practices

<CardGroup>
  <Card title="Individual Accounts" icon="users">
    Create separate administrator accounts for each person who needs access. Never share credentials. Individual accounts provide accountability through audit logs and allow targeted access revocation.
  </Card>

  <Card title="Strong Passwords" icon="lock">
    Use passwords that exceed the minimum requirements. Combine uppercase, lowercase, numbers, and special characters. Consider using a password manager to generate and store credentials securely.
  </Card>

  <Card title="Scale Gradually" icon="trending-up">
    When scaling workers, increase in small increments and monitor resource utilization after each change. Doubling worker counts without verifying CPU and memory headroom can degrade performance.
  </Card>

  <Card title="Maintenance Windows" icon="clock">
    Schedule worker scaling during planned maintenance windows. Worker count changes require service restarts, which cause brief interruptions to API availability.
  </Card>
</CardGroup>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="User cannot log in to the Dashboard" icon="alert-triangle">
    **Cause**: Incorrect credentials, disabled account, or the user is not assigned to any project.

    **Resolution**: Verify the username and password in the Management tool. Ensure the user account is enabled and assigned to at least one project with a valid role. Reset the password if necessary.
  </Accordion>

  <Accordion title="Account creation fails with username validation error" icon="alert-triangle">
    **Cause**: The username does not meet the requirements (3-32 characters, alphanumeric only) or uses the reserved name `admin`.

    **Resolution**: Choose a different username that meets the character requirements and is not reserved.
  </Accordion>

  <Accordion title="API becomes unresponsive after scaling workers" icon="alert-triangle">
    **Cause**: Worker count exceeds available CPU cores, causing excessive context switching and memory pressure.

    **Resolution**: Reduce the worker count to match the number of available CPU cores. Monitor memory usage to ensure the node has sufficient RAM for all worker processes.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Cloud Fleet" icon="globe" href="/deployment/cloud-fleet">
    View the interactive topology map of your deployed infrastructure and service status
  </Card>

  <Card title="Dashboard User Guide" icon="monitor" href="/services/dashboard/user-guide">
    Learn to manage instances, volumes, networks, and images through the Xloud Dashboard
  </Card>
</CardGroup>


# Operations
Source: https://docs.xloud.tech/deployment/operations

Deploy, upgrade, reconfigure, and manage your cloud infrastructure with guided playbooks

Operations is the command center for your cloud deployment. Every infrastructure action — bootstrap, prechecks, deploy, reconfigure, upgrade, stop, and destroy — runs through this tool. It wraps all deployment playbooks into a visual wizard with a live terminal, real-time progress tracking, and searchable task history.

<Note>
  **Prerequisites**

  * [Bootstrap](/deployment/bootstrap) completed with all dependency groups passing
  * [Hosts](/deployment/hosts) configured with SSH connectivity to all target nodes
  * [Configuration](/deployment/configuration) saved with networking, storage, and service settings
  * [Images](/deployment/images) loaded and available in the local registry
</Note>

***

## Deployment Wizard

The Operations wizard guides you through four steps to execute any deployment action. Each step validates input before allowing you to proceed.

<Steps>
  <Step title="Select Action" icon="list">
    Select the deployment action to execute from the available categories: Deploy, Manage, Backend, or Advanced. Each action is described with its purpose and expected duration.
  </Step>

  <Step title="Select Target Hosts" icon="server">
    Select which servers to target:

    * **All Hosts** — execute across the entire cluster
    * **Specific Hosts** — select individual servers via a checklist
    * **Host Patterns** — target groups such as all compute nodes or all storage nodes

    <Tip>For initial deployment, select All Hosts. For targeted maintenance or upgrades, select specific hosts to minimize disruption.</Tip>
  </Step>

  <Step title="Select Services (Optional)" icon="layers">
    Narrow the scope to specific cloud services or deploy all enabled services. Selecting individual services is useful for targeted reconfiguration or troubleshooting without affecting the rest of the cluster.
  </Step>

  <Step title="Run Deployment" icon="play">
    Review the action summary and click **Run** to execute. Destructive actions (Stop, Destroy) require explicit confirmation before execution begins.

    <Warning>Once a deployment action starts, interrupting it mid-execution can leave services in an inconsistent state. Allow the action to complete fully before taking further steps.</Warning>
  </Step>
</Steps>

***

## Available Actions

<Tabs>
  <Tab title="Deploy" icon="rocket">
    Actions for initial deployment and readiness validation.

    | Action                | Purpose                                                                                  | Typical Duration |
    | --------------------- | ---------------------------------------------------------------------------------------- | ---------------- |
    | **Bootstrap Servers** | Prepare target servers — install Docker, configure networking, set up prerequisites      | 5-15 min         |
    | **Prechecks**         | Validate that everything is ready — ports, services, configurations, and connectivity    | 2-5 min          |
    | **Deploy**            | Deploy all enabled cloud services as Docker containers across the cluster                | 30-90 min        |
    | **Post-Deploy**       | Create the admin user, default networks, base flavors, and initial project configuration | 5-10 min         |

    <Note>
      Additional deploy actions are available via the command selector: **Deploy Bifrost**
      (bare-metal provisioning service) and **Deploy Servers** (provision bare-metal servers
      through Bifrost). These are not shown as wizard cards but can be selected from the
      action dropdown.
    </Note>

    <Warning>Always run Prechecks before Deploy. It catches configuration issues, port conflicts, and missing dependencies that would otherwise cause failures 30+ minutes into deployment.</Warning>
  </Tab>

  <Tab title="Manage" icon="settings">
    Actions for ongoing lifecycle management after initial deployment.

    | Action          | Purpose                                                                     |
    | --------------- | --------------------------------------------------------------------------- |
    | **Pull Images** | Pre-pull container images to all nodes before an upgrade or redeployment    |
    | **Reconfigure** | Apply configuration changes to running services without a full redeployment |
    | **Upgrade**     | Rolling upgrade to a new release version with minimal downtime              |

    <Note>
      Additional manage actions are available via the command selector: **Upgrade Bifrost**
      (upgrade the bare-metal provisioning service).
    </Note>

    \| **Stop** | Stop all containers across the cluster (causes full downtime, requires confirmation) |
    \| **Destroy** | Remove all containers, configurations, and data completely (requires confirmation) |

    <Danger>Destroy is irreversible. It removes all containers, configuration files, and data from every target node. This action cannot be undone. Ensure you have verified backups before executing.</Danger>
  </Tab>

  <Tab title="Backend" icon="database">
    Database and message queue maintenance operations.

    | Action               | Purpose                                                                   |
    | -------------------- | ------------------------------------------------------------------------- |
    | **MariaDB Backup**   | Create a full database backup of the Galera cluster for disaster recovery |
    | **MariaDB Recovery** | Recover a broken or corrupted database from a previous backup             |
    | **RabbitMQ Reset**   | Reset message queue state when queues are stuck or consumers are stalled  |
    | **RabbitMQ Upgrade** | Upgrade the RabbitMQ version to match a new release                       |

    <Warning>MariaDB Recovery overwrites the current database state with backup data. Any changes made after the backup was taken are lost.</Warning>
  </Tab>

  <Tab title="Advanced" icon="sliders">
    Infrastructure-level and diagnostic operations.

    | Action                   | Purpose                                                                                            |
    | ------------------------ | -------------------------------------------------------------------------------------------------- |
    | **Deploy Containers**    | Deploy only the container infrastructure layer without configuring services                        |
    | **Octavia Certificates** | Generate or renew TLS certificates for the Xloud Load Balancer service                             |
    | **Generate Config**      | Generate all service configuration files without deploying — useful for pre-review                 |
    | **Validate Config**      | Check configuration files for syntax errors and missing required values                            |
    | **Gather Facts**         | Collect system information (hardware, network, OS) from all target hosts                           |
    | **Nova Libvirt Cleanup** | Clean up stale libvirt resources — orphaned domains, volumes, and network filters on compute nodes |

    <Info>Generate Config and Validate Config are non-destructive — they do not modify any running services. Use them freely for pre-deployment review.</Info>
  </Tab>
</Tabs>

***

## Command Modifiers

Actions in the Operations wizard support optional modifiers that control execution behavior.

| Modifier            | Description                                                                                                                                                                                     | Applicable Actions                    |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| **Dry-run mode**    | Simulate the action without making changes. Shows what tasks would execute and which hosts would be affected.                                                                                   | All deployment and management actions |
| **Backup mode**     | Create a backup before executing. Supports **full** (complete state snapshot) and **incremental** (changes since last backup) modes.                                                            | MariaDB Backup                        |
| **Offline mode**    | Execute without internet access, using pre-staged packages and images.                                                                                                                          | Bootstrap Servers                     |
| **Verbose logging** | Increase Ansible output verbosity from level 0 (default) through level 4 (maximum debug output). Higher levels include connection debugging, module arguments, and full task execution details. | All actions                           |

<Tip>
  Use **dry-run mode** before any production change. It validates the action plan without modifying the cluster, letting you catch configuration issues before they affect running services.
</Tip>

***

## Live Terminal

Every action execution streams full terminal output in real time, providing complete visibility into each playbook task as it runs.

<CardGroup>
  <Card title="Real-Time Metrics" icon="activity">
    Live counters for task success, warnings, failures, and overall completion percentage update as each task finishes.
  </Card>

  <Card title="Searchable History" icon="search">
    All task output is archived and searchable. Filter by task name, host, or status to quickly locate specific events across long deployment runs.
  </Card>
</CardGroup>

<AccordionGroup>
  <Accordion title="Understanding Terminal Output" icon="terminal">
    Each line in the terminal represents an Ansible task. Tasks display their target host, task name, and result status:

    * **ok** — Task completed successfully, no changes needed
    * **changed** — Task completed successfully and modified the system
    * **failed** — Task encountered an error (deployment may continue or stop depending on the task)
    * **skipped** — Task was not applicable to the current configuration
  </Accordion>

  <Accordion title="Quick Guide" icon="info">
    The Operations tool includes a Quick Guide button that opens a recommended workflow tutorial. This guide walks through the standard deployment sequence and explains when to use each action.
  </Accordion>
</AccordionGroup>

***

## Recommended Workflow

Follow this sequence for a successful first deployment. Each step must complete without errors before proceeding to the next.

<Steps>
  <Step title="Run Prechecks" icon="shield-check">
    Execute Prechecks against all target hosts. This validates port availability, service connectivity, configuration consistency, and Docker readiness across the cluster.
  </Step>

  <Step title="Fix Reported Issues" icon="wrench">
    Review any failures or warnings from the Prechecks output. Common issues include missing Docker configuration, incorrect NIC assignments, and port conflicts. Resolve every failure before continuing.
  </Step>

  <Step title="Run Deploy" icon="play">
    Execute the Deploy action against all hosts. This provisions every enabled cloud service as Docker containers, configures HAProxy load balancing, and starts all services.

    <Info>Deployment duration scales with cluster size and network speed. A 3-node cluster typically completes in 30-45 minutes. Larger clusters with 10+ nodes may take up to 90 minutes.</Info>
  </Step>

  <Step title="Run Post-Deploy" icon="flag">
    Execute Post-Deploy to create the admin user account, default tenant networks, base compute flavors, and initial project structure. This step finalizes the environment for production use.
  </Step>

  <Step title="Verify via Dashboard and CLI" icon="circle-check">
    Log in to the **Xloud Dashboard** (`https://connect.<your-domain>`) with the admin credentials created during Post-Deploy. Verify that all services are listed and operational.

    ```bash title="Verify service endpoints" theme={null}
    source openrc.sh
    openstack service list
    openstack compute service list
    openstack network agent list
    ```

    <Check>All services report status **up** — the deployment is complete and operational.</Check>
  </Step>
</Steps>

<Tip>
  For configuration changes after initial deployment, use **Reconfigure** instead of a full redeployment. Reconfigure applies changes incrementally to running services without downtime, completing in minutes instead of 30-90 minutes.
</Tip>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Prechecks report port conflicts" icon="alert-triangle">
    **Cause**: Another service or previous deployment remnant is using a port required by a cloud service.

    **Resolution**: Identify the conflicting process using `ss -tlnp | grep <port>` on the affected host. Stop or remove the conflicting service. If a previous deployment was not fully destroyed, run Destroy before redeploying.
  </Accordion>

  <Accordion title="Deploy fails with image pull errors" icon="alert-triangle">
    **Cause**: Container images are not available in the local registry, or Docker cannot reach the registry.

    **Resolution**: Return to the [Images](/deployment/images) tool and verify that all required images are loaded and pushed to the local registry. Confirm that `docker-registry:4000` is listed as an insecure registry in the Docker daemon configuration on every node.
  </Accordion>

  <Accordion title="Post-Deploy fails to create admin user" icon="alert-triangle">
    **Cause**: The Xloud Identity service is not yet fully initialized when Post-Deploy runs.

    **Resolution**: Wait 30-60 seconds after Deploy completes before running Post-Deploy. The identity service needs time to complete database migrations and start accepting requests. Re-run Post-Deploy after the brief delay.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Management" icon="users" href="/deployment/management">
    Create users, projects, quotas, and credentials for your deployed cloud
  </Card>

  <Card title="Cloud Fleet" icon="globe" href="/deployment/cloud-fleet">
    View the interactive topology map of your deployed infrastructure
  </Card>
</CardGroup>


# XDeploy Key
Source: https://docs.xloud.tech/deployment/xdeploy-key

Activate the XDeploy management platform with a hardware-bound license key

XDeploy Key licenses the management platform itself. This is the first step when installing XDeploy on a new server. Without a valid key, deployment tools and configuration interfaces are restricted. The key is bound to your server's hardware fingerprint to prevent unauthorized copying or redistribution.

<Note>
  **Prerequisites**

  * XDeploy installed on the management server
  * Access to [license.xloud.tech](https://license.xloud.tech) for key file submission
  * Server hardware finalized (CPU, motherboard, and disks should not change after activation)
</Note>

<Warning>
  This licenses the **XDeploy management tool** — the interface used to deploy and manage your cloud. To license the deployed cloud cluster, use [Cluster License](/deployment/cluster-license) instead.
</Warning>

***

## Activation Status

The status indicator at the top of the XDeploy Key page shows the current activation state of the management platform.

| Status       | Meaning                                              |
| ------------ | ---------------------------------------------------- |
| **Active**   | Valid key installed, all XDeploy tools are unlocked  |
| **Inactive** | No key installed, deployment tools are restricted    |
| **Expired**  | Key has passed its expiration date, renewal required |

Use the **Verify** button to re-check key validity against the current hardware fingerprint. This is useful after hardware maintenance to confirm the key is still valid.

<Info>
  The activation status is checked automatically each time XDeploy loads. Manual verification is only needed when troubleshooting or after hardware changes.
</Info>

***

## Activation Process

<Steps>
  <Step title="Request Key" icon="download">
    Click **Request Key** to collect the hardware fingerprint from the management server. The tool gathers:

    | Data Point      | Purpose                                      |
    | --------------- | -------------------------------------------- |
    | **CPU Sockets** | Physical processor count for license binding |
    | **Total Cores** | Aggregate core count across all sockets      |
    | **Hostname**    | Server identity for license tracking         |

    The collected data is encrypted and packaged into a downloadable `.xlic` request file. Save this file to your local machine.

    <Tip>
      The request file contains only hardware identifiers. No passwords, network configurations, or sensitive data are included in the request.
    </Tip>
  </Step>

  <Step title="Submit to Portal" icon="upload">
    Upload the `.xlic` request file to [license.xloud.tech](https://license.xloud.tech). The portal validates the request and generates a signed key file.

    <Info>
      Key generation is typically immediate for customers with active support agreements. Contact [support@xloud.tech](mailto:support@xloud.tech) if your request is pending for more than 24 hours.
    </Info>
  </Step>

  <Step title="Upload Key" icon="circle-check">
    Return to the XDeploy Key page and drag the signed `.xlic` key file into the upload area. The maximum file size is 50 KB.

    The tool validates the digital signature, displays a preview of the key details (customer name, expiration date, licensed features), and installs the key.

    <Check>
      XDeploy activated. All platform tools are now unlocked and ready for use.
    </Check>
  </Step>
</Steps>

***

## Key Management

Two management actions are available for an installed key:

<CardGroup>
  <Card title="Verify" icon="shield-check">
    Re-validates the installed key against the current hardware fingerprint. Use after hardware maintenance, motherboard replacement, or CPU changes to confirm the key remains valid.
  </Card>

  <Card title="Remove" icon="trash">
    Deletes the installed key from the management server. Requires explicit confirmation before proceeding. After removal, XDeploy reverts to the restricted state until a new key is installed.
  </Card>
</CardGroup>

<Danger>
  Removing the key deactivates XDeploy immediately. All deployment tools, configuration interfaces, and management workflows become restricted. You will need a new key to resume using the platform. Only remove the key if you are decommissioning the server or transferring the license.
</Danger>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Key upload rejected with invalid signature" icon="alert-triangle">
    **Cause**: The `.xlic` file was corrupted during download or transfer.

    **Resolution**: Re-download the signed key file from the license portal. Transfer using a binary-safe method (SCP, SFTP, or direct browser download). Do not open the file in a text editor, as this can alter binary content. Upload the file again.
  </Accordion>

  <Accordion title="Verification fails after hardware maintenance" icon="alert-triangle">
    **Cause**: The hardware fingerprint changed due to CPU, motherboard, or disk replacement.

    **Resolution**: Generate a new key request with the updated hardware fingerprint. Submit the new request to [license.xloud.tech](https://license.xloud.tech) and install the returned signed key.
  </Accordion>

  <Accordion title="XDeploy shows Inactive despite key upload" icon="alert-triangle">
    **Cause**: The key file may have been uploaded for a different server, or the key has expired.

    **Resolution**: Check the key details in the upload preview — confirm the hostname and hardware identifiers match the current server. If the key is expired, request a renewal through the license portal or contact [support@xloud.tech](mailto:support@xloud.tech).
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Bootstrap" icon="check" href="/deployment/bootstrap">
    Validate hardware requirements and install deployment dependencies
  </Card>

  <Card title="Cluster License" icon="key" href="/deployment/cluster-license">
    License the deployed cloud cluster for production use
  </Card>
</CardGroup>


# XSDS Storage
Source: https://docs.xloud.tech/deployment/xsds

Deploy and manage distributed storage for block, object, and file services

XSDS manages the distributed storage cluster that provides block storage (virtual machine disks), object storage (S3-compatible), and file storage (shared filesystems). Data is replicated across multiple nodes for reliability and high availability. This tool handles bootstrapping the initial storage cluster, configuring storage tiers by media type, editing the cluster configuration file, and viewing deployment logs.

<Note>
  **Prerequisites**

  * Hosts configured with **XSDS-Bootstrap** and **XSDS** roles in [Hosts](/deployment/hosts)
  * Dedicated storage disks available on at least one node (separate from the root filesystem)
  * Network connectivity between all storage nodes on the storage network plane
</Note>

***

## XSDS Tabs

The XSDS module is organized into four tabs, each covering a distinct aspect of storage cluster management.

<Tabs>
  <Tab title="Bootstrap Configuration" icon="play">
    The Bootstrap Configuration tab handles the initial cluster creation on the first designated node. This is a one-time operation that creates the monitor daemon, manager daemon, and cluster identity. All subsequent nodes join this cluster.

    ***

    ### Container Image

    | Field                    | Description                                                                                                                          |
    | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
    | **XSDS Container Image** | Select the storage daemon container image version from the dropdown. The available images are loaded from the local Docker registry. |

    ***

    ### Bootstrap Settings

    | Setting                  | Description                                                             |
    | ------------------------ | ----------------------------------------------------------------------- |
    | **Monitor IP Address**   | IP address of the first monitor node (auto-detected from inventory)     |
    | **Bootstrap Hostname**   | Hostname of the initial node (auto-detected from inventory)             |
    | **Public Network CIDR**  | CIDR for client-to-storage communication (e.g., `10.0.0.0/24`)          |
    | **Cluster Network CIDR** | CIDR for storage-to-storage replication traffic (e.g., `172.16.1.0/24`) |

    <Tip>
      Keeping replication traffic on a separate cluster network prevents it from competing with client I/O. This separation is critical for production performance --- without it, heavy replication during recovery events can saturate the client-facing network and degrade VM disk performance.
    </Tip>

    ***

    ### Custom Bootstrap Attributes

    <Accordion title="Custom Bootstrap Attributes" icon="settings">
      Additional flags that modify bootstrap behavior for non-standard environments. Expand this section to configure advanced options:

      | Flag                     | Purpose                                                                              |
      | ------------------------ | ------------------------------------------------------------------------------------ |
      | `--allow-overwrite`      | Permit re-bootstrapping on a node that was previously initialized                    |
      | `--skip-pull`            | Skip container image download if the image is already present locally                |
      | `--skip-firewalld`       | Do not attempt to configure firewall rules (use when firewall is managed externally) |
      | `--single-host-defaults` | Apply defaults suitable for single-node development deployments                      |
    </Accordion>

    ***

    ### Bootstrap Command

    The tab displays a preview of the full bootstrap command that will be executed based on your configuration. Review the command before proceeding.

    Click **Run Bootstrap Command** to execute the bootstrap operation. Progress streams live to the Logs tab.

    <Warning>
      Bootstrap is a destructive operation on the target node. Running it on a node that already has a storage cluster without `--allow-overwrite` will fail. Running it with `--allow-overwrite` will destroy the existing cluster data on that node.
    </Warning>
  </Tab>

  <Tab title="Storage Tiers" icon="layers">
    The Storage Tiers tab manages storage tiers by grouping disks by media type (NVMe, SSD, HDD) and creating separate CRUSH rules, pools, and volume types for each tier. This enables offering multiple volume types within the same cluster.

    ***

    ### Actions

    | Button                  | Description                                                                                                                      |
    | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
    | **Detect Tiers**        | Query the storage cluster for all available device classes. The system inspects every OSD and reports its underlying media type. |
    | **Apply Configuration** | Creates the CRUSH rules, storage pools, and registers corresponding volume types based on detected tiers.                        |

    Click **Detect Tiers** to scan the cluster. After reviewing the results, click **Apply Configuration** to create the tier infrastructure.

    ***

    ### Information Cards

    The right side of the tab displays reference information:

    <CardGroup>
      <Card title="Storage Tiering" icon="layers">
        Explains what storage tiering does --- grouping disks by media type and creating separate pools so tenants can select performance tiers when creating volumes.
      </Card>

      <Card title="What Gets Configured" icon="settings">
        Details the resources created when applying tiers: CRUSH rules pinned to device classes, dedicated storage pools per tier, volume types registered in the block storage service, and Prometheus metrics for tier monitoring.
      </Card>

      <Card title="Requirements" icon="circle-check">
        Lists prerequisites for tier configuration: a running storage cluster with at least one OSD, and mixed media types (NVMe, SSD, HDD) for multiple tiers.
      </Card>
    </CardGroup>

    <Tip>
      Without tiers, all data lands on whatever disks are available. With tiers, you can offer **Fast SSD Storage** and **Standard HDD Storage** as distinct volume types --- giving tenants control over performance and cost trade-offs.
    </Tip>
  </Tab>

  <Tab title="Config File" icon="file-text">
    The Config File tab provides a direct editor for the initial storage cluster configuration file (`ceph.conf`). Use this to review and modify low-level cluster settings before or after bootstrap.

    ***

    ### Configuration Editor

    | Element                        | Description                                                                    |
    | ------------------------------ | ------------------------------------------------------------------------------ |
    | **Initial Ceph Configuration** | Header indicating this is the base cluster configuration                       |
    | **Reload**                     | Button to reload the configuration from the server, discarding unsaved changes |
    | **Save Configuration**         | Button to write the edited configuration back to the server                    |
    | **Configuration Content**      | Text area displaying the full `ceph.conf` file                                 |

    The configuration content is organized into standard sections:

    | Section          | Contents                                                                   |
    | ---------------- | -------------------------------------------------------------------------- |
    | `[global]`       | Cluster identity, FSID, authentication mode, placement group settings      |
    | Networking       | Public and cluster network CIDRs, bind addresses                           |
    | Container Images | Storage daemon container image references                                  |
    | Security         | Authentication settings, keyring paths                                     |
    | Pool Defaults    | Default replication size, minimum replication, placement group auto-tuning |

    <Warning>
      Modifying the cluster configuration directly can impact cluster stability. Only edit settings you understand. Incorrect values for replication size, network CIDRs, or authentication can make the cluster inaccessible.
    </Warning>

    <Tip>
      Use the Reload button before making changes to ensure you are editing the latest version of the configuration. After saving, verify the cluster remains healthy by checking the cluster status.
    </Tip>
  </Tab>

  <Tab title="Logs" icon="clock">
    The Logs tab displays deployment and operation logs for all XSDS activities --- bootstrap commands, tier detection, configuration changes, and error output.

    | Column        | Description                                                                               |
    | ------------- | ----------------------------------------------------------------------------------------- |
    | **Timestamp** | Date and time of each log entry                                                           |
    | **Operation** | The XSDS operation that generated the log (bootstrap, tier detection, configuration save) |
    | **Output**    | Command output, status messages, and any error details                                    |

    <Info>
      Review the Logs tab after every XSDS operation to confirm success. Bootstrap operations stream output in real time. If an operation fails, the error details displayed here are the primary diagnostic resource.
    </Info>
  </Tab>
</Tabs>

***

## Validation

After configuring XSDS, verify that the storage cluster is operational:

<Steps>
  <Step title="Verify Bootstrap Completion" icon="circle-check">
    Check the Logs tab for a successful bootstrap message. The bootstrap command should complete without errors, confirming that the monitor and manager daemons are running.
  </Step>

  <Step title="Verify Storage Tiers" icon="layers">
    Switch to the Storage Tiers tab and click **Detect Tiers**. Confirm that the detected device classes match your physical hardware inventory. If tiers have been applied, verify that the corresponding volume types are registered.
  </Step>

  <Step title="Review Configuration" icon="file-text">
    Open the Config File tab and verify that the cluster configuration reflects your network CIDRs, replication settings, and container image versions.
  </Step>

  <Step title="Test Volume Creation" icon="hard-drive">
    Create a test volume using each configured volume type to confirm that data placement follows the tier rules. Verify the volume is accessible and writable from a compute instance.
  </Step>
</Steps>

<Check>
  Storage cluster bootstrapped, tiers configured, and test volumes created successfully.
</Check>

***

## Next Steps

<CardGroup>
  <Card title="Storage Tiers" icon="layers" href="/services/storage/storage-tiers">
    Manage volume types and storage tier policies for tenants
  </Card>

  <Card title="Cloud Fleet" icon="globe" href="/deployment/cloud-fleet">
    Visualize your entire infrastructure topology including storage nodes
  </Card>
</CardGroup>


# Ansible Integration
Source: https://docs.xloud.tech/integrations/ansible

Automate instance configuration, patch management, compliance enforcement, and operational workflows on Xloud using Ansible modules and dynamic inventory.

## Overview

Ansible integrates with Xloud through two complementary mechanisms: the `openstack.cloud`
collection for infrastructure automation, and SSH/WinRM-based playbooks for instance
configuration. Ansible operates agentlessly — no software is installed on your managed instances
beyond a working SSH daemon and Python interpreter.

The Xloud dynamic inventory plugin sources instance metadata directly from the compute API,
automatically organizing hosts by project, availability zone, image, and custom metadata tags.

<Note>
  **Prerequisites**

  * Ansible 2.12 or later installed
  * `openstack.cloud` collection: `ansible-galaxy collection install openstack.cloud`
  * Xloud application credentials or `openrc` file sourced in the shell
  * `openstacksdk` Python library: `pip install openstacksdk`
</Note>

***

## Dynamic Inventory

The `openstack.cloud.openstack` inventory plugin generates a live host list from the Xloud
compute API. Hosts are grouped by instance metadata, eliminating the need to maintain static
inventory files.

```yaml title="inventory/openstack.yml" theme={null}
plugin: openstack.cloud.openstack
auth:
  auth_url: "https://api.<your-domain>:5000/v3"
  username: "{{ lookup('env', 'OS_USERNAME') }}"
  password: "{{ lookup('env', 'OS_PASSWORD') }}"
  project_name: "{{ lookup('env', 'OS_PROJECT_NAME') }}"
  user_domain_name: Default
  project_domain_name: Default
expand_hostvars: true
fail_on_errors: false
groups:
  web_servers: "'web' in name"
  app_servers: "'app' in name"
  db_servers: "'db' in name"
compose:
  ansible_host: access_ipv4
```

Test inventory resolution:

```bash title="List dynamic inventory hosts" theme={null}
ansible-inventory -i inventory/openstack.yml --list
```

***

## Playbook Examples

### OS Bootstrap

Bootstrap a newly provisioned instance with required packages, users, and firewall rules:

```yaml title="playbooks/bootstrap.yml" theme={null}
---
- name: Bootstrap new instances
  hosts: all
  become: true
  vars:
    admin_users:
      - name: deploy
        groups: sudo
        ssh_key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"

  tasks:
    - name: Update package cache and upgrade
      apt:
        update_cache: true
        upgrade: dist
        cache_valid_time: 3600

    - name: Install required packages
      apt:
        name:
          - curl
          - vim
          - htop
          - unattended-upgrades
          - ufw
        state: present

    - name: Create admin users
      user:
        name: "{{ item.name }}"
        groups: "{{ item.groups }}"
        shell: /bin/bash
        create_home: true
      loop: "{{ admin_users }}"

    - name: Add SSH authorized keys
      authorized_key:
        user: "{{ item.name }}"
        key: "{{ item.ssh_key }}"
        state: present
      loop: "{{ admin_users }}"

    - name: Configure UFW default deny
      ufw:
        state: enabled
        direction: incoming
        policy: deny

    - name: Allow SSH
      ufw:
        rule: allow
        port: "22"
        proto: tcp
```

### Patch Management

Apply security patches across all instances in a project:

```yaml title="playbooks/patch.yml" theme={null}
---
- name: Apply security patches
  hosts: all
  become: true
  serial: "25%"

  tasks:
    - name: Update package cache
      apt:
        update_cache: true
        cache_valid_time: 0

    - name: Apply security updates only
      apt:
        upgrade: safe
        update_cache: false

    - name: Check if reboot is required
      stat:
        path: /var/run/reboot-required
      register: reboot_required

    - name: Reboot if kernel was updated
      reboot:
        reboot_timeout: 300
        msg: "Rebooting after kernel patch"
      when: reboot_required.stat.exists

    - name: Verify instance is responsive after reboot
      wait_for_connection:
        delay: 10
        timeout: 120
```

### CIS Compliance Enforcement

Apply CIS baseline hardening to Linux instances:

```yaml title="playbooks/cis-harden.yml" theme={null}
---
- name: Apply CIS Level 1 baseline
  hosts: all
  become: true

  tasks:
    - name: Disable unused filesystems
      lineinfile:
        path: /etc/modprobe.d/disable-filesystems.conf
        line: "install {{ item }} /bin/true"
        create: true
        mode: "0644"
      loop:
        - cramfs
        - freevxfs
        - jffs2
        - hfs
        - hfsplus
        - udf

    - name: Set password minimum length
      lineinfile:
        path: /etc/security/pwquality.conf
        regexp: "^minlen"
        line: "minlen = 14"

    - name: Set SSH MaxAuthTries
      lineinfile:
        path: /etc/ssh/sshd_config
        regexp: "^MaxAuthTries"
        line: "MaxAuthTries 4"
      notify: Restart SSH

    - name: Disable root SSH login
      lineinfile:
        path: /etc/ssh/sshd_config
        regexp: "^PermitRootLogin"
        line: "PermitRootLogin no"
      notify: Restart SSH

    - name: Enable auditd
      service:
        name: auditd
        state: started
        enabled: true

  handlers:
    - name: Restart SSH
      service:
        name: sshd
        state: restarted
```

### Infrastructure Management via Xloud Modules

Create and manage Xloud resources directly from playbooks:

```yaml title="playbooks/provision.yml" theme={null}
---
- name: Provision application tier
  hosts: localhost
  gather_facts: false

  tasks:
    - name: Create application network
      openstack.cloud.network:
        state: present
        name: app-network
        external: false

    - name: Create subnet
      openstack.cloud.subnet:
        state: present
        network_name: app-network
        name: app-subnet
        cidr: 10.200.0.0/24
        dns_nameservers:
          - 8.8.8.8

    - name: Launch application instances
      openstack.cloud.server:
        state: present
        name: "app-{{ item }}"
        image: Ubuntu-22.04
        flavor: m1.medium
        key_name: deployer-key
        network: app-network
        security_groups:
          - default
          - app-sg
        wait: true
      loop: "{{ range(1, 4) | list }}"
      register: created_servers

    - name: Add new instances to in-memory inventory
      add_host:
        hostname: "{{ item.server.access_ipv4 }}"
        groups: new_instances
      loop: "{{ created_servers.results }}"
```

***

## Credential Management

<Tip>
  Store secrets used in playbooks in Xloud Key Manager (Barbican) or HashiCorp Vault.
  Retrieve them at runtime using the `community.general.hashi_vault` or
  `openstack.cloud.identity_user` lookup plugins rather than hardcoding in vars files.
</Tip>

```yaml title="group_vars/all/vault.yml" theme={null}
# Encrypted with ansible-vault
db_password: !vault |
  $ANSIBLE_VAULT;1.1;AES256
  ...encrypted blob...
```

```bash title="Run playbook with vault password" theme={null}
ansible-playbook -i inventory/openstack.yml playbooks/bootstrap.yml \
  --vault-password-file ~/.vault-pass
```

***

## Running Playbooks

<Steps>
  <Step title="Source credentials" icon="key">
    ```bash title="Load Xloud credentials" theme={null}
    source admin-openrc.sh
    ```
  </Step>

  <Step title="Test connectivity" icon="wifi">
    ```bash title="Ping all dynamic inventory hosts" theme={null}
    ansible -i inventory/openstack.yml all -m ping
    ```

    <Check>All targeted instances return a `pong` response.</Check>
  </Step>

  <Step title="Run a playbook" icon="play">
    ```bash title="Run bootstrap playbook" theme={null}
    ansible-playbook -i inventory/openstack.yml playbooks/bootstrap.yml \
      --limit web_servers \
      --diff \
      --check
    ```

    Remove `--check` to apply changes. `--diff` shows what would change on each host.
  </Step>

  <Step title="Verify results" icon="circle-check">
    ```bash title="Check instance facts" theme={null}
    ansible -i inventory/openstack.yml web_servers -m setup \
      -a "filter=ansible_distribution*"
    ```

    <Check>Playbook completes with no failed tasks. Verify changes on instances via SSH or the Dashboard console.</Check>
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Terraform Integration" href="/integrations/terraform">
    Use Terraform for provisioning and Ansible for post-provision configuration
  </Card>

  <Card title="Wazuh Integration" href="/integrations/wazuh">
    Deploy Wazuh agents using Ansible playbooks for SIEM and compliance monitoring
  </Card>

  <Card title="Key Manager" href="/services/key-manager/store-secrets">
    Store playbook secrets in Xloud Key Manager for secure credential retrieval
  </Card>

  <Card title="Auto-Scaling" href="/services/orchestration/autoscaling">
    Bootstrap auto-scaled instances using Ansible cloud-init integration
  </Card>
</CardGroup>


# Grafana Dashboards
Source: https://docs.xloud.tech/integrations/grafana

Configure Grafana data sources for Prometheus and Xloud services, import pre-built operational dashboards, and build custom visualizations for compute, storage, and networking.

## Overview

Grafana connects to Prometheus and other Xloud data sources to provide operational dashboards
for infrastructure metrics, service health, storage utilization, and network performance.
Grafana is included in the XIMP monitoring stack and is pre-configured with a Prometheus
data source pointing to the cluster's Prometheus instance.

<Note>
  **Prerequisites**

  * Grafana 9.0 or later (included in XIMP stack)
  * Prometheus deployed and scraping targets ([Prometheus integration](/integrations/prometheus))
  * Grafana admin credentials (default: sourced from XDeploy configuration)
  * Network access from the Grafana host to Prometheus on port 9090
</Note>

***

## Data Source Configuration

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open data source settings" icon="database">
        Log in to Grafana and navigate to **Configuration → Data Sources → Add data source**.
        Select **Prometheus** from the time series databases section.
      </Step>

      <Step title="Configure connection" icon="settings">
        | Field           | Value                   | Description                                           |
        | --------------- | ----------------------- | ----------------------------------------------------- |
        | Name            | `Xloud Prometheus`      | Display name in dashboard queries                     |
        | URL             | `http://10.0.1.71:9090` | Prometheus server address                             |
        | Scrape interval | `15s`                   | Must match `global.scrape_interval` in prometheus.yml |
        | HTTP Method     | `POST`                  | Required for long queries                             |

        Leave authentication blank if Prometheus has no auth configured. If basic auth is
        enabled, enter credentials under the **Auth** section.
      </Step>

      <Step title="Test and save" icon="circle-check">
        Click **Save & Test**. Grafana sends a test query to Prometheus.

        <Check>Status shows **Data source is working** with a sample metric count.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI (API)" icon="terminal">
    Provision data sources programmatically using the Grafana HTTP API — useful for
    automated deployments:

    ```bash title="Add Prometheus data source via API" theme={null}
    curl -X POST http://admin:$GRAFANA_PASS@localhost:3000/api/datasources \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Xloud Prometheus",
        "type": "prometheus",
        "url": "http://10.0.1.71:9090",
        "access": "proxy",
        "isDefault": true,
        "jsonData": {
          "httpMethod": "POST",
          "scrapeInterval": "15s"
        }
      }'
    ```

    <Check>Response returns `{"message": "Datasource added", "id": 1}` with the new data source ID.</Check>
  </Tab>
</Tabs>

***

## Pre-Built Dashboard Templates

### Node Exporter Full Dashboard

The Node Exporter Full dashboard (Grafana ID `1860`) provides comprehensive per-host metrics
including CPU, memory, disk I/O, network throughput, and system load.

<Tabs>
  <Tab title="Import via ID" icon="download">
    <Steps>
      <Step title="Open import dialog" icon="folder-open">
        Navigate to **Dashboards → Import**. Enter dashboard ID `1860` in the
        **Import via grafana.com** field and click **Load**.
      </Step>

      <Step title="Select data source" icon="database">
        Select **Xloud Prometheus** from the data source dropdown. Set the dashboard
        name and folder, then click **Import**.
      </Step>
    </Steps>

    <Check>The Node Exporter Full dashboard appears with live data for all scraped instances.</Check>
  </Tab>

  <Tab title="Import via JSON" icon="file-code">
    For air-gapped environments, download and import the dashboard JSON directly:

    ```bash title="Import dashboard JSON via API" theme={null}
    curl -X POST http://admin:$GRAFANA_PASS@localhost:3000/api/dashboards/import \
      -H "Content-Type: application/json" \
      -d @node-exporter-full.json
    ```
  </Tab>
</Tabs>

### Recommended Dashboard IDs

| Dashboard          | Grafana ID | Description                                            |
| ------------------ | ---------- | ------------------------------------------------------ |
| Node Exporter Full | `1860`     | Complete per-host metrics — CPU, memory, disk, network |
| Ceph Cluster       | `2842`     | Ceph OSD, pool, and IOPS metrics                       |
| Prometheus Stats   | `2`        | Prometheus internal metrics and scrape health          |
| Alertmanager       | `9578`     | Alert routing and notification delivery status         |

***

## Custom Dashboard Configuration

### Infrastructure Overview Panel

Create a summary row with stat panels for key fleet metrics:

```json title="CPU Usage Stat Panel (panel JSON fragment)" theme={null}
{
  "type": "stat",
  "title": "Average CPU Usage",
  "targets": [
    {
      "expr": "100 - avg(rate(node_cpu_seconds_total{mode='idle'}[5m])) * 100",
      "legendFormat": "CPU %"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "percent",
      "thresholds": {
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 60 },
          { "color": "red", "value": 80 }
        ]
      }
    }
  }
}
```

### Auto-Scaling Group Size Panel

Track the current size of Orchestration auto-scaling groups using a time series panel:

```promql title="Auto-scaling group size query" theme={null}
# Requires the Heat exporter or custom metric pushed from Orchestration stack outputs
xloud_asg_current_size{stack="web-asg-stack"}
```

<Tip>
  Export stack outputs to Prometheus using a cron job that calls the Orchestration API and
  pushes a gauge metric via the Prometheus Pushgateway. This provides real-time ASG size
  visibility in Grafana without a dedicated Heat exporter.
</Tip>

***

## Alerting in Grafana

Grafana can evaluate alert rules against Prometheus queries and route notifications
independently of Alertmanager. Use this for dashboard-level alerts that notify specific
teams via Slack, email, or PagerDuty.

```yaml title="Grafana alert rule example" theme={null}
# Configured via Grafana UI: Alerting → Alert rules → New alert rule
# Expression: avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0.8
# For: 2m
# Labels: severity=warning, team=ops
# Annotations: summary="High CPU detected on {{ $labels.instance }}"
```

Configure notification channels:

<Steps>
  <Step title="Add a contact point" icon="bell">
    Navigate to **Alerting → Contact points → Add contact point**.
    Select the notification type (Email, Slack, PagerDuty, Webhook) and configure
    the destination.
  </Step>

  <Step title="Create a notification policy" icon="route">
    Navigate to **Alerting → Notification policies**. Create a policy that matches
    your alert labels (e.g., `severity=critical`) and routes to the appropriate
    contact point.
  </Step>

  <Step title="Create an alert rule" icon="circle-x">
    Navigate to **Alerting → Alert rules → New alert rule**. Select the Prometheus
    data source, write the PromQL expression, set evaluation parameters, and assign
    the notification policy.
  </Step>
</Steps>

***

## Dashboard Variables

Use template variables to make dashboards interactive across multiple instances, projects,
and availability zones:

```promql title="Instance variable query" theme={null}
label_values(node_uname_info, instance)
```

```promql title="Job variable query" theme={null}
label_values(up, job)
```

Add variables under **Dashboard Settings → Variables → Add variable**. Set query type
to **Query**, select the Prometheus data source, and enter the label values query.

***

## Next Steps

<CardGroup>
  <Card title="Prometheus Integration" href="/integrations/prometheus">
    Configure Prometheus scrape targets and alert rules that feed Grafana data sources
  </Card>

  <Card title="XIMP Monitoring" href="/services/monitoring/user-guide/dashboards">
    Explore the built-in XIMP monitoring stack that includes pre-configured Grafana
  </Card>

  <Card title="Auto-Scaling" href="/services/orchestration/autoscaling">
    Visualize auto-scaling group size changes on Grafana time series dashboards
  </Card>

  <Card title="Wazuh SIEM" href="/integrations/wazuh">
    Integrate Wazuh security events into Grafana for unified security dashboards
  </Card>
</CardGroup>


# Integrations
Source: https://docs.xloud.tech/integrations/index

Connect Xloud Cloud Platform with third-party tools for infrastructure automation, monitoring, security, and observability.

<p>
  Xloud Cloud Platform integrates with industry-standard tools across infrastructure automation,
  monitoring, observability, and security. Use these integrations to build fully automated
  provisioning pipelines, gain deep operational visibility, and enforce security compliance
  across your cloud environment.
</p>

***

<CardGroup>
  <Card title="Terraform" icon="layers" href="/integrations/terraform">
    Provision and manage Xloud infrastructure declaratively using the Terraform provider for
    compute, networking, and storage.
  </Card>

  <Card title="Ansible" icon="terminal" href="/integrations/ansible">
    Automate OS configuration, patch management, and compliance enforcement across instances
    using Ansible modules and dynamic inventory.
  </Card>

  <Card title="Prometheus" icon="chart-line" href="/integrations/prometheus">
    Collect infrastructure and service metrics from Xloud, configure alerting rules, and
    wire alert webhooks into orchestration scaling policies.
  </Card>

  <Card title="Grafana" icon="gauge" href="/integrations/grafana">
    Build operational dashboards on top of Prometheus data sources with pre-built dashboard
    templates for compute, storage, and networking.
  </Card>

  <Card title="Wazuh" icon="shield-check" href="/integrations/wazuh">
    Deploy Wazuh agents on Xloud instances for real-time threat detection, log collection,
    vulnerability scanning, and SIEM integration.
  </Card>
</CardGroup>

***

<CardGroup>
  <Card title="Automation Tools" icon="wrench">
    **Terraform** and **Ansible** cover the full infrastructure lifecycle — from provisioning
    new instances and networks to day-2 configuration, patch management, and compliance
    enforcement. Both tools integrate natively with the Xloud API.
  </Card>

  <Card title="Observability Stack" icon="activity">
    **Prometheus** collects metrics from Xloud services and instances. **Grafana** provides
    visualization dashboards. Together they form a production-grade observability layer that
    also drives auto-scaling signal delivery to Xloud Orchestration.
  </Card>

  <Card title="Security & Compliance" icon="lock">
    **Wazuh** provides a unified SIEM, intrusion detection, and compliance monitoring layer
    across all instances. Agent deployment is automated using Ansible, with log forwarding
    integrated into the central Wazuh manager.
  </Card>

  <Card title="API & CLI Access" icon="code">
    All Xloud services expose a REST API. The `openstack` CLI and Terraform provider interact
    directly with these APIs using application credentials or token-based authentication
    from Xloud Identity.
  </Card>
</CardGroup>


# Prometheus Integration
Source: https://docs.xloud.tech/integrations/prometheus

Configure Prometheus to scrape metrics from Xloud services and instances, set up service discovery, define alert rules, and wire Alertmanager webhooks into Orchestration scaling policies.

## Overview

Prometheus is the primary metrics backend for Xloud environments. It collects time-series
metrics from compute nodes, storage clusters, and deployed services through scrape targets
and service discovery. Alertmanager receives rule evaluation results from Prometheus and
routes alert notifications — including webhook signals that trigger Xloud Orchestration
auto-scaling policies.

Prometheus replaces legacy telemetry stacks (Ceilometer/Aodh) for metric collection and
alarm-based scaling in Xloud deployments.

<Note>
  **Prerequisites**

  * Prometheus 2.40 or later deployed (included in XIMP monitoring stack)
  * Alertmanager 0.25 or later
  * Node exporter deployed on all instances to be monitored
  * Network access from the Prometheus host to scrape targets on port 9100 (node exporter)
</Note>

***

## Architecture

```mermaid theme={null}
graph TD
    NE[Node Exporter<br/>port 9100] -->|scrape| PROM[Prometheus]
    CE[Ceph Exporter<br/>port 9095] -->|scrape| PROM
    PROM -->|evaluate rules| AM[Alertmanager]
    AM -->|webhook POST| SCALE[Orchestration<br/>Signal URL]
    AM -->|email / Slack| OPS[Operations Team]
    PROM -->|query| GF[Grafana]
    style PROM fill:#197560,color:#fff
    style AM fill:#145C4C,color:#fff
```

***

## Prometheus Configuration

### Base Configuration

```yaml title="prometheus.yml" theme={null}
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s
  external_labels:
    environment: "production"
    region: "RegionOne"

rule_files:
  - "/etc/prometheus/rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "alertmanager:9093"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets:
          - "10.0.1.71:9100"
          - "10.0.1.72:9100"
          - "10.0.1.75:9100"

  - job_name: "ceph"
    static_configs:
      - targets: ["10.0.1.71:9095"]
```

### Service Discovery via Xloud API

Use the `openstack_sd_configs` scrape configuration to automatically discover instances
by project and assign labels from instance metadata:

```yaml title="scrape-config-discovery.yml" theme={null}
scrape_configs:
  - job_name: "xloud_instances"
    openstack_sd_configs:
      - identity_endpoint: "https://api.<your-domain>:5000/v3"
        username: "prometheus"
        password: "{{ OS_PASSWORD }}"
        domain_name: Default
        project_name: monitoring
        region: RegionOne
        role: instance
        port: 9100
        tls_config:
          insecure_skip_verify: false

    relabel_configs:
      - source_labels: [__meta_openstack_instance_name]
        target_label: instance
      - source_labels: [__meta_openstack_project_id]
        target_label: project
      - source_labels: [__meta_openstack_tag_role]
        target_label: role
      - source_labels: [__meta_openstack_instance_status]
        regex: ACTIVE
        action: keep
```

<Tip>
  Tag instances with `role=web`, `role=app`, or `role=db` via instance metadata to enable
  role-based Prometheus label filtering and targeted alert rule evaluation.
</Tip>

***

## Alert Rules

### Infrastructure Alert Rules

```yaml title="/etc/prometheus/rules/infrastructure.yml" theme={null}
groups:
  - name: infrastructure
    interval: 30s
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is unreachable"
          description: "Prometheus has not received a scrape response for 2 minutes."

      - alert: HighCpuUsage
        expr: >
          100 - (avg by (instance) (
            rate(node_cpu_seconds_total{mode="idle"}[2m])
          ) * 100) > 80
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.1f\" }}% — above 80% threshold."

      - alert: LowCpuUsage
        expr: >
          100 - (avg by (instance) (
            rate(node_cpu_seconds_total{mode="idle"}[10m])
          ) * 100) < 20
        for: 10m
        labels:
          severity: info
        annotations:
          summary: "Low CPU on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.1f\" }}% — below 20% threshold."

      - alert: HighMemoryUsage
        expr: >
          (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value | printf \"%.1f\" }}%."

      - alert: DiskSpaceLow
        expr: >
          (node_filesystem_avail_bytes{mountpoint="/"} /
           node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Root filesystem has {{ $value | printf \"%.1f\" }}% space remaining."
```

### Auto-Scaling Alert Rules

Wire these alert rules into Alertmanager webhook receivers to drive Xloud Orchestration
scaling policies:

```yaml title="/etc/prometheus/rules/autoscaling.yml" theme={null}
groups:
  - name: autoscaling
    rules:
      - alert: ScaleOutWeb
        expr: >
          avg(rate(node_cpu_seconds_total{mode!="idle",role="web"}[2m])) > 0.80
        for: 2m
        labels:
          severity: warning
          action: scale_out
          tier: web
        annotations:
          summary: "Web tier CPU high — scale out"

      - alert: ScaleInWeb
        expr: >
          avg(rate(node_cpu_seconds_total{mode!="idle",role="web"}[10m])) < 0.20
        for: 10m
        labels:
          severity: info
          action: scale_in
          tier: web
        annotations:
          summary: "Web tier CPU low — scale in"
```

***

## Alertmanager Configuration

```yaml title="alertmanager.yml" theme={null}
global:
  resolve_timeout: 5m

route:
  receiver: "default"
  group_by: ["alertname", "tier"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        action: scale_out
        tier: web
      receiver: "web-scale-out"
      repeat_interval: 2m

    - match:
        action: scale_in
        tier: web
      receiver: "web-scale-in"
      repeat_interval: 12m

    - match:
        severity: critical
      receiver: "ops-critical"

receivers:
  - name: "default"
    email_configs:
      - to: "ops@example.com"
        from: "alertmanager@xloud.tech"
        smarthost: "smtp.xloud.tech:587"

  - name: "web-scale-out"
    webhook_configs:
      - url: "<scale_out_url from orchestration stack output>"
        send_resolved: false

  - name: "web-scale-in"
    webhook_configs:
      - url: "<scale_in_url from orchestration stack output>"
        send_resolved: false

  - name: "ops-critical"
    email_configs:
      - to: "oncall@example.com"
        from: "alertmanager@xloud.tech"
        smarthost: "smtp.xloud.tech:587"

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["instance"]
```

***

## Useful Queries

| Query                                                         | Purpose                                  |
| ------------------------------------------------------------- | ---------------------------------------- |
| `up`                                                          | Check which scrape targets are reachable |
| `rate(node_cpu_seconds_total{mode!="idle"}[5m])`              | CPU utilization per core                 |
| `node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes` | Memory availability ratio                |
| `node_filesystem_avail_bytes{mountpoint="/"}`                 | Root disk free bytes                     |
| `rate(node_network_receive_bytes_total[5m])`                  | Network ingress rate                     |
| `node_load1`                                                  | 1-minute load average                    |

***

## Validation

<Tabs>
  <Tab title="Prometheus UI" icon="gauge">
    Navigate to `http://<prometheus-host>:9090`:

    1. Open **Status → Targets** — all scrape targets show **UP** state
    2. Open **Alerts** — configured rules appear with their evaluation state
    3. Run a query: enter `up` in the expression bar and click **Execute**

    <Check>All expected targets appear with state `UP` and no scrape errors.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Check Prometheus health" theme={null}
    curl -s http://localhost:9090/-/healthy
    ```

    ```bash title="Query active alerts via API" theme={null}
    curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | .labels'
    ```

    ```bash title="Check Alertmanager status" theme={null}
    curl -s http://localhost:9093/-/healthy
    ```

    <Check>All endpoints return `Healthy` status and alert list matches configured rules.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Grafana Dashboards" href="/integrations/grafana">
    Build operational dashboards using Prometheus as a data source
  </Card>

  <Card title="Auto-Scaling" href="/services/orchestration/autoscaling">
    Wire Alertmanager webhooks into Orchestration scaling policy signal URLs
  </Card>

  <Card title="Wazuh SIEM" href="/integrations/wazuh">
    Complement Prometheus metrics with Wazuh security event monitoring
  </Card>

  <Card title="XIMP Monitoring" href="/services/monitoring/user-guide/dashboards">
    Explore the built-in XIMP monitoring stack that includes Prometheus
  </Card>
</CardGroup>


# Terraform Integration
Source: https://docs.xloud.tech/integrations/terraform

Use the Terraform provider for Xloud to provision and manage compute instances, networks, volumes, and stacks as infrastructure-as-code.

## Overview

The Terraform provider for Xloud uses the OpenStack provider (`hashicorp/openstack`) to manage
compute, networking, storage, and identity resources. You store infrastructure definitions in
version-controlled `.tf` files, enabling repeatable deployments, drift detection, and change
review workflows via standard CI/CD tooling.

<Note>
  **Prerequisites**

  * Terraform 1.3 or later installed ([terraform.io](https://terraform.io))
  * Xloud application credentials or an `openrc` file from the Dashboard
  * Network and security group resources created in your project (or managed via Terraform)
</Note>

***

## Provider Configuration

<Tabs>
  <Tab title="Environment Variables" icon="terminal">
    Source your credentials file before running Terraform commands. The provider reads
    standard `OS_*` environment variables:

    ```bash title="Load Xloud credentials" theme={null}
    source admin-openrc.sh
    ```

    ```hcl title="provider.tf" theme={null}
    terraform {
      required_providers {
        openstack = {
          source  = "terraform-provider-openstack/openstack"
          version = "~> 2.0"
        }
      }
    }

    provider "openstack" {
      # Credentials are read from OS_* environment variables
    }
    ```
  </Tab>

  <Tab title="Inline Configuration" icon="file-code">
    For CI/CD pipelines where environment variables are injected as secrets:

    ```hcl title="provider.tf" theme={null}
    terraform {
      required_providers {
        openstack = {
          source  = "terraform-provider-openstack/openstack"
          version = "~> 2.0"
        }
      }
    }

    provider "openstack" {
      auth_url    = "https://api.<your-domain>:5000/v3"
      region      = "RegionOne"
      tenant_name = var.project_name
      user_name   = var.username
      password    = var.password
    }
    ```

    <Warning>
      Store credentials in CI/CD secret variables, not in `.tf` files. Never commit
      credentials to source control.
    </Warning>
  </Tab>
</Tabs>

***

## Resource Examples

### Compute Instance

```hcl title="compute.tf" theme={null}
data "openstack_compute_flavor_v2" "small" {
  name = "m1.small"
}

data "openstack_images_image_v2" "ubuntu" {
  name        = "Ubuntu-22.04"
  most_recent = true
}

data "openstack_networking_network_v2" "private" {
  name = "private"
}

resource "openstack_compute_keypair_v2" "deployer" {
  name       = "deployer-key"
  public_key = file("~/.ssh/id_rsa.pub")
}

resource "openstack_compute_instance_v2" "web" {
  name            = "web-server-01"
  image_id        = data.openstack_images_image_v2.ubuntu.id
  flavor_id       = data.openstack_compute_flavor_v2.small.id
  key_pair        = openstack_compute_keypair_v2.deployer.name
  security_groups = ["default", "web-sg"]

  network {
    name = data.openstack_networking_network_v2.private.name
  }

  user_data = <<-EOF
    #!/bin/bash
    apt-get update -y
    apt-get install -y nginx
    systemctl enable --now nginx
  EOF
}

output "instance_ip" {
  value = openstack_compute_instance_v2.web.access_ip_v4
}
```

### Network and Router

```hcl title="networking.tf" theme={null}
resource "openstack_networking_network_v2" "app_net" {
  name           = "app-network"
  admin_state_up = true
}

resource "openstack_networking_subnet_v2" "app_subnet" {
  name            = "app-subnet"
  network_id      = openstack_networking_network_v2.app_net.id
  cidr            = "10.100.0.0/24"
  ip_version      = 4
  dns_nameservers = ["8.8.8.8", "8.8.4.4"]
}

resource "openstack_networking_router_v2" "app_router" {
  name                = "app-router"
  admin_state_up      = true
  external_network_id = var.external_network_id
}

resource "openstack_networking_router_interface_v2" "app_router_iface" {
  router_id = openstack_networking_router_v2.app_router.id
  subnet_id = openstack_networking_subnet_v2.app_subnet.id
}
```

### Block Storage Volume

```hcl title="storage.tf" theme={null}
resource "openstack_blockstorage_volume_v3" "data_vol" {
  name        = "data-volume-01"
  size        = 100
  volume_type = "ceph-ssd"
  description = "Application data volume"
}

resource "openstack_compute_volume_attach_v2" "data_attach" {
  instance_id = openstack_compute_instance_v2.web.id
  volume_id   = openstack_blockstorage_volume_v3.data_vol.id
}
```

### Floating IP

```hcl title="floating-ip.tf" theme={null}
resource "openstack_networking_floatingip_v2" "web_fip" {
  pool = "external"
}

resource "openstack_compute_floatingip_associate_v2" "web_fip_assoc" {
  floating_ip = openstack_networking_floatingip_v2.web_fip.address
  instance_id = openstack_compute_instance_v2.web.id
}

output "public_ip" {
  value = openstack_networking_floatingip_v2.web_fip.address
}
```

***

## State Management

<Tabs>
  <Tab title="Remote State (Recommended)" icon="database">
    Store Terraform state remotely to enable collaboration and prevent state file conflicts.
    Use an S3-compatible backend pointed at Xloud Object Storage:

    ```hcl title="backend.tf" theme={null}
    terraform {
      backend "s3" {
        bucket                      = "terraform-state"
        key                         = "prod/web-tier/terraform.tfstate"
        region                      = "us-east-1"
        endpoint                    = "https://object.<your-domain>"
        access_key                  = var.swift_access_key
        secret_key                  = var.swift_secret_key
        skip_credentials_validation = true
        skip_metadata_api_check     = true
        skip_region_validation      = true
        force_path_style            = true
      }
    }
    ```
  </Tab>

  <Tab title="Local State" icon="hard-drive">
    For single-operator or development use, Terraform defaults to local state storage in
    `terraform.tfstate`. Commit this file to a private repository or use
    `.gitignore` to exclude it from shared repositories.

    <Warning>
      Local state is not suitable for team workflows. Multiple operators running
      `terraform apply` concurrently against local state will corrupt state.
      Use remote state for any shared environment.
    </Warning>
  </Tab>
</Tabs>

***

## Workflow

<Steps>
  <Step title="Initialize the working directory" icon="folder">
    ```bash title="Initialize Terraform" theme={null}
    terraform init
    ```

    Downloads the provider plugin and configures the backend.
  </Step>

  <Step title="Review the execution plan" icon="eye">
    ```bash title="Show plan" theme={null}
    terraform plan -out=tfplan
    ```

    Review all resources that will be created, modified, or destroyed before applying.
  </Step>

  <Step title="Apply the configuration" icon="play">
    ```bash title="Apply configuration" theme={null}
    terraform apply tfplan
    ```

    <Check>Resources are created and outputs are printed. Verify instances appear in the Dashboard under **Project → Compute → Instances**.</Check>
  </Step>

  <Step title="Destroy when done" icon="trash-2">
    ```bash title="Destroy all resources" theme={null}
    terraform destroy
    ```

    All resources defined in the configuration are removed. Use for development or ephemeral environments.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Ansible Integration" href="/integrations/ansible">
    Combine Terraform provisioning with Ansible for post-provision configuration management
  </Card>

  <Card title="Auto-Scaling with Orchestration" href="/services/orchestration/autoscaling">
    Deploy auto-scaling stacks via Terraform and wire Prometheus alerts for dynamic scaling
  </Card>

  <Card title="Block Storage" href="/services/storage/create-volume">
    Create and attach persistent volumes to Terraform-managed instances
  </Card>

  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Generate non-expiring credentials for use in Terraform pipelines
  </Card>
</CardGroup>


# Wazuh SIEM Integration
Source: https://docs.xloud.tech/integrations/wazuh

Deploy Wazuh agents on Xloud instances for real-time threat detection, file integrity monitoring, log collection, vulnerability scanning, and centralized SIEM visibility.

## Overview

Wazuh provides a unified security information and event management (SIEM) platform for your Xloud
environment. Agents deployed on your compute instances forward security events, system logs, and
file integrity alerts to a central Wazuh manager. The manager correlates events across all
monitored instances, applies detection rules, and generates alerts for security incidents,
compliance violations, and vulnerability findings.

<Note>
  **Prerequisites**

  * Wazuh manager 4.7 or later deployed (standalone or cluster)
  * Network access from all instances to the Wazuh manager on TCP port 1514 (agent enrollment)
    and TCP port 1515 (agent registration)
  * `sudo` or root access on instances for agent installation
  * Ansible for automated bulk deployment (recommended for more than 5 instances)
</Note>

***

## Architecture

```mermaid theme={null}
graph TD
    A1[Instance: web-01<br/>Wazuh Agent] -->|TLS 1514| MGR[Wazuh Manager]
    A2[Instance: app-01<br/>Wazuh Agent] -->|TLS 1514| MGR
    A3[Instance: db-01<br/>Wazuh Agent] -->|TLS 1514| MGR
    MGR -->|index events| IDX[Wazuh Indexer<br/>OpenSearch]
    IDX -->|query| DASH[Wazuh Dashboard]
    MGR -->|alerts| EMAIL[Email / Slack]
    style MGR fill:#197560,color:#fff
    style DASH fill:#145C4C,color:#fff
```

### Components

| Component           | Role                                                                                             |
| ------------------- | ------------------------------------------------------------------------------------------------ |
| **Wazuh Agent**     | Installed on each monitored instance — collects logs, monitors files, and reports to the manager |
| **Wazuh Manager**   | Receives agent data, evaluates detection rules, and generates security alerts                    |
| **Wazuh Indexer**   | OpenSearch-based index for storing and querying security events                                  |
| **Wazuh Dashboard** | Web UI for alert review, compliance reports, and agent management                                |

***

## Agent Deployment

### Manual Installation (Single Instance)

<Tabs>
  <Tab title="Linux (Ubuntu/Debian)" icon="terminal">
    <Steps>
      <Step title="Add Wazuh repository" icon="download">
        ```bash title="Add Wazuh GPG key and repository" theme={null}
        curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | \
          gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg \
          --import && chmod 644 /usr/share/keyrings/wazuh.gpg

        echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] \
          https://packages.wazuh.com/4.x/apt/ stable main" | \
          sudo tee /etc/apt/sources.list.d/wazuh.list

        sudo apt-get update
        ```
      </Step>

      <Step title="Install the agent" icon="package">
        ```bash title="Install Wazuh agent" theme={null}
        WAZUH_MANAGER="<wazuh-manager-ip>" \
        WAZUH_AGENT_NAME="$(hostname)" \
        sudo apt-get install -y wazuh-agent
        ```

        Replace `<wazuh-manager-ip>` with the IP address of your Wazuh manager.
      </Step>

      <Step title="Enable and start the agent" icon="play">
        ```bash title="Start Wazuh agent" theme={null}
        sudo systemctl daemon-reload
        sudo systemctl enable wazuh-agent
        sudo systemctl start wazuh-agent
        ```

        <Check>Agent registers with the manager. Verify in the Wazuh Dashboard under **Agents** — the instance appears with status **Active**.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Linux (RHEL/Rocky)" icon="terminal">
    <Steps>
      <Step title="Add Wazuh repository" icon="download">
        ```bash title="Add Wazuh RPM repository" theme={null}
        sudo rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH

        cat > /etc/yum.repos.d/wazuh.repo << 'EOF'
        [wazuh]
        gpgcheck=1
        gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
        enabled=1
        name=EL-$releasever - Wazuh
        baseurl=https://packages.wazuh.com/4.x/yum/
        protect=1
        EOF
        ```
      </Step>

      <Step title="Install and start" icon="play">
        ```bash title="Install Wazuh agent" theme={null}
        WAZUH_MANAGER="<wazuh-manager-ip>" \
        WAZUH_AGENT_NAME="$(hostname)" \
        sudo yum install -y wazuh-agent

        sudo systemctl daemon-reload
        sudo systemctl enable --now wazuh-agent
        ```

        <Check>Agent registers with the manager and appears **Active** in the Dashboard.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Bulk Deployment via Ansible

Deploy Wazuh agents to all Xloud instances using the Ansible dynamic inventory:

```yaml title="playbooks/wazuh-deploy.yml" theme={null}
---
- name: Deploy Wazuh agent to all instances
  hosts: all
  become: true
  vars:
    wazuh_manager_ip: "10.0.1.71"
    wazuh_version: "4.7"

  tasks:
    - name: Add Wazuh GPG key (Debian/Ubuntu)
      apt_key:
        url: https://packages.wazuh.com/key/GPG-KEY-WAZUH
        state: present
      when: ansible_os_family == "Debian"

    - name: Add Wazuh repository (Debian/Ubuntu)
      apt_repository:
        repo: >
          deb https://packages.wazuh.com/4.x/apt/ stable main
        state: present
        filename: wazuh
      when: ansible_os_family == "Debian"

    - name: Install Wazuh agent (Debian/Ubuntu)
      apt:
        name: wazuh-agent
        state: present
        update_cache: true
      environment:
        WAZUH_MANAGER: "{{ wazuh_manager_ip }}"
        WAZUH_AGENT_NAME: "{{ inventory_hostname }}"
      when: ansible_os_family == "Debian"

    - name: Enable and start Wazuh agent
      systemd:
        name: wazuh-agent
        enabled: true
        state: started
        daemon_reload: true

    - name: Verify agent is running
      command: systemctl is-active wazuh-agent
      register: agent_status
      changed_when: false

    - name: Confirm agent status
      assert:
        that: agent_status.stdout == "active"
        fail_msg: "Wazuh agent is not running on {{ inventory_hostname }}"
```

Run the playbook using the Xloud dynamic inventory:

```bash title="Deploy Wazuh agents to all instances" theme={null}
ansible-playbook -i inventory/openstack.yml playbooks/wazuh-deploy.yml
```

***

## Log Collection Configuration

Configure the agent to forward specific log files to the Wazuh manager for centralized
analysis:

```xml title="/var/ossec/etc/ossec.conf (log collection section)" theme={null}
<ossec_config>
  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/nginx/access.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/nginx/error.log</location>
  </localfile>

  <localfile>
    <log_format>json</log_format>
    <location>/var/log/app/*.json</location>
  </localfile>
</ossec_config>
```

***

## File Integrity Monitoring

Wazuh monitors filesystem paths for unauthorized modifications — files added, deleted, or
modified outside of expected change windows trigger alerts:

```xml title="/var/ossec/etc/ossec.conf (FIM section)" theme={null}
<ossec_config>
  <syscheck>
    <frequency>3600</frequency>
    <alert_new_files>yes</alert_new_files>

    <directories check_all="yes" report_changes="yes" realtime="yes">
      /etc
    </directories>
    <directories check_all="yes" report_changes="yes">
      /usr/bin
      /usr/sbin
      /bin
      /sbin
    </directories>

    <ignore>/etc/mtab</ignore>
    <ignore>/etc/resolv.conf</ignore>
    <ignore type="sregex">.log$|.swp$</ignore>
  </syscheck>
</ossec_config>
```

***

## Compliance Reporting

Wazuh includes pre-built compliance rule mappings for common frameworks. Enable compliance
scanning in the agent configuration:

| Framework        | Coverage               | Wazuh Rule Group         |
| ---------------- | ---------------------- | ------------------------ |
| CIS Ubuntu 22.04 | Level 1 and Level 2    | `cis_ubuntu_linux_22-04` |
| PCI DSS 3.2.1    | Requirements 6, 10, 11 | `pci_dss`                |
| HIPAA            | Security rule subset   | `hipaa`                  |
| NIST 800-53      | Control families       | `nist_800_53`            |

View compliance dashboards in the Wazuh Dashboard under **Security → Regulatory Compliance**.

***

## Verification

<Tabs>
  <Tab title="Wazuh Dashboard" icon="gauge">
    Navigate to the Wazuh Dashboard at `http://<wazuh-dashboard-host>:5601`:

    1. Open **Agents** — all deployed instances appear with status **Active**
    2. Open **Security Events** — incoming events from agents are visible in real time
    3. Open **Integrity Monitoring** — file change events appear per monitored path
    4. Open **Regulatory Compliance** — compliance scores per instance

    <Check>All agents show **Active** status. Events are flowing from monitored instances.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Check agent registration from the Wazuh manager:

    ```bash title="List all registered agents" theme={null}
    sudo /var/ossec/bin/agent_control -lc
    ```

    Check agent connectivity status:

    ```bash title="Show agent detail" theme={null}
    sudo /var/ossec/bin/agent_control -i <agent-id>
    ```

    Test rule evaluation with a sample log entry:

    ```bash title="Test Wazuh rule engine" theme={null}
    sudo /var/ossec/bin/wazuh-logtest
    ```

    <Check>All agents appear with status `Active`. Log test confirms rules are evaluated correctly.</Check>
  </Tab>
</Tabs>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Agent shows Disconnected in Wazuh Dashboard" icon="wifi-off">
    **Cause**: The agent cannot reach the Wazuh manager on TCP port 1514, or the agent
    service stopped.

    **Resolution**:

    ```bash title="Check agent service status" theme={null}
    sudo systemctl status wazuh-agent
    ```

    ```bash title="Test connectivity to manager" theme={null}
    nc -zv <wazuh-manager-ip> 1514
    ```

    Verify that the security group for the instance allows outbound TCP 1514 to the manager.
  </Accordion>

  <Accordion title="Agent registered but no events in Dashboard" icon="clock">
    **Cause**: Log collection paths do not exist, or the agent configuration has a syntax
    error.

    **Resolution**:

    ```bash title="Validate agent configuration" theme={null}
    sudo /var/ossec/bin/wazuh-logtest -V
    ```

    ```bash title="Restart agent after config change" theme={null}
    sudo systemctl restart wazuh-agent
    ```

    Check `/var/ossec/logs/ossec.log` on the instance for parsing errors.
  </Accordion>

  <Accordion title="Bulk deployment fails on some instances" icon="circle-x">
    **Cause**: Package manager repository not reachable from instance (outbound internet
    blocked), or wrong OS family detected.

    **Resolution**: Ensure instances have outbound HTTP/HTTPS access to
    `packages.wazuh.com`, or host the Wazuh packages internally and update the repository
    URL in the playbook. Use `--limit` to re-run the playbook on failed hosts only.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Ansible Integration" href="/integrations/ansible">
    Automate Wazuh agent deployment and configuration updates using Ansible playbooks
  </Card>

  <Card title="Prometheus Integration" href="/integrations/prometheus">
    Complement Wazuh security events with infrastructure metrics from Prometheus
  </Card>

  <Card title="Grafana Dashboards" href="/integrations/grafana">
    Build unified security and operations dashboards combining Wazuh and Prometheus data
  </Card>

  <Card title="Key Manager" href="/services/key-manager/store-secrets">
    Store Wazuh registration keys securely in Xloud Key Manager
  </Card>
</CardGroup>


# Xloud
Source: https://docs.xloud.tech/introduction

User guides, admin references, CLI documentation, and resources for the Xloud Cloud Platform.

<div>
  <div>
    <p>
      Compute  ·  Storage  ·  Networking  ·  HCI  ·  Monitoring  ·  Disaster Recovery
    </p>
  </div>

  <div>
    <img alt="Xloud Cloud Platform Services" />
  </div>
</div>

***

## Featured Services

<CardGroup>
  <Card title="Compute" icon="cpu" href="/services/compute">
    Virtual machines on bare-metal hypervisors
  </Card>

  <Card title="Networking" icon="network" href="/services/networking/index">
    SDN, floating IPs, security groups
  </Card>

  <Card title="Block Storage" icon="hard-drive" href="/services/storage/index">
    Persistent volumes, snapshots, and backups
  </Card>
</CardGroup>

***

## Xloud Products

<Tabs>
  <Tab title="Infrastructure" icon="boxes">
    <CardGroup>
      <Card title="XAVS — Advanced Virtualization Suite" icon="server" href="/products/xavs">
        Manage VMs and containers on a single optimized platform.
      </Card>

      <Card title="XPCI — Private Cloud Infrastructure" icon="cloud" href="/products/xpci">
        Full-featured private cloud with all enterprise services.
      </Card>

      <Card title="XHCI — Hyperconverged Infrastructure" icon="boxes" href="/products/xhci">
        Compute, storage, and networking in one system.
      </Card>
    </CardGroup>
  </Tab>

  <Tab title="Virtualization" icon="server">
    <CardGroup>
      <Card title="Hypervisor" icon="cpu" href="/services/compute/hypervisor">
        Hardware-assisted virtualization with near-bare-metal performance
      </Card>

      <Card title="XOS — Operating System" icon="terminal" href="/products/xos">
        Purpose-built Linux distribution for Xloud private cloud deployments
      </Card>
    </CardGroup>
  </Tab>

  <Tab title="Storage" icon="database">
    <CardGroup>
      <Card title="XSDS — Software-Defined Storage" icon="database" href="/products/xsds">
        Fault-tolerant distributed storage. Unified block, object, and file with self-healing.
      </Card>

      <Card title="XNAS — NAS Storage" icon="hard-drive" href="/products/xnas">
        SMB, NFS, AFP, FTP with snapshots, HA, thin provisioning, deduplication
      </Card>

      <Card title="XNexus — Unified Storage" icon="layers" href="/products/xnexus">
        Unified storage platform — coming soon
      </Card>
    </CardGroup>
  </Tab>

  <Tab title="Operations" icon="activity">
    <CardGroup>
      <Card title="XIMP — Infrastructure Monitoring" icon="activity" href="/products/ximp">
        End-to-end monitoring, APM, log management, and security
      </Card>

      <Card title="XDR — Disaster Recovery" icon="rotate-ccw" href="/products/xdr">
        Failover automation, replication, RPO/RTO management
      </Card>
    </CardGroup>
  </Tab>
</Tabs>

***

## Services

**Compute & Orchestration**

| Service                | Description                                             | Docs                                                                                                |
| ---------------------- | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Compute**            | Virtual machines, live migration, hypervisor management | [User Guide](/services/compute/user-guide) · [Admin Guide](/services/compute/admin-guide)           |
| **Kubernetes**         | Managed Kubernetes clusters                             | [User Guide](/services/kubernetes/user-guide) · [Admin Guide](/services/kubernetes/admin-guide)     |
| **Instance HA**        | Automated instance recovery on host failure             | [User Guide](/services/instance-ha/user-guide) · [Admin Guide](/services/instance-ha/admin-guide)   |
| **Resource Optimizer** | Workload placement and consolidation                    | [User Guide](/services/optimization/user-guide) · [Admin Guide](/services/optimization/admin-guide) |

**Storage**

| Service                      | Description                                           | Docs                                                                                                    |
| ---------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **Block Storage**            | Persistent volumes, snapshots, backups, storage tiers | [User Guide](/services/storage/user-guide) · [Admin Guide](/services/storage/admin-guide)               |
| **Object Storage**           | S3-compatible distributed object store                | [User Guide](/services/object-storage/user-guide) · [Admin Guide](/services/object-storage/admin-guide) |
| **Software-Defined Storage** | Unified block, object, and file storage               | [User Guide](/services/sds/user-guide) · [Admin Guide](/services/sds/admin-guide)                       |
| **Image Service**            | OS images, snapshots, and image sharing               | [User Guide](/services/images/user-guide) · [Admin Guide](/services/images/admin-guide)                 |

**Networking & Security**

| Service               | Description                                         | Docs                                                                                                  |
| --------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Networking**        | SDN, tenant networks, security groups, floating IPs | [User Guide](/services/networking/user-guide) · [Admin Guide](/services/networking/admin-guide)       |
| **Load Balancer**     | L4/L7 load balancing with health checks and TLS     | [User Guide](/services/load-balancer/user-guide) · [Admin Guide](/services/load-balancer/admin-guide) |
| **DNS**               | Managed DNS zones and record management             | [User Guide](/services/dns/user-guide) · [Admin Guide](/services/dns/admin-guide)                     |
| **Identity & Access** | Authentication, RBAC, projects, multi-tenancy       | [User Guide](/services/identity/user-guide) · [Admin Guide](/services/identity/admin-guide)           |
| **Key Manager**       | Secrets, certificates, encryption key management    | [User Guide](/services/key-manager/user-guide) · [Admin Guide](/services/key-manager/admin-guide)     |

***

## Release Notes

<Update label="March 2026" description="Latest Releases">
  | Product     | Version     | Status                                   |
  | ----------- | ----------- | ---------------------------------------- |
  | **XAVS**    | 2025.1 LTS  | Stable — Long-term support               |
  | **XPCI**    | 2025.1      | Stable — Full private cloud platform     |
  | **XDeploy** | 2025.1-beta | Beta — Deployment & lifecycle management |
  | **XOS**     | 3.1         | Stable — Base OS with security hardening |

  [View full changelog and version history →](/changelog)
</Update>

***

## Resources

<CardGroup>
  <Card title="Blogs" icon="file-text" href="https://blog.xloud.tech/">
    Articles, best practices, platform updates
  </Card>

  <Card title="Contact Support" icon="headphones" href="mailto:support@xloud.tech">
    24/7/365 multilingual support
  </Card>

  <Card title="About Xloud" icon="building-2" href="https://xloud.tech/about">
    Xloud Technologies — Made in India
  </Card>
</CardGroup>


# Xloud Products
Source: https://docs.xloud.tech/products/index

Infrastructure solutions for virtualization, private cloud, and hyper-converged environments. Choose the right platform for your deployment.

<p>Overview</p>

Xloud Technologies delivers a portfolio of purpose-built infrastructure products designed for enterprises, service providers, and data centers. Each product is engineered to integrate seamlessly with the Xloud platform — from single-site virtualization deployments to multi-site private cloud environments.

***

<p>Infrastructure</p>

<CardGroup>
  <Card title="XAVS" icon="server" href="/products/xavs">
    **Advanced Virtualization Suite** — Manage VMs and containers on a single optimized platform with bare-metal performance.
  </Card>

  <Card title="XPCI" icon="cloud" href="/products/xpci">
    **Private Cloud Infrastructure** — Full-featured private cloud with the complete suite of enterprise-grade services.
  </Card>

  <Card title="XHCI" icon="boxes" href="/products/xhci">
    **Hyper-Converged Infrastructure** — Consolidate compute, storage, and networking into one cohesive, scalable system.
  </Card>
</CardGroup>

***

<p>Virtualization</p>

<CardGroup>
  <Card title="Hypervisor" icon="cpu" href="/services/compute/hypervisor">
    **Compute Hypervisor** — Bare-metal virtualization with hardware-assisted acceleration and NUMA topology awareness.
  </Card>

  <Card title="XOS" icon="terminal" href="/products/xos">
    **Xloud Operating System** — Purpose-built Linux distribution pre-configured for all Xloud cluster nodes.
  </Card>
</CardGroup>

***

<p>Storage</p>

<CardGroup>
  <Card title="XSDS" icon="database" href="/products/xsds">
    **Software-Defined Storage** — Fault-tolerant distributed storage with unified block, object, and file access.
  </Card>

  <Card title="XNAS" icon="hard-drive" href="/products/xnas">
    **NAS Storage** — SMB, NFS, AFP, and FTP with snapshots, HA, thin provisioning, and deduplication.
  </Card>

  <Card title="XNexus" icon="layers" href="/products/xnexus">
    **Unified Storage** — Next-generation unified storage platform for modern infrastructure needs.
  </Card>
</CardGroup>

***

<p>Operations</p>

<CardGroup>
  <Card title="XIMP" icon="activity" href="/products/ximp">
    **Infrastructure Monitoring** — End-to-end observability with metrics, logs, APM, and security analytics.
  </Card>

  <Card title="XDR" icon="rotate-ccw" href="/products/xdr">
    **Disaster Recovery** — Automated failover, replication, and RPO/RTO management across sites.
  </Card>
</CardGroup>

***

<p>Product Comparison</p>

The following table shows which services are included in each core infrastructure product.

| Service                      | XAVS | XPCI | XHCI |
| ---------------------------- | :--: | :--: | :--: |
| Compute (Virtual Machines)   |   ✓  |   ✓  |   ✓  |
| Block Storage                |   ✓  |   ✓  |   ✓  |
| Networking (SDN)             |   ✓  |   ✓  |   ✓  |
| Identity & Access Management |   ✓  |   ✓  |   ✓  |
| Image Service                |   ✓  |   ✓  |   ✓  |
| Load Balancer                |   ✓  |   ✓  |   —  |
| DNS Service                  |   —  |   ✓  |   —  |
| Key Manager                  |   —  |   ✓  |   —  |
| Object Storage               |   —  |   ✓  |   —  |
| Software-Defined Storage     |   —  |   ✓  |   ✓  |
| Infrastructure Monitoring    |   —  |   ✓  |   —  |
| Disaster Recovery            |   —  |   ✓  |   —  |
| Instance High Availability   |   —  |   ✓  |   —  |
| Kubernetes (K8SaaS)          |   —  |   ✓  |   —  |
| Resource Optimizer           |   —  |   ✓  |   —  |

<Note>
  XHCI includes Xloud Distributed Storage (XSDS) as an integrated component. XPCI includes all services from XAVS plus the full suite of platform services. Contact [sales@xloud.tech](mailto:sales@xloud.tech) to discuss which product best fits your requirements.
</Note>

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Quick Start" icon="rocket" href="/cli-setup">
    Deploy your first Xloud environment with XDeploy in under an hour
  </Card>

  <Card title="Services Overview" icon="layers" href="/services/compute">
    Browse documentation for all individual platform services
  </Card>

  <Card title="Contact Sales" icon="building-2" href="mailto:sales@xloud.tech">
    Discuss product licensing and deployment options with our team
  </Card>

  <Card title="Xloud.tech" icon="globe" href="https://xloud.tech">
    Learn more about Xloud Technologies and our full product portfolio
  </Card>
</CardGroup>


# XAVS — Advanced Virtualization Suite
Source: https://docs.xloud.tech/products/xavs

Manage virtual machines and containers on a single optimized platform. Bare-metal performance, live migration, and enterprise-grade security.

XAVS (Advanced Virtualization Suite) is Xloud's flagship virtualization product. It enables you to manage VMs and containers on a single, unified platform with bare-metal performance, live migration, and enterprise-grade security.

<Card title="XAVS — Advanced Virtualization Suite" icon="external-link" href="https://xloud.tech/xavs">
  Product details, hardware compatibility, performance benchmarks, and datasheet on xloud.tech
</Card>

***

<p>XAVS Documentation</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/compute/user-guide">
    Launch, manage, resize, migrate, and snapshot virtual machine instances.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/compute/admin-guide">
    Configure hypervisor hosts, schedulers, flavors, quotas, and compute policies.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/compute/cli-reference">
    Full CLI command reference for compute, flavors, aggregates, and migrations.
  </Card>

  <Card title="Deployment Guide" icon="rocket" href="/deployment">
    Deploy XAVS using XDeploy — the Xloud cluster provisioning and lifecycle tool.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Launch Instances" icon="play" href="/services/compute/launch-instance">
    Create virtual machines from OS images, configure networking, and attach storage in one workflow.
  </Card>

  <Card title="Live Migration" icon="arrow-right-left" href="/services/compute/live-migration">
    Move running workloads between compute nodes with zero downtime — including vTPM-secured instances.
  </Card>

  <Card title="Live vCPU & RAM Scaling" icon="trending-up" href="/services/compute/live-resize">
    Dynamically adjust vCPU and memory on running instances without rebooting.
  </Card>

  <Card title="Snapshots & Backup" icon="camera" href="/services/compute/snapshots">
    Point-in-time instance snapshots with filesystem consistency and granular file-level restore.
  </Card>
</CardGroup>

***

<p>Related Services</p>

<CardGroup>
  <Card title="Block Storage" icon="hard-drive" href="/services/storage/index">
    Persistent volumes and snapshots backed by the XSDS distributed storage backend
  </Card>

  <Card title="Networking" icon="network" href="/services/networking/index">
    Tenant networks, floating IPs, and security groups for XAVS compute instances
  </Card>

  <Card title="XSDS Storage" icon="database" href="/products/xsds">
    The distributed storage layer that powers XAVS block volumes and image storage
  </Card>
</CardGroup>


# XDR — Disaster Recovery
Source: https://docs.xloud.tech/products/xdr

Protect business continuity with automated failover and replication.

XDR delivers automated disaster recovery for private cloud workloads — with continuous replication, defined RPO/RTO targets, and one-click failover across sites.

<Card title="XDR — Disaster Recovery" icon="external-link" href="https://xloud.tech/disaster-recovery">
  Product details, RPO/RTO benchmarks, and deployment options on xloud.tech
</Card>

***

<p>XDR Documentation</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/disaster-recovery/user-guide/protection-plans">
    Protection plans, failover, failback, and DR testing
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/disaster-recovery/admin-guide/architecture">
    Replication configuration, recovery plans, and automation
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Protection Plans" icon="shield-check" href="/services/disaster-recovery/user-guide/protection-plans">
    Define which workloads are protected, their RPO targets, and recovery group ordering.
  </Card>

  <Card title="Automated Failover" icon="zap" href="/services/disaster-recovery/user-guide/failover">
    One-click or fully automated failover to the DR site with pre-configured runbooks.
  </Card>

  <Card title="Failback" icon="rotate-ccw" href="/services/disaster-recovery/user-guide/failback">
    Resynchronize and fail back to the primary site after recovery without data loss.
  </Card>

  <Card title="DR Testing" icon="flask-conical" href="/services/disaster-recovery/user-guide/test-dr">
    Test failover in an isolated environment without impacting production replication.
  </Card>
</CardGroup>

***

<p>Related Products</p>

<CardGroup>
  <Card title="XPCI" icon="cloud" href="/products/xpci">
    XDR is included in XPCI — the full private cloud platform
  </Card>

  <Card title="XSDS" icon="database" href="/products/xsds">
    XDR replicates Ceph-backed block volumes and object storage
  </Card>

  <Card title="Contact Sales" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about XDR deployment and licensing
  </Card>
</CardGroup>


# XHCI — Hyper-Converged Infrastructure
Source: https://docs.xloud.tech/products/xhci

Consolidate compute, storage, and networking into one cohesive, scalable system. Simplified operations with enterprise-grade reliability.

XHCI (Hyper-Converged Infrastructure) consolidates compute, storage, and networking into a single platform — eliminating the operational complexity of managing separate silos and enabling horizontal scaling from 3 nodes to hundreds.

<Card title="XHCI — Hyper-Converged Infrastructure" icon="external-link" href="https://xloud.tech/hyperconverged">
  Product details, hardware compatibility matrix, and datasheet on xloud.tech
</Card>

***

<p>XHCI Documentation</p>

<CardGroup>
  <Card title="Deployment Guide" icon="rocket" href="/deployment">
    Deploy XHCI using XDeploy — configure nodes, storage, and networking from one interface.
  </Card>

  <Card title="Compute Admin Guide" icon="cpu" href="/services/compute/admin-guide">
    Configure hypervisor hosts, schedulers, and compute resources across converged nodes.
  </Card>

  <Card title="Storage Admin Guide" icon="database" href="/services/sds/admin-guide/architecture">
    Manage the integrated Ceph storage layer — pools, CRUSH maps, and capacity planning.
  </Card>

  <Card title="Networking Admin Guide" icon="network" href="/services/networking/admin-guide">
    Configure OVN-based tenant networking, provider networks, and security groups.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Converged Compute & Storage" icon="layers" href="/services/compute/architecture">
    Every node runs both hypervisor and Ceph OSD — adding a node expands compute and storage simultaneously.
  </Card>

  <Card title="Self-Healing Storage" icon="shield-check" href="/services/sds/admin-guide/architecture">
    Ceph continuously monitors and rebalances data. Single-node failures do not cause data loss.
  </Card>

  <Card title="Software-Defined Networking" icon="network" href="/services/networking/user-guide">
    OVN-based tenant networks, floating IPs, and security groups across the converged cluster.
  </Card>

  <Card title="Live Migration" icon="arrow-right-left" href="/services/compute/live-migration">
    Move running workloads between converged nodes with zero downtime using shared Ceph storage.
  </Card>
</CardGroup>

***

<p>Related Products</p>

<CardGroup>
  <Card title="XAVS" icon="server" href="/products/xavs">
    Advanced Virtualization Suite — the compute platform running on XHCI nodes
  </Card>

  <Card title="XSDS" icon="database" href="/products/xsds">
    Software-Defined Storage — the Ceph storage layer integrated in every XHCI node
  </Card>

  <Card title="Contact Sales" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about XHCI sizing and deployment options
  </Card>
</CardGroup>


# XIMP — Infrastructure Monitoring Platform
Source: https://docs.xloud.tech/products/ximp

End-to-end monitoring across infrastructure, applications, and networks.

XIMP provides unified observability across your entire infrastructure — from hypervisor metrics and application traces to network flows and security events — with real-time alerting and log analytics.

<Card title="XIMP — Infrastructure Monitoring Platform" icon="external-link" href="https://xloud.tech/XIMP">
  Product details, datasheet, and feature overview on xloud.tech
</Card>

***

<p>XIMP Documentation</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/monitoring/user-guide/dashboards">
    Dashboards, metrics, alerts, log analytics, and network monitoring
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/monitoring/admin-guide/architecture">
    Agent configuration, metric collectors, log collection, and retention policies
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Infrastructure Dashboards" icon="gauge" href="/services/monitoring/user-guide/dashboards">
    Pre-built Grafana dashboards for host metrics, hypervisor utilization, and cluster health.
  </Card>

  <Card title="Log Analytics" icon="file-text" href="/services/monitoring/user-guide/log-analytics">
    Centralized log collection, full-text search, and pattern detection across all nodes via OpenSearch.
  </Card>

  <Card title="Alerting" icon="bell" href="/services/monitoring/user-guide/alert-rules">
    Rule-based alerts with multi-channel delivery — email, Slack, webhooks, and PagerDuty.
  </Card>

  <Card title="Network Monitoring" icon="network" href="/services/monitoring/user-guide/network-monitoring">
    Flow analysis, bandwidth utilization, and interface health tracking.
  </Card>
</CardGroup>

***

<p>Related Products</p>

<CardGroup>
  <Card title="XPCI" icon="cloud" href="/products/xpci">
    XIMP is included in XPCI — the full private cloud platform
  </Card>

  <Card title="Security" icon="shield" href="/security/index">
    Wazuh SIEM and security alerting integrated into the XIMP monitoring stack
  </Card>

  <Card title="Contact Sales" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about XIMP deployment and licensing
  </Card>
</CardGroup>


# xInsight-AI — Intelligent Cloud Operations
Source: https://docs.xloud.tech/products/xinsight

Natural language cloud management with on-premises AI inference, 42 operational tools, and role-based access control.

xInsight-AI is Xloud's AI-powered cloud operations assistant — manage your entire infrastructure through natural language instead of navigating dashboards or memorizing CLI commands. All AI processing runs on-premises; data never leaves your cluster.

<Card title="xInsight-AI" icon="external-link" href="https://xloud.tech/contact">
  Contact the Xloud team for xInsight-AI deployment and licensing
</Card>

***

<p>Key Capabilities</p>

<Info>**Xloud-Developed** — xInsight-AI is developed by Xloud.</Info>

<CardGroup>
  <Card title="Natural Language Operations" icon="message-circle">
    Manage compute, networking, storage, and identity resources through conversation. 42 operational tools available across all service domains.
  </Card>

  <Card title="Three Operating Modes" icon="sliders">
    **Ask** — read-only queries and status checks. **Plan** — propose changes for your approval before execution. **Agent** — execute changes directly with appropriate guardrails.
  </Card>

  <Card title="On-Premises Inference" icon="lock">
    All AI processing runs locally on the cluster. No external API calls, no cloud dependencies, no data leaving your infrastructure.
  </Card>

  <Card title="Role-Based Access Control" icon="shield">
    Integrated with the platform identity service. xInsight-AI respects project scoping, admin roles, and policy enforcement for every operation.
  </Card>
</CardGroup>

***

<p>Related Products</p>

<CardGroup>
  <Card title="XPCI" icon="cloud" href="/products/xpci">
    xInsight-AI manages the full XPCI private cloud platform
  </Card>

  <Card title="XIMP" icon="activity" href="/products/ximp">
    Query monitoring data and alerts through natural language
  </Card>

  <Card title="Contact Sales" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about xInsight-AI deployment
  </Card>
</CardGroup>


# XMS — Xloud Migration Suite
Source: https://docs.xloud.tech/products/xms

Agentless workload migration from multiple source platforms to Xloud with pre-migration assessment and incremental sync.

XMS (Xloud Migration Suite) is Xloud's agentless workload migration platform — supporting V2V migrations from VMware vSphere, Hyper-V, Nutanix AHV, AWS EC2, Azure, KVM/libvirt, and other source platforms with pre-migration assessment and incremental sync.

<CardGroup>
  <Card title="Migration Service Docs" icon="book-open" href="/services/migration">
    Full user guide, admin guide, and CLI reference for the Migration service
  </Card>

  <Card title="Contact Xloud" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about migration planning and licensing
  </Card>
</CardGroup>

***

## Key Capabilities

<Info>**Xloud-Developed** — XMS is developed by Xloud.</Info>

<CardGroup>
  <Card title="Multi-Platform Sources" icon="server">
    VMware vSphere, Microsoft Hyper-V, Nutanix AHV, AWS EC2, Microsoft Azure, KVM/libvirt, Proxmox VE, oVirt/RHV, GCP Compute, and Oracle Cloud.
  </Card>

  <Card title="Pre-Migration Assessment" icon="chart-line">
    Readiness assessment across compute, storage, network, and driver dimensions. Each workload receives a Ready, Conditional, or Not Ready verdict with specific risk flags.
  </Card>

  <Card title="Incremental Sync" icon="rotate-cw">
    Changed Block Tracking (CBT) minimizes data transfer. Only modified blocks are replicated, reducing cutover downtime to minutes.
  </Card>

  <Card title="Automatic Conversion" icon="cpu">
    VirtIO driver injection, boot loader adjustment, and VMware Tools removal applied automatically during image conversion — no manual guest preparation.
  </Card>
</CardGroup>

***

## Migration Workflow

<Steps>
  <Step title="Discovery" icon="search">
    Agentless scan of source environment — inventories all VMs, disks, networks, and dependencies.
  </Step>

  <Step title="Assessment" icon="chart-bar">
    Pre-migration readiness assessment. Each workload receives a Ready, Conditional, or Not Ready verdict with specific risk flags.
  </Step>

  <Step title="Incremental Sync" icon="rotate-cw">
    Initial full copy followed by incremental delta syncs using Changed Block Tracking.
  </Step>

  <Step title="Cutover" icon="play">
    Final sync, source VM shutdown, automatic driver injection and boot loader repair, target VM start.
  </Step>

  <Step title="Validation" icon="check">
    Post-migration checks. Source VM preserved for a rollback window (typically 7-30 days per your change management policy) before decommissioning.
  </Step>
</Steps>

***

## Related Products

<CardGroup>
  <Card title="XPCI" icon="cloud" href="/products/xpci">
    XMS migrates workloads into the XPCI private cloud platform
  </Card>

  <Card title="XIMP" icon="activity" href="/products/ximp">
    Monitor migrated workloads with XIMP infrastructure monitoring
  </Card>

  <Card title="Contact Sales" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about migration planning and licensing
  </Card>
</CardGroup>


# XNAS — NAS Storage
Source: https://docs.xloud.tech/products/xnas

Network-attached storage with SMB, NFS, AFP, and FTP support.

XNAS provides enterprise network-attached storage with broad protocol support — enabling seamless file sharing across Windows, Linux, and macOS environments with built-in high availability and data protection.

<Card title="XNAS — NAS Storage" icon="external-link" href="https://xloud.tech/xnas">
  Product details, protocol support matrix, and datasheet on xloud.tech
</Card>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Multi-Protocol File Sharing" icon="share-2">
    SMB, NFS, AFP, and FTP support for cross-platform access from Windows, Linux, and macOS.
  </Card>

  <Card title="Snapshots & Cloning" icon="camera">
    Point-in-time snapshots and instant volume cloning for backup and recovery.
  </Card>

  <Card title="High Availability" icon="heart-pulse">
    Active-active clustering with automatic failover and no single point of failure.
  </Card>

  <Card title="Thin Provisioning & Deduplication" icon="layers">
    Maximize storage efficiency with inline deduplication and thin provisioning.
  </Card>
</CardGroup>

***

<p>Related Products</p>

<CardGroup>
  <Card title="XSDS" icon="database" href="/products/xsds">
    Software-Defined Storage for block and object workloads
  </Card>

  <Card title="XNexus" icon="layers" href="/products/xnexus">
    Unified storage platform — coming soon
  </Card>

  <Card title="Contact Sales" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about XNAS deployment and licensing
  </Card>
</CardGroup>


# XNexus — Unified Storage
Source: https://docs.xloud.tech/products/xnexus

Unified storage platform — coming soon.

XNexus is Xloud's next-generation unified storage platform, bringing together block, file, and object access under a single management plane. Full documentation will be available at general release.

<Note>
  For product details and early access information, visit the [XNexus product page on xloud.tech](https://xloud.tech/xnexus).
</Note>


# XOS — Xloud Operating System
Source: https://docs.xloud.tech/products/xos

Purpose-built Linux distribution for Xloud private cloud nodes.

XOS is a purpose-built Linux distribution, pre-configured and hardened for Xloud private cloud deployments. Every cluster node in an XAVS, XPCI, or XHCI environment runs XOS as its base operating system.

<Card title="XOS — Xloud Operating System" icon="external-link" href="https://xloud.tech/xos">
  Product details and datasheet on xloud.tech
</Card>

XOS ships with the complete Xloud runtime pre-installed — all required system services, container runtimes, storage drivers, and the XDeploy management agent. This ensures every node starts from a known, validated baseline. Security-hardened out of the box with immutable configuration, restricted SSH access, and pre-configured firewall policies.

***

<p>Getting Started</p>

<CardGroup>
  <Card title="Deployment Guide" icon="rocket" href="/deployment">
    Deploy XOS-based clusters using XDeploy
  </Card>

  <Card title="Security Hardening" icon="shield" href="/security/hardening-guide">
    XOS security hardening, audit policies, and compliance configuration
  </Card>

  <Card title="Contact Sales" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about XOS licensing and deployment
  </Card>
</CardGroup>


# XPCI — Private Cloud Infrastructure
Source: https://docs.xloud.tech/products/xpci

Full-featured private cloud platform with enterprise-grade services, multi-tenancy, and complete infrastructure automation.

XPCI (Private Cloud Infrastructure) is Xloud's comprehensive private cloud product. It delivers the full suite of enterprise-grade services — compute, storage, networking, identity, monitoring, and disaster recovery — in a single unified deployment.

<Card title="XPCI — Private Cloud Infrastructure" icon="external-link" href="https://xloud.tech/private-cloud">
  Product details, sizing guides, and deployment options on xloud.tech
</Card>

***

<p>XPCI Documentation</p>

<CardGroup>
  <Card title="Deployment Guide" icon="rocket" href="/deployment">
    Deploy the full XPCI stack using XDeploy — from bare hardware to production in hours.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/admin-guide">
    Administer every XPCI service — compute, storage, networking, identity, monitoring, and more.
  </Card>

  <Card title="User Guide" icon="book-open" href="/user-guide">
    Create and manage cloud resources — instances, volumes, networks, and applications.
  </Card>

  <Card title="Security" icon="shield" href="/security/index">
    Security hardening, compliance, and audit guidance for XPCI environments.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Compute" icon="cpu" href="/services/compute/user-guide">
    Full virtual machine lifecycle — launch, resize, migrate, snapshot, and recover instances.
  </Card>

  <Card title="Software-Defined Storage" icon="database" href="/services/sds/index">
    Distributed block, object, and file storage with self-healing and petabyte scalability.
  </Card>

  <Card title="Networking" icon="network" href="/services/networking/user-guide">
    Tenant networks, security groups, floating IPs, load balancing, and DNS management.
  </Card>

  <Card title="Monitoring & Observability" icon="activity" href="/services/monitoring/user-guide/dashboards">
    Infrastructure metrics, log analytics, security alerting, and real-time dashboards.
  </Card>

  <Card title="Horizontal Scaling" icon="maximize" href="/services/compute/compute-hosts">
    Add compute and storage nodes with zero downtime and zero data migration. Single control plane scales from 3 to 500+ nodes. Proven in production at 8,000+ node deployments worldwide.
  </Card>

  <Card title="Multi-Hypervisor Management" icon="layers" href="/services/compute/hypervisor">
    Manage both native hypervisor and VMware ESXi compute nodes from a single dashboard and API. Gradual VMware-to-native migration path from one management console.
  </Card>
</CardGroup>

<Info>**Xloud-Developed** — Horizontal Scaling and Multi-Hypervisor Management are developed by Xloud and ship with XAVS / XPCI.</Info>

***

<p>Related Products</p>

<CardGroup>
  <Card title="XAVS" icon="server" href="/products/xavs">
    Advanced Virtualization Suite — the compute layer inside XPCI
  </Card>

  <Card title="XSDS" icon="layers" href="/products/xsds">
    Software-Defined Storage — the storage layer inside XPCI
  </Card>

  <Card title="Contact Sales" icon="phone" href="https://xloud.tech/contact">
    Talk to the Xloud team about XPCI sizing and deployment options
  </Card>
</CardGroup>


# XSDS — Software-Defined Storage
Source: https://docs.xloud.tech/products/xsds

Fault-tolerant distributed storage with unified block, object, and file access.

XSDS delivers enterprise-grade distributed storage built on Ceph — providing unified block, object, and file access with self-healing data protection and petabyte-scale capacity.

<Note>
  For full product details, visit the [XSDS product page on xloud.tech](https://xloud.tech/ceph).
</Note>

***

<p>Service Guides</p>

<CardGroup>
  <Card title="Block Storage" icon="hard-drive" href="/services/storage/index">
    Persistent volumes, snapshots, and backups
  </Card>

  <Card title="Object Storage" icon="box" href="/services/object-storage">
    S3-compatible object containers and access control
  </Card>

  <Card title="Software-Defined Storage" icon="database" href="/services/sds/index">
    Ceph cluster management and pool configuration
  </Card>

  <Card title="Image Service" icon="disc" href="/services/images/index">
    OS image catalog backed by distributed storage
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Unified Storage" icon="layers">
    Block, object, and file access from a single distributed cluster
  </Card>

  <Card title="Self-Healing" icon="shield-check">
    Automatic data rebalancing and recovery from drive or node failures
  </Card>

  <Card title="Petabyte Scale" icon="trending-up">
    Linear scale-out from terabytes to petabytes by adding nodes
  </Card>

  <Card title="Multi-Protocol" icon="network">
    RBD, S3, Swift, CephFS, NFS, and iSCSI in one platform
  </Card>

  <Card title="Erasure Coding" icon="binary">
    Configurable erasure coding for efficient fault-tolerant capacity
  </Card>

  <Card title="Storage Tiering" icon="arrow-up-down">
    Automatic hot/warm/cold tiering across NVMe, SSD, and HDD pools
  </Card>
</CardGroup>


# API Security
Source: https://docs.xloud.tech/security/api-security

Secure the Xloud API layer with token authentication, application credentials, rate limiting, CORS, RBAC policy enforcement, and mutual TLS between services.

## Overview

All Xloud platform services expose REST APIs secured by Xloud Identity (Keystone). Authentication is token-based with configurable expiry, and the platform enforces authorization through a role-based access control (RBAC) policy engine. This page covers the full API security stack: authentication flows, application credentials, and rate limiting. It also covers audit logging, CORS configuration, and service-to-service mutual TLS.

<Note>
  **Prerequisites**

  * An active Xloud account with the `member` or `admin` role
  * CLI tools installed: `openstack` CLI ([setup guide](/cli-setup))
  * For application credentials: access to **Project → Identity → Application Credentials**
</Note>

***

## Token-Based Authentication

All API requests require a valid token issued by Xloud Identity. Tokens are scoped to a project and carry the user's role assignments for that project.

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant Keystone as Xloud Identity
    participant NovaAPI as Service API
    Client->>Keystone: POST /v3/auth/tokens (credentials)
    Keystone-->>Client: X-Subject-Token header
    Client->>NovaAPI: GET /v2.1/servers (X-Auth-Token: <token>)
    NovaAPI->>Keystone: GET /v3/auth/tokens (validate)
    Keystone-->>NovaAPI: Token valid + roles
    NovaAPI-->>Client: 200 OK + response
```

### Token Scopes

| Scope          | Description                    | Use Case                        |
| -------------- | ------------------------------ | ------------------------------- |
| Project-scoped | Bound to a specific project    | Standard user operations        |
| Domain-scoped  | Bound to a domain              | Domain administrator operations |
| System-scoped  | Platform-wide admin operations | Infrastructure management       |
| Unscoped       | No project or domain binding   | Token exchange only             |

<Tabs>
  <Tab title="CLI" icon="terminal">
    ```bash title="Authenticate and get a token" theme={null}
    source admin-openrc.sh

    # Verify the active token
    openstack token issue
    ```

    ```bash title="Expected output" theme={null}
    +------------+----------------------------------------------------------+
    | Field      | Value                                                    |
    +------------+----------------------------------------------------------+
    | expires    | 2025-03-18T11:00:00+0000                                 |
    | id         | gAAAAABm...                                              |
    | project_id | a1b2c3d4...                                              |
    | user_id    | e5f6g7h8...                                              |
    +------------+----------------------------------------------------------+
    ```

    ```bash title="Authenticate via API directly" theme={null}
    curl -si -X POST https://<vip>:5000/v3/auth/tokens \
      -H "Content-Type: application/json" \
      -d '{
        "auth": {
          "identity": {
            "methods": ["password"],
            "password": {
              "user": {
                "name": "admin",
                "domain": {"name": "Default"},
                "password": "<password>"
              }
            }
          },
          "scope": {
            "project": {"name": "admin", "domain": {"name": "Default"}}
          }
        }
      }' | grep -i x-subject-token
    ```
  </Tab>

  <Tab title="Dashboard" icon="gauge">
    The Xloud Dashboard handles token acquisition automatically at login. Session tokens are stored server-side and refreshed transparently.

    To inspect the active session token:

    1. Navigate to **Project → API Access**.
    2. Click **Download OpenStack RC File** to export credentials for CLI use.
    3. Click **View Credentials** to see the active API endpoint list.

    <Tip>Download the RC file and source it in your shell to configure CLI access with the same credentials used in the Dashboard session.</Tip>
  </Tab>
</Tabs>

***

## Application Credentials

Application credentials allow automation scripts and CI/CD pipelines to authenticate without embedding a username and password. They are scoped to a project, have configurable expiry, and can be restricted to specific API operations using access rules.

<Warning>
  Never store your account password in scripts or configuration files. Use application credentials instead. Application credentials can be revoked individually without changing the account password.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Application Credentials" icon="key">
        Navigate to **Project → Identity → Application Credentials** and click **Create Application Credential**.
      </Step>

      <Step title="Configure the credential" icon="settings">
        | Field           | Recommended Value          | Notes                                                  |
        | --------------- | -------------------------- | ------------------------------------------------------ |
        | Name            | `ci-deployment-prod`       | Descriptive name identifying the use case              |
        | Secret          | Auto-generated             | Store the secret securely — it is shown only once      |
        | Expiration Date | Set an expiry              | Required for compliance environments                   |
        | Roles           | Select only required roles | Follow least-privilege                                 |
        | Unrestricted    | No                         | Leave unchecked to allow role inheritance restrictions |

        <Warning>The secret is displayed only once after creation. Store it immediately in a secrets manager or CI/CD vault. It cannot be retrieved again.</Warning>
      </Step>

      <Step title="Download the RC file" icon="download">
        Click **Download openrc file**. Source this file in your automation environment to authenticate using the application credential.

        <Check>The downloaded RC file uses `OS_AUTH_TYPE=v3applicationcredential` — no password is stored in plaintext.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create application credential" theme={null}
    openstack application credential create \
      --description "CI/CD deployment automation" \
      --expiration "2026-01-01T00:00:00" \
      ci-deployment-prod
    ```

    ```bash title="Create with access rules (least privilege)" theme={null}
    openstack application credential create \
      --description "Read-only monitoring credential" \
      --access-rules '[
        {"service": "compute", "method": "GET", "path": "/v2.1/servers"},
        {"service": "identity", "method": "GET", "path": "/v3/projects"}
      ]' \
      monitoring-readonly
    ```

    ```bash title="Use application credential in scripts" theme={null}
    export OS_AUTH_TYPE=v3applicationcredential
    export OS_AUTH_URL=https://<vip>:5000/v3
    export OS_APPLICATION_CREDENTIAL_ID=<id>
    export OS_APPLICATION_CREDENTIAL_SECRET=<secret>
    openstack server list
    ```
  </Tab>
</Tabs>

***

## RBAC Policy Enforcement

Xloud enforces access control using oslo.policy rules. Every API operation checks the caller's token against the service's policy file before executing.

### Default Role Hierarchy

| Role                  | Scope             | Permissions                                           |
| --------------------- | ----------------- | ----------------------------------------------------- |
| `admin`               | System or project | Full access to all operations                         |
| `member`              | Project           | Standard create/read/update/delete within the project |
| `reader`              | Project           | Read-only access to project resources                 |
| `heat_stack_owner`    | Project           | Manage orchestration stacks                           |
| `load-balancer_admin` | Project           | Manage load balancers                                 |

### Custom Policy Overrides

```bash title="View current compute policy" theme={null}
docker exec nova_api cat /etc/nova/policy.yaml 2>/dev/null || \
  docker exec nova_api cat /etc/nova/policy.json
```

```yaml title="Example: restrict live migration to system-admin only" theme={null}
# /etc/xavs/nova-api/policy.yaml
"os_compute_api:os-migrate-server:migrate_live": "role:admin and system_scope:all"
```

<Info>
  Xloud uses the standard RBAC model. Custom policy overrides should be placed in service-specific policy files and deployed via the XAVS config overlay mechanism. Do not modify policy files directly inside containers — changes are lost on restart.
</Info>

***

## API Rate Limiting

Rate limiting protects the platform from abuse and ensures fair resource allocation between projects. Limits are enforced at the HAProxy layer and within individual services.

| Limit Type                | Default               | Configurable |
| ------------------------- | --------------------- | ------------ |
| Compute API (per user)    | 50 POST / minute      | Yes          |
| Compute API (per project) | 200 requests / minute | Yes          |
| Identity token issuance   | 100 / minute          | Yes          |
| Image upload              | 10 / hour             | Yes          |

```yaml title="Configure compute rate limits" theme={null}
# /etc/xavs/globals.d/_60_rate_limits.yml
nova_api_rate_limits: |
  (POST, "*", .*, 50, MINUTE);
  (GET, "*", .*, 300, MINUTE)
```

***

## CORS Configuration

Cross-Origin Resource Sharing (CORS) controls which origins can make browser-based API requests. Configure allowed origins to match your Dashboard and any custom web applications.

```yaml title="Restrict CORS origins" theme={null}
# /etc/xavs/globals.d/_60_cors.yml
keystone_cors_allowed_origin: "https://connect.<your-domain>"
nova_cors_allowed_origin: "https://connect.<your-domain>"
neutron_cors_allowed_origin: "https://connect.<your-domain>"
```

<Warning>
  Do not set `allowed_origin: "*"` in production. This allows any website to make authenticated API calls on behalf of a user with an active session cookie, enabling cross-site request forgery (CSRF) attacks.
</Warning>

***

## Service-to-Service Authentication (Mutual TLS)

Platform services authenticate to each other using service user accounts. When internal TLS is enabled, these connections also use mutual TLS certificate validation.

```mermaid theme={null}
graph LR
    Nova -->|mTLS + service token| Neutron
    Nova -->|mTLS + service token| Cinder[Block Storage]
    Nova -->|mTLS + service token| Glance[Image Service]
    Neutron -->|mTLS + service token| Keystone[Identity]
```

Service accounts are created during deployment with minimal permissions scoped to inter-service operations only. Do not use these accounts for manual operations.

```bash title="List service users" theme={null}
openstack user list --domain service
```

***

## Audit Logging for API Calls

All API calls are recorded in the audit log with the caller identity, token scope, target resource, and operation result. See the [Compliance and Auditing](/security/compliance) page for log format, retention, and aggregation configuration.

```bash title="View recent API audit events" theme={null}
docker exec keystone grep "req-" /var/log/kolla/keystone/keystone.log | tail -20
```

***

## Next Steps

<CardGroup>
  <Card title="Compliance and Auditing" href="/security/compliance">
    Audit log format, retention, and compliance framework mapping
  </Card>

  <Card title="Identity and Access" href="/services/identity/index">
    Users, projects, domains, and federation configuration
  </Card>

  <Card title="Infrastructure Security" href="/security/infrastructure">
    TLS configuration and certificate management
  </Card>

  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Detailed application credential management guide
  </Card>
</CardGroup>


# Compliance and Auditing
Source: https://docs.xloud.tech/security/compliance

Configure audit logging, log aggregation, and retention for Xloud Platform. Map platform controls to SOC 2, ISO 27001, HIPAA, PCI-DSS, and GDPR frameworks.

## Overview

Xloud Platform generates structured audit logs for all API calls, authentication events, and administrative operations. Logs follow the CADF (Cloud Audit Data Federation) standard. You can ship them to external SIEM systems, log aggregation pipelines, or retain them locally with configurable retention policies. This page covers audit log configuration, aggregation, and framework-by-framework compliance mapping.

<Note>
  **Prerequisites**

  * Administrator role in Xloud Identity
  * For log aggregation: XIMP (Infrastructure Monitoring) enabled with the centralized logging add-on
  * For compliance reports: access to the audit log pipeline or SIEM tool receiving Xloud events
</Note>

***

## Audit Logging

### CADF Event Structure

Every platform event produces a CADF-formatted audit record:

```json title="Example CADF audit event" theme={null}
{
  "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
  "id": "evt-a1b2c3d4-...",
  "eventTime": "2025-03-18T09:30:00.000000+00:00",
  "action": "create",
  "outcome": "success",
  "initiator": {
    "typeURI": "service/security/account/user",
    "id": "user-id-...",
    "name": "admin",
    "project_id": "project-id-...",
    "host": {
      "address": "10.0.0.100",
      "agent": "python-keystoneclient"
    }
  },
  "target": {
    "typeURI": "compute/machine",
    "id": "server-id-...",
    "name": "prod-web-01"
  },
  "observer": {
    "typeURI": "service/compute",
    "id": "nova-api-host-01"
  },
  "reason": {
    "reasonCode": "200",
    "reasonType": "HTTP"
  }
}
```

### Enable Audit Middleware

Audit middleware is enabled per service. By default, all Xloud API services have audit middleware active. To verify:

```bash title="Check audit middleware status" theme={null}
docker exec nova_api grep -i "audit" /etc/nova/nova.conf | grep -v "^#"
```

To enable or reconfigure audit middleware explicitly:

```yaml title="/etc/xavs/globals.d/_60_audit.yml" theme={null}
nova_audit_events: "all"
keystone_audit_events: "all"
neutron_audit_events: "all"
cinder_audit_events: "all"
glance_audit_events: "all"
```

***

## Log Aggregation and Retention

<Tabs>
  <Tab title="Centralized Logging (XIMP)" icon="gauge">
    <Steps>
      <Step title="Enable central logging" icon="settings">
        In XDeploy, navigate to **Configuration → Global Settings** and enable **Central Logging**. This activates Fluentd on all nodes and deploys OpenSearch as the log aggregation backend.
      </Step>

      <Step title="Configure retention policy" icon="clock">
        Navigate to **XIMP → Log Management → Index Policies**. Set the retention period:

        | Log Type              | Minimum Retention | Recommended |
        | --------------------- | ----------------- | ----------- |
        | Authentication events | 90 days           | 1 year      |
        | API audit events      | 90 days           | 1 year      |
        | Service logs          | 30 days           | 90 days     |
        | Security events       | 1 year            | 3 years     |

        <Warning>Some compliance frameworks require specific minimum retention periods. PCI-DSS requires 1 year with 3 months immediately available. HIPAA requires 6 years for audit logs. Configure retention before collecting audit data.</Warning>
      </Step>

      <Step title="Configure SIEM forwarding" icon="arrow-right">
        To forward audit events to an external SIEM:

        Navigate to **XIMP → Integrations → Log Forwarding** and configure the SIEM endpoint, authentication, and event filter.

        <Check>Events appear in the SIEM within the configured polling interval (typically 30–60 seconds).</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```yaml title="Enable centralized logging" theme={null}
    # /etc/xavs/globals.d/_60_logging.yml
    enable_central_logging: "yes"
    opensearch_log_retention_days: 365
    fluentd_syslog_port: 5140
    ```

    ```bash title="Deploy logging stack" theme={null}
    xavs-ansible deploy --tags fluentd,opensearch,opensearch-dashboards
    ```

    ```bash title="Query audit logs via OpenSearch API" theme={null}
    curl -sk -u admin:<password> \
      "https://10.0.1.71:9200/audit-*/_search" \
      -H "Content-Type: application/json" \
      -d '{
        "query": {
          "bool": {
            "filter": [
              {"term": {"action": "delete"}},
              {"range": {"eventTime": {"gte": "now-24h"}}}
            ]
          }
        },
        "sort": [{"eventTime": "desc"}],
        "size": 50
      }' | python3 -m json.tool
    ```
  </Tab>
</Tabs>

***

## Compliance Framework Mapping

The following table maps Xloud platform controls to requirements in major compliance frameworks.

| Control                    | SOC 2        | ISO 27001 | HIPAA               | PCI-DSS  | GDPR    |
| -------------------------- | ------------ | --------- | ------------------- | -------- | ------- |
| TLS for all API endpoints  | CC6.1        | A.10.1.1  | § 164.312(e)        | 4.1      | Art. 32 |
| Token-based authentication | CC6.1, CC6.2 | A.9.4.2   | § 164.312(d)        | 8.3      | Art. 32 |
| RBAC policy enforcement    | CC6.3        | A.9.4.1   | § 164.312(a)        | 7.1      | Art. 25 |
| Audit logging (CADF)       | CC7.2        | A.12.4.1  | § 164.312(b)        | 10.2     | Art. 30 |
| Log retention ≥ 1 year     | CC7.2        | A.12.4.1  | § 164.312(b)        | 10.7     | Art. 30 |
| Volume encryption (LUKS)   | CC6.7        | A.10.1.1  | § 164.312(a)(2)(iv) | 3.4, 3.5 | Art. 32 |
| Key management (Barbican)  | CC6.7        | A.10.1.2  | § 164.312(e)(2)     | 3.6      | Art. 32 |
| Network segmentation       | CC6.6        | A.13.1.1  | § 164.312(a)        | 1.1      | Art. 25 |
| Security group enforcement | CC6.6        | A.13.1.3  | § 164.312(a)        | 1.2      | Art. 32 |
| Change tracking            | CC8.1        | A.12.1.2  | § 164.312(b)        | 6.4.5    | Art. 30 |
| Vulnerability scanning     | CC7.1        | A.12.6.1  | § 164.308(a)(8)     | 6.3.3    | Art. 32 |

***

## Security Scanning and Vulnerability Management

<Steps>
  <Step title="Schedule regular vulnerability scans" icon="search">
    Integrate Xloud with your vulnerability scanning tool (Wazuh, OpenVAS, Qualys, or similar). Scan all compute nodes and control plane hosts at minimum monthly.

    ```bash title="Run Lynis system audit" theme={null}
    lynis audit system --quick --report-file /var/log/lynis-report-$(date +%Y%m%d).txt
    ```
  </Step>

  <Step title="Review and triage findings" icon="list">
    Prioritize findings by CVSS score:

    | Severity | CVSS Range | Target Remediation |
    | -------- | ---------- | ------------------ |
    | Critical | 9.0–10.0   | 24 hours           |
    | High     | 7.0–8.9    | 7 days             |
    | Medium   | 4.0–6.9    | 30 days            |
    | Low      | 0.1–3.9    | 90 days            |
  </Step>

  <Step title="Patch and re-scan" icon="rotate-cw">
    Apply patches via the standard XOS update process and re-scan to confirm remediation.

    ```bash title="Apply security updates on all nodes" theme={null}
    xavs-ansible deploy --tags common --limit control,compute
    ```

    <Check>Re-scan confirms the vulnerability is resolved. Update the compliance tracking document with the remediation date.</Check>
  </Step>
</Steps>

***

## Change Tracking

All infrastructure changes made through XDeploy, xavs-ansible, and the Xloud API are recorded with timestamps, user identity, and the before/after state.

```bash title="View recent deployment history" theme={null}
xavs-ansible facts --list-hosts all
cat /var/log/xavs/ansible.log | grep -E "PLAY|TASK|fatal" | tail -50
```

For platform API changes, query the audit log:

```bash title="Filter audit events for a specific user" theme={null}
# Via OpenSearch
curl -sk -u admin:<password> \
  "https://10.0.1.71:9200/audit-*/_search" \
  -d '{"query": {"term": {"initiator.name": "admin"}}, "size": 100}'
```

***

## Incident Response

<AccordionGroup>
  <Accordion title="Suspected credential compromise" icon="key">
    **Immediate actions**:

    1. Revoke the compromised credential or token immediately:
       ```bash theme={null}
       openstack user password set --password <new-strong-password> <user>
       # Revoke all active tokens for the user:
       openstack token revoke $(openstack token issue -c id -f value)
       ```
    2. Review audit logs for the compromised account in the 30 days prior to detection.
    3. Identify all resources created or modified with the compromised credential.
    4. Rotate any application credentials created by the affected account.
    5. File an incident report with exact timeline, affected resources, and remediation actions taken.
  </Accordion>

  <Accordion title="Unauthorized API access detected" icon="circle-x">
    **Investigation steps**:

    1. Identify the source IP and token from the audit log.
    2. Check whether the token is still valid:
       ```bash theme={null}
       openstack token validate <token-id>
       ```
    3. Revoke the token if it remains active.
    4. Review HAProxy access logs for all requests from the source IP.
    5. Determine whether a firewall block is warranted at the network perimeter.
  </Accordion>

  <Accordion title="Encryption key suspected compromised" icon="lock">
    **Immediate actions**:

    1. Identify all volumes and objects using the compromised key.
    2. Snapshot all affected volumes immediately.
    3. Create new encrypted volumes with a fresh key, copy data, and delete old volumes.
    4. Delete the compromised key from Xloud Key Management after verifying all data has been migrated.
    5. Document the scope of exposure and notify relevant parties per your incident response policy.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Hardening Guide" href="/security/hardening-guide">
    Pre-deployment hardening checklist to meet baseline compliance requirements
  </Card>

  <Card title="Infrastructure Security" href="/security/infrastructure">
    TLS configuration for SOC 2 and PCI-DSS encryption-in-transit controls
  </Card>

  <Card title="Data Security" href="/security/data-security">
    Volume and object encryption for data-at-rest compliance requirements
  </Card>

  <Card title="Monitoring (XIMP)" href="/services/monitoring/index">
    Log aggregation, alerting, and security event dashboards
  </Card>
</CardGroup>


# Data Security
Source: https://docs.xloud.tech/security/data-security

Encrypt block volumes with LUKS, protect object storage at rest and in transit, manage encryption keys with Xloud Key Management, and configure secure deletion.

## Overview

Xloud protects data at every stage: in transit between services, at rest on storage backends, and during backup operations. Volume encryption uses LUKS with keys managed exclusively by Xloud Key Management (Barbican). Object storage supports server-side encryption. This page covers configuration for all data protection mechanisms available in the platform.

<Note>
  **Prerequisites**

  * Administrator role in Xloud Identity
  * Xloud Key Management enabled (`enable_barbican: "yes"` in XDeploy configuration)
  * For encrypted volumes: an encrypted volume type configured in Block Storage
  * For object storage encryption: access to the object storage admin API
</Note>

***

## Data Protection Architecture

```mermaid theme={null}
graph TD
    VM[Virtual Machine] -->|Encrypted channel| Volume[Encrypted Volume LUKS]
    Volume -->|Ceph encryption at rest| Ceph[Distributed Storage]
    Ceph -->|Key reference| Barbican[Key Management]
    Barbican -->|HSM or DB backend| Keys[Encryption Keys]
    VM -->|HTTPS| ObjectStorage[Object Storage]
    ObjectStorage -->|SSE-C or SSE-KMS| ObjKeys[Object Encryption Keys]
    Backups[Volume Backups] -->|Encrypted| BackupStore[Backup Repository]
```

| Layer               | Mechanism                           | Key Storage                   |
| ------------------- | ----------------------------------- | ----------------------------- |
| Block volume (LUKS) | AES-256-XTS                         | Xloud Key Management          |
| Ceph OSD at rest    | AES-256-GCM                         | Ceph key management daemon    |
| Object storage SSE  | AES-256-CBC                         | Per-object or per-bucket keys |
| Database            | MariaDB TDE or encrypted tablespace | External KMS                  |
| Backup              | Inherited from volume encryption    | Same key as source volume     |

***

## Volume Encryption (LUKS)

<Badge>GA</Badge>

LUKS encryption wraps every I/O operation at the compute host before data reaches the storage network. The compute host fetches the encryption key from Xloud Key Management at volume attachment time and never writes it to disk.

### Create an Encrypted Volume Type

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open Volume Types" icon="database">
        Navigate to **Admin → Volume → Volume Types** and click **Create Volume Type**.
      </Step>

      <Step title="Define encryption parameters" icon="key">
        After creating the type, select **View Encryption** and click **Create Encryption**:

        | Field            | Value                                       | Notes                          |
        | ---------------- | ------------------------------------------- | ------------------------------ |
        | Provider Class   | `nova.volume.encryptors.luks.LuksEncryptor` | LUKS v2 encryptor              |
        | Control Location | `front-end`                                 | Encryption at the compute host |
        | Cipher           | `aes-xts-plain64`                           | AES-256 in XTS mode            |
        | Key Size         | `256`                                       | 256-bit AES key                |

        <Check>The volume type now shows **Encrypted: Yes** in the type list.</Check>
      </Step>

      <Step title="Create an encrypted volume" icon="plus">
        Navigate to **Project → Volumes → Create Volume** and select the encrypted type. The platform generates a unique encryption key in Xloud Key Management and associates it with the volume.

        <Warning>Once a volume is created with encryption enabled, the encryption cannot be removed without creating a new unencrypted volume and migrating the data. Plan your volume type strategy before provisioning production volumes.</Warning>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create encrypted volume type" theme={null}
    openstack volume type create encrypted-nvme \
      --description "NVMe-backed encrypted volumes"

    openstack volume type set encrypted-nvme \
      --encryption-provider nova.volume.encryptors.luks.LuksEncryptor \
      --encryption-cipher aes-xts-plain64 \
      --encryption-key-size 256 \
      --encryption-control-location front-end
    ```

    ```bash title="Provision an encrypted volume" theme={null}
    openstack volume create \
      --size 200 \
      --type encrypted-nvme \
      --description "Production database volume" \
      db-vol-01
    ```

    ```bash title="Verify encryption is active" theme={null}
    openstack volume show db-vol-01 --column encrypted --format value
    # Returns: True

    openstack volume show db-vol-01 --column "volume_image_metadata" --format json
    ```
  </Tab>
</Tabs>

***

## Key Management Integration (Barbican)

Xloud Key Management stores, rotates, and controls access to all encryption keys. Each encrypted volume has a unique key. Key access is audited and can be restricted by ACL.

### Key Operations

<Tabs>
  <Tab title="CLI" icon="terminal">
    ```bash title="List secrets managed by Key Management" theme={null}
    openstack secret list --limit 10
    ```

    ```bash title="Retrieve key metadata (not the key material)" theme={null}
    openstack secret get <secret-href>
    ```

    ```bash title="Create a named secret for application use" theme={null}
    openstack secret store \
      --name "app-encryption-key" \
      --secret-type "symmetric" \
      --payload-content-type "application/octet-stream" \
      --payload "$(openssl rand -base64 32)"
    ```

    ```bash title="Set an ACL on a secret" theme={null}
    openstack acl user add \
      --user <user-id> \
      --project <project-id> \
      <secret-href>
    ```
  </Tab>

  <Tab title="Dashboard" icon="gauge">
    Navigate to **Project → Key Manager → Secrets** to view and manage secrets associated with your project.

    * **Store** — Store a new secret (symmetric key, certificate, passphrase, or opaque blob)
    * **View ACL** — Control which users and projects can access each secret
    * **Delete** — Remove a secret and revoke access to associated encrypted resources

    <Warning>Deleting an encryption key for an in-use volume renders that volume permanently inaccessible. Key deletion is irreversible. Always verify a key is no longer in use before deleting it.</Warning>
  </Tab>
</Tabs>

### Key Rotation

```bash title="Rotate an encryption key (create new, re-encrypt)" theme={null}
# Step 1: Create a new key
NEW_KEY=$(openstack secret store \
  --name "rotated-key-$(date +%Y%m)" \
  --secret-type symmetric \
  --payload "$(openssl rand -base64 32)" \
  -c "Secret href" -f value)

# Step 2: Migrate volume to new key (requires volume to be detached)
openstack volume migrate <volume-id> --host <new-host-with-new-key>
```

<Tip>Key rotation requires migrating the volume. Schedule rotations during maintenance windows. Xloud Key Management retains old keys until explicitly deleted to support rollback scenarios.</Tip>

***

## Object Storage Encryption

Object storage encryption protects data at rest in the object store. Two modes are supported:

| Mode    | Description                            | Key Location     |
| ------- | -------------------------------------- | ---------------- |
| SSE-C   | Customer-provided key sent per request | Client-managed   |
| SSE-KMS | Keys managed by Xloud Key Management   | Platform-managed |

```bash title="Upload object with SSE-C encryption" theme={null}
# Generate a 256-bit key
KEY=$(openssl rand -base64 32)

openstack object create my-container my-file.dat \
  --object-name encrypted-file.dat \
  -H "X-Object-Sysmeta-Crypto-Key: $KEY" \
  -H "X-Object-Sysmeta-Crypto-Etag: yes"
```

<Info>
  With SSE-C, the client is responsible for storing and supplying the encryption key on every request. If the key is lost, the object is permanently unreadable. Use SSE-KMS for platform-managed key storage and rotation.
</Info>

***

## Encrypted Backups

Volume backups inherit the encryption state of their source volume. Backups of encrypted volumes are stored encrypted in the backup repository.

```bash title="Create encrypted backup" theme={null}
openstack volume backup create \
  --name db-vol-backup-$(date +%Y%m%d) \
  --force \
  db-vol-01
```

```bash title="Verify backup encryption" theme={null}
openstack volume backup show db-vol-backup-20251201 \
  --column "availability_zone" \
  --column "volume_id" \
  --column "status" \
  --format table
```

<Warning>
  Backup keys are tied to the same Xloud Key Management secret as the source volume. If the source volume's encryption key is deleted, the backup becomes permanently unreadable. Do not delete keys for volumes that have active backups.
</Warning>

***

## Data-at-Rest Encryption for Databases

Platform databases (MariaDB) use encrypted tablespaces for sensitive configuration and credential storage. XDeploy configures this during deployment, so you do not need manual intervention.

```yaml title="Enable database encryption (XDeploy configuration)" theme={null}
mariadb_enable_encryption: "yes"
mariadb_encryption_key_management: "file"
```

For production deployments, use an external key management server:

```yaml title="External KMS for database encryption" theme={null}
mariadb_encryption_key_management: "hashicorp_vault"
mariadb_encryption_vault_addr: "https://vault.internal:8200"
mariadb_encryption_vault_token: "{{ vault_token }}"
```

***

## Secure Deletion and Data Scrubbing

When a volume is deleted, the underlying storage blocks are marked for reclamation. For regulatory compliance requiring secure deletion:

```bash title="Zero-fill a volume before deletion" theme={null}
# Attach volume to a temporary instance and zero it
# Inside the instance:
sudo dd if=/dev/zero of=/dev/vdb bs=4M status=progress
sudo sync

# Then detach and delete the volume
openstack volume detach <instance-id> <volume-id>
openstack volume delete <volume-id>
```

<Info>
  For Ceph-backed storage, the storage layer performs data scrubbing through Ceph's object deletion and placement group (PG) scrub process. Immediately after deletion, the data may remain on OSDs until the next scrub cycle. For environments requiring immediate data erasure, use LUKS encryption. Deleting the key renders the data unreadable without waiting for the scrub cycle.
</Info>

***

## Next Steps

<CardGroup>
  <Card title="Key Manager Service" href="/services/key-manager/store-secrets">
    Detailed guide for storing and managing secrets in Xloud Key Management
  </Card>

  <Card title="Volume Encryption (Admin)" href="/services/storage/encryption">
    Administrative guide for volume encryption and volume type management
  </Card>

  <Card title="VM Security" href="/security/vm-security">
    vTPM, Secure Boot, and hypervisor-level protection for virtual machines
  </Card>

  <Card title="Compliance" href="/security/compliance">
    Audit logging and compliance framework requirements for encrypted data
  </Card>
</CardGroup>


# Security Hardening Guide
Source: https://docs.xloud.tech/security/hardening-guide

Step-by-step pre-deployment security hardening for Xloud Platform — OS hardening, SSH configuration, service minimization, database and message queue hardening.

## Overview

This guide provides a structured hardening walkthrough for Xloud Platform nodes. Apply these controls before the first production deployment. The checklist at the end of this page provides a verification reference for compliance audits.

Hardening operates at four levels: the host operating system (XOS), the Xloud platform services, supporting infrastructure components (database, message queue), and the metadata service.

<Info>**Xloud-Developed** — XAVS ships with automated hardening roles that enforce CIS benchmarks out of the box. Automated controls include SSH hardening and allowlisting, audit logging configuration, Docker security benchmarks, and AppArmor profile enforcement. These roles run during initial deployment and can be re-applied at any time via `xavs-ansible reconfigure --tags hardening`.</Info>

<Note>
  **Prerequisites**

  * XOS installed on all control and compute nodes
  * XDeploy access with bootstrap credentials
  * SSH access to all cluster nodes
  * A change management window — some steps require service restarts
</Note>

<Warning>
  Apply hardening before deploying production workloads. Some controls (such as disabling unused kernel modules) require a reboot. Schedule accordingly.
</Warning>

***

## OS Hardening

<Steps>
  <Step title="Configure SSH" icon="terminal">
    Restrict SSH to key-based authentication and disable direct root login on all nodes:

    ```bash title="Harden SSH configuration" theme={null}
    cat >> /etc/ssh/sshd_config << 'EOF'
    PermitRootLogin no
    PasswordAuthentication no
    PubkeyAuthentication yes
    AuthorizedKeysFile .ssh/authorized_keys
    X11Forwarding no
    AllowTcpForwarding no
    MaxAuthTries 3
    ClientAliveInterval 300
    ClientAliveCountMax 2
    LoginGraceTime 30
    EOF

    systemctl restart sshd
    ```

    <Check>Verify: `ssh root@<node>` is rejected. Key-based login as the `xloud` user succeeds.</Check>
  </Step>

  <Step title="Enable automatic security updates" icon="rotate-cw">
    Configure unattended upgrades for security patches:

    ```bash title="Configure unattended upgrades" theme={null}
    apt-get install -y unattended-upgrades apt-listchanges

    cat > /etc/apt/apt.conf.d/50unattended-upgrades << 'EOF'
    Unattended-Upgrade::Allowed-Origins {
      "${distro_id}:${distro_codename}-security";
    };
    Unattended-Upgrade::AutoFixInterruptedDpkg "true";
    Unattended-Upgrade::MinimalSteps "true";
    Unattended-Upgrade::Mail "security@your-org.com";
    Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
    Unattended-Upgrade::Remove-Unused-Dependencies "true";
    EOF

    systemctl enable unattended-upgrades
    systemctl start unattended-upgrades
    ```
  </Step>

  <Step title="Configure host firewall (ufw)" icon="shield">
    Restrict inbound traffic to required service ports only:

    ```bash title="Configure ufw on control nodes" theme={null}
    ufw default deny incoming
    ufw default allow outgoing
    ufw allow from 10.0.0.0/8 to any port 22    # SSH from management network
    ufw allow from 10.0.0.0/8 to any port 443   # HTTPS API
    ufw allow from 10.0.0.0/8 to any port 5000  # Keystone
    ufw allow from 10.0.0.0/8 to any port 8774  # Nova API
    ufw allow from 10.0.0.0/8 to any port 9696  # Neutron
    ufw enable
    ```

    ```bash title="Configure ufw on compute nodes" theme={null}
    ufw default deny incoming
    ufw default allow outgoing
    ufw allow from 10.0.0.0/8 to any port 22      # SSH
    ufw allow from 10.0.0.0/8 to any port 16509   # libvirt
    ufw allow from 10.0.0.0/8 to any port 49152:49215/tcp  # live migration
    ufw enable
    ```
  </Step>

  <Step title="Disable unused kernel modules" icon="cpu">
    Remove attack surface by disabling kernel modules that are not required:

    ```bash title="Disable unused modules" theme={null}
    cat > /etc/modprobe.d/hardening.conf << 'EOF'
    # Disable uncommon filesystems
    install cramfs /bin/true
    install freevxfs /bin/true
    install jffs2 /bin/true
    install hfs /bin/true
    install hfsplus /bin/true
    install squashfs /bin/true
    install udf /bin/true

    # Disable uncommon network protocols
    install dccp /bin/true
    install sctp /bin/true
    install rds /bin/true
    install tipc /bin/true
    EOF
    ```
  </Step>

  <Step title="Configure kernel security parameters" icon="settings">
    ```bash title="Apply kernel hardening via sysctl" theme={null}
    cat > /etc/sysctl.d/99-hardening.conf << 'EOF'
    # Network hardening
    net.ipv4.conf.all.accept_source_route = 0
    net.ipv4.conf.default.accept_source_route = 0
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.default.accept_redirects = 0
    net.ipv4.conf.all.secure_redirects = 0
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.icmp_echo_ignore_broadcasts = 1
    net.ipv4.icmp_ignore_bogus_error_responses = 1
    net.ipv4.tcp_syncookies = 1
    net.ipv4.conf.all.log_martians = 1

    # Memory hardening
    kernel.randomize_va_space = 2
    kernel.dmesg_restrict = 1
    kernel.kptr_restrict = 2
    fs.protected_hardlinks = 1
    fs.protected_symlinks = 1
    EOF

    sysctl -p /etc/sysctl.d/99-hardening.conf
    ```
  </Step>

  <Step title="Enable AppArmor enforcement" icon="shield-check">
    AppArmor profiles for hypervisor processes ship in enforce mode on XOS. Verify and enable:

    ```bash title="Verify AppArmor status" theme={null}
    aa-status | grep -E "profiles|enforce|complain"
    ```

    Switch complain-mode profiles to enforce:

    ```bash title="Enforce all complain-mode profiles" theme={null}
    for profile in $(aa-status --json | python3 -c "
    import sys, json
    d = json.load(sys.stdin)
    for p in d.get('complain', []): print(p)
    "); do
      aa-enforce /etc/apparmor.d/$profile 2>/dev/null || true
    done
    ```

    <Check>Run `aa-status` and confirm no profiles remain in complain mode for hypervisor-related processes.</Check>
  </Step>
</Steps>

***

## Service Hardening

<Steps>
  <Step title="Minimize running services" icon="x-circle">
    Disable services that are not required on each node type:

    ```bash title="Disable unused services on control nodes" theme={null}
    # Services not needed on a dedicated control plane node
    systemctl disable --now cups bluetooth avahi-daemon
    systemctl mask cups bluetooth avahi-daemon
    ```

    ```bash title="Disable unused services on compute nodes" theme={null}
    systemctl disable --now cups bluetooth
    systemctl mask cups bluetooth
    ```
  </Step>

  <Step title="Set file permission baselines" icon="file-lock">
    ```bash title="Restrict sensitive configuration files" theme={null}
    chmod 600 /etc/xavs/passwords.yml
    chmod 750 /etc/xavs/
    find /etc/xavs/certificates/ -type f -name "*.key" -exec chmod 600 {} \;
    ```
  </Step>

  <Step title="Harden the metadata service" icon="server">
    The instance metadata service is accessible at `169.254.169.254` from all VMs. Enable the shared secret to prevent unauthorized cross-tenant metadata access:

    ```yaml title="/etc/xavs/globals.d/_60_metadata.yml" theme={null}
    neutron_metadata_proxy_shared_secret: "<generate with: openssl rand -hex 32>"
    nova_metadata_enabled: "yes"
    ```

    ```bash title="Deploy metadata hardening" theme={null}
    xavs-ansible reconfigure --tags nova,neutron
    ```
  </Step>

  <Step title="Configure audit daemon (auditd)" icon="list">
    ```bash title="Install and configure auditd" theme={null}
    apt-get install -y auditd audispd-plugins

    cat >> /etc/audit/rules.d/hardening.rules << 'EOF'
    # Monitor privileged command execution
    -a always,exit -F arch=b64 -S execve -F euid=0 -k privileged
    # Monitor file permission changes on sensitive paths
    -w /etc/passwd -p wa -k identity
    -w /etc/shadow -p wa -k identity
    -w /etc/xavs/passwords.yml -p rwa -k xavs-passwords
    # Monitor network configuration changes
    -a always,exit -F arch=b64 -S sethostname -S setdomainname -k network-config
    EOF

    augenrules --load
    systemctl enable auditd
    systemctl start auditd
    ```
  </Step>
</Steps>

***

## Database Hardening

<Steps>
  <Step title="Remove anonymous and test accounts" icon="user-x">
    ```bash title="Audit MariaDB accounts" theme={null}
    docker exec mariadb mysql -uroot -p"${MYSQL_ROOT_PASSWORD}" \
      -e "SELECT user, host, password FROM mysql.user ORDER BY user;"
    ```

    Remove any anonymous accounts and the test database if present.
  </Step>

  <Step title="Restrict remote root access" icon="lock">
    ```bash title="Disable remote root login" theme={null}
    docker exec mariadb mysql -uroot -p"${MYSQL_ROOT_PASSWORD}" \
      -e "DELETE FROM mysql.user WHERE User='root' AND Host != 'localhost'; FLUSH PRIVILEGES;"
    ```
  </Step>

  <Step title="Enable binary log encryption" icon="database">
    ```yaml title="/etc/xavs/globals.d/_60_db_security.yml" theme={null}
    mariadb_enable_binlog: "yes"
    mariadb_binlog_format: "ROW"
    mariadb_expire_logs_days: "7"
    ```
  </Step>
</Steps>

***

## Message Queue Hardening

<Steps>
  <Step title="Enable RabbitMQ TLS" icon="lock">
    ```yaml title="/etc/xavs/globals.d/_60_rabbitmq.yml" theme={null}
    rabbitmq_enable_tls: "yes"
    rabbitmq_management_enabled: "yes"
    ```
  </Step>

  <Step title="Remove default guest account" icon="user-x">
    The `guest` account in RabbitMQ is restricted to `localhost` by default in the Xloud deployment. Verify it is not accessible remotely:

    ```bash title="Verify guest user restrictions" theme={null}
    docker exec rabbitmq rabbitmqctl list_users
    # guest user should not appear, or should be tagged as non-admin
    ```
  </Step>

  <Step title="Set queue and message TTLs" icon="clock">
    Configure message TTLs to prevent message queue buildup from accumulating sensitive data:

    ```yaml title="RabbitMQ TTL policy" theme={null}
    rabbitmq_message_ttl: "3600000"
    rabbitmq_queue_ttl: "86400000"
    ```
  </Step>
</Steps>

***

## Pre-Deployment Hardening Checklist

Use this checklist to verify hardening is complete before going to production.

| Category    | Control                    | Verification Command                                                                                        |
| ----------- | -------------------------- | ----------------------------------------------------------------------------------------------------------- |
| SSH         | Root login disabled        | `grep PermitRootLogin /etc/ssh/sshd_config`                                                                 |
| SSH         | Password auth disabled     | `grep PasswordAuthentication /etc/ssh/sshd_config`                                                          |
| OS          | Auto-updates active        | `systemctl is-enabled unattended-upgrades`                                                                  |
| OS          | Host firewall enabled      | `ufw status`                                                                                                |
| OS          | ASLR enabled               | `sysctl kernel.randomize_va_space`                                                                          |
| OS          | AppArmor enforcing         | `aa-status --json \| python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('enforce',[])))"` |
| Services    | Unused services disabled   | `systemctl list-units --state=active --type=service`                                                        |
| TLS         | External TLS active        | `openssl s_client -connect <vip>:443 \| openssl x509 -noout -enddate`                                       |
| TLS         | Internal TLS active        | `grep "enable_tls_internal" /etc/xavs/globals.d/_60_tls.yml`                                                |
| Certs       | Certificate not expired    | `openssl x509 -in /etc/xavs/certificates/haproxy.crt -noout -enddate`                                       |
| Permissions | Passwords files restricted | `stat -c "%a %U %G" /etc/xavs/passwords.yml`                                                                |
| Metadata    | Shared secret set          | `grep "neutron_metadata_proxy_shared_secret" /etc/xavs/globals.d/`                                          |
| Database    | No anonymous users         | `docker exec mariadb mysql -e "SELECT user,host FROM mysql.user WHERE user=''"`                             |
| Audit       | auditd running             | `systemctl is-active auditd`                                                                                |
| Audit       | Audit rules loaded         | `auditctl -l \| wc -l`                                                                                      |

***

## Next Steps

<CardGroup>
  <Card title="Infrastructure Security" href="/security/infrastructure">
    TLS configuration and certificate management
  </Card>

  <Card title="Compliance" href="/security/compliance">
    Map hardening controls to SOC 2, ISO 27001, and PCI-DSS requirements
  </Card>

  <Card title="Network Security" href="/security/network-security">
    Security groups, FWaaS, and network segmentation
  </Card>

  <Card title="Troubleshooting" href="/security/hardening-guide">
    Diagnose and resolve common security configuration issues
  </Card>
</CardGroup>


# Security
Source: https://docs.xloud.tech/security/index

Comprehensive security documentation for Xloud Platform — infrastructure hardening, VM isolation, API protection, data encryption, and compliance.

<p>Defense in Depth</p>

Xloud Platform enforces security at every layer of the stack. From hypervisor isolation and encrypted data channels to role-based access control and audit logging, the platform is designed so that no single control failure exposes workloads. This layered approach — commonly called defense in depth — means each security boundary operates independently and reinforces the others.

The sections below cover every security domain: infrastructure TLS, VM isolation, API authentication, data encryption, network segmentation, compliance frameworks, pre-deployment hardening, and troubleshooting.

***

## Xloud SIEM — Unified Security Operations

<CardGroup>
  <Card title="Xloud SIEM" icon="shield-alert" href="/security/xloud-siem">
    The integrated Security Information and Event Management layer — Wazuh (HIDS), Lynis
    (auditing), and OpenSCAP (compliance) unified in the **Security Posture** and
    **Alerts** pages in Monitor Center. Start here for a single-pane view of the entire
    cluster's security posture.
  </Card>
</CardGroup>

***

<p>Infrastructure Security</p>

<CardGroup>
  <Card title="Infrastructure Security" icon="server" href="/security/infrastructure">
    TLS configuration for all platform services, certificate management, HAProxy termination, and endpoint hardening.
  </Card>

  <Card title="Hardening Guide" icon="shield-check" href="/security/hardening-guide">
    Pre-deployment OS hardening, service minimization, database and message queue hardening, and a step-by-step checklist.
  </Card>
</CardGroup>

***

<p>Virtual Machine Security</p>

<CardGroup>
  <Card title="VM Security" icon="cpu" href="/security/vm-security">
    Hypervisor isolation, security groups, vTPM, encrypted volumes, Secure Boot, anti-affinity, and live migration TLS.
  </Card>

  <Card title="Network Security" icon="network" href="/security/network-security">
    Security groups, FWaaS, port security, anti-spoofing, VLAN/VXLAN segmentation, and VPN as a Service.
  </Card>
</CardGroup>

***

<p>API and Data</p>

<CardGroup>
  <Card title="API Security" icon="lock" href="/security/api-security">
    Token authentication, application credentials, rate limiting, CORS, RBAC policy enforcement, and mutual TLS.
  </Card>

  <Card title="Data Security" icon="database" href="/security/data-security">
    Volume encryption (LUKS), object storage encryption, key management integration, encrypted backups, and secure deletion.
  </Card>
</CardGroup>

***

<p>Compliance and Operations</p>

<CardGroup>
  <Card title="Compliance and Auditing" icon="clipboard-list" href="/security/compliance">
    Audit logging, log retention, SOC 2 / ISO 27001 / HIPAA / PCI-DSS / GDPR frameworks, and incident response.
  </Card>

  <Card title="Security Troubleshooting" icon="wrench" href="/security/hardening-guide">
    TLS errors, 401/403 authentication failures, security group rule issues, audit log gaps, and encryption failures.
  </Card>
</CardGroup>

***

<p>Security Tools — Xloud SIEM</p>

These three scanners run in parallel inside Xloud SIEM. Their combined results appear in
the **Security Posture** and **Alerts** pages in the Dashboard's Monitor Center.

<CardGroup>
  <Card title="Wazuh" icon="shield-alert" href="/security/wazuh">
    Host intrusion detection, file integrity monitoring, vulnerability assessment, and compliance reporting — deployed across all VMs.
  </Card>

  <Card title="Lynis" icon="list-checks" href="/security/lynis">
    OS security auditing with a hardening index score, actionable remediation suggestions, and fleet-wide sweep support.
  </Card>

  <Card title="OpenSCAP" icon="clipboard-check" href="/security/openscap">
    SCAP-based compliance scanning against CIS Benchmarks, DISA STIGs, PCI-DSS, and HIPAA profiles with automated remediation playbooks.
  </Card>
</CardGroup>

***

<p>Security Architecture</p>

The following table summarizes the security controls enforced at each layer of the Xloud platform.

| Layer         | Controls                                                                                               |
| ------------- | ------------------------------------------------------------------------------------------------------ |
| Hypervisor    | Process isolation, seccomp profiles, AppArmor confinement, dedicated service users, live migration TLS |
| Networking    | Security groups (stateful), FWaaS, port security, anti-spoofing, VLAN/VXLAN isolation                  |
| Control Plane | TLS on all APIs, token-based authentication, RBAC policy enforcement, rate limiting                    |
| Storage       | LUKS volume encryption, Ceph encryption at rest, key management via Xloud Key Management               |
| Audit         | CADF event logging, centralized log aggregation, immutable audit trails                                |
| Host Security | Integrated security platform (intrusion detection + FIM), system auditing, SCAP compliance scanning    |

<Note>
  Xloud follows a shared responsibility model. The platform enforces infrastructure-level controls. Workload owners are responsible for securing applications running inside virtual machines.
</Note>

***

<p>Xloud Security Platform Capabilities</p>

<Info>**Xloud-Developed** — This capability is developed by Xloud and ships with XAVS.</Info>

The following security capabilities are built into the Xloud platform and deploy automatically as part of XAVS. Each capability is production-ready and requires no third-party licensing.

<CardGroup>
  <Card title="Integrated SIEM" icon="radar">
    Full security information and event management built into the platform. Agent-based monitoring on all nodes with real-time alerting and log correlation.
  </Card>

  <Card title="Triple Compliance Scanning" icon="clipboard-list">
    Three independent scanners running in parallel: SCA benchmarks, system audit, and SCAP profiles. CIS Level 1 and Level 2 benchmarks included.
  </Card>

  <Card title="OS Hardening (CIS Benchmark)" icon="shield-check">
    Automated CIS benchmark hardening: SSH controls, audit logging, Docker security benchmarks, AppArmor profiles, and SSH allowlisting.
  </Card>

  <Card title="Security Operations Dashboard" icon="layout-dashboard">
    Auto-deployed monitoring dashboard with panels for agent status, API health, credential recovery events, certificate expiry, scan results, and cluster health.
  </Card>

  <Card title="Self-Healing Credentials" icon="key">
    Three-layer automated credential recovery: post-deployment enforcement, periodic watchdog (5-minute intervals), and filesystem guardian (10-minute intervals). Recovery time under 5 minutes with zero human intervention.
  </Card>

  <Card title="Active Response" icon="shield-alert">
    Automated threat response: firewall blocking, host denial, and account disabling on SSH brute force detection.
  </Card>

  <Card title="Certificate Expiry Monitoring" icon="calendar-clock">
    Automated certificate lifecycle monitoring with 30-day warning and 7-day critical alerts across all platform services.
  </Card>

  <Card title="Custom Detection Rules" icon="scan-search">
    Platform-specific detection rules for container lifecycle events including start, stop, crash, and resource limit triggers.
  </Card>

  <Card title="Default Alert Rules" icon="bell">
    12 pre-configured alert rules across 4 groups: node alerts (disk, memory, CPU), service alerts, storage alerts, and infrastructure alerts.
  </Card>

  <Card title="SBOM CI Pipeline" icon="package-search">
    Supply chain security: container image vulnerability scanning, software bill of materials generation (SPDX and CycloneDX formats), and cryptographic image signing.
  </Card>
</CardGroup>

***

<p>Quick Links</p>

<CardGroup>
  <Card title="Enable TLS" icon="lock" href="/security/infrastructure#tls-configuration">
    Configure TLS for all platform services
  </Card>

  <Card title="Security Groups" icon="shield" href="/security/network-security#security-groups">
    Create and manage stateful firewall rules
  </Card>

  <Card title="Encrypt Volumes" icon="hard-drive" href="/security/data-security#volume-encryption">
    Enable LUKS encryption for block storage
  </Card>

  <Card title="Hardening Checklist" icon="list-checks" href="/security/hardening-guide#checklist">
    Pre-deployment security verification
  </Card>
</CardGroup>


# Infrastructure Security
Source: https://docs.xloud.tech/security/infrastructure

Configure TLS for all Xloud platform services, manage certificates, harden service endpoints, and secure HAProxy termination for internal and external traffic.

## Overview

Xloud platform services communicate over encrypted channels by default. TLS protects your API endpoints, inter-service traffic, database connections, and message queue channels. This page covers certificate provisioning and TLS configuration at the internal and external layers. It also covers HAProxy termination and endpoint hardening for production deployments.

<Note>
  **Prerequisites**

  * XDeploy access with administrator privileges
  * SSL/TLS certificates (CA-signed, self-signed, or Let's Encrypt)
  * Access to the XDeploy configuration interface or `/etc/xavs/globals.d/` on the bootstrap node
</Note>

***

## TLS Architecture

Xloud separates TLS configuration into three independent scopes:

| Scope            | Description                                              | Applies To                                             |
| ---------------- | -------------------------------------------------------- | ------------------------------------------------------ |
| **External TLS** | Traffic between clients and the platform API / Dashboard | Public-facing endpoints, HAProxy VIP                   |
| **Internal TLS** | Service-to-service traffic within the cluster            | API ↔ database, API ↔ message queue, service ↔ service |
| **Backend TLS**  | HAProxy to upstream service connections                  | HAProxy → Keystone, Nova API, Neutron, etc.            |

```mermaid theme={null}
graph LR
    Client -->|External TLS| HAProxy
    HAProxy -->|Backend TLS| NovaAPI[Nova API]
    HAProxy -->|Backend TLS| Keystone
    HAProxy -->|Backend TLS| Neutron
    NovaAPI -->|Internal TLS| RabbitMQ
    NovaAPI -->|Internal TLS| MariaDB
    NovaAPI -->|Internal TLS| Memcached
```

<Warning>
  Enabling TLS requires all services to be restarted. Plan a maintenance window for initial TLS enablement on an existing cluster. New clusters should have TLS configured before the first deployment.
</Warning>

***

## TLS Configuration

<Tabs>
  <Tab title="XDeploy" icon="gauge">
    <Steps>
      <Step title="Open Global Configuration" icon="settings">
        Log in to XDeploy and navigate to **Configuration → Global Settings**.
      </Step>

      <Step title="Enable TLS Scopes" icon="lock">
        Locate the **Security** section and enable each TLS scope appropriate for your deployment:

        | Setting             | Recommended Value | Description                         |
        | ------------------- | ----------------- | ----------------------------------- |
        | Enable External TLS | Yes               | Encrypts client-to-HAProxy traffic  |
        | Enable Internal TLS | Yes               | Encrypts service-to-service traffic |
        | Enable Backend TLS  | Yes               | Encrypts HAProxy-to-service traffic |

        <Tip>For greenfield deployments, enable all three scopes from the start. For existing clusters, enable external TLS first, then internal TLS in a second maintenance window.</Tip>
      </Step>

      <Step title="Configure Certificate Source" icon="certificate">
        Select the certificate source for external TLS:

        * **Self-signed**: Xloud generates certificates automatically using the internal CA
        * **CA-signed**: Upload your organization's certificate and private key
        * **Let's Encrypt**: Provide a domain name and contact email for automatic provisioning

        Upload the certificate bundle if using CA-signed certificates.
      </Step>

      <Step title="Apply Configuration" icon="play">
        Click **Save and Deploy**. XDeploy runs the TLS configuration playbook across all nodes.

        <Check>All services restart with TLS enabled. HAProxy health checks confirm green status.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create TLS globals override" icon="file">
        Create or edit the TLS configuration file:

        ```bash title="Create TLS configuration" theme={null}
        cat > /etc/xavs/globals.d/_60_tls.yml << 'EOF'
        kolla_enable_tls_external: "yes"
        kolla_enable_tls_internal: "yes"
        kolla_enable_tls_backend: "yes"
        EOF
        ```
      </Step>

      <Step title="Configure external certificates" icon="certificate">
        For CA-signed certificates, place your files in the correct locations:

        ```bash title="Install CA-signed certificate" theme={null}
        # Copy certificate and key to XAVS certificates directory
        cp your-cert.crt /etc/xavs/certificates/haproxy.crt
        cp your-key.key  /etc/xavs/certificates/haproxy.key
        cp your-ca.crt   /etc/xavs/certificates/ca/xloud-ca.crt
        chmod 600 /etc/xavs/certificates/haproxy.key
        ```

        For self-signed certificates, run the certificate generation utility:

        ```bash title="Generate self-signed certificates" theme={null}
        xavs-ansible certificates
        ```
      </Step>

      <Step title="Deploy TLS configuration" icon="play">
        ```bash title="Deploy with TLS enabled" theme={null}
        xavs-ansible deploy --tags haproxy,certificates
        ```

        After the initial certificate deployment, reconfigure all services:

        ```bash title="Reconfigure all services" theme={null}
        xavs-ansible reconfigure
        ```

        <Check>All service endpoints respond on HTTPS. Run `openssl s_client -connect <VIP>:443` to verify certificate validity.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Certificate Management

### Self-Signed Certificates

Xloud uses an internal CA to generate self-signed certificates for all services. You must distribute the CA certificate to all clients that communicate with the platform.

```bash title="Export internal CA certificate" theme={null}
cat /etc/xavs/certificates/ca/xloud-ca.crt
```

Import this CA certificate into your browser, operating system trust store, or client configuration to avoid certificate validation errors.

### CA-Signed Certificates

<Steps>
  <Step title="Generate a CSR" icon="file-text">
    ```bash title="Generate certificate signing request" theme={null}
    openssl req -new -newkey rsa:4096 -nodes \
      -keyout haproxy.key \
      -out haproxy.csr \
      -subj "/CN=<your-vip-hostname>/O=Your Organization/C=IN" \
      -addext "subjectAltName=DNS:<hostname>,IP:<vip-ip>"
    ```
  </Step>

  <Step title="Submit CSR to your CA" icon="building">
    Submit `haproxy.csr` to your certificate authority. The CA returns a signed certificate (`haproxy.crt`) and the CA chain (`ca-chain.crt`).
  </Step>

  <Step title="Install the signed certificate" icon="circle-check">
    ```bash title="Install certificate files" theme={null}
    cp haproxy.crt      /etc/xavs/certificates/haproxy.crt
    cp haproxy.key      /etc/xavs/certificates/haproxy.key
    cp ca-chain.crt     /etc/xavs/certificates/ca/xloud-ca.crt
    xavs-ansible deploy --tags haproxy
    ```
  </Step>
</Steps>

### Certificate Renewal

<Warning>
  Certificates must be renewed before expiry. Monitor certificate expiration and plan renewals at least 30 days in advance. An expired certificate causes authentication failures across all platform services.
</Warning>

```bash title="Check certificate expiration" theme={null}
openssl x509 -in /etc/xavs/certificates/haproxy.crt -noout -enddate
```

***

## HAProxy TLS Termination

HAProxy terminates external TLS at the VIP and forwards requests to upstream services. The configuration supports both TLS termination (backend plain) and TLS pass-through (backend TLS).

| Mode          | Description                                   | Use Case                               |
| ------------- | --------------------------------------------- | -------------------------------------- |
| Termination   | HAProxy decrypts; backend receives plain HTTP | Default — simplifies backend config    |
| Re-encryption | HAProxy decrypts; re-encrypts to backend      | Maximum security; backend TLS required |
| Pass-through  | HAProxy forwards encrypted bytes unchanged    | End-to-end mTLS for specific services  |

The default configuration uses TLS termination for external traffic and re-encryption for backend traffic when `kolla_enable_tls_backend: "yes"` is set.

***

## Service Endpoint Hardening

<AccordionGroup>
  <Accordion title="Disable unused API versions" icon="code">
    Restrict API endpoints to the minimum required versions. For Xloud Compute, disable legacy v2.0 and enforce v2.1:

    ```yaml title="/etc/xavs/globals.d/_60_endpoint_hardening.yml" theme={null}
    nova_api_enabled_apis: "osapi_compute,metadata"
    ```
  </Accordion>

  <Accordion title="Restrict cipher suites" icon="lock">
    Enforce modern cipher suites and disable weak protocols. Add to the HAProxy global configuration:

    ```yaml title="Cipher suite hardening" theme={null}
    kolla_tls_min_version: "TLSv1.2"
    kolla_tls_ciphers: "ECDHE+AESGCM:ECDHE+CHACHA20:!aNULL:!MD5:!DSS"
    ```

    This disables TLS 1.0, TLS 1.1, and all weak cipher suites including RC4, 3DES, and export ciphers.
  </Accordion>

  <Accordion title="Enable HSTS headers" icon="shield">
    For Dashboard (Horizon) endpoints, enable HTTP Strict Transport Security to prevent protocol downgrade attacks:

    ```yaml title="HSTS configuration" theme={null}
    horizon_enable_hsts: "yes"
    horizon_hsts_max_age: 31536000
    ```
  </Accordion>

  <Accordion title="Configure Keystone token expiry" icon="clock">
    Reduce the default token lifetime to limit the window of exposure for compromised tokens:

    ```yaml title="Token expiry configuration" theme={null}
    keystone_token_expiration: 3600
    keystone_allow_expired_window: 300
    ```

    The default token lifetime is 3600 seconds (1 hour). Reduce to 1800 for sensitive environments.
  </Accordion>
</AccordionGroup>

***

## Validation

Verify TLS is active across all platform endpoints:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to the XDeploy **Services** view. All service health indicators should show green. Click on any service to view its TLS status.

    <Check>All services show a valid certificate with a matching hostname and a future expiry date.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Verify external TLS" theme={null}
    openssl s_client -connect <vip-hostname>:443 -servername <vip-hostname> 2>/dev/null \
      | openssl x509 -noout -subject -dates
    ```

    ```bash title="Test API endpoint" theme={null}
    curl -sk https://<vip-hostname>:5000/v3 | python3 -m json.tool
    ```

    ```bash title="Check all service ports" theme={null}
    for port in 443 5000 8774 9292 9696 8004 9511; do
      echo -n "Port $port: "
      openssl s_client -connect <vip>:$port -servername <vip-hostname> \
        -brief 2>/dev/null | grep -E "Protocol|Cipher"
    done
    ```

    <Check>Each port responds with a valid TLS handshake and a modern cipher suite (TLS 1.2 or 1.3).</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Hardening Guide" href="/security/hardening-guide">
    OS-level hardening, SSH configuration, and service minimization to complement TLS
  </Card>

  <Card title="API Security" href="/security/api-security">
    Token authentication, application credentials, and RBAC enforcement
  </Card>

  <Card title="Compliance" href="/security/compliance">
    Audit logging, log retention, and compliance framework mapping
  </Card>

  <Card title="Network Security" href="/security/network-security">
    Security groups, FWaaS, and network segmentation
  </Card>
</CardGroup>


# Lynis
Source: https://docs.xloud.tech/security/lynis

Run Lynis security audits on Xloud virtual machines and host nodes to detect misconfigurations, hardening gaps, and compliance deviations with remediation guidance.

## Overview

Lynis is an open-source security auditing tool that performs in-depth system scans directly on Linux hosts — no agent required. It checks over 300 security controls covering kernel hardening, authentication configuration, filesystem permissions, network services, software patches, and logging posture. Each scan produces a hardening index score and a prioritized list of remediation suggestions.

Xloud bundles Lynis in XOS and makes it available via the XDeploy automation pipeline for both individual node audits and fleet-wide compliance sweeps.

<Info>**Xloud-Developed** — Lynis is one of three independent scanners in [Xloud SIEM](/security/xloud-siem) — Wazuh, Lynis, and OpenSCAP run in parallel across all nodes for layered compliance coverage. Results are aggregated on the **Security Posture** page in Monitor Center.</Info>

<Note>
  **Prerequisites**

  * SSH access to the target host (or run directly on the node)
  * Lynis installed (pre-installed on XOS nodes; install via `apt install lynis` on guest VMs)
  * Root or sudo access on the target system
</Note>

***

## How Lynis Works

Lynis runs as a shell script directly on the host. It does not require a daemon, network connection, or external service. It tests the live system state — not a snapshot — and reports findings immediately.

```mermaid theme={null}
graph LR
    A[Run lynis audit system] --> B[300+ Security Tests]
    B --> C[Hardening Index Score]
    B --> D[Warnings & Suggestions]
    D --> E[Remediation Steps]
    C --> F[Compliance Report]
    style A fill:#197560,color:#fff
    style C fill:#145C4C,color:#fff
```

| Phase               | What Lynis Checks                                                   |
| ------------------- | ------------------------------------------------------------------- |
| **Boot & Services** | GRUB password, bootloader permissions, running services, inetd      |
| **Kernel**          | Kernel parameters (sysctl), loaded modules, ASLR, core dumps        |
| **Authentication**  | PAM configuration, password policies, sudo rules, SSH settings      |
| **File Systems**    | Mount options (noexec, nosuid), world-writable files, SUID binaries |
| **Networking**      | Open ports, firewall status, TCP wrappers, IPv6 configuration       |
| **Logging**         | Syslog daemon, log rotation, audit daemon (auditd) status           |
| **Software**        | Package manager integrity, outdated packages, compiler availability |
| **Malware**         | Rootkit indicators, suspicious files, integrity tool presence       |

***

## Run a Security Audit

<Tabs>
  <Tab title="Single Host" icon="terminal">
    <Steps>
      <Step title="Run the full system audit" icon="play">
        ```bash title="Run Lynis audit" theme={null}
        lynis audit system
        ```

        Lynis runs all tests interactively and prints results to stdout. The full report is saved to `/var/log/lynis.log` and the report data to `/var/log/lynis-report.dat`.
      </Step>

      <Step title="Review the hardening index" icon="bar-chart">
        At the end of the scan output, Lynis shows:

        ```
        Hardening index : 72 [##############      ]
        Tests performed : 238
        Plugins enabled : 2
        ```

        Scores above 80 indicate a well-hardened system. Scores below 60 indicate significant gaps.
      </Step>

      <Step title="Review warnings and suggestions" icon="circle-x">
        ```bash title="Filter warnings from the log" theme={null}
        grep "^\\[WARNING\\]" /var/log/lynis.log
        ```

        ```bash title="Filter suggestions" theme={null}
        grep "^\\[SUGGESTION\\]" /var/log/lynis.log
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Non-Interactive (CI/CD)" icon="settings">
    Run Lynis in non-interactive mode for automated pipelines:

    ```bash title="Non-interactive audit with exit code" theme={null}
    lynis audit system --non-interactive --quiet --logfile /tmp/lynis.log
    echo "Exit code: $?"
    ```

    Exit codes:

    | Code     | Meaning                         |
    | -------- | ------------------------------- |
    | `0`      | Audit completed successfully    |
    | `1`      | Audit aborted                   |
    | `64–127` | Warnings found (non-zero count) |

    ```bash title="Extract hardening score from report" theme={null}
    grep "hardening_index" /var/log/lynis-report.dat | cut -d= -f2
    ```
  </Tab>

  <Tab title="Fleet Audit (Ansible)" icon="layers">
    Run Lynis across all instances in a project:

    ```yaml title="ansible/playbooks/lynis-audit.yml" theme={null}
    ---
    - name: Run Lynis security audit
      hosts: all
      become: true
      tasks:
        - name: Install Lynis
          apt:
            name: lynis
            state: present
            update_cache: true

        - name: Run Lynis audit
          command: lynis audit system --non-interactive --quiet
          register: lynis_result
          changed_when: false

        - name: Fetch hardening index
          command: grep "hardening_index" /var/log/lynis-report.dat
          register: hardening_score
          changed_when: false

        - name: Display score
          debug:
            msg: "{{ inventory_hostname }}: {{ hardening_score.stdout }}"

        - name: Fetch report
          fetch:
            src: /var/log/lynis-report.dat
            dest: "reports/{{ inventory_hostname }}-lynis.dat"
            flat: true
    ```

    ```bash title="Run the fleet audit" theme={null}
    xavs-ansible run --playbook lynis-audit.yml --limit web-tier
    ```
  </Tab>
</Tabs>

***

## Common Findings and Fixes

<AccordionGroup>
  <Accordion title="SSH configuration weaknesses" icon="terminal">
    **Finding**: Lynis warns that root login is permitted or password authentication is enabled.

    ```bash title="Harden SSH" theme={null}
    cat >> /etc/ssh/sshd_config << 'EOF'
    PermitRootLogin no
    PasswordAuthentication no
    MaxAuthTries 3
    X11Forwarding no
    AllowTcpForwarding no
    EOF
    systemctl restart sshd
    ```
  </Accordion>

  <Accordion title="Kernel parameters not hardened" icon="cpu">
    **Finding**: ASLR disabled, IP forwarding enabled unnecessarily, or core dumps allowed.

    ```bash title="Apply kernel hardening via sysctl" theme={null}
    cat >> /etc/sysctl.d/99-xloud-hardening.conf << 'EOF'
    kernel.randomize_va_space = 2
    kernel.dmesg_restrict = 1
    kernel.kptr_restrict = 2
    fs.suid_dumpable = 0
    net.ipv4.conf.all.log_martians = 1
    net.ipv4.conf.all.rp_filter = 1
    net.ipv4.tcp_syncookies = 1
    EOF
    sysctl --system
    ```
  </Accordion>

  <Accordion title="Audit daemon not running" icon="clipboard-list">
    **Finding**: `auditd` not installed or not running — system activity is not being logged.

    ```bash title="Install and enable auditd" theme={null}
    apt install auditd audispd-plugins -y
    systemctl enable --now auditd
    ```

    ```bash title="Add basic audit rules" theme={null}
    cat >> /etc/audit/rules.d/xloud.rules << 'EOF'
    -w /etc/passwd -p wa -k identity
    -w /etc/shadow -p wa -k identity
    -w /etc/sudoers -p wa -k sudo
    -w /var/log/auth.log -p wa -k auth
    -a always,exit -F arch=b64 -S execve -k exec
    EOF
    augenrules --load
    ```
  </Accordion>

  <Accordion title="World-writable files detected" icon="file-x">
    **Finding**: Files or directories are world-writable, creating privilege escalation risk.

    ```bash title="Find world-writable files" theme={null}
    find / -xdev -type f -perm -002 -not -path "/proc/*" 2>/dev/null

    # Remove world-write permission
    chmod o-w <FILE_PATH>
    ```
  </Accordion>

  <Accordion title="Compiler available on production host" icon="code">
    **Finding**: Build tools (`gcc`, `cc`) present on a production node — unnecessary attack surface.

    ```bash title="Remove compilers from production hosts" theme={null}
    apt remove --purge gcc g++ make build-essential -y
    apt autoremove -y
    ```
  </Accordion>
</AccordionGroup>

***

## Hardening Index Targets

| Environment                         | Target Score | Notes                              |
| ----------------------------------- | ------------ | ---------------------------------- |
| Development VMs                     | 60+          | Baseline — essential controls only |
| Staging / Test                      | 70+          | Near-production hardening          |
| Production workloads                | 80+          | Full hardening applied             |
| Regulated environments (PCI, HIPAA) | 85+          | Compliance-grade hardening         |

<Tip>
  Run Lynis immediately after provisioning a new node and again after applying the hardening guide. Use the score delta to confirm controls are applied correctly.
</Tip>

***

## Scheduled Audits

Run Lynis on a schedule to detect configuration drift:

```bash title="/etc/cron.weekly/lynis-audit" theme={null}
#!/bin/bash
lynis audit system --non-interactive --quiet \
  --logfile /var/log/lynis-$(date +%Y%m%d).log \
  --report-file /var/log/lynis-report-$(date +%Y%m%d).dat
```

```bash theme={null}
chmod +x /etc/cron.weekly/lynis-audit
```

***

## Next Steps

<CardGroup>
  <Card title="Xloud SIEM Overview" href="/security/xloud-siem">
    Back to the unified Xloud SIEM hub — Security Posture and Alerts dashboards
  </Card>

  <Card title="Wazuh HIDS" href="/security/wazuh">
    Add real-time intrusion detection and file integrity monitoring on top of periodic Lynis audits
  </Card>

  <Card title="OpenSCAP" href="/security/openscap">
    Perform SCAP-based compliance scans against CIS, STIG, and PCI-DSS profiles
  </Card>

  <Card title="Compliance" href="/security/compliance">
    Map Lynis findings to SOC 2, ISO 27001, and HIPAA compliance requirements
  </Card>
</CardGroup>


# Network Security
Source: https://docs.xloud.tech/security/network-security

Enforce network security with stateful security groups, FWaaS, port security, anti-spoofing, VLAN/VXLAN segmentation, and VPN as a Service on Xloud.

## Overview

Xloud Networking enforces security at the network plane without requiring agents inside your virtual machines. Stateful security groups operate at the virtual switch level, and FWaaS policies apply at the virtual router. Port security prevents spoofing attacks, and VLAN/VXLAN segmentation isolates tenant traffic at the data plane. This agentless model provides consistent enforcement regardless of guest OS configuration.

<Note>
  **Prerequisites**

  * An active Xloud project with at least one network and subnet
  * `member` or `admin` role in Xloud Identity
  * For FWaaS: XPCI license with `enable_neutron_fwaas: "yes"` in XDeploy configuration
  * For VPNaaS: `enable_neutron_vpnaas: "yes"` in XDeploy configuration
</Note>

***

## Network Security Architecture

```mermaid theme={null}
graph TD
    Internet -->|External traffic| Router[Virtual Router]
    Router -->|FWaaS L3 policy| FW[Firewall Rules]
    FW -->|Allow/Deny| Network[Tenant Network]
    Network -->|Security Groups L4| Port1[VM Port A]
    Network -->|Security Groups L4| Port2[VM Port B]
    Port1 -->|Port Security Anti-spoofing| VM1[Virtual Machine 1]
    Port2 -->|Port Security Anti-spoofing| VM2[Virtual Machine 2]
```

| Layer            | Control                 | Enforcement Point   |
| ---------------- | ----------------------- | ------------------- |
| L3 routing       | Firewall as a Service   | Virtual router      |
| L3/L4 filtering  | Security groups         | Virtual switch port |
| L2 anti-spoofing | Port security           | Virtual switch port |
| L2 isolation     | VLAN/VXLAN segmentation | Overlay network     |
| L3 remote access | VPN as a Service        | Virtual router      |

***

## Security Groups

Security groups implement stateful L3/L4 packet filtering. Each rule specifies a direction (ingress/egress), protocol, port range, and a source/destination specifier (CIDR or another security group). The firewall automatically permits return traffic for allowed sessions.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open Security Groups" icon="shield">
        Navigate to **Project → Network → Security Groups**. The `default` security group allows all outbound traffic and permits inbound traffic only from other members of the same group.
      </Step>

      <Step title="Create a purpose-specific group" icon="plus">
        Click **Create Security Group**. Give it a meaningful name such as `database-servers` or `load-balancer-frontend`.
      </Step>

      <Step title="Add rules" icon="list">
        Click **Manage Rules** → **Add Rule**. Define each rule:

        | Field     | Description                                                                          |
        | --------- | ------------------------------------------------------------------------------------ |
        | Rule      | Protocol preset (SSH, HTTP, HTTPS) or Custom                                         |
        | Direction | Ingress (inbound) or Egress (outbound)                                               |
        | Open Port | Port or port range                                                                   |
        | Remote    | CIDR for IP-based restriction, or another Security Group for group-based restriction |

        <Tip>Use security group references as the Remote source instead of CIDR ranges when possible. This allows membership-based access that scales automatically as instances are added to the referenced group.</Tip>
      </Step>

      <Step title="Apply to instance" icon="link">
        In **Project → Compute → Instances**, open the instance menu and select **Edit Security Groups**. Assign the new group and remove overly permissive groups.

        <Check>Test connectivity from an allowed host and confirm blocked traffic is dropped silently.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a tiered security group" theme={null}
    # Create the group
    openstack security group create database-servers \
      --description "MySQL access from app tier only"

    # Allow MySQL from app-servers group only (no CIDR needed)
    openstack security group rule create database-servers \
      --protocol tcp \
      --dst-port 3306 \
      --remote-group app-servers \
      --ingress

    # Allow monitoring from management CIDR
    openstack security group rule create database-servers \
      --protocol tcp \
      --dst-port 9104 \
      --remote-ip 10.10.0.0/24 \
      --ingress

    # Deny all other inbound (implicit — no additional rule needed)
    ```

    ```bash title="List all rules in a group" theme={null}
    openstack security group rule list database-servers --long
    ```

    ```bash title="Apply to a running instance" theme={null}
    openstack server add security group my-db-server database-servers
    openstack server remove security group my-db-server default
    ```
  </Tab>
</Tabs>

***

## Firewall as a Service (FWaaS)

<Badge>Enterprise</Badge>

FWaaS applies stateless or stateful L3 firewall policies at the virtual router. Unlike security groups (which operate per port), FWaaS policies apply to all traffic traversing a router. This makes them suitable for north-south perimeter control and micro-segmentation between subnets.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create firewall rules" icon="list">
        Navigate to **Project → Network → Firewalls → Rules** and click **Add Rule**:

        | Field            | Example        |
        | ---------------- | -------------- |
        | Name             | `block-telnet` |
        | Protocol         | TCP            |
        | Destination Port | 23             |
        | Action           | Deny           |
        | Enabled          | Yes            |
      </Step>

      <Step title="Create a firewall policy" icon="clipboard-list">
        Navigate to **Firewall Policies → Create Policy**. Add the rules in priority order (first match wins).
      </Step>

      <Step title="Create a firewall group" icon="shield">
        Navigate to **Firewall Groups → Create Group**. Associate the ingress and egress policies, then attach the firewall group to one or more router ports.

        <Check>Test blocked traffic — packets matching deny rules are dropped at the router without reaching the destination VM's security group layer.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create firewall rules" theme={null}
    openstack firewall group rule create \
      --name block-telnet \
      --protocol tcp \
      --destination-port 23 \
      --action deny \
      --enabled

    openstack firewall group rule create \
      --name allow-https \
      --protocol tcp \
      --destination-port 443 \
      --action allow \
      --enabled
    ```

    ```bash title="Create policy and group" theme={null}
    openstack firewall group policy create perimeter-ingress \
      --firewall-rule block-telnet \
      --firewall-rule allow-https

    openstack firewall group create \
      --name perimeter-fw \
      --ingress-firewall-policy perimeter-ingress \
      --port <router-port-id>
    ```
  </Tab>
</Tabs>

***

## Port Security and Anti-Spoofing

Port security prevents virtual machines from injecting packets with spoofed source MAC or IP addresses. This blocks ARP poisoning (spoofing MAC-to-IP mappings), DHCP starvation (exhausting IP address leases), and IP spoofing attacks.

Port security is enabled by default on all ports. To verify:

```bash title="Check port security status" theme={null}
openstack port show <port-id> --column port_security_enabled --format value
# Returns: True
```

### Allowed Address Pairs

For use cases requiring secondary IPs (virtual IPs, HAProxy floating IPs, Keepalived), add allowed address pairs:

```bash title="Add allowed address pair for VIP" theme={null}
openstack port set <port-id> \
  --allowed-address ip-address=192.168.10.100,mac-address=fa:16:3e:xx:xx:xx
```

<Warning>
  Disabling port security (`openstack port set --no-port-security-enabled`) removes all anti-spoofing controls and bypasses security group enforcement on that port. Only disable port security when explicitly required (e.g., NFV workloads managing their own forwarding).
</Warning>

***

## Network Segmentation (VLAN and VXLAN)

Tenant networks are isolated at the data plane using VLAN tags (provider networks) or VXLAN encapsulation (overlay networks). Each tenant network has a unique segmentation ID that prevents cross-tenant traffic even on shared physical infrastructure.

| Network Type | Segmentation                                              | Isolation Scope                             |
| ------------ | --------------------------------------------------------- | ------------------------------------------- |
| VLAN         | 802.1Q tag (1–4094)                                       | Hardware-enforced on physical switches      |
| VXLAN        | 24-bit VXLAN Network Identifier (VNI), up to 16M segments | Software-defined, scales across hypervisors |
| GRE          | Tunnel key                                                | Point-to-point overlay                      |

```bash title="Create an isolated tenant network" theme={null}
openstack network create \
  --provider-network-type vxlan \
  --description "Isolated production network" \
  prod-internal-net

openstack subnet create \
  --network prod-internal-net \
  --subnet-range 172.16.10.0/24 \
  --dns-nameserver 8.8.8.8 \
  --no-dhcp-dns \
  prod-internal-subnet
```

***

## DDoS Protection

Xloud Networking provides rate limiting at the port and router levels to mitigate volumetric attacks:

```bash title="Apply QoS bandwidth limit to a port" theme={null}
openstack qos policy create ddos-protection

openstack qos rule create \
  --type bandwidth-limit \
  --max-kbps 100000 \
  --max-burst-kbps 200000 \
  ddos-protection

openstack port set <port-id> --qos-policy ddos-protection
```

For infrastructure-level DDoS protection, use the XIMP monitoring integration to detect anomalous traffic patterns and trigger automated response playbooks.

***

## VPN as a Service

<Badge>Enterprise</Badge>

VPN as a Service extends tenant networks to remote sites over IPsec tunnels without exposing public floating IPs.

```bash title="Create an IPsec VPN connection" theme={null}
# Create IKE policy
openstack vpn ikepolicy create \
  --ike-version v2 \
  --encryption-algorithm aes-256 \
  --auth-algorithm sha-256 \
  --pfs group14 \
  ike-aes256-sha256

# Create IPsec policy
openstack vpn ipsecpolicy create \
  --transform-protocol esp \
  --encryption-algorithm aes-256 \
  --auth-algorithm sha-256 \
  --pfs group14 \
  ipsec-aes256-sha256

# Create VPN service on the router
openstack vpn service create \
  --router <router-id> \
  --subnet <local-subnet-id> \
  prod-vpn-service

# Create site connection
openstack vpn ipsec site connection create \
  --vpnservice prod-vpn-service \
  --ikepolicy ike-aes256-sha256 \
  --ipsecpolicy ipsec-aes256-sha256 \
  --peer-address <remote-gateway-ip> \
  --peer-cidr <remote-subnet-cidr> \
  --psk "<pre-shared-key>" \
  site-to-hq
```

***

## Next Steps

<CardGroup>
  <Card title="VM Security" href="/security/vm-security">
    Hypervisor isolation, vTPM, and anti-affinity workload placement
  </Card>

  <Card title="Infrastructure Security" href="/security/infrastructure">
    TLS configuration and endpoint hardening
  </Card>

  <Card title="Networking Service" href="/services/networking/index">
    Complete networking service documentation with user and admin guides
  </Card>

  <Card title="Hardening Guide" href="/security/hardening-guide">
    Pre-deployment hardening checklist for compute and network nodes
  </Card>
</CardGroup>


# OpenSCAP
Source: https://docs.xloud.tech/security/openscap

Scan Xloud virtual machines and host nodes against CIS Benchmarks, DISA STIGs, and PCI-DSS profiles using OpenSCAP and the SCAP Security Guide.

## Overview

OpenSCAP is the open standard for automated security compliance scanning. It evaluates your system against machine-readable SCAP (Security Content Automation Protocol) content — including CIS Benchmarks, DISA STIGs, PCI-DSS, HIPAA, and ANSSI profiles. Each scan produces detailed HTML and XML reports that map every test to a specific compliance requirement.

Xloud Platform ships OpenSCAP tooling on XOS and supports fleet-wide compliance scanning via the XDeploy automation pipeline. You can forward scan reports to SIEM systems or store them as audit artifacts for regulatory reviews.

<Info>**Xloud-Developed** — OpenSCAP is one of three independent scanners in [Xloud SIEM](/security/xloud-siem) — Wazuh, Lynis, and OpenSCAP run in parallel across all nodes for layered compliance coverage. Results are aggregated on the **Security Posture** page in Monitor Center.</Info>

<Note>
  **Prerequisites**

  * `openscap-scanner` and `scap-security-guide` packages installed (pre-installed on XOS nodes)
  * Guest VMs: `apt install openscap-scanner ssg-debderived` on Ubuntu/Debian
  * Root access on the target system
  * Target profile selected from the SCAP Security Guide (SSG)
</Note>

***

## Available Profiles

The SCAP Security Guide ships dozens of profiles for common compliance frameworks. Key profiles for Xloud environments:

| Profile ID                                               | Framework         | Target        |
| -------------------------------------------------------- | ----------------- | ------------- |
| `xccdf_org.ssgproject.content_profile_cis_level1_server` | CIS Level 1       | Ubuntu Server |
| `xccdf_org.ssgproject.content_profile_cis_level2_server` | CIS Level 2       | Ubuntu Server |
| `xccdf_org.ssgproject.content_profile_pci-dss`           | PCI-DSS v3.2.1    | Ubuntu Server |
| `xccdf_org.ssgproject.content_profile_hipaa`             | HIPAA             | Ubuntu Server |
| `xccdf_org.ssgproject.content_profile_anssi_bp28_high`   | ANSSI BP-028 HIGH | Ubuntu Server |
| `xccdf_org.ssgproject.content_profile_stig`              | DISA STIG         | RHEL-based    |

```bash title="List all available profiles for your OS" theme={null}
oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml | grep "Profile:"
```

***

## Run a Compliance Scan

<Tabs>
  <Tab title="Single Host" icon="terminal">
    <Steps>
      <Step title="Identify the SCAP content file" icon="file">
        ```bash title="Locate SSG content for Ubuntu 22.04" theme={null}
        ls /usr/share/xml/scap/ssg/content/ | grep ubuntu22
        # Output: ssg-ubuntu2204-ds.xml
        ```
      </Step>

      <Step title="Run the scan against a profile" icon="play">
        ```bash title="Scan against CIS Level 1 Server profile" theme={null}
        oscap xccdf eval \
          --profile xccdf_org.ssgproject.content_profile_cis_level1_server \
          --results /tmp/results-cis-l1.xml \
          --report /tmp/report-cis-l1.html \
          /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml
        ```

        The scan evaluates each rule and produces:

        * `results-cis-l1.xml` — machine-readable XCCDF results
        * `report-cis-l1.html` — human-readable HTML report
      </Step>

      <Step title="Review the HTML report" icon="eye">
        Copy the report to a location accessible from a browser:

        ```bash title="Copy report to web-accessible path" theme={null}
        cp /tmp/report-cis-l1.html /var/www/html/scap-report.html
        ```

        The report shows each rule with a **pass**, **fail**, or **not applicable** result, linked to the compliance requirement ID and remediation guidance.

        <Check>Check the score at the top of the report. A score above 80% indicates strong compliance posture for that profile.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="With Remediation" icon="wrench">
    OpenSCAP can generate an Ansible remediation playbook for all failing rules:

    <Steps>
      <Step title="Generate remediation playbook" icon="layers">
        ```bash title="Generate Ansible remediation from scan results" theme={null}
        oscap xccdf generate fix \
          --fix-type ansible \
          --output /tmp/remediation-cis-l1.yml \
          --result-id "" \
          /tmp/results-cis-l1.xml
        ```
      </Step>

      <Step title="Review and apply the playbook" icon="play">
        ```bash title="Apply remediations" theme={null}
        ansible-playbook /tmp/remediation-cis-l1.yml \
          -i localhost, --connection local \
          --become
        ```

        <Warning>
          Review the generated playbook before applying. Some remediations (e.g., disabling USB storage or changing kernel parameters) may affect running services. Apply during a maintenance window.
        </Warning>
      </Step>

      <Step title="Re-scan to verify" icon="refresh-cw">
        ```bash title="Re-scan after remediation" theme={null}
        oscap xccdf eval \
          --profile xccdf_org.ssgproject.content_profile_cis_level1_server \
          --results /tmp/results-cis-l1-post.xml \
          --report /tmp/report-cis-l1-post.html \
          /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml
        ```

        <Check>Score should improve. Compare with the pre-remediation report to confirm fixes applied successfully.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Fleet Scan (Ansible)" icon="layers">
    Run OpenSCAP across all instances and collect reports centrally:

    ```yaml title="ansible/playbooks/openscap-scan.yml" theme={null}
    ---
    - name: OpenSCAP compliance scan
      hosts: all
      become: true
      vars:
        scap_profile: xccdf_org.ssgproject.content_profile_cis_level1_server
        scap_content: /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml
        report_dir: /var/log/scap

      tasks:
        - name: Install OpenSCAP and SSG
          apt:
            name:
              - openscap-scanner
              - ssg-debderived
            state: present
            update_cache: true

        - name: Create report directory
          file:
            path: "{{ report_dir }}"
            state: directory
            mode: "0750"

        - name: Run SCAP scan
          command: >
            oscap xccdf eval
            --profile {{ scap_profile }}
            --results {{ report_dir }}/results-{{ inventory_hostname }}.xml
            --report {{ report_dir }}/report-{{ inventory_hostname }}.html
            {{ scap_content }}
          register: scap_result
          failed_when: scap_result.rc > 2
          changed_when: false

        - name: Fetch results
          fetch:
            src: "{{ report_dir }}/results-{{ inventory_hostname }}.xml"
            dest: "scap-results/{{ inventory_hostname }}-results.xml"
            flat: true
    ```

    ```bash title="Run the fleet scan" theme={null}
    xavs-ansible run --playbook openscap-scan.yml
    ```
  </Tab>
</Tabs>

***

## Interpreting Results

Each rule in the HTML report maps to a specific compliance control:

| Result             | Meaning                            | Action                         |
| ------------------ | ---------------------------------- | ------------------------------ |
| **Pass**           | System meets the requirement       | No action needed               |
| **Fail**           | Requirement not met                | Apply remediation              |
| **Not Applicable** | Rule does not apply to this system | Document exemption             |
| **Not Checked**    | Rule requires manual verification  | Perform manual check           |
| **Error**          | Scan could not evaluate the rule   | Check for missing dependencies |

### Score Interpretation

| Score Range | Compliance Posture                              |
| ----------- | ----------------------------------------------- |
| 90–100%     | Excellent — minimal gaps                        |
| 80–89%      | Good — a few controls need attention            |
| 70–79%      | Moderate — hardening required before production |
| Below 70%   | Poor — significant remediation needed           |

***

## Scheduled Scanning

Run scans on a weekly schedule and archive results:

```bash title="/etc/cron.weekly/openscap-scan" theme={null}
#!/bin/bash
DATE=$(date +%Y%m%d)
REPORT_DIR="/var/log/scap"
PROFILE="xccdf_org.ssgproject.content_profile_cis_level1_server"
CONTENT="/usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml"

mkdir -p "$REPORT_DIR"

oscap xccdf eval \
  --profile "$PROFILE" \
  --results "$REPORT_DIR/results-$DATE.xml" \
  --report "$REPORT_DIR/report-$DATE.html" \
  "$CONTENT"

# Keep 90 days of reports
find "$REPORT_DIR" -name "*.xml" -mtime +90 -delete
find "$REPORT_DIR" -name "*.html" -mtime +90 -delete
```

```bash theme={null}
chmod +x /etc/cron.weekly/openscap-scan
```

***

## Profile Selection Guide

<AccordionGroup>
  <Accordion title="Which profile should I use?" icon="help-circle">
    | Workload Type                | Recommended Profile               |
    | ---------------------------- | --------------------------------- |
    | General production instances | CIS Level 1 Server                |
    | High-security workloads      | CIS Level 2 Server                |
    | Payment card environments    | PCI-DSS                           |
    | Healthcare data              | HIPAA                             |
    | Government / defense         | ANSSI BP-028 HIGH or DISA STIG    |
    | Development and staging      | CIS Level 1 (relaxed enforcement) |

    Start with CIS Level 1 for all new deployments. Escalate to Level 2 or framework-specific profiles for regulated workloads.
  </Accordion>

  <Accordion title="Custom profile development" icon="settings">
    You can create tailored profiles by extending existing SSG content using SCAP Workbench or editing the XCCDF XML directly. Custom profiles allow you to:

    * Disable rules that conflict with your application requirements
    * Add organization-specific controls
    * Override severity levels for risk-accepted findings

    Store custom profiles in `/etc/scap/custom-profiles/` and reference them with `--profile-id` in scan commands.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Xloud SIEM Overview" href="/security/xloud-siem">
    Back to the unified Xloud SIEM hub — Security Posture and Alerts dashboards
  </Card>

  <Card title="Wazuh HIDS" href="/security/wazuh">
    Complement SCAP scans with continuous real-time host intrusion detection
  </Card>

  <Card title="Lynis Auditing" href="/security/lynis">
    Run OS security audits with hardening index scoring
  </Card>

  <Card title="Compliance Frameworks" href="/security/compliance">
    Map SCAP results to SOC 2, ISO 27001, and HIPAA audit requirements
  </Card>
</CardGroup>


# Virtual Machine Security
Source: https://docs.xloud.tech/security/vm-security

Secure virtual machines at the hypervisor level with isolation controls, vTPM, encrypted volumes, Secure Boot, anti-affinity, and live migration TLS.

## Overview

Xloud enforces virtual machine security at the hypervisor layer — below the guest OS — through a combination of isolation controls, hardware security features, and network policy enforcement. This agentless model means security policies apply regardless of what runs inside the VM, making it suitable for your regulated environments, multi-tenant deployments, and zero-trust architectures.

<Note>
  **Prerequisites**

  * Xloud Platform with XAVS or XPCI
  * Administrator or project-member role in Xloud Identity
  * For vTPM: XPCI license with Barbican key management enabled
  * For Secure Boot: UEFI-compatible image with the appropriate properties set
</Note>

***

## Hypervisor Isolation

Each virtual machine runs in a fully isolated compute process. The hypervisor enforces the following isolation primitives:

| Control              | Description                                                                |
| -------------------- | -------------------------------------------------------------------------- |
| Process isolation    | Each VM runs as a separate process owned by a dedicated service account    |
| Namespace separation | Virtual network and storage namespaces are per-tenant                      |
| seccomp filtering    | Restricts system calls available to the hypervisor process                 |
| AppArmor confinement | Mandatory access control profile limits hypervisor file and network access |
| Resource limits      | CPU pinning and memory hard limits prevent noisy-neighbor interference     |

<Info>
  Xloud uses dedicated service users for the compute hypervisor process. These accounts have no login shell and no home directory. The AppArmor profile runs in complain mode by default. You can switch it to enforce mode using the hardening guide.
</Info>

***

## Security Groups

Security groups provide stateful L3/L4 firewalling for virtual machine network interfaces. The hypervisor enforces rules at the virtual switch layer — traffic that does not match an allow rule is silently dropped before reaching the VM.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open Security Groups" icon="shield">
        Navigate to **Project → Network → Security Groups** in the Xloud Dashboard.
      </Step>

      <Step title="Create a new security group" icon="plus">
        Click **Create Security Group**. Provide a name (e.g., `web-servers`) and an optional description.
      </Step>

      <Step title="Add inbound rules" icon="arrow-right">
        Click **Manage Rules** on the new group, then **Add Rule**:

        | Field      | Example Value   | Notes                                  |
        | ---------- | --------------- | -------------------------------------- |
        | Rule       | Custom TCP Rule | Or use presets: SSH, HTTP, HTTPS       |
        | Direction  | Ingress         | Inbound to the VM                      |
        | Port Range | 443             | Single port or range (e.g., 8000-8080) |
        | Remote     | CIDR            | Restrict by IP range                   |
        | CIDR       | 10.0.0.0/8      | Limit to internal network only         |

        <Warning>The default security group permits all outbound traffic and blocks all inbound traffic. Always create explicit inbound rules for required ports.</Warning>
      </Step>

      <Step title="Attach to an instance" icon="link">
        Navigate to **Project → Compute → Instances**, open the instance, and select **Edit Security Groups** from the **Actions** menu. Add the new group and remove the `default` group if not needed.

        <Check>The security group is listed in the instance details. Test connectivity using `nc` or `curl` from an allowed source.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create security group" theme={null}
    openstack security group create web-servers \
      --description "Inbound HTTPS from internal network"
    ```

    ```bash title="Add inbound HTTPS rule" theme={null}
    openstack security group rule create web-servers \
      --protocol tcp \
      --dst-port 443 \
      --remote-ip 10.0.0.0/8 \
      --ingress
    ```

    ```bash title="Add inbound SSH rule (restricted)" theme={null}
    openstack security group rule create web-servers \
      --protocol tcp \
      --dst-port 22 \
      --remote-ip 192.168.100.0/24 \
      --ingress
    ```

    ```bash title="Attach security group to instance" theme={null}
    openstack server add security group my-instance web-servers
    ```
  </Tab>
</Tabs>

***

## vTPM (Virtual Trusted Platform Module)

<Badge>Enterprise</Badge>

Virtual TPM provides a hardware-backed cryptographic identity to virtual machines. Xloud Key Management (Barbican) encrypts vTPM state using a stored key, and the state is portable across live migrations.

### Use Cases

* Full-disk encryption (BitLocker, LUKS with TPM unsealing)
* Measured boot and attestation
* Certificate and secret sealing to the platform

### Enable vTPM on an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create a vTPM-enabled flavor" icon="cpu">
        Navigate to **Admin → Compute → Flavors**. Create or edit a flavor and set the extra specification:

        | Key              | Value     |
        | ---------------- | --------- |
        | `hw:tpm_model`   | `tpm-crb` |
        | `hw:tpm_version` | `2.0`     |
      </Step>

      <Step title="Launch instance with vTPM flavor" icon="play">
        Launch the instance using the vTPM-enabled flavor. The hypervisor provisions a software TPM process backed by the platform key management service.

        <Check>Inside the VM, verify TPM presence: `ls /dev/tpm0` or `tpm2_getcap properties-fixed`.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create vTPM flavor" theme={null}
    openstack flavor create --vcpus 4 --ram 8192 --disk 50 secure-vm-medium
    openstack flavor set secure-vm-medium \
      --property hw:tpm_model=tpm-crb \
      --property hw:tpm_version=2.0
    ```

    ```bash title="Launch instance with vTPM" theme={null}
    openstack server create \
      --flavor secure-vm-medium \
      --image ubuntu-24.04 \
      --network private-net \
      my-tpm-instance
    ```

    ```bash title="Verify vTPM inside VM" theme={null}
    # Run inside the virtual machine
    ls -la /dev/tpm*
    tpm2_getcap properties-fixed | grep TPMVendorID
    ```
  </Tab>
</Tabs>

***

## Encrypted Volumes

You can encrypt block storage volumes using LUKS (Linux Unified Key Setup). Xloud Key Management manages the encryption keys and never stores them on the compute node.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create an encrypted volume type" icon="key">
        Navigate to **Admin → Volume → Volume Types**. Create a new type (e.g., `encrypted-ssd`) and configure the encryption provider:

        | Field            | Value                                       |
        | ---------------- | ------------------------------------------- |
        | Provider         | `nova.volume.encryptors.luks.LuksEncryptor` |
        | Cipher           | `aes-xts-plain64`                           |
        | Key Size         | `256`                                       |
        | Control Location | `front-end`                                 |
      </Step>

      <Step title="Create an encrypted volume" icon="hard-drive">
        Navigate to **Project → Volumes → Create Volume**. Select the encrypted volume type. The system generates and stores the encryption key in Xloud Key Management automatically.

        <Check>The volume details page shows **Encryption** as **Yes**.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create encrypted volume type" theme={null}
    openstack volume type create encrypted-ssd

    openstack volume type set encrypted-ssd \
      --encryption-provider nova.volume.encryptors.luks.LuksEncryptor \
      --encryption-cipher aes-xts-plain64 \
      --encryption-key-size 256 \
      --encryption-control-location front-end
    ```

    ```bash title="Create encrypted volume" theme={null}
    openstack volume create \
      --size 100 \
      --type encrypted-ssd \
      my-encrypted-volume
    ```

    ```bash title="Verify encryption is active" theme={null}
    openstack volume show my-encrypted-volume -c encrypted -f value
    # Returns: True
    ```
  </Tab>
</Tabs>

***

## Secure Boot (UEFI)

Secure Boot prevents unsigned bootloaders and kernels from loading on virtual machines. It is supported for images that include UEFI firmware.

```bash title="Enable Secure Boot on an image" theme={null}
openstack image set --property hw_firmware_type=uefi \
  --property hw_machine_type=q35 \
  --property os_secure_boot=required \
  ubuntu-24.04-secure
```

```bash title="Verify Secure Boot properties" theme={null}
openstack image show ubuntu-24.04-secure \
  -c properties -f json | python3 -m json.tool
```

<Tip>Use `os_secure_boot=optional` to allow instances to boot with or without Secure Boot. Use `required` to enforce it — instances will fail to boot if the image does not support UEFI Secure Boot.</Tip>

***

## Anti-Affinity for Workload Isolation

Server groups with anti-affinity rules distribute your critical workloads across separate physical hosts, preventing a single hypervisor failure from taking down multiple replicas simultaneously.

```bash title="Create anti-affinity server group" theme={null}
openstack server group create \
  --policy anti-affinity \
  production-web-tier
```

```bash title="Launch instances in the group" theme={null}
openstack server create \
  --flavor m1.large \
  --image ubuntu-24.04 \
  --network private-net \
  --hint group=<server-group-id> \
  web-node-01
```

<Warning>
  Anti-affinity placement may fail if there are insufficient compute hosts. Xloud returns a scheduling error rather than placing instances on the same host when the policy cannot be satisfied.
</Warning>

***

## Live Migration Security

Live migrations transmit VM memory state across the network. Xloud encrypts migration streams using TLS to prevent eavesdropping on in-transit memory contents.

```yaml title="/etc/xavs/globals.d/_60_migration_security.yml" theme={null}
nova_live_migration_tunnelled: "yes"
nova_console_allowed_origins: "https://<dashboard-hostname>"
```

<Info>
  When `nova_live_migration_tunnelled` is enabled, migration traffic is routed through the compute service's encrypted channel rather than directly between hypervisors. This adds a small latency overhead but ensures the migration stream is protected end-to-end.
</Info>

***

## Next Steps

<CardGroup>
  <Card title="Network Security" href="/security/network-security">
    Security groups, FWaaS micro-segmentation, and VLAN/VXLAN isolation
  </Card>

  <Card title="Data Security" href="/security/data-security">
    Volume encryption, key management, and encrypted backups
  </Card>

  <Card title="Hardening Guide" href="/security/hardening-guide">
    AppArmor enforcement, SSH hardening, and hypervisor node hardening
  </Card>

  <Card title="Security Groups (Compute)" href="/services/compute/security-groups">
    Detailed security group management for compute instances
  </Card>
</CardGroup>


# Wazuh
Source: https://docs.xloud.tech/security/wazuh

Deploy Wazuh agents across Xloud virtual machines for real-time threat detection, file integrity monitoring, vulnerability assessment, and compliance auditing.

## Overview

Wazuh is the host-based intrusion detection and security monitoring platform bundled with Xloud Platform. It provides host-level visibility into what is happening inside each of your virtual machines through a lightweight agent. The Wazuh manager runs as a centralized service. Agents deploy to each instance and stream security events, file changes, and vulnerability data back for real-time correlation.

Xloud ships this platform pre-integrated with XDeploy, so you can mass-deploy agents across projects using the standard automation pipeline.

<Info>**Xloud-Developed** — This capability is developed by Xloud and ships with XAVS. The integrated security platform is surfaced in the Xloud Dashboard as the **Security Posture** page in Monitor Center, providing a unified view of agent status, threat alerts, and compliance results across the cluster. See [Xloud SIEM](/security/xloud-siem) for the full overview.</Info>

<Tip>
  **XDeploy GUI** — Enable Wazuh (with Lynis auditing, OpenSCAP compliance, and OS hardening) through the [XDeploy Configuration](/deployment/configuration) interface under **XDeploy → Security → HIDS**. No manual file editing required.
</Tip>

<Note>
  **Prerequisites**

  * Wazuh Manager deployed (enabled via XDeploy → Security → HIDS)
  * Network reachability from guest VMs to the Wazuh Manager on ports 1514/1515
  * Agent registration token available from the Wazuh Manager dashboard
</Note>

***

## Architecture

```mermaid theme={null}
graph LR
    A[VM Instance] -->|Syslog / Events| W[Wazuh Agent]
    W -->|TCP 1514| M[Wazuh Manager]
    M -->|Index| E[OpenSearch / Wazuh Dashboard]
    M -->|Alerts| AL[Alertmanager / SIEM]
    style M fill:#197560,color:#fff
    style W fill:#145C4C,color:#fff
```

| Component           | Role                                                                              |
| ------------------- | --------------------------------------------------------------------------------- |
| **Wazuh Agent**     | Collects logs, file events, process activity, and vulnerability data from each VM |
| **Wazuh Manager**   | Correlates events, applies detection rules, triggers alerts                       |
| **Wazuh Dashboard** | OpenSearch-based UI for alert triage, compliance reports, and forensic queries    |
| **Ruleset**         | MITRE ATT\&CK-mapped detection rules — over 3,000 out of the box                  |

***

## Capabilities

<CardGroup>
  <Card title="File Integrity Monitoring" icon="file-scan">
    Track every create, modify, and delete on monitored paths. Alert on unauthorized changes to `/etc/passwd`, SSH keys, cron files, and application configs.
  </Card>

  <Card title="Intrusion Detection" icon="shield-alert">
    Real-time log analysis against MITRE ATT\&CK-mapped rules. Detects brute-force attempts, privilege escalation, rootkits, and lateral movement.
  </Card>

  <Card title="Vulnerability Assessment" icon="bug">
    Continuous scan of installed packages against CVE databases. Reports vulnerable packages per host with severity scores and remediation guidance.
  </Card>

  <Card title="Compliance Auditing" icon="clipboard-list">
    Built-in checks for PCI-DSS, HIPAA, NIST 800-53, CIS benchmarks, and GDPR. Generates per-host compliance reports with pass/fail details.
  </Card>
</CardGroup>

***

## Deploy Wazuh Agent

<Tabs>
  <Tab title="Linux (Ansible)" icon="terminal">
    Use the bundled Ansible role to deploy agents across all instances in a project:

    ```bash title="Deploy Wazuh agents via xavs-ansible" theme={null}
    xavs-ansible deploy --tags wazuh-agent \
      --extra-vars "wazuh_manager_ip=<MANAGER_IP> wazuh_registration_token=<TOKEN>"
    ```

    The role installs the agent, registers it with the manager, and starts the `wazuh-agent` service automatically.

    <Check>Agent appears in the Wazuh Dashboard under **Agents** within 60 seconds of deployment.</Check>
  </Tab>

  <Tab title="Linux (Manual)" icon="terminal">
    <Steps>
      <Step title="Add the Wazuh repository" icon="package">
        ```bash title="Add repository and install agent" theme={null}
        curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --dearmor \
          -o /usr/share/keyrings/wazuh.gpg

        echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] \
          https://packages.wazuh.com/4.x/apt/ stable main" \
          > /etc/apt/sources.list.d/wazuh.list

        apt-get update && apt-get install -y wazuh-agent
        ```
      </Step>

      <Step title="Configure the manager address" icon="settings">
        ```bash title="Set Wazuh Manager IP" theme={null}
        sed -i 's|MANAGER_IP|<YOUR_MANAGER_IP>|g' /var/ossec/etc/ossec.conf
        ```
      </Step>

      <Step title="Register and start the agent" icon="play">
        ```bash title="Register agent and start service" theme={null}
        /var/ossec/bin/agent-auth -m <MANAGER_IP> -p 1515
        systemctl enable --now wazuh-agent
        ```

        <Check>Agent registers and begins streaming events to the manager.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Windows" icon="monitor">
    <Steps>
      <Step title="Download and install the MSI" icon="download">
        Download the Wazuh Windows agent MSI from the Wazuh Manager dashboard under **Agents → Deploy new agent**.
      </Step>

      <Step title="Install with manager address" icon="settings">
        ```powershell title="Silent install with manager configuration" theme={null}
        msiexec /i wazuh-agent.msi /q `
          WAZUH_MANAGER="<MANAGER_IP>" `
          WAZUH_REGISTRATION_SERVER="<MANAGER_IP>" `
          WAZUH_AGENT_NAME="<HOSTNAME>"
        ```
      </Step>

      <Step title="Start the agent service" icon="play">
        ```powershell title="Start Wazuh agent" theme={null}
        NET START WazuhSvc
        ```

        <Check>Agent appears in the Wazuh Dashboard within 60 seconds.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## File Integrity Monitoring Configuration

Configure which paths are monitored for changes in `/var/ossec/etc/ossec.conf`:

```xml title="/var/ossec/etc/ossec.conf — FIM configuration" theme={null}
<syscheck>
  <frequency>3600</frequency>

  <!-- Critical system files -->
  <directories check_all="yes" realtime="yes">/etc/passwd,/etc/shadow,/etc/group</directories>
  <directories check_all="yes" realtime="yes">/etc/ssh</directories>
  <directories check_all="yes" realtime="yes">/root/.ssh</directories>

  <!-- Application configs -->
  <directories check_all="yes">/etc/nginx</directories>
  <directories check_all="yes">/etc/apache2</directories>

  <!-- Ignore transient paths -->
  <ignore>/etc/mtab</ignore>
  <ignore>/etc/hosts.deny</ignore>
</syscheck>
```

| Option            | Description                                                         |
| ----------------- | ------------------------------------------------------------------- |
| `realtime="yes"`  | Alert immediately on change (inotify-based), not at next scan cycle |
| `check_all="yes"` | Monitor permissions, ownership, size, MD5, SHA1, SHA256, and mtime  |
| `frequency`       | Scan interval in seconds for non-realtime paths                     |

***

## Vulnerability Assessment

Wazuh continuously scans installed packages against NVD and vendor CVE feeds. Results appear in the Dashboard under **Vulnerability Detector**.

```bash title="Trigger an on-demand vulnerability scan" theme={null}
/var/ossec/bin/wazuh-control restart
```

| Severity | CVSS Score Range | Action                                         |
| -------- | ---------------- | ---------------------------------------------- |
| Critical | 9.0–10.0         | Immediate patching required                    |
| High     | 7.0–8.9          | Patch within 7 days                            |
| Medium   | 4.0–6.9          | Patch within 30 days                           |
| Low      | 0.1–3.9          | Track and remediate at next maintenance window |

***

## Compliance Reports

Wazuh ships with built-in compliance checks. Enable a framework in `ossec.conf`:

```xml title="Enable PCI-DSS compliance checks" theme={null}
<rootcheck>
  <system_audit>/var/ossec/etc/shared/pci_dss_reqs.txt</system_audit>
</rootcheck>
```

Available compliance frameworks:

| Framework     | File                                                  |
| ------------- | ----------------------------------------------------- |
| PCI-DSS 3.2.1 | `pci_dss_reqs.txt`                                    |
| HIPAA         | `hipaa_reqs.txt`                                      |
| NIST 800-53   | `nist800_53_reqs.txt`                                 |
| GDPR          | `gdpr_reqs.txt`                                       |
| CIS Benchmark | `cis_debian_linux_rcl.txt` / `cis_rhel_linux_rcl.txt` |

Reports are accessible in the Wazuh Dashboard under **Regulatory Compliance**.

***

## Alert Integration

Forward Wazuh alerts to external systems:

<Tabs>
  <Tab title="Slack / Webhook" icon="bell">
    ```xml title="/var/ossec/etc/ossec.conf — webhook integration" theme={null}
    <integration>
      <name>slack</name>
      <hook_url>https://hooks.slack.com/services/YOUR/WEBHOOK/URL</hook_url>
      <level>10</level>
      <alert_format>json</alert_format>
    </integration>
    ```

    Alerts at level 10 and above (high severity) are forwarded automatically.
  </Tab>

  <Tab title="Syslog / SIEM" icon="server">
    ```xml title="/var/ossec/etc/ossec.conf — syslog forwarding" theme={null}
    <syslog_output>
      <level>9</level>
      <server>10.0.1.100</server>
      <port>514</port>
      <format>default</format>
    </syslog_output>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Xloud SIEM Overview" href="/security/xloud-siem">
    Back to the unified Xloud SIEM hub — Security Posture and Alerts dashboards
  </Card>

  <Card title="Lynis Security Auditing" href="/security/lynis">
    Run automated OS security audits and generate hardening recommendations
  </Card>

  <Card title="OpenSCAP Compliance Scanning" href="/security/openscap">
    Scan instances against CIS, STIG, and PCI-DSS profiles using SCAP content
  </Card>

  <Card title="Compliance and Auditing" href="/security/compliance">
    Understand audit logging and compliance frameworks supported by Xloud
  </Card>
</CardGroup>


# Xloud SIEM
Source: https://docs.xloud.tech/security/xloud-siem

Unified security operations on Xloud Platform — Wazuh, Lynis, and OpenSCAP in a single Security Posture view with live Alerts in Monitor Center.

<Info>**Xloud-Developed** — Xloud SIEM is the integrated Security Information and Event Management layer built into the Xloud Platform. It stitches together Wazuh, Lynis, and OpenSCAP into a single dashboard surface and correlates findings against your actual cluster inventory.</Info>

## What is Xloud SIEM?

Xloud SIEM is the integrated security operations layer on the Xloud Platform. It runs
three independent scanners in parallel — **Wazuh** for host intrusion detection,
**Lynis** for OS-level auditing, and **OpenSCAP** for CIS / STIG compliance — and surfaces
the combined results in two Dashboard views:

* **Security Posture** — a single pane for agent inventory, vulnerabilities, alerts,
  compliance scores, encryption status, and microsegmentation.
* **Alerts** — active security and infrastructure alerts with rules, history, and
  silences.

<CardGroup>
  <Card title="Wazuh (HIDS)" icon="shield-alert" href="/security/wazuh">
    Host intrusion detection, file integrity, vulnerability assessment, and rule-based
    threat correlation across every VM.
  </Card>

  <Card title="Lynis" icon="list-checks" href="/security/lynis">
    300+ on-host security audits with a hardening index score per node and prioritized
    remediation guidance.
  </Card>

  <Card title="OpenSCAP" icon="clipboard-check" href="/security/openscap">
    SCAP-based compliance scanning — CIS Benchmarks, DISA STIGs, PCI-DSS, HIPAA, ANSSI
    profiles with pass/fail reports.
  </Card>
</CardGroup>

<Note>
  **Prerequisites** — Xloud SIEM requires Wazuh to be enabled on the cluster
  (XDeploy → Security → HIDS). When Wazuh is disabled, **Monitor Center → Security Posture**
  shows an empty state asking you to enable it.
</Note>

***

## Video Walkthrough

<iframe title="How to Review Security Posture on Xloud" />

***

## The Two Dashboard Views

Everything Xloud SIEM exposes in the Dashboard lives in **Monitor Center** (admin view only):

| Page                 | What it shows                                                                                                                                             |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Security Posture** | Agent fleet, live alerts, CIS compliance %, vulnerability CVEs, volume encryption status, security-group risk scoring, and cluster health — all in 8 tabs |
| **Alerts**           | Active Alerts, Alert Rules, History, and Silences — unified for both infrastructure (Prometheus) and security (Wazuh) sources                             |

<Warning>
  Both pages are in the **administrator view only**. Open the Dashboard as an admin and
  expand **Monitor Center** to reach them.
</Warning>

***

## Security Posture — 8 Tabs

The Security Posture page is the headline view for Xloud SIEM. Every tab aggregates data
across the cluster and links back to raw Wazuh, Lynis, or OpenSCAP output.

<AccordionGroup>
  <Accordion title="1. Overview" icon="gauge">
    Four top cards show **Active Agents** (active/total), **Cluster Nodes** with manager
    version, **CIS Compliance** as a single %, and **Manager** type. Below that: per-agent
    SCA chart, multi-layer compliance bar chart (Lynis vs OpenSCAP vs Wazuh SCA), and two
    stacked progress views for Lynis Hardening Index and OpenSCAP CIS Score per node.
  </Accordion>

  <Accordion title="2. Agents" icon="users">
    Table of every deployed Wazuh agent with ID, Name, IP, Status (active, disconnected,
    pending, never connected), OS, Version, SCA score %, Groups, and Last Seen timestamp.
    Filterable by Name and Status.
  </Accordion>

  <Accordion title="3. Alerts" icon="bell">
    Live stream of Wazuh alerts filtered by time window (1h / 6h / 24h / 3d / 7d) and
    minimum severity (All 3+ / Medium 5+ / High 8+ / Critical 12+). Columns: Time,
    Severity tag, Rule ID, Description, Agent, Source IP.
  </Accordion>

  <Accordion title="4. Compliance" icon="clipboard-check">
    Per-node matrix combining **Lynis Score** (out of 100), Warnings, Suggestions,
    **OpenSCAP** %, pass/fail counts, and **Wazuh SCA** %. Click **Why?** on any row to
    open a modal listing the exact Lynis findings — warnings and suggestions with test
    IDs and remediation text.
  </Accordion>

  <Accordion title="5. Encryption" icon="lock">
    Lists every instance with encryption status — Encrypted (all volumes), Unencrypted,
    or No volumes. Shows each attached volume with its encryption flag and size, so you
    can spot mixed-state VMs immediately.
  </Accordion>

  <Accordion title="6. Microsegmentation" icon="network">
    Four sub-tabs for the micro-segmentation view:

    * **Security Groups** — every project security group with rule count, ingress / egress
      counts, a risk score (0-100) with color bar, risk level tag, Wide-Open flag, and
      suggested fixes.
    * **Flow Map** — allowed VM-to-VM flows listing Source VM, Destination VM, Protocol,
      Ports, and Via Security Group.
    * **VM Mapping** — each instance with its Status, Host, IPs, attached Security Groups,
      and Tags.
    * **Tag Groups** — resources clustered by tag, showing Resource Type, Tag, Count, and
      up to 5 example resources per group.
  </Accordion>

  <Accordion title="7. Vulnerabilities" icon="bug">
    Six counter cards (Total CVEs, Critical, High, Medium, Low, Solved) plus a severity
    donut and a "Vulnerabilities by Agent" bar chart. Searchable CVE table with columns:
    CVE, Severity, Package, Version, Agent, Status, Description.
  </Accordion>

  <Accordion title="8. Cluster" icon="server">
    Wazuh cluster topology — Cluster Status, Manager version, Cluster Nodes (master vs
    worker count), and Total Agents. Two donut charts break down node types and agent
    status. Bottom table lists each cluster node with Name, Type, Version, IP, and Status.
  </Accordion>
</AccordionGroup>

<Tip>
  The top-right **Export Report** button downloads a CSV snapshot of the entire Security
  Posture view. The **Wazuh Dashboard** button opens the native Wazuh UI in a new tab for
  deeper investigation.
</Tip>

***

## Alerts — Unified Security and Infrastructure View

The **Alerts** page (**Monitor Center → Alerts**) consolidates every alert into one
interface, regardless of source:

| Tab               | Purpose                                                                                                                                                           |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Active Alerts** | Currently firing alerts with a **Security** tag (Wazuh-sourced) or **Infrastructure** tag (Prometheus-sourced), severity (critical / warning / info), and context |
| **Alert Rules**   | View and edit the rules driving each alert — thresholds, evaluation intervals, notification channels                                                              |
| **History**       | Historical alert timeline for audit, incident reviews, and trend analysis                                                                                         |
| **Silences**      | Active silences that suppress noisy alerts during maintenance windows                                                                                             |

<Info>
  Wazuh-detected threats and infrastructure metric alerts (Prometheus) land in the same
  table, so operators get a single place to triage events — no tool switching.
</Info>

***

## For Users

Most Xloud users interact with Xloud SIEM indirectly — their VMs are scanned
automatically by the platform's security suite. If you are a non-admin user:

<CardGroup>
  <Card title="See what's scanned on your VMs" icon="eye">
    Ask your administrator for a Security Posture export. Every VM's compliance score,
    encryption status, and live alerts are captured in the report.
  </Card>

  <Card title="Run a manual audit" icon="play">
    If you need an ad-hoc Lynis or OpenSCAP run inside a VM you own, see the
    [Lynis](/security/lynis) and [OpenSCAP](/security/openscap) user guides for
    self-service commands.
  </Card>
</CardGroup>

***

## For Administrators

Xloud SIEM is designed to work out of the box once Wazuh is enabled at deploy time.
Typical admin workflows:

<Steps>
  <Step title="Enable Xloud SIEM at deploy time" icon="toggle-right">
    In XDeploy → Configuration → Monitoring & Logging, toggle **Enable Security Suite**.
    The suite activates Wazuh (HIDS), Lynis (auditing), and OpenSCAP (compliance) on every
    cluster node.
  </Step>

  <Step title="Deploy agents to tenant VMs" icon="server">
    Use the bundled `xavs-ansible` role to mass-deploy Wazuh agents across projects. See
    the [Wazuh](/security/wazuh) page for the exact command and options.
  </Step>

  <Step title="Review Security Posture daily" icon="clipboard-list">
    Open **Monitor Center → Security Posture**, review the Overview tab for trend
    changes, and drill into any node with falling compliance scores using the **Why?**
    link in the Compliance tab.
  </Step>

  <Step title="Triage alerts in real time" icon="bell">
    Use **Monitor Center → Alerts → Active Alerts**. The Security tag isolates
    Wazuh-originated threats from infrastructure noise.
  </Step>

  <Step title="Export for audit or compliance reviews" icon="download">
    Use the **Export Report** button on Security Posture to produce a timestamped CSV
    snapshot of the cluster's full security state.
  </Step>
</Steps>

***

## How the Three Scanners Differ

Each scanner attacks a different attack surface — they are complementary, not redundant.

| Capability                    |  Wazuh  |  Lynis  | OpenSCAP |
| ----------------------------- | :-----: | :-----: | :------: |
| Agent-based live monitoring   |   Yes   |    —    |     —    |
| On-host audit script          |    —    |   Yes   |    Yes   |
| Real-time alerts              |   Yes   |    —    |     —    |
| File integrity monitoring     |   Yes   |    —    |     —    |
| CVE / vulnerability scanning  |   Yes   |    —    |     —    |
| Hardening index score         | Partial |   Yes   |     —    |
| CIS Benchmark profiles        |   Yes   | Partial |    Yes   |
| DISA STIG / PCI-DSS profiles  |    —    |    —    |    Yes   |
| MITRE ATT\&CK mapping         |   Yes   |    —    |     —    |
| XML / HTML compliance reports |    —    |    —    |    Yes   |

Together they give you defense-in-depth: Wazuh watches for active attacks, Lynis catches
misconfiguration drift, OpenSCAP proves regulatory conformance.

***

## Tool Deep-Dives

<CardGroup>
  <Card title="Wazuh" icon="shield-alert" href="/security/wazuh">
    Architecture, agent deployment via Ansible or manual steps, detection rules, File
    Integrity Monitoring configuration, and the Wazuh Dashboard.
  </Card>

  <Card title="Lynis" icon="list-checks" href="/security/lynis">
    How the script runs, score interpretation, warnings vs suggestions, per-node and
    fleet-wide sweeps, and remediation workflows.
  </Card>

  <Card title="OpenSCAP" icon="clipboard-check" href="/security/openscap">
    Available profiles (CIS L1 / L2, PCI-DSS, HIPAA, ANSSI, STIG), how to run a scan,
    reading the XML/HTML reports, and applying the remediation playbooks.
  </Card>
</CardGroup>

***

## Common Tasks

<AccordionGroup>
  <Accordion title="Check cluster-wide CIS compliance in 30 seconds" icon="clock">
    Open **Monitor Center → Security Posture → Overview**. The top-row **CIS Compliance**
    card shows the cluster-wide average across all scanned agents.
  </Accordion>

  <Accordion title="Find out why one node's compliance score dropped" icon="trending-down">
    In the **Compliance** tab, locate the node and click **Why?** next to its row. A modal
    lists every Lynis warning and suggestion with test IDs and remediation text.
  </Accordion>

  <Accordion title="Spot unencrypted volumes across the fleet" icon="lock-open">
    Open the **Encryption** tab. Any instance tagged **Unencrypted** has at least one
    volume without encryption enabled.
  </Accordion>

  <Accordion title="Find high-risk security groups" icon="triangle-alert">
    Open the **Microsegmentation → Security Groups** tab. Sort by Risk Score descending.
    Groups tagged **Wide-Open** are the most urgent to review.
  </Accordion>

  <Accordion title="Export a snapshot for audit" icon="file-text">
    Click **Export Report** on any Security Posture tab. The download is a timestamped CSV
    with agent, compliance, and vulnerability data for the entire cluster.
  </Accordion>

  <Accordion title="Investigate a live Wazuh alert" icon="search">
    In **Security Posture → Alerts**, expand the row to see the rule description and source
    IP. Click **Wazuh Dashboard** in the top-right to jump to the native UI for deeper
    forensic queries.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Wazuh Deep Dive" icon="shield-alert" href="/security/wazuh">
    Architecture, agent deployment, ruleset, File Integrity Monitoring
  </Card>

  <Card title="Lynis Deep Dive" icon="list-checks" href="/security/lynis">
    Run audits, interpret the hardening index, fleet sweeps
  </Card>

  <Card title="OpenSCAP Deep Dive" icon="clipboard-check" href="/security/openscap">
    Compliance profiles, scan commands, report formats, remediation
  </Card>

  <Card title="Compliance and Auditing" icon="gavel" href="/security/compliance">
    SOC 2, ISO 27001, HIPAA, PCI-DSS, GDPR frameworks at the platform level
  </Card>
</CardGroup>


# Xloud Compute
Source: https://docs.xloud.tech/services/compute

Xloud Compute provides resizable virtual machine instances running on bare-metal hypervisors in your private cloud infrastructure.

Resizable virtual machine instances running on bare-metal hypervisors in your private cloud.

<Note>
  Compute is included in [XAVS](/products/xavs), [XPCI](/products/xpci), and [XHCI](/products/xhci). See product pages on [xloud.tech](https://xloud.tech/xavs) for specifications and datasheets.
</Note>

***

<p>Guides & References</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/compute/user-guide">
    Instances, flavors, images, key pairs, security groups — launch, manage, and scale VMs.
  </Card>

  <Card title="Admin Guide" icon="shield" href="/services/compute/admin-guide">
    Compute hosts, hypervisors, quotas, scheduling, and cluster maintenance.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/compute/cli-reference">
    `openstack server` commands for instance management from the command line.
  </Card>

  <Card title="Hypervisor (KVM)" icon="cpu" href="/services/compute/hypervisor">
    KVM configuration, CPU modes, backing storage, nested virtualization.
  </Card>
</CardGroup>

***

<p>Key Features</p>

<CardGroup>
  <Card title="Bare-Metal Hypervisor" icon="cpu" href="/services/compute/hypervisor">
    Runs directly on physical hardware — maximum performance for enterprise workloads.
  </Card>

  <Card title="Live Migration" icon="arrow-right-left" href="/services/compute/live-migration">
    Move running workloads between hosts with zero downtime.
  </Card>

  <Card title="Live vCPU/RAM Scaling" icon="move-vertical" href="/services/compute/live-resize">
    Add or reduce vCPU and RAM on a running instance — no reboot, no downtime.
  </Card>

  <Card title="Snapshots & Backup" icon="copy" href="/services/compute/snapshots">
    Automated snapshots and restore points with minimal overhead.
  </Card>

  <Card title="Security" icon="shield" href="/services/compute/security-hardening">
    VM encryption, distributed firewalls, TPM 2.0 support.
  </Card>

  <Card title="Disaster Recovery" icon="rotate-ccw" href="/services/disaster-recovery/index">
    One-click failover/failback with application-aware recovery.
  </Card>
</CardGroup>

***

<p>Compute Components</p>

| Component            | Description                                                                            |
| -------------------- | -------------------------------------------------------------------------------------- |
| **Hypervisor**       | Bare-metal virtualization layer on physical hardware                                   |
| **Scheduler**        | Places instances on optimal hosts based on resources, affinity, and availability zones |
| **Conductor**        | Mediates database interactions and long-running operations                             |
| **API**              | RESTful endpoint for instance lifecycle, flavors, key pairs, server groups             |
| **Metadata Service** | Provides instance config (hostname, SSH keys, user data) at boot                       |
| **Console (VNC)**    | Browser-based console access to running instances                                      |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Block Storage" icon="hard-drive" href="/services/storage/index">
    Persistent volumes for compute instances
  </Card>

  <Card title="Networking" icon="network" href="/services/networking/index">
    Virtual networks, floating IPs, security groups
  </Card>

  <Card title="Image Service" icon="disc" href="/services/images/index">
    OS images and snapshots for launching instances
  </Card>

  <Card title="Identity & Access" icon="fingerprint" href="/services/identity/index">
    Authentication and RBAC for compute operations
  </Card>

  <Card title="Load Balancer" icon="combine" href="/services/load-balancer">
    Distribute traffic across instances
  </Card>

  <Card title="Instance HA" icon="heart-pulse" href="/services/instance-ha/index">
    Automatic failover on host failure
  </Card>
</CardGroup>


# Compute Administration Guide
Source: https://docs.xloud.tech/services/compute/admin-guide

Deploy, configure, and maintain Xloud Compute services across your infrastructure.

Manage compute hosts, flavors, quotas, scheduling, live migration, security, and advanced hypervisor features.

***

<CardGroup>
  <Card title="Architecture" icon="workflow" href="/services/compute/architecture">
    Service topology and components
  </Card>

  <Card title="Compute Hosts" icon="server" href="/services/compute/compute-hosts">
    Hypervisor host management
  </Card>

  <Card title="Flavors" icon="sliders" href="/services/compute/flavors">
    Instance type management
  </Card>

  <Card title="Quotas" icon="chart-pie" href="/services/compute/quotas">
    Per-project resource limits
  </Card>

  <Card title="Scheduling" icon="shuffle" href="/services/compute/scheduling">
    Filters, weights, and placement
  </Card>

  <Card title="Live Migration" icon="arrow-right-left" href="/services/compute/live-migration">
    Zero-downtime instance moves
  </Card>

  <Card title="Console Access" icon="monitor" href="/services/compute/console-access">
    VNC, SPICE, and serial consoles
  </Card>

  <Card title="Security" icon="shield" href="/services/compute/security-hardening">
    TLS, metadata, rate limiting
  </Card>

  <Card title="Live Scaling" icon="sliders" href="/services/compute/live-resize-admin">
    Configure live vCPU/RAM scaling
  </Card>

  <Card title="Advanced" icon="cpu" href="/services/compute/advanced-features">
    CPU pin, hugepages, GPU, vTPM
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/compute/troubleshooting">
    Common issues and fixes
  </Card>
</CardGroup>


# Advanced Compute Features
Source: https://docs.xloud.tech/services/compute/advanced-features

Configure CPU pinning, huge pages, GPU passthrough, vTPM, UEFI Secure Boot, and PCI passthrough for high-performance workloads.

## Overview

Xloud Compute exposes advanced hypervisor capabilities for workloads that require
dedicated hardware resources, hardware-enforced security, or accelerated computing.
These features are activated through flavor extra specs and require corresponding
host-level configuration on participating compute nodes.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials sourced (`source openrc.sh`)
  * Host-level hardware features configured through XDeploy (IOMMU, huge pages, VFIO)
  * Relevant scheduler filters active in the scheduler configuration
</Note>

***

## Features

<AccordionGroup>
  <Accordion title="NUMA-Aware Scheduling" icon="cpu">
    <Badge icon="building">Enterprise</Badge>

    <Info>**Xloud-Developed** — NUMA-aware scheduling with per-flavor granularity is developed by Xloud and ships with XAVS / XPCI.</Info>

    NUMA-aware scheduling ensures that an instance's vCPUs and memory are allocated
    from the same physical NUMA cell on the host, avoiding cross-node memory access
    penalties. The `NUMATopologyFilter` in the scheduler evaluates each host's NUMA
    topology and rejects placements that would split an instance across cells.

    Unlike VMware's cluster-wide EVC setting, Xloud provides **per-flavor NUMA
    granularity** — each flavor can define its own NUMA cell count and CPU/memory
    distribution, allowing mixed workload profiles on the same compute cluster.

    ```bash title="Create a NUMA-aware flavor (2 NUMA nodes, 8 vCPUs)" theme={null}
    openstack flavor create \
      --vcpus 8 \
      --ram 16384 \
      --disk 100 \
      numa.8xlarge

    openstack flavor set numa.8xlarge \
      --property hw:numa_nodes=2
    ```

    | Extra Spec       | Values       | Description                                                             |
    | ---------------- | ------------ | ----------------------------------------------------------------------- |
    | `hw:numa_nodes`  | Integer      | Number of guest NUMA nodes (vCPUs and memory split evenly across nodes) |
    | `hw:numa_cpus.N` | CPU list     | Pin specific vCPUs to NUMA node N (e.g., `0,1,2,3`)                     |
    | `hw:numa_mem.N`  | Integer (MB) | Memory allocation for NUMA node N                                       |

    <Tip>
      For most workloads, setting `hw:numa_nodes=1` is sufficient to ensure all
      resources are allocated from a single NUMA cell. Use `hw:numa_nodes=2` only for
      large instances that exceed a single cell's capacity.
    </Tip>
  </Accordion>

  <Accordion title="CPU Pinning" icon="microchip">
    <Badge icon="building">Enterprise</Badge>

    <Info>**Xloud-Developed** — CPU pinning with mixed-policy support is developed by Xloud and ships with XAVS / XPCI.</Info>

    CPU pinning dedicates physical CPU threads exclusively to a single instance,
    eliminating scheduling jitter and improving performance predictability for
    latency-sensitive workloads such as real-time databases, financial applications,
    and telco VNFs.

    Xloud supports three CPU policy modes:

    * **`dedicated`** — all vCPUs are pinned to exclusive physical threads
    * **`shared`** — vCPUs float across all available host CPUs (default behavior)
    * **`mixed`** — some vCPUs are pinned while others float, enabling a balance between deterministic performance for critical threads and flexible scheduling for background threads

    **Host requirements**: NUMA topology support must be enabled on the compute node.
    The `NUMATopologyFilter` must be active in the scheduler filter chain.

    Create a flavor with dedicated CPU policy:

    ```bash title="Create CPU-pinned flavor" theme={null}
    openstack flavor create \
      --vcpus 8 \
      --ram 16384 \
      --disk 100 \
      rt.8xlarge
    ```

    <CodeGroup>
      ```bash title="Dedicated pinning (all vCPUs pinned)" theme={null}
      openstack flavor set rt.8xlarge \
        --property hw:cpu_policy=dedicated \
        --property hw:cpu_thread_policy=prefer
      ```

      ```bash title="Mixed pinning (pin specific vCPUs only)" theme={null}
      openstack flavor set rt.8xlarge \
        --property hw:cpu_policy=mixed \
        --property hw:cpu_dedicated_mask=0-3 \
        --property hw:cpu_thread_policy=prefer
      ```
    </CodeGroup>

    | Extra Spec              | Values                         | Description                                                     |
    | ----------------------- | ------------------------------ | --------------------------------------------------------------- |
    | `hw:cpu_policy`         | `dedicated`, `shared`, `mixed` | Controls whether vCPUs are pinned to exclusive physical threads |
    | `hw:cpu_dedicated_mask` | CPU mask (e.g., `0-3`)         | When `mixed` policy is used, specifies which vCPUs are pinned   |
    | `hw:cpu_thread_policy`  | `prefer`, `isolate`, `require` | Controls whether sibling hyperthreads are used                  |
    | `hw:numa_nodes`         | Integer                        | Number of NUMA nodes to expose inside the instance              |

    <Tip>
      Configure CPU pinning host requirements on participating nodes through XDeploy
      under **Compute → Advanced → NUMA Configuration**. Only hosts with NUMA support
      enabled accept CPU-pinned instances.
    </Tip>
  </Accordion>

  <Accordion title="CPU Feature Masking (EVC Equivalent)" icon="shield">
    <Badge icon="building">Enterprise</Badge>

    <Info>**Xloud-Developed** — Per-flavor CPU feature masking is developed by Xloud and ships with XAVS / XPCI.</Info>

    CPU feature masking normalizes the CPU instruction set exposed to instances,
    enabling live migration across compute nodes with different CPU generations. This
    is the Xloud equivalent of VMware's Enhanced vMotion Compatibility (EVC), with a
    key difference: Xloud applies masking **per-flavor or per-image** rather than at
    the cluster level, allowing different workloads to use different CPU baselines on
    the same cluster.

    Set a common CPU model baseline on the flavor:

    ```bash title="Set CPU model for cross-generation migration" theme={null}
    openstack flavor set <flavor-name> \
      --property hw:cpu_mode=custom \
      --property hw:cpu_model=Cascadelake-Server-noTSX
    ```

    | Extra Spec     | Values                                     | Description                                                                    |
    | -------------- | ------------------------------------------ | ------------------------------------------------------------------------------ |
    | `hw:cpu_mode`  | `host-model`, `host-passthrough`, `custom` | `custom` enables explicit CPU model selection                                  |
    | `hw:cpu_model` | CPU model name                             | Target CPU generation baseline (e.g., `Cascadelake-Server-noTSX`, `IvyBridge`) |

    <Note>
      When using `host-model` or `host-passthrough`, the instance exposes the host's
      native CPU features. Live migration to a host with a different CPU generation
      will fail if the destination lacks required features. Use `custom` mode with an
      explicit model to guarantee migration compatibility.
    </Note>
  </Accordion>

  <Accordion title="Huge Pages" icon="layers">
    <Badge icon="building">Enterprise</Badge>

    <Info>**Xloud-Developed** — Huge page management with per-flavor page size selection is developed by Xloud and ships with XAVS / XPCI.</Info>

    Huge pages reduce TLB (Translation Lookaside Buffer) pressure for memory-intensive
    workloads. The hypervisor pre-allocates huge page pools on the host at boot time.
    Xloud supports two page sizes:

    * **2 MB huge pages** — suitable for most workloads including databases, application servers, and general-purpose memory optimization. Lower host memory fragmentation risk.
    * **1 GB huge pages** — optimal for HPC, in-memory analytics, and latency-sensitive network functions (DPDK, VNFs) where TLB coverage per entry is critical.

    **Host requirements**: Huge page pools must be pre-allocated on the compute node.
    Configure pool sizes through XDeploy under **Compute → Advanced → Memory
    Configuration**.

    <CodeGroup>
      ```bash title="Request 2 MB huge pages (general workloads)" theme={null}
      openstack flavor set <flavor-name> \
        --property hw:mem_page_size=2MB
      ```

      ```bash title="Request 1 GB huge pages (HPC / DPDK)" theme={null}
      openstack flavor set <flavor-name> \
        --property hw:mem_page_size=1GB
      ```

      ```bash title="Let the scheduler choose available page size" theme={null}
      openstack flavor set <flavor-name> \
        --property hw:mem_page_size=any
      ```
    </CodeGroup>

    Instances using huge pages are scheduled only onto hosts where the required huge
    page pool is allocated. The scheduler rejects hosts without sufficient huge pages
    of the requested size.

    <Tip>
      Use `hw:mem_page_size=any` to allow the scheduler to place the instance on a
      host with either 2 MB or 1 GB pages, maximizing scheduling flexibility.
    </Tip>
  </Accordion>

  <Accordion title="GPU Passthrough" icon="monitor">
    <Badge icon="building">Enterprise</Badge>

    GPU passthrough exposes a physical GPU device directly to an instance with
    near-native performance. The GPU is bound to the VFIO driver on the host and
    assigned exclusively to one instance at a time. Use this for AI/ML training,
    3D rendering, and GPU-accelerated simulation workloads.

    **Host requirements**: IOMMU must be enabled in host BIOS and OS. The GPU must
    be bound to the VFIO driver. Configure this through XDeploy under **Compute →
    Hardware → GPU Configuration**.

    List available PCI resource providers to find the GPU alias:

    ```bash title="List resource providers" theme={null}
    openstack resource provider list
    ```

    ```bash title="Show inventory for a resource provider" theme={null}
    openstack resource provider inventory list <resource-provider-uuid>
    ```

    Request the GPU in a flavor using the configured device alias:

    ```bash title="Add GPU alias to flavor" theme={null}
    openstack flavor set <flavor-name> \
      --property pci_passthrough:alias=nvidia-a100:1
    ```

    <Note>
      All compute nodes exposing the same GPU type must use the same alias name. The
      `PciPassthroughFilter` must be active in the scheduler filter chain to route
      GPU-requesting instances to hosts with available devices.
    </Note>
  </Accordion>

  <Accordion title="vTPM (Virtual Trusted Platform Module)" icon="shield-check">
    <Badge icon="building">Enterprise</Badge>

    <Info>**Xloud-Developed** — vTPM with live migration support and Dashboard integration is developed by Xloud and ships with XAVS / XPCI.</Info>

    vTPM provides a software-emulated TPM chip inside the instance, enabling disk
    encryption (BitLocker, LUKS), measured boot attestation, and secure credential
    storage. Xloud supports **live migration of vTPM instances** with automatic secret
    transfer via [Xloud Key Management](/services/key-manager/admin-guide) — the
    encrypted TPM state is seamlessly transferred to the destination host without
    manual intervention.

    **Supported models:**

    | Model                                   | Use Case                                                                             |
    | --------------------------------------- | ------------------------------------------------------------------------------------ |
    | `tpm-crb` (Command Response Buffer)     | Recommended for TPM 2.0. Modern interface used by Windows 11, RHEL 9+, Ubuntu 22.04+ |
    | `tpm-tis` (TPM Interface Specification) | Legacy interface for TPM 1.2 compatibility and older operating systems               |

    **Host requirements**: [Xloud Key Management](/services/key-manager/admin-guide)
    must be enabled for secret storage. The `swtpm` software TPM emulator must be
    installed on all compute nodes.

    **Provisioning methods:**

    <CodeGroup>
      ```bash title="Via flavor extra specs" theme={null}
      openstack flavor set <flavor-name> \
        --property hw:tpm_version=2.0 \
        --property hw:tpm_model=tpm-crb
      ```

      ```bash title="Via image properties" theme={null}
      openstack image set <image-id> \
        --property hw_tpm_version=2.0 \
        --property hw_tpm_model=tpm-crb
      ```
    </CodeGroup>

    In the Dashboard, vTPM can be enabled through the **flavor extra specs panel** or
    the **image admin form**. The instance detail page displays the vTPM status when
    attached.

    | Extra Spec / Image Property         | Values               | Description                |
    | ----------------------------------- | -------------------- | -------------------------- |
    | `hw:tpm_version` / `hw_tpm_version` | `1.2`, `2.0`         | vTPM specification version |
    | `hw:tpm_model` / `hw_tpm_model`     | `tpm-tis`, `tpm-crb` | Virtual TPM hardware model |

    <Warning>
      vTPM state is encrypted at rest using a secret stored in Xloud Key Management.
      Ensure Xloud Key Management is enabled and healthy before provisioning vTPM
      instances. If the Key Management service is unavailable, vTPM instances cannot
      start or be live-migrated.
    </Warning>
  </Accordion>

  <Accordion title="UEFI Boot and Secure Boot" icon="hard-drive">
    UEFI boot is required for Secure Boot, vTPM, and GPT-partitioned disk layouts.
    Enable UEFI at the image level or as a flavor property. Secure Boot adds an
    additional layer by verifying the bootloader signature at startup.

    Set UEFI firmware type on an image:

    ```bash title="Enable UEFI boot on an image" theme={null}
    openstack image set \
      --property hw_firmware_type=uefi \
      <image-id>
    ```

    Enable Secure Boot through a flavor property:

    ```bash title="Require Secure Boot via flavor" theme={null}
    openstack flavor set <flavor-name> \
      --property os:secure_boot=required
    ```

    <Note>
      Secure Boot requires a signed bootloader in the guest OS. Unsigned kernels and
      bootloaders will fail to start with Secure Boot enabled. Verify guest OS
      compatibility before deploying Secure Boot in production.
    </Note>
  </Accordion>

  <Accordion title="PCI Passthrough" icon="cpu">
    <Badge icon="building">Enterprise</Badge>

    PCI passthrough grants exclusive access to any host PCI device — network adapters,
    accelerators, storage controllers, or FPGAs — to a single instance. The device is
    isolated from the host OS using IOMMU groups, providing hardware-level isolation
    and near-native performance.

    **Host requirements**: IOMMU must be enabled. Device aliases must be configured
    and consistent across all nodes exposing the same device type. Configure through
    XDeploy under **Compute → Hardware → PCI Passthrough**.

    ```bash title="Request a PCI device via flavor" theme={null}
    openstack flavor set <flavor-name> \
      --property pci_passthrough:alias=<device-alias>:1
    ```

    Replace `<device-alias>` with the alias name configured in XDeploy. The number
    after the colon specifies how many devices to attach (typically `1`).

    <Warning>
      PCI-attached devices cannot be live-migrated. Instances with PCI passthrough are
      limited to cold migration only. Factor this constraint into your maintenance
      and availability planning.
    </Warning>
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Flavor Management" href="/services/compute/flavors">
    Apply extra specs to flavors to activate advanced hardware features for tenants.
  </Card>

  <Card title="Security Hardening" href="/services/compute/security-hardening">
    Combine vTPM and UEFI Secure Boot with compute control plane hardening.
  </Card>

  <Card title="Admin Guide" href="/services/compute/admin-guide">
    Return to the Compute Administration Guide index.
  </Card>
</CardGroup>


# Compute Service Architecture
Source: https://docs.xloud.tech/services/compute/architecture

Understand how Xloud Compute components interact — API, Scheduler, Conductor, Compute Agent, and Placement — to deliver virtual machine lifecycle management.

## Overview

Xloud Compute follows a distributed, service-oriented architecture. API requests enter
through a central API tier, are routed through the Scheduler and Placement services to
select the optimal host, and are executed by Compute Agents running on every hypervisor
node. Each agent communicates directly with the local hypervisor to manage instance
lifecycle operations — creation, power state, migration, and console access.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Service Topology

The diagram below illustrates how requests flow from the end user through the Xloud
Compute control plane to the hypervisor layer.

```mermaid theme={null}
graph TD
    U([User / Dashboard]) --> API[Compute API<br/>:8774]
    API --> SCH[Scheduler]
    SCH -->|Placement decision| CN1[Compute Node 1]
    SCH -->|Placement decision| CN2[Compute Node 2]
    SCH -->|Placement decision| CN3[Compute Node N]
    CN1 --> HV1[Hypervisor<br/>Host 1]
    CN2 --> HV2[Hypervisor<br/>Host 2]
    CN3 --> HV3[Hypervisor<br/>Host N]
    HV1 --> VM1[VMs]
    HV2 --> VM2[VMs]
    HV3 --> VM3[VMs]
    API --> PL[Placement API<br/>:8778]
    PL --> SCH
    API --> CON[Conductor]
    CON --> DB[(Database)]
    style API fill:#197560,color:#fff
    style SCH fill:#197560,color:#fff
    style PL fill:#3F8F7E,color:#fff
    style CON fill:#3F8F7E,color:#fff
```

***

## Service Components

| Component         | Port     | Runs On               | Description                                                                     |
| ----------------- | -------- | --------------------- | ------------------------------------------------------------------------------- |
| **Compute API**   | 8774     | Controller nodes      | REST API endpoint for all instance lifecycle operations                         |
| **Placement API** | 8778     | Controller nodes      | Tracks resource inventory and allocation across all compute hosts               |
| **Scheduler**     | Internal | Controller nodes      | Selects the optimal compute node for each new instance request                  |
| **Conductor**     | Internal | Controller nodes      | Orchestrates multi-step operations; acts as a database proxy for compute agents |
| **Compute Agent** | Internal | Every hypervisor node | Manages instance lifecycle on the local hypervisor                              |
| **Console Proxy** | 6080     | Controller nodes      | VNC console proxy for browser-based instance access                             |

<Note>
  The Conductor service decouples compute agents from direct database access. All database
  writes from hypervisor nodes flow through the Conductor, which enforces access control
  and serializes state transitions.
</Note>

***

## Service Lifecycle

The following table describes the expected runtime state of each service on a healthy
cluster.

| Service           | Expected State                   | Managed By     |
| ----------------- | -------------------------------- | -------------- |
| `nova-api`        | Running, listening on `:8774`    | XDeploy / XAVS |
| `placement-api`   | Running, listening on `:8778`    | XDeploy / XAVS |
| `nova-scheduler`  | Running                          | XDeploy / XAVS |
| `nova-conductor`  | Running                          | XDeploy / XAVS |
| `nova-compute`    | Running on every hypervisor node | XDeploy / XAVS |
| `nova-novncproxy` | Running, listening on `:6080`    | XDeploy / XAVS |

<Tabs>
  <Tab title="XDeploy" icon="server">
    In XDeploy, navigate to **Operations** and run **Prechecks** to validate all
    Compute service components across the cluster. The output shows:

    * Per-service status on every node
    * Last heartbeat timestamp
    * Service version and host assignment

    Use **Reconfigure** from Operations to recover a failed component without
    a full node reboot.
  </Tab>

  <Tab title="CLI" icon="terminal">
    Verify that all Compute services are in an `up` state:

    ```bash title="List all compute services" theme={null}
    openstack compute service list
    ```

    Expected output shows all services with `State: up` and `Status: enabled`.

    ```bash title="Filter for degraded services" theme={null}
    openstack compute service list | grep -v "up"
    ```

    <Check>All services report `State: up`. Any service showing `down` requires investigation on the host where it runs.</Check>
  </Tab>
</Tabs>

***

## Request Flow: Instance Creation

Understanding the creation flow helps diagnose failures at each stage.

<Steps>
  <Step title="API receives the request" icon="arrow-down-to-line">
    The Compute API validates the request, checks quota, and creates an instance record
    in the database with status `BUILD`.
  </Step>

  <Step title="Placement selects resources" icon="target">
    The Placement API queries resource inventories to find hosts with sufficient vCPU,
    RAM, and disk. It returns a list of allocation candidates.
  </Step>

  <Step title="Scheduler filters and ranks" icon="filter">
    The Scheduler applies the configured filter chain to eliminate ineligible hosts,
    then ranks remaining candidates using weighers. The top-ranked host is selected.
  </Step>

  <Step title="Conductor orchestrates the build" icon="git-merge">
    The Conductor sends a `build_instance` RPC call to the Compute Agent on the selected
    host. It monitors progress and updates the database with state transitions.
  </Step>

  <Step title="Compute Agent creates the instance" icon="play">
    The Compute Agent on the target hypervisor node provisions the instance — downloading
    the image, allocating network interfaces, attaching volumes, and starting the virtual
    machine. The instance transitions from `BUILD` to `ACTIVE`.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    Manage hypervisor nodes — list, inspect, enable, and disable hosts.
  </Card>

  <Card title="Scheduling" href="/services/compute/scheduling">
    Configure filters, weighers, host aggregates, and availability zones.
  </Card>

  <Card title="Admin Guide" href="/services/compute/admin-guide">
    Return to the Compute Administration Guide index.
  </Card>
</CardGroup>


# Availability Zones
Source: https://docs.xloud.tech/services/compute/availability-zones

Partition your compute cluster into independent fault domains. Select availability zones during instance launch.

## Overview

Availability zones partition the compute cluster into independent fault domains. Each
zone typically represents a separate rack, power circuit, or physical location. Placing
instances across multiple zones protects against localized hardware failures.

<Note>
  **Prerequisites**

  * At least one availability zone configured by your administrator
  * Zones are configured through [Host Aggregates](/services/compute/scheduling)
</Note>

***

## Select an Availability Zone at Launch

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    During instance creation, the **Available Zone** field appears in **Step 1
    (Base Config)** of the wizard.

    | User Type          | Options Shown                                                                                                               |
    | ------------------ | --------------------------------------------------------------------------------------------------------------------------- |
    | **Regular users**  | Dropdown of available zone names                                                                                            |
    | **Administrators** | Grouped options — **Available Zones** (zone names) plus **Pin to Host** (`zone:hostname` entries for direct host placement) |

    <Tip>
      If you do not select a specific zone, the scheduler automatically chooses
      the zone with the most available resources. This is recommended for most
      workloads.
    </Tip>

    For high availability, launch replicated instances in different zones using a
    [Server Group](/services/compute/server-groups) with `anti-affinity` policy.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="List availability zones" theme={null}
      openstack availability zone list --compute
      ```

      ```bash title="Launch in a specific zone" theme={null}
      openstack server create \
        --image <IMAGE_ID> \
        --flavor <FLAVOR> \
        --network <NETWORK> \
        --availability-zone <ZONE_NAME> \
        my-instance
      ```

      ```bash title="Launch on a specific host (admin)" theme={null}
      openstack server create \
        --image <IMAGE_ID> \
        --flavor <FLAVOR> \
        --network <NETWORK> \
        --availability-zone <ZONE_NAME>:<HOSTNAME> \
        my-instance
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Zone Management

Availability zones are managed through Host Aggregates by administrators. See
[Compute Scheduling](/services/compute/scheduling) for creating aggregates with
zone assignments.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Host Aggregates** in the admin sidebar. The
    **Availability Zones** tab shows a read-only list of all configured zones.

    To create a new zone, create a Host Aggregate and assign it a new availability
    zone name during creation.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List zones" theme={null}
      openstack availability zone list --compute
      ```

      ```bash title="Create zone via aggregate" theme={null}
      openstack aggregate create --zone new-zone my-aggregate
      openstack aggregate add host my-aggregate <HOSTNAME>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Select an availability zone in the instance create wizard
  </Card>

  <Card title="Server Groups" href="/services/compute/server-groups">
    Combine zones with anti-affinity for maximum resilience
  </Card>

  <Card title="Scheduling" href="/services/compute/scheduling">
    Create host aggregates and manage zone assignments
  </Card>

  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    View which hosts belong to which zones
  </Card>
</CardGroup>


# Bare Metal Provisioning
Source: https://docs.xloud.tech/services/compute/bare-metal

Provision physical servers as cloud resources using the Dashboard's 3-step wizard or CLI. Register nodes and configure IPMI.

## Overview

Xloud Bare Metal provisioning treats physical servers as first-class cloud resources.
Administrators enroll nodes by registering their management interface credentials, then
users can provision bare metal instances through the same workflow as virtual machines.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator access to the Xloud Dashboard (admin view)
  * Bare metal service (Ironic) enabled on the platform
  * IPMI access to the physical servers
  * Deploy kernel and ramdisk images uploaded to the image service
</Note>

***

## View Bare Metal Nodes

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Bare Metal Nodes** in the admin sidebar. This page
    is only visible when the Ironic endpoint is enabled.

    | Column                   | Description                                                   |
    | ------------------------ | ------------------------------------------------------------- |
    | **Node ID/Name**         | Node identifier (clickable to view details)                   |
    | **Ironic Instance Name** | Name of the instance provisioned on this node                 |
    | **Power State**          | Power on, power off, or unknown                               |
    | **Provision State**      | Available, active, deploying, etc.                            |
    | **Maintained**           | Whether the node is in maintenance mode (with reason tooltip) |
    | **Number of Ports**      | Network ports registered for this node                        |
    | **Driver**               | Management driver (e.g., `ipmi`)                              |
    | **Created At**           | Registration timestamp                                        |

    Filter by **Name**, **Power State**, or **Provision State**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="List bare metal nodes" theme={null}
      openstack baremetal node list
      ```

      ```bash title="Show node details" theme={null}
      openstack baremetal node show <NODE_ID>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Register a Bare Metal Node

The Dashboard provides a 3-step wizard for node registration.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Create Node wizard">
        Navigate to **Compute > Bare Metal Nodes**. Click **Create Node**.
      </Step>

      <Step title="Step 1 — Node Info">
        | Field               | Type            | Required | Description                                               |
        | ------------------- | --------------- | -------- | --------------------------------------------------------- |
        | **Name**            | Text            | No       | Optional node name                                        |
        | **Driver**          | Dropdown        | Yes      | Management driver (`ipmi`)                                |
        | **Properties**      | Key-value pairs | Yes      | Must include: `cpus`, `memory_mb`, `local_gb`, `cpu_arch` |
        | **Extra**           | Key-value pairs | No       | Additional metadata                                       |
        | **Standard Traits** | Multi-select    | No       | Standard resource traits                                  |
        | **Custom Traits**   | Dynamic list    | No       | Custom traits (must match `CUSTOM_[A-Z0-9_]{1,248}`)      |
        | **Resource Class**  | Text            | No       | Node resource class                                       |

        <Tip>
          Required property keys (`cpus`, `memory_mb`, `local_gb`, `cpu_arch`) can be
          auto-detected using the **Inspect** action after registration.
        </Tip>
      </Step>

      <Step title="Step 2 — Drive Info (Interfaces)">
        Configure hardware interfaces:

        | Field                 | Default    | Options         |
        | --------------------- | ---------- | --------------- |
        | **Boot Interface**    | pxe        | pxe, ipxe, fake |
        | **Console Interface** | no-console | no-console      |
        | **Network Interface** | noop       | flat, noop      |
        | **RAID Interface**    | no-raid    | no-raid, agent  |
        | **Storage Interface** | noop       | noop            |
        | **Vendor Interface**  | ipmitool   | ipmitool        |
      </Step>

      <Step title="Step 3 — Driver Info (IPMI Credentials)">
        | Field              | Type         | Required | Description                |
        | ------------------ | ------------ | -------- | -------------------------- |
        | **Deploy Kernel**  | Image select | Yes      | Kernel image (AKI format)  |
        | **Deploy Ramdisk** | Image select | Yes      | Ramdisk image (ARI format) |
        | **IPMI Address**   | IP input     | Yes      | BMC/IPMI management IP     |
        | **IPMI Port**      | Number       | No       | IPMI port (max: 65535)     |
        | **IPMI Username**  | Text         | Yes      | BMC login username         |
        | **IPMI Password**  | Text         | Yes      | BMC login password         |

        **Advanced IPMI options** (click "More" to expand):

        | Field                     | Default       | Options             |
        | ------------------------- | ------------- | ------------------- |
        | **IPMI Bridge**           | no            | no                  |
        | **IPMI Privilege Level**  | ADMINISTRATOR | ADMINISTRATOR, USER |
        | **IPMI Protocol Version** | 2.0           | 1.5, 2.0            |
      </Step>

      <Step title="Create the node">
        Click **Confirm**. The node appears in the list with provision state
        `Enroll`.

        <Check>Node appears in Bare Metal Nodes list. Use **Inspect** to auto-detect hardware properties.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Register a bare metal node" theme={null}
    openstack baremetal node create \
      --driver ipmi \
      --name my-bare-metal \
      --property cpus=32 \
      --property memory_mb=131072 \
      --property local_gb=1000 \
      --property cpu_arch=x86_64 \
      --driver-info ipmi_address=192.168.1.100 \
      --driver-info ipmi_username=admin \
      --driver-info ipmi_password=password \
      --driver-info deploy_kernel=<KERNEL_IMAGE_ID> \
      --driver-info deploy_ramdisk=<RAMDISK_IMAGE_ID>
    ```
  </Tab>
</Tabs>

***

## Node Management Actions

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    The following actions are available from the node row's **More** dropdown:

    | Action                | Description                                              |
    | --------------------- | -------------------------------------------------------- |
    | **Edit**              | Update node properties, interfaces, and IPMI credentials |
    | **Power On**          | Power on the physical server                             |
    | **Power Off**         | Power off the physical server                            |
    | **Inspect**           | Auto-detect hardware properties from the BMC             |
    | **Set Maintenance**   | Put the node in maintenance mode                         |
    | **Clear Maintenance** | Remove the node from maintenance mode                    |
    | **Set Boot Device**   | Configure the next boot device                           |
    | **Create Port**       | Register a network port for the node                     |
    | **Create Port Group** | Create a port group (bonded interfaces)                  |
    | **Delete**            | Remove the node from the platform                        |

    The first row action is **Manage State** for provisioning lifecycle transitions.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Power on" theme={null}
      openstack baremetal node power on <NODE_ID>
      ```

      ```bash title="Power off" theme={null}
      openstack baremetal node power off <NODE_ID>
      ```

      ```bash title="Inspect hardware" theme={null}
      openstack baremetal node inspect <NODE_ID>
      ```

      ```bash title="Set maintenance" theme={null}
      openstack baremetal node maintenance set <NODE_ID> --reason "Hardware upgrade"
      ```

      ```bash title="Clear maintenance" theme={null}
      openstack baremetal node maintenance unset <NODE_ID>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Node Detail

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a node name to open the detail page. Three tabs are available:

    **Base Info tab**:

    * Base Info: Chassis ID, Resource Class, Maintenance status and reason
    * Driver: All driver info key-value pairs (IPMI credentials masked)
    * Boot Device: Current boot device and persistence setting
    * Properties: All node properties
    * Traits: Standard and custom traits
    * Interface Validation: Table showing each interface's validation status

    **Ports tab** — Network ports registered for this node (with CRUD actions)

    **Port Groups tab** — Bonded port groups (with CRUD actions)
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Show node detail" theme={null}
      openstack baremetal node show <NODE_ID>
      ```

      ```bash title="List node ports" theme={null}
      openstack baremetal port list --node <NODE_ID>
      ```

      ```bash title="Validate node interfaces" theme={null}
      openstack baremetal node validate <NODE_ID>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Provision bare metal instances through the instance create wizard
  </Card>

  <Card title="Flavors" href="/services/compute/flavors">
    Create bare metal flavors with resource class matching
  </Card>

  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    Monitor bare metal hypervisor resources
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve bare metal provisioning failures
  </Card>
</CardGroup>


# Block Device Mapping
Source: https://docs.xloud.tech/services/compute/block-device-mapping

Control how storage volumes are presented to Xloud Compute instances at launch. Configure boot-from-volume, system disks, data disks, and CD-ROM devices.

## Overview

Block device mapping controls how storage is attached to an instance at launch time.
The Dashboard's Instance Create wizard provides a visual interface for configuring
the system disk, additional data disks, and CD-ROM devices. Understanding these
options helps you choose the right storage configuration for your workloads.

<Note>
  **Prerequisites**

  * Block storage service enabled on the platform
  * Available volume types configured by your administrator
  * Sufficient volume and storage quota
</Note>

***

## Storage Options in the Create Wizard

The **Step 1 (Base Config)** of the Instance Create wizard provides these storage
configuration options:

### Boot Source and System Disk

| Source                | Boot From Volume | Result                                                           |
| --------------------- | ---------------- | ---------------------------------------------------------------- |
| **Image**             | Yes (default)    | Creates a new persistent system disk from the image              |
| **Image**             | No               | Boots directly from image (ephemeral root disk — lost on delete) |
| **Instance Snapshot** | Automatic        | System disk configuration inherited from snapshot                |
| **Bootable Volume**   | N/A              | Boots from existing volume (count limited to 1)                  |

<Warning>
  Instances booted without a persistent volume (**Boot From Volume = No**) lose
  their root disk when deleted. Always use boot-from-volume for production workloads
  that need data persistence.
</Warning>

### System Disk Configuration

When **Boot From Volume** is set to Yes, configure the system disk:

| Field                     | Description                                                                     |
| ------------------------- | ------------------------------------------------------------------------------- |
| **Volume Type**           | Select from available storage backends (e.g., SSD, HDD tiers)                   |
| **Size (GiB)**            | Minimum is the largest of: flavor disk size, image minimum disk, and image size |
| **Delete on Termination** | Whether to delete the boot volume when the instance is deleted                  |

### Data Disks

Click **Add Data Disks** to attach additional persistent volumes at launch. Each
data disk has the same configuration options as the system disk (Volume Type, Size,
Delete on Termination).

<Tip>
  Limit the number of attached disks to 16 or fewer for optimal I/O performance.
  The dashboard shows a recommendation when adding data disks.
</Tip>

### CD-ROM Device

The wizard supports attaching a CD-ROM device at launch time:

| CD-ROM Source | Description                                          |
| ------------- | ---------------------------------------------------- |
| **None**      | No CD-ROM attached (default)                         |
| **Image**     | Mount an image (e.g., ISO) as a virtual CD-ROM drive |
| **Volume**    | Mount an existing volume as a CD-ROM drive           |

<Info>
  **Xloud-Developed** — The CD-ROM attachment feature in the instance create wizard
  is developed by Xloud and ships with XAVS / XPCI.
</Info>

***

## CLI Block Device Mapping

<Tabs>
  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Boot from image with persistent volume" theme={null}
      openstack server create \
        --image <IMAGE_ID> \
        --boot-from-volume 50 \
        --flavor <FLAVOR> \
        --network <NETWORK> \
        my-instance
      ```

      ```bash title="Boot from existing volume" theme={null}
      openstack server create \
        --volume <VOLUME_ID> \
        --flavor <FLAVOR> \
        --network <NETWORK> \
        my-instance
      ```

      ```bash title="Advanced block device mapping" theme={null}
      openstack server create \
        --block-device source=image,id=<IMAGE_ID>,dest=volume,size=50,bootindex=0,shutdown=remove \
        --block-device source=blank,dest=volume,size=100,shutdown=preserve \
        --flavor <FLAVOR> \
        --network <NETWORK> \
        my-instance
      ```
    </CodeGroup>

    **Block device mapping parameters**:

    | Parameter   | Values                                 | Description                                             |
    | ----------- | -------------------------------------- | ------------------------------------------------------- |
    | `source`    | `image`, `volume`, `snapshot`, `blank` | Source type for the block device                        |
    | `dest`      | `volume`, `local`                      | Destination — persistent volume or ephemeral local disk |
    | `id`        | UUID                                   | Source resource ID (for image, volume, or snapshot)     |
    | `size`      | Integer (GiB)                          | Volume size                                             |
    | `bootindex` | `0` = boot, `-1` = non-boot            | Boot order (`0` for system disk)                        |
    | `shutdown`  | `remove`, `preserve`                   | Delete behavior on instance termination                 |
  </Tab>
</Tabs>

***

## Volume Type Selection

When creating system or data disks, the available volume types depend on your
platform's storage backend configuration. Common tiers include:

| Tier        | Typical Backend             | Use Case                         |
| ----------- | --------------------------- | -------------------------------- |
| **SSD**     | Ceph SSD pool or local NVMe | Databases, high-IOPS workloads   |
| **HDD**     | Ceph HDD pool or NFS        | Archival, log storage, bulk data |
| **Default** | Platform default tier       | General-purpose workloads        |

<Note>
  Your administrator configures available volume types through
  [XDeploy](/deployment). The volume types shown in the instance wizard
  match those configured for the platform.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Create an instance with the full 4-step wizard including disk configuration
  </Card>

  <Card title="Create a Volume" href="/services/storage/create-volume">
    Pre-create volumes to attach as bootable or data disks
  </Card>

  <Card title="Volume Types" href="/services/storage/volume-types">
    Understand available storage tiers and their characteristics
  </Card>

  <Card title="Instance Snapshots" href="/services/compute/snapshots">
    Create snapshots of instances with their storage configuration
  </Card>
</CardGroup>


# Compute CLI Reference
Source: https://docs.xloud.tech/services/compute/cli-reference

CLI commands for managing Xloud Compute instances — create, list, start, stop, resize, migrate, and manage keypairs.

## Overview

The `openstack server` command group manages the full lifecycle of compute instances. Commands cover instance creation, power operations, resize, migration, console access, keypair management, and security group assignment.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Project membership with appropriate role (Member for user operations, Admin for host-level ops)
</Note>

***

## Instances

### List and Inspect

<CodeGroup>
  ```bash title="List all instances in your project" theme={null}
  openstack server list
  ```

  ```bash title="Filter by status" theme={null}
  openstack server list --status ACTIVE
  openstack server list --status SHUTOFF
  openstack server list --status ERROR
  ```

  ```bash title="Show all tenants (admin)" theme={null}
  openstack server list --all-projects
  ```

  ```bash title="Show instance details" theme={null}
  openstack server show <server>
  ```

  ```bash title="JSON output for scripting" theme={null}
  openstack server show <server> --format json
  ```
</CodeGroup>

### Create

<CodeGroup>
  ```bash title="Basic instance" theme={null}
  openstack server create \
    --flavor m1.small \
    --image Ubuntu-22.04 \
    --network private \
    --key-name my-keypair \
    my-instance
  ```

  ```bash title="With user-data script" theme={null}
  openstack server create \
    --flavor m1.medium \
    --image Ubuntu-22.04 \
    --network private \
    --key-name my-keypair \
    --user-data ./bootstrap.sh \
    my-instance
  ```

  ```bash title="With security group and boot volume" theme={null}
  openstack server create \
    --flavor m1.large \
    --image Ubuntu-22.04 \
    --network private \
    --key-name my-keypair \
    --security-group web-servers \
    --boot-from-volume 50 \
    my-instance
  ```

  ```bash title="In specific availability zone" theme={null}
  openstack server create \
    --flavor m1.small \
    --image Ubuntu-22.04 \
    --network private \
    --availability-zone zone-a \
    my-instance
  ```
</CodeGroup>

### Power Operations

<CodeGroup>
  ```bash title="Stop / Start / Reboot" theme={null}
  openstack server stop <server>
  openstack server start <server>
  openstack server reboot <server>
  openstack server reboot --hard <server>
  ```

  ```bash title="Suspend / Resume / Pause" theme={null}
  openstack server suspend <server>
  openstack server resume <server>
  openstack server pause <server>
  openstack server unpause <server>
  ```

  ```bash title="Rescue / Unrescue" theme={null}
  openstack server rescue <server>
  openstack server unrescue <server>
  ```
</CodeGroup>

### Resize and Migrate

<CodeGroup>
  ```bash title="Resize instance" theme={null}
  openstack server resize --flavor m1.large <server>
  openstack server resize confirm <server>
  openstack server resize revert <server>
  ```

  ```bash title="Live migration (admin)" theme={null}
  openstack server migrate --live-migration <server>
  openstack server migrate --live-migration --host <dest-host> <server>
  ```

  ```bash title="Cold migration (admin)" theme={null}
  openstack server migrate <server>
  ```
</CodeGroup>

### Snapshots and Backups

<CodeGroup>
  ```bash title="Create snapshot" theme={null}
  openstack server image create --name my-snapshot <server>
  ```

  ```bash title="List backups" theme={null}
  openstack server backup list <server>
  ```

  ```bash title="Create backup" theme={null}
  openstack server backup create \
    --name weekly-backup \
    --type weekly \
    --rotate 4 \
    <server>
  ```
</CodeGroup>

### Delete

```bash title="Delete instance" theme={null}
openstack server delete <server>
```

```bash title="Delete multiple instances" theme={null}
openstack server delete <server1> <server2> <server3>
```

***

## Keypairs

<CodeGroup>
  ```bash title="List keypairs" theme={null}
  openstack keypair list
  ```

  ```bash title="Create keypair (generates private key)" theme={null}
  openstack keypair create my-keypair
  openstack keypair create my-keypair > my-keypair.pem
  chmod 600 my-keypair.pem
  ```

  ```bash title="Import existing public key" theme={null}
  openstack keypair create --public-key ~/.ssh/id_rsa.pub my-keypair
  ```

  ```bash title="Show keypair" theme={null}
  openstack keypair show my-keypair
  ```

  ```bash title="Delete keypair" theme={null}
  openstack keypair delete my-keypair
  ```
</CodeGroup>

***

## Flavors

<CodeGroup>
  ```bash title="List flavors" theme={null}
  openstack flavor list
  openstack flavor list --public
  ```

  ```bash title="Show flavor details" theme={null}
  openstack flavor show m1.small
  ```

  ```bash title="Create custom flavor (admin)" theme={null}
  openstack flavor create \
    --ram 4096 \
    --vcpus 2 \
    --disk 40 \
    --public \
    m1.custom
  ```
</CodeGroup>

***

## Console Access

<CodeGroup>
  ```bash title="Get VNC console URL" theme={null}
  openstack console url show <server>
  ```

  ```bash title="Get serial console" theme={null}
  openstack console url show --serial <server>
  ```

  ```bash title="View console log" theme={null}
  openstack console log show <server>
  openstack console log show --lines 50 <server>
  ```
</CodeGroup>

***

## Server Groups (Anti-Affinity)

<CodeGroup>
  ```bash title="List server groups" theme={null}
  openstack server group list
  ```

  ```bash title="Create anti-affinity group" theme={null}
  openstack server group create --policy anti-affinity my-group
  ```

  ```bash title="Create affinity group" theme={null}
  openstack server group create --policy affinity my-group
  ```

  ```bash title="Launch instance in server group" theme={null}
  openstack server create \
    --flavor m1.small \
    --image Ubuntu-22.04 \
    --network private \
    --hint group=<group-id> \
    my-instance
  ```

  ```bash title="Delete server group" theme={null}
  openstack server group delete my-group
  ```
</CodeGroup>

***

## Common Options

| Option                     | Description                                |
| -------------------------- | ------------------------------------------ |
| `--format json`            | Output as JSON                             |
| `--format yaml`            | Output as YAML                             |
| `--format value -c <col>`  | Extract a single field                     |
| `--os-project-name <proj>` | Override project for this command          |
| `--all-projects`           | Show resources across all projects (admin) |
| `--wait`                   | Block until the operation completes        |

***

## Next Steps

<CardGroup>
  <Card title="Launch Instance Guide" href="/services/compute/launch-instance">
    Step-by-step walkthrough for launching your first instance
  </Card>

  <Card title="Block Storage CLI" href="/services/storage/cli-reference">
    Volume create, attach, and snapshot commands
  </Card>
</CardGroup>


# Clone an Instance
Source: https://docs.xloud.tech/services/compute/clone-instance

Create an exact copy of a running instance including disks, network, and security configuration using the Dashboard or CLI.

## Overview

Cloning creates an exact copy of an existing instance — including its root disk, attached
volumes, network configuration, security groups, and key pair. The clone can optionally
be placed on a different network or availability zone. This is useful for scaling
horizontally, creating test environments from production instances, or duplicating
pre-configured application stacks.

<Info>**Xloud-Developed** — Instance cloning is developed by Xloud and ships with XAVS / XPCI.</Info>

<Note>
  **Prerequisites**

  * An instance in `Active`, `Stopped`, or `Shutoff` status
  * Sufficient quota for the new instance (vCPU, RAM, disk)
</Note>

***

## Clone an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Clone dialog">
        Navigate to **Compute > Instances**. Click the **More** dropdown on the
        instance row, then select **Clone Instance** under the **Clone & Template**
        group.

        <Note>
          Clone Instance is only available for instances in `Active`, `Stopped`,
          or `Shutoff` status.
        </Note>
      </Step>

      <Step title="Configure the clone">
        | Field              | Type      | Required | Default          | Description                                  |
        | ------------------ | --------- | -------- | ---------------- | -------------------------------------------- |
        | **Instance**       | Read-only | —        | Source name      | The instance being cloned                    |
        | **Name**           | Text      | Yes      | `{source}-clone` | Display name for the cloned instance         |
        | **Network**        | Dropdown  | No       | Same as source   | Optionally place on a different network      |
        | **Available Zone** | Dropdown  | No       | Same as source   | Optionally place in a different zone         |
        | **Auto Start**     | Checkbox  | No       | Checked          | Start the clone automatically after creation |
        | **Description**    | Text area | No       | —                | Optional notes                               |

        <Tip>
          Leave **Network** and **Available Zone** empty to use the same settings
          as the source instance. Override them only when you need the clone on a
          different network segment or fault domain.
        </Tip>
      </Step>

      <Step title="Create the clone">
        Click **Confirm**. The clone appears in the instance list with status
        `Build`, transitioning to `Active` when ready (if Auto Start is checked).

        <Check>Cloned instance is `Active` with the same configuration as the source.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    Clone uses the Xloud-developed Nova API extension:

    ```bash title="Clone an instance" theme={null}
    curl -X POST -H "X-Auth-Token: $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "xloud-clone": {
          "name": "my-instance-clone",
          "auto_start": true
        }
      }' \
      "$NOVA_ENDPOINT/v2.1/servers/<INSTANCE_ID>/action"
    ```

    ```bash title="Clone to a different network" theme={null}
    curl -X POST -H "X-Auth-Token: $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "xloud-clone": {
          "name": "my-instance-clone",
          "auto_start": true,
          "network_id": "<TARGET_NETWORK_ID>",
          "availability_zone": "az2"
        }
      }' \
      "$NOVA_ENDPOINT/v2.1/servers/<INSTANCE_ID>/action"
    ```

    <Note>
      Clone uses the Xloud-developed `xloud-clone` Nova server action. The standard
      `openstack` CLI does not have a native clone command.
    </Note>
  </Tab>
</Tabs>

***

## What Gets Cloned

| Component             | Cloned | Notes                                  |
| --------------------- | ------ | -------------------------------------- |
| Root disk             | Yes    | Full copy of the boot volume           |
| Attached volumes      | Yes    | Copies of all data volumes             |
| Network configuration | Yes    | Same network unless overridden         |
| Security groups       | Yes    | Same security group assignments        |
| Key pair              | Yes    | Same SSH key pair                      |
| Flavor                | Yes    | Same vCPU, RAM, disk profile           |
| Metadata              | Yes    | Instance metadata preserved            |
| User data             | No     | Cloud-init scripts are not re-executed |

***

## Next Steps

<CardGroup>
  <Card title="VM Templates" href="/services/compute/vm-templates">
    Convert instances to reusable templates for standardized deployments
  </Card>

  <Card title="Instance Snapshots" href="/services/compute/snapshots">
    Create point-in-time snapshots without full cloning
  </Card>

  <Card title="Create an Instance" href="/services/compute/launch-instance">
    Launch new instances from scratch using the 4-step wizard
  </Card>

  <Card title="Server Groups" href="/services/compute/server-groups">
    Use anti-affinity to distribute clones across hosts
  </Card>
</CardGroup>


# Cloud-init and Provisioning Scripts
Source: https://docs.xloud.tech/services/compute/cloud-init

Bootstrap virtual machines with Bash, PowerShell, cloud-config, or any cloud-init compatible script at first boot, and re-run scripts on running and discovered instances as part of an operational workflow.

## Overview

Xloud Compute supports automatic execution of customer-supplied scripts
at virtual machine launch time and at any later point during the
machine's lifecycle. The mechanism is the industry-standard **cloud-init**
agent on Linux guests and **Cloudbase-Init** on Windows guests — both
consume the same Xloud metadata service so the same launch flow works
for every supported OS.

<Info>
  **Two capabilities, one mechanism**

  1. **At provisioning** — the hypervisor executes Bash or PowerShell
     scripts during virtual machine creation to automate system
     bootstrapping operations (installing packages, configuring services,
     joining clusters, registering with monitoring, etc).

  2. **On provisioned and discovered instances** — the same scripting
     mechanism can be invoked on already-running and previously-imported
     virtual machines as part of an operational workflow (re-bootstrap,
     re-key, rotate credentials, patch in place, run a one-off task).
</Info>

<Note>
  **Prerequisites**

  * A guest image with `cloud-init` (Linux) or `Cloudbase-Init` (Windows)
    pre-installed. The Xloud public image catalog ships with both.
  * A network or metadata route from the instance to the Xloud metadata
    service (link-local `169.254.169.254` over the management network is
    enabled by default).
  * For post-provisioning execution: SSH (Linux) or WinRM (Windows)
    reachability from the orchestration host, or an attached qemu-guest-
    agent channel.
</Note>

***

## Supported Script Formats

The user-data payload is read once by the agent on first boot. The
**first line** of the payload tells the agent how to interpret the rest.

| Header line                                                    | Treated as                                              | Runs on                  |
| -------------------------------------------------------------- | ------------------------------------------------------- | ------------------------ |
| `#!/bin/bash` (or `#!/bin/sh`, `#!/usr/bin/env python3`, etc.) | A shebang script                                        | Linux (cloud-init)       |
| `#cloud-config`                                                | Declarative YAML — packages, users, files, run commands | Linux (cloud-init)       |
| `#include`                                                     | Fetches a script from a URL                             | Linux (cloud-init)       |
| `Content-Type: multipart/mixed` (MIME)                         | Multiple payloads in one upload                         | Linux (cloud-init)       |
| `#ps1` or `#ps1_sysnative`                                     | PowerShell script                                       | Windows (Cloudbase-Init) |
| `#cmd`                                                         | Classic Windows command prompt batch                    | Windows (Cloudbase-Init) |
| `#cloud-config`                                                | Declarative YAML (subset supported)                     | Windows (Cloudbase-Init) |

<Tip>
  When you need to run more than one script type at first boot — for
  example a `#cloud-config` to install packages **plus** a Bash script
  to run after — wrap them in a MIME multipart payload. The Dashboard
  accepts the full multipart blob in the User Data box.
</Tip>

***

## Provisioning a VM with a Script

The user-data input lives in the **System Config** step of the launch
wizard, under **Advanced Options → User Data**, and is mirrored by the
`--user-data` flag on the CLI.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the launch wizard">
        Navigate to **Compute → Instances** and click
        **Create Instance**. Complete the **Base Config** and
        **Network Config** steps as normal.
      </Step>

      <Step title="Expand Advanced Options">
        On the **System Config** step, scroll to the **Advanced Options**
        toggle and turn it on. The **User Data** textarea becomes
        visible.
      </Step>

      <Step title="Paste or upload the script">
        Paste the script directly into the textarea, or click the upload
        icon and choose a local `.sh`, `.yaml`, `.ps1`, or `.cmd` file.

        | Limit                | Value                                              |
        | -------------------- | -------------------------------------------------- |
        | Maximum payload size | 16 KB (before base64 encoding)                     |
        | Encoding             | UTF-8, ASCII safe                                  |
        | Forbidden characters | None — full Unicode allowed inside the script body |

        The Dashboard rejects payloads above the limit before submit.
      </Step>

      <Step title="Confirm and launch">
        Continue to **Confirm Config**, review, and click **Confirm**.
        The script begins executing inside the guest as soon as
        cloud-init or Cloudbase-Init reaches its `final` stage —
        typically within 30–90 seconds of the VM going `Active`.

        <Check>Cloud-init logs at `/var/log/cloud-init.log` (Linux) or `C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log` (Windows) confirm successful execution.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="Launch with a Bash bootstrap script" theme={null}
    openstack server create \
      --image <IMAGE_ID> \
      --flavor <FLAVOR_NAME> \
      --network <NETWORK_ID> \
      --key-name <KEYPAIR_NAME> \
      --user-data /path/to/bootstrap.sh \
      my-linux-vm
    ```

    ```bash title="Launch with a cloud-config YAML" theme={null}
    openstack server create \
      --image <IMAGE_ID> \
      --flavor <FLAVOR_NAME> \
      --network <NETWORK_ID> \
      --user-data /path/to/cloud-init.yaml \
      my-linux-vm
    ```

    ```bash title="Launch a Windows VM with a PowerShell bootstrap" theme={null}
    openstack server create \
      --image <WINDOWS_IMAGE_ID> \
      --flavor <FLAVOR_NAME> \
      --network <NETWORK_ID> \
      --user-data /path/to/bootstrap.ps1 \
      my-windows-vm
    ```

    The CLI auto-detects the format from the file's first line and
    base64-encodes the payload before submission, so you do not need to
    encode it yourself.
  </Tab>
</Tabs>

***

## Linux Bootstrap Examples (Bash and cloud-config)

### Bash script — install and start nginx

```bash title="bootstrap.sh" theme={null}
#!/bin/bash
set -euxo pipefail

# Wait for the package manager lock to clear
while pgrep -a apt > /dev/null; do sleep 2; done

apt-get update
apt-get install -y nginx

systemctl enable --now nginx

# Drop a marker file so monitoring can confirm bootstrap completed
echo "$(date -u +%FT%TZ) cloud-init bootstrap complete" > /var/log/xloud-bootstrap.done
```

### cloud-config YAML — declarative bootstrap

```yaml title="cloud-init.yaml" theme={null}
#cloud-config
package_update: true
package_upgrade: true

packages:
  - nginx
  - curl
  - git

users:
  - name: ops
    groups: sudo
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... ops@xloud

write_files:
  - path: /etc/xloud/site.conf
    permissions: "0644"
    content: |
      backend = production
      region  = ap-south-1

runcmd:
  - systemctl enable --now nginx
  - curl -fsSL https://repo.xloud.tech/setup.sh | bash
  - echo "bootstrap-done" > /var/log/xloud-bootstrap.done

final_message: "Xloud bootstrap finished after $UPTIME seconds"
```

<Tip>
  cloud-config is preferred over raw Bash for routine bootstrapping —
  it is declarative, idempotent, and the `packages:` and `runcmd:` keys
  read straightforwardly in change reviews. Use Bash only when you need
  imperative control flow (loops, retries, conditional logic).
</Tip>

***

## Windows Bootstrap Examples (PowerShell and cmd)

### PowerShell script — install IIS and write a marker

```powershell title="bootstrap.ps1" theme={null}
#ps1_sysnative
$ErrorActionPreference = "Stop"

# Install the Web Server role with management tools
Install-WindowsFeature -Name Web-Server -IncludeManagementTools

# Replace the default page so health checks see the right banner
Set-Content -Path C:\inetpub\wwwroot\index.html `
  -Value "<h1>Xloud-managed IIS host</h1>" -Encoding UTF8

# Disk-level marker the monitoring agent picks up
$now = Get-Date -Format "yyyy-MM-ddTHH:mm:ssZ"
"$now Xloud bootstrap complete" | Out-File C:\xloud-bootstrap.done -Encoding UTF8
```

<Note>
  `#ps1_sysnative` runs the script through the native 64-bit PowerShell
  on 64-bit Windows guests, avoiding the 32-bit WoW64 redirector. Use
  `#ps1` if you specifically need the 32-bit engine.
</Note>

### Classic command prompt — register the VM with internal DNS

```bat title="bootstrap.cmd" theme={null}
#cmd
@echo off
nslookup %COMPUTERNAME%.corp.local
nltest /dsregdns
echo Xloud bootstrap complete > C:\xloud-bootstrap.done
```

### cloud-config on Windows — restricted subset

```yaml title="cloud-init.yaml" theme={null}
#cloud-config
users:
  - name: admin
    passwd: ChangeMeAtFirstLogin!
    groups: Administrators

set_hostname: web-01.corp.local

runcmd:
  - powershell -Command "Install-WindowsFeature Web-Server"
```

<Warning>
  Not all `cloud-config` keys are honored by Cloudbase-Init. `users`,
  `set_hostname`, `runcmd`, and `write_files` are well supported.
  `apt`-specific keys, systemd unit management, and Linux-only modules
  are ignored. Test your YAML on a throwaway VM before rolling it out
  to a production fleet.
</Warning>

***

## Running Scripts on Provisioned or Discovered Instances

For operational workflows that fire **after** a VM has been provisioned
— or on instances that were imported into Xloud through migration or
discovery — the same scripting mechanism is available. Choose the path
that fits your security and connectivity model.

<AccordionGroup>
  <Accordion title="Rebuild with new user-data (Bash or PowerShell)">
    The Compute API's **rebuild** action re-applies a fresh user-data
    payload on the next boot of an existing instance. The instance
    keeps its UUID, IP, security groups, and metadata; only the disk
    image and user-data are re-applied.

    ```bash title="Re-run bootstrap on an existing instance" theme={null}
    openstack server rebuild <INSTANCE_ID> \
      --image <SAME_IMAGE_ID> \
      --user-data /path/to/new-bootstrap.sh
    ```

    Works for both Linux and Windows guests. The user-data format is
    the same as at first launch — Bash, cloud-config, PowerShell, or
    cmd, identified by the first line of the payload.
  </Accordion>

  <Accordion title="Ansible against the instance IP">
    The recommended path for ongoing operational workflows. The
    instance's floating IP (or fixed IP if the orchestrator runs inside
    the cloud) is the target. Ansible's `shell` module handles Bash
    payloads on Linux, and `win_shell` / `win_command` handle Bash-via-
    WSL or PowerShell on Windows.

    ```yaml title="ops-playbook.yaml" theme={null}
    - hosts: web-tier
      tasks:
        - name: Rotate the deploy key
          ansible.builtin.shell: |
            sudo /opt/xloud/rotate-key.sh
          when: ansible_os_family != "Windows"

        - name: Rotate the deploy key on Windows hosts
          ansible.windows.win_powershell:
            script: C:\Program Files\Xloud\rotate-key.ps1
          when: ansible_os_family == "Windows"
    ```

    XAVS Deployment Automation already includes the inventory plugin
    that pulls instance lists straight from Xloud, so a fresh
    `ansible-playbook` run picks up newly-launched VMs without manual
    inventory editing.
  </Accordion>

  <Accordion title="qemu-guest-agent (hypervisor-side execution)">
    When an instance has the **qemu-guest-agent** package installed and
    the `hw_qemu_guest_agent=yes` property set on its source image, the
    administrator can send commands directly through the virtio-serial
    channel — no SSH or WinRM is required. This is the path that
    satisfies "execute scripts on a discovered VM" when the VM has no
    public network reachability.

    ```bash title="Execute a one-shot command via the guest agent" theme={null}
    virsh qemu-agent-command instance-00000abc \
      '{"execute":"guest-exec","arguments":{
         "path":"/bin/sh",
         "arg":["-c","systemctl restart nginx"],
         "capture-output":true}}'
    ```

    The same channel supports `guest-exec` on Windows guests for
    PowerShell payloads.
  </Accordion>

  <Accordion title="Re-run cloud-init on the next boot">
    For situations where the VM already has cloud-init installed and
    you just want it to re-process its user-data (for example, after
    editing the metadata via API), connect to the guest and clear the
    state:

    ```bash title="Linux (cloud-init)" theme={null}
    sudo cloud-init clean --logs
    sudo reboot
    ```

    On the next boot the agent re-fetches user-data from the metadata
    service and re-runs the bootstrap. Cloudbase-Init exposes a similar
    `cloudbase-init --reset-service-password` style flow.
  </Accordion>
</AccordionGroup>

***

## Verification

| Layer                                  | How to check                                                                | Expected                                  |
| -------------------------------------- | --------------------------------------------------------------------------- | ----------------------------------------- |
| **Cloud-init invoked**                 | `sudo cloud-init status --long` (Linux)                                     | `status: done`                            |
| **Cloudbase-Init invoked**             | Inspect `cloudbase-init.log`                                                | Final line reads `Plugins execution done` |
| **User-data was visible to the guest** | `curl http://169.254.169.254/openstack/latest/user_data` from inside the VM | Returns the exact payload you submitted   |
| **Script ran**                         | Check the marker file written by your script                                | File exists and is recent                 |
| **Errors during run**                  | `sudo cloud-init analyze show` (Linux) or `cloudbase-init.log` (Windows)    | No `ERROR` lines                          |

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="The script never ran">
    The guest image probably has cloud-init or Cloudbase-Init disabled
    or uninstalled. Boot a fresh VM from the same image, run
    `which cloud-init` (Linux) or check for the **Cloudbase-Init**
    service in `services.msc` (Windows). If absent, rebuild the image
    or pick one from the Xloud public catalog.
  </Accordion>

  <Accordion title="The script ran but failed silently">
    Cloud-init swallows non-zero exit codes from the user script by
    default. Set `set -euxo pipefail` at the top of a Bash payload and
    `$ErrorActionPreference = "Stop"` at the top of a PowerShell payload
    to force errors to surface. Then re-run with the rebuild action
    above and check the log files listed in **Verification**.
  </Accordion>

  <Accordion title="Payload too large — 16 KB limit">
    The metadata service caps user-data at 16 KB raw (before base64).
    For larger payloads, host the script on an internal URL and use
    `#include https://repo.internal/bootstrap.sh` as the user-data; the
    agent fetches and executes it inline.
  </Accordion>

  <Accordion title="PowerShell complains about execution policy">
    Cloudbase-Init bypasses execution policy when running scripts it
    fetches from the metadata service, so this is rare. If you see
    `cannot be loaded because running scripts is disabled`, the script
    is being launched by something *other* than Cloudbase-Init — check
    Scheduled Tasks or third-party agents that may have been baked into
    the image.
  </Accordion>

  <Accordion title="The user-data ran once but I want it to run on every boot">
    Cloud-init runs `runcmd` and shell scripts once per **instance ID**.
    For per-boot execution use `bootcmd:` in `#cloud-config`, or
    schedule a systemd unit that fires on boot. On Windows, set the
    Cloudbase-Init plugin policy to `BootStatusPolicy = AlwaysRun` in
    `cloudbase-init.conf`.
  </Accordion>
</AccordionGroup>

***

## Security Considerations

* **Treat user-data as code, not config.** It runs as root on Linux and
  as Administrator on Windows. Anyone with `compute:create_server`
  permission can submit it.
* **Do not embed secrets in user-data.** The metadata service is reachable
  by every process inside the guest. Use Xloud Key Management for
  secrets and fetch them at runtime over an authenticated channel.
* **Network reachability of the metadata service.** Restrict the
  metadata route in tenant security groups when an instance is moved
  to a hardened production segment — cloud-init will not need it
  after first boot.
* **Audit script content.** XAVS Deployment Automation's `--check`
  mode renders the playbook diff before execution. Use it for any
  multi-host operational workflow.

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" icon="server" href="/services/compute/launch-instance">
    Full launch-wizard reference including every System Config field.
  </Card>

  <Card title="Linux VM Guide" icon="terminal" href="/services/compute/linux-vm-guide">
    Boot, login, and lifecycle for Linux guests.
  </Card>

  <Card title="Windows VM Guide" icon="monitor" href="/services/compute/windows-vm-guide">
    Cloudbase-Init, RDP setup, and Windows-specific notes.
  </Card>

  <Card title="Instance Snapshots" icon="camera" href="/services/compute/snapshots">
    Capture a known-good state after bootstrap finishes.
  </Card>

  <Card title="Instance Rollback" icon="rotate-ccw" href="/services/compute/instance-rollback">
    Revert a bad bootstrap to a clean snapshot.
  </Card>

  <Card title="Instance Tagging" icon="tag" href="/services/compute/instance-tagging">
    Tag VMs so operational scripts can target them by group.
  </Card>
</CardGroup>


# Compute Hosts
Source: https://docs.xloud.tech/services/compute/compute-hosts

List, inspect, enable, and disable hypervisor hosts in Xloud Compute. Monitor real-time vCPU, memory, and instance utilization across the cluster.

## Overview

The Compute Hosts page provides a real-time view of all hypervisor nodes in the cluster.
Administrators use it to monitor resource utilization, identify overcommitted hosts, plan
capacity, and manage host availability for maintenance operations.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator access to the Xloud Dashboard (admin view)
</Note>

***

## View Hypervisors

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Hypervisors** in the admin sidebar. The page has two tabs:

    **Hypervisor tab** — lists all compute nodes:

    | Column                      | Description                                                    |
    | --------------------------- | -------------------------------------------------------------- |
    | **Hostname**                | Hypervisor hostname (clickable to view details)                |
    | **Type**                    | Hypervisor type (e.g., QEMU, Ironic)                           |
    | **VCPU (Core)**             | Used/Total vCPUs with progress bar (shows `-` for bare metal)  |
    | **Configured Memory (GiB)** | Used/Total memory with progress bar (shows `-` for bare metal) |
    | **Instances**               | Number of running instances on this host                       |

    Filter by **Hostname** or **Type**.

    **Compute Host tab** — lists compute services:

    | Column                | Description                                        |
    | --------------------- | -------------------------------------------------- |
    | **Host**              | Service hostname                                   |
    | **Availability Zone** | Zone assignment                                    |
    | **Service Status**    | Enabled or Disabled (with disabled reason tooltip) |
    | **Service State**     | Up or Down                                         |
    | **Last Updated**      | Last heartbeat timestamp                           |

    Filter by **Host**, **Service Status**, or **Service State**.

    **Compute Host actions**:

    | Action      | Description                                                    |
    | ----------- | -------------------------------------------------------------- |
    | **Disable** | Disable the compute service (prevents new instance scheduling) |
    | **Enable**  | Re-enable a disabled compute service                           |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="List all hypervisors" theme={null}
      openstack hypervisor list
      ```

      ```bash title="Show hypervisor details" theme={null}
      openstack hypervisor show <HOSTNAME>
      ```

      ```bash title="List compute services" theme={null}
      openstack compute service list
      ```

      ```bash title="Disable a compute host" theme={null}
      openstack compute service set --disable --disable-reason "Maintenance" <HOST> nova-compute
      ```

      ```bash title="Enable a compute host" theme={null}
      openstack compute service set --enable <HOST> nova-compute
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Hypervisor Detail

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a hypervisor hostname in the list to open the detail page.

    The header shows: **Hostname**, **Type**, **VCPU** (used/total), **Memory**
    (used/total), and **VGPU** (used/total, if GPU resources exist).

    The **Members** tab shows all instances running on this hypervisor.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show hypervisor stats" theme={null}
    openstack hypervisor show <HOSTNAME>
    ```

    ```bash title="List instances on a hypervisor" theme={null}
    openstack server list --all-projects --host <HOSTNAME>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Live Migration" href="/services/compute/live-migration">
    Move instances off a host before maintenance
  </Card>

  <Card title="Scheduling" href="/services/compute/scheduling">
    Configure host aggregates and scheduling policies
  </Card>

  <Card title="Availability Zones" href="/services/compute/availability-zones">
    Organize hosts into fault domains
  </Card>

  <Card title="Quotas" href="/services/compute/quotas">
    Set per-project resource limits
  </Card>
</CardGroup>


# Remote Console Access
Source: https://docs.xloud.tech/services/compute/console-access

Access instances via browser-based VNC console from the Dashboard. Use for out-of-band management and boot-time debugging.

## Overview

The Xloud Dashboard provides browser-based VNC console access to instances. The console
opens in a new browser tab and provides direct keyboard and mouse interaction with the
guest OS — useful for troubleshooting network issues, configuring instances that cannot
be reached via SSH, or observing the boot process.

<Note>
  **Prerequisites**

  * An instance in `Active` status
  * The instance must not be a bare metal instance
  * A modern web browser with JavaScript enabled
</Note>

***

## Open the Console

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Instances">
        Navigate to **Compute > Instances** in the sidebar.
      </Step>

      <Step title="Open the console">
        Click the **Console** action on the instance row — this is the **first
        action** (directly visible button, not under the More dropdown).

        A confirmation dialog appears. Click **Confirm** to open the VNC console
        in a new browser tab.

        <Note>
          The Console action is only available for instances in `Active` status.
          Bare metal instances do not support VNC console access.
        </Note>
      </Step>

      <Step title="Interact with the instance">
        The console provides full keyboard and mouse input. Use it to:

        * Log in to the guest OS when SSH is unavailable
        * Observe boot messages and GRUB configuration
        * Debug network configuration issues
        * Access Windows instances via the graphical interface

        <Tip>
          For Windows instances, the VNC console provides RDP-like graphical access
          directly in the browser without requiring a separate RDP client.
        </Tip>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="Get VNC console URL" theme={null}
    openstack console url show <INSTANCE_ID>
    ```

    Open the returned URL in a browser to access the VNC console.

    ```bash title="View serial console log" theme={null}
    openstack console log show <INSTANCE_ID> | tail -50
    ```

    <Tip>
      Use the serial console log to review boot messages and diagnose startup
      issues without opening a VNC session.
    </Tip>
  </Tab>
</Tabs>

***

## Instance Detail — Logs Tab

The instance detail page includes a **Logs** tab that displays the serial console
output. Navigate to **Compute > Instances**, click the instance name, and select
the **Logs** tab. This provides the same information as `openstack console log show`
without needing CLI access.

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Create a new instance with the appropriate configuration
  </Card>

  <Card title="Manage IP Addresses" href="/services/compute/manage-ips">
    Configure floating IPs for SSH access alongside console
  </Card>

  <Card title="Reboot an Instance" href="/services/compute/reboot-instance">
    Restart an unresponsive instance before using the console
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve console connection issues and black screens
  </Card>
</CardGroup>


# Flavor Management
Source: https://docs.xloud.tech/services/compute/flavors

Create and manage compute flavors using the 2-step wizard. Define vCPU, RAM, disk, GPU, NUMA, and access control.

## Overview

Flavors define the virtual hardware profile for instances — vCPU count, memory allocation,
root disk size, and optional hardware extensions like GPU and NUMA pinning. Administrators
manage the flavor catalog. Users select flavors when launching instances.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator access to the Xloud Dashboard (admin view)
</Note>

***

## View Flavors

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Flavors** in the admin sidebar. The list shows:

    | Column                         | Description                                                |
    | ------------------------------ | ---------------------------------------------------------- |
    | **ID/Name**                    | Flavor identifier (clickable to view details)              |
    | **Category**                   | General Purpose, Compute Optimized, Memory Optimized, etc. |
    | **CPU**                        | Number of vCPUs                                            |
    | **Memory**                     | RAM allocation (formatted as GiB)                          |
    | **Internal Network Bandwidth** | Network throughput in Gbps                                 |
    | **Ephemeral Disk**             | Temporary scratch disk size (GiB)                          |
    | **Storage IOPS**               | I/O operations per second limit                            |
    | **Public**                     | Whether the flavor is available to all projects            |

    Filter by **Name**, **CPU**, **Memory**, or **Category**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="List all flavors" theme={null}
      openstack flavor list --all
      ```

      ```bash title="Show flavor details" theme={null}
      openstack flavor show <FLAVOR_NAME>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Create a Flavor

The Dashboard provides a 2-step wizard: **Params Setting** and **Access Type Setting**.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Create Flavor wizard">
        Navigate to **Compute > Flavors** in the admin sidebar.
        Click **Create Flavor**.
      </Step>

      <Step title="Step 1 — Params Setting">
        Configure the virtual hardware profile.

        **Architecture** (required) — Select the hardware architecture:

        | Architecture            | Description                      |
        | ----------------------- | -------------------------------- |
        | X86 Architecture        | Standard x86-64 virtual machines |
        | Heterogeneous Computing | GPU-accelerated instances        |
        | Bare Metal              | Physical server provisioning     |
        | ARM Architecture        | ARM-based instances              |

        **Category** (required) — Depends on the selected architecture:

        * X86: General Purpose, Compute Optimized, Memory Optimized, Big Data, Local SSD, High Clock Speed
        * Heterogeneous: GPU Compute, Visualization Compute

        **Basic parameters**:

        | Field            | Type   | Required | Description                |
        | ---------------- | ------ | -------- | -------------------------- |
        | **Name**         | Text   | Yes      | Flavor name                |
        | **vCPUs**        | Number | Yes      | Virtual CPU count (min: 1) |
        | **Memory (GiB)** | Number | Yes      | RAM allocation (min: 1)    |

        **Hot-Add Configuration** (hidden for Bare Metal):

        | Field                    | Description                                         |
        | ------------------------ | --------------------------------------------------- |
        | **Enable Hot-Add**       | Yes/No — enables live vCPU/RAM scaling              |
        | **Minimum CPU**          | Minimum vCPUs at boot (required if Hot-Add enabled) |
        | **Minimum Memory (GiB)** | Minimum RAM at boot (required if Hot-Add enabled)   |

        <Info>
          **Xloud-Developed** — Hot-Add configuration for live vCPU and RAM scaling
          is developed by Xloud and ships with XAVS / XPCI.
        </Info>

        **Storage & Bandwidth** (varies by architecture):

        | Field              | Visibility                        | Description                       |
        | ------------------ | --------------------------------- | --------------------------------- |
        | **Bandwidth**      | Hidden for Bare Metal             | Internal network bandwidth (Gbps) |
        | **Ephemeral Disk** | When applicable                   | Temporary scratch disk (GiB)      |
        | **Root Disk**      | Hidden when block storage enabled | Root disk size (GiB)              |
        | **IOPS**           | Hidden for Bare Metal             | Storage I/O limit                 |

        **NUMA Configuration** (for non-compute-optimized, non-bare-metal):

        | Field                | Description                        |
        | -------------------- | ---------------------------------- |
        | **NUMA Nodes**       | Number of NUMA nodes               |
        | **Memory Page Size** | large, small, any, or custom value |

        **GPU Parameters** (only for GPU categories):

        | Field          | Description                       |
        | -------------- | --------------------------------- |
        | **GPU Type**   | Select from configured GPU models |
        | **GPU Number** | Number of GPUs (min: 1)           |

        **Compute Optimized Parameters** (only for Compute Optimized category):

        | Field                 | Description                                                         |
        | --------------------- | ------------------------------------------------------------------- |
        | **NUMA Nodes**        | Per-node CPU and RAM allocation (total must match flavor vCPUs/RAM) |
        | **CPU Policy**        | `dedicated` or `shared`                                             |
        | **CPU Thread Policy** | `prefer`, `isolate`, or `require`                                   |
        | **Memory Page Size**  | `large`, `small`, `any`, or custom                                  |
      </Step>

      <Step title="Step 2 — Access Type Setting">
        Configure which projects can use this flavor.

        | Option             | Description                               |
        | ------------------ | ----------------------------------------- |
        | **Public**         | Available to all projects on the platform |
        | **Access Control** | Restricted to selected projects only      |

        When **Access Control** is selected, a project table appears. Select one
        or more projects to grant access.
      </Step>

      <Step title="Create the flavor">
        Click **Confirm** to create the flavor.

        <Check>Flavor appears in the list and is available for instance creation.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Create a basic flavor" theme={null}
      openstack flavor create \
        --vcpus 4 --ram 8192 --disk 80 \
        --public \
        m1.large
      ```

      ```bash title="Create a private flavor" theme={null}
      openstack flavor create \
        --vcpus 8 --ram 16384 --disk 160 \
        --private \
        m1.xlarge

      # Grant access to specific project
      openstack flavor set --project <PROJECT_ID> m1.xlarge
      ```

      ```bash title="Create flavor with extra specs (NUMA, CPU pinning)" theme={null}
      openstack flavor create \
        --vcpus 4 --ram 8192 --disk 0 \
        --public \
        compute-optimized.large

      openstack flavor set \
        --property hw:cpu_policy=dedicated \
        --property hw:cpu_thread_policy=prefer \
        --property hw:mem_page_size=large \
        --property hw:numa_nodes=1 \
        compute-optimized.large
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Flavor Detail

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a flavor name in the list. The detail page shows:

    **Detail tab** — Flavor specifications:

    * Base Info: Network Bandwidth, Ephemeral Disk, IOPS, NUMA config
    * GPU Info (if applicable): GPU type and count
    * Compute Optimized Info (if applicable): Per-NUMA-node CPU/RAM, CPU/thread policy, page size
    * Extra Specs: Full JSON of all extra specifications

    **Instances tab** — All instances currently using this flavor.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Show flavor details" theme={null}
      openstack flavor show <FLAVOR_NAME>
      ```

      ```bash title="List extra specs" theme={null}
      openstack flavor show <FLAVOR_NAME> -c properties
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Delete a Flavor

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click the **Delete** action (first row action) on the flavor row. Confirm deletion.

    <Danger>
      Existing instances using the deleted flavor continue running but cannot be
      resized. Create a replacement flavor before deleting.
    </Danger>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a flavor" theme={null}
    openstack flavor delete <FLAVOR_NAME>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Use flavors when creating instances in the 4-step wizard
  </Card>

  <Card title="Live vCPU/RAM Scaling" href="/services/compute/live-resize-admin">
    Configure Hot-Add flavors for zero-downtime scaling
  </Card>

  <Card title="Quotas" href="/services/compute/quotas">
    Set per-project limits on vCPU, RAM, and instance counts
  </Card>

  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    Monitor hypervisor resources and host availability
  </Card>
</CardGroup>


# Hypervisor Configuration
Source: https://docs.xloud.tech/services/compute/hypervisor

Configure the native hypervisor driver for Xloud Compute. Covers image formats, CPU modes, backing storage, nested virtualization, and performance tuning.

## Overview

Xloud Compute uses a native hypervisor as its default virtualization layer, managed through the libvirt driver. The native hypervisor provides hardware-assisted virtualization on x86, ARM, and POWER architectures, delivering near-bare-metal performance for virtual machine workloads.

The libvirt driver translates Xloud Compute API requests into hypervisor operations on each compute node. Understanding these configuration options helps you tune workload placement, CPU topology, storage performance, and security posture.

<Note>
  **Prerequisites**

  * Administrator access to the Xloud platform and XDeploy
  * Compute nodes running XOS with `libvirt` and `qemu-kvm` installed
  * Verify hardware virtualization support: `grep -o 'vmx\|svm' /proc/cpuinfo | head -1`
</Note>

***

## Supported Image Formats

The following disk image formats are supported by the native hypervisor driver. The format is detected automatically from the image metadata stored in the Xloud Image Service.

| Format  | Name                  | Description                                                                    | Recommended Use                                          |
| ------- | --------------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------- |
| `raw`   | Raw disk image        | Flat binary representation of disk contents. No overhead from format features. | Maximum I/O performance, RBD-backed volumes              |
| `qcow2` | QEMU Copy-on-Write v2 | Supports snapshots, compression, and copy-on-write. More flexible than raw.    | Local storage, development environments                  |
| `qed`   | QEMU Enhanced Disk    | Optimized for sparse images with faster lookup tables than qcow2.              | Legacy workloads; qcow2 is preferred for new deployments |
| `vmdk`  | VMware Disk           | VMware-compatible format. Supported for import/migration scenarios.            | VM migrations from VMware environments                   |

<Tip>
  For production deployments using XSDS (distributed storage) as the storage backend, use `raw` format images. The distributed storage layer handles copy-on-write natively, making qcow2 overhead unnecessary and counterproductive.
</Tip>

***

## Hardware Requirements

<Tabs>
  <Tab title="x86 / x86_64" icon="microchip">
    x86 is the primary supported architecture for Xloud Compute.

    **CPU Requirements**

    * Intel VT-x (`vmx` flag in `/proc/cpuinfo`) or AMD-V (`svm` flag)
    * For optimal performance: Intel VT-d or AMD-Vi (IOMMU) for PCI passthrough

    **Verification**

    ```bash title="Check virtualization support" theme={null}
    grep -oE 'vmx|svm' /proc/cpuinfo | sort -u
    ```

    ```bash title="Check IOMMU (for PCI passthrough)" theme={null}
    dmesg | grep -i iommu
    ```

    **BIOS/UEFI Settings**

    * Enable **Intel VT-x / AMD-V** in BIOS
    * Enable **Intel VT-d / AMD-Vi** if PCI passthrough is required
    * Enable **Hyper-Threading** for improved vCPU density (optional)

    <Check>Both `vmx` or `svm` flag and IOMMU should be present for full feature support.</Check>
  </Tab>

  <Tab title="ARM / AArch64" icon="cpu">
    ARM 64-bit (AArch64) is supported for compute nodes running compatible hardware.

    **CPU Requirements**

    * ARMv8-A or later with hardware virtualization extensions (EL2)
    * PSCI firmware support for CPU hotplug operations

    **Verification**

    ```bash title="Check ARM virtualization support" theme={null}
    grep -i 'Features' /proc/cpuinfo | grep -i asimd
    ```

    <Warning>
      Not all ARM platforms support all hypervisor features. Verify your hardware's EL2 support before deploying production workloads.
    </Warning>
  </Tab>

  <Tab title="POWER / ppc64le" icon="zap">
    IBM POWER (ppc64le) is supported with the native hypervisor driver for POWER.

    **CPU Requirements**

    * POWER8 or later for full hardware virtualization
    * POWER7 supports paravirtualized mode (reduced performance)

    **Verification**

    ```bash title="Check POWER virtualization mode" theme={null}
    grep -i 'platform' /proc/cpuinfo
    ```

    <Note>Full hardware virtualization on POWER requires bare-metal POWER hardware — nested virtualization is not supported in this mode.</Note>
  </Tab>
</Tabs>

***

## Backing Storage Options

The hypervisor driver supports multiple backing storage configurations for instance disks. The backing storage determines how ephemeral instance disks are stored on compute nodes.

| Storage Type         | Description                                   | Pros                                           | Cons                                      |
| -------------------- | --------------------------------------------- | ---------------------------------------------- | ----------------------------------------- |
| **QCOW** (local)     | qcow2 files on compute node local storage     | Simple setup, copy-on-write snapshots          | No live migration, no fault tolerance     |
| **Flat** (raw local) | Raw image files on compute node local storage | Maximum local I/O performance                  | No live migration, no fault tolerance     |
| **LVM**              | Logical volumes on compute node volume groups | Better I/O than file-backed, thin provisioning | Complex setup, no live migration          |
| **RBD** (XSDS)       | Distributed block device via network          | Live migration, fault tolerance, snapshots     | Requires XSDS distributed storage cluster |

<Warning>
  Local storage backends (QCOW, Flat, LVM) do not support live migration. Use RBD-backed storage when live migration between compute nodes is required.
</Warning>

The storage backend is configured per compute node in the Nova configuration, managed by XDeploy via the xavs-ansible deployment playbooks.

***

## CPU Configuration Modes

The CPU mode controls how CPU features and topology are presented to virtual machines. This affects live migration compatibility and performance.

| CPU Mode           | Description                                   |        Live Migration       | Performance | Use Case                                              |
| ------------------ | --------------------------------------------- | :-------------------------: | :---------: | ----------------------------------------------------- |
| `host-passthrough` | Exposes exact host CPU model and all features |   Requires identical CPUs   |     Best    | Homogeneous clusters, bare-metal benchmarks           |
| `host-model`       | Snapshots the host CPU model at VM launch     |  Restricted to similar CPUs | Near-native | Clusters with similar CPU generations                 |
| `custom`           | Specifies an explicit baseline CPU model      | Cross-generation compatible |   Reduced   | Mixed-CPU clusters, live migration across generations |
| `none`             | QEMU default — minimal feature set            |          Compatible         |    Lowest   | Legacy compatibility only; not recommended            |

<Tip>
  For mixed-CPU clusters (e.g., nodes with Intel Icelake and Cascadelake CPUs), use `cpu_mode = custom` with a common baseline model such as `Cascadelake-Server-noTSX`. This ensures live migration succeeds across all nodes in the cluster.
</Tip>

### Configuring CPU Mode

CPU mode is set in the Nova compute configuration, which XDeploy manages via `globals.d` overrides:

```yaml title="/etc/xavs/globals.d/_50_compute.yml" theme={null}
nova_cpu_mode: "custom"
nova_cpu_model: "Cascadelake-Server-noTSX"
```

Apply the change with:

```bash title="Apply compute configuration" theme={null}
xavs-ansible deploy -t nova
```

***

## Nested Virtualization

Nested virtualization allows virtual machines to run their own hypervisors (e.g., for CI/CD pipelines, hypervisor testing, or running Kubernetes with virtualization-backed nodes).

### Enabling Nested Virtualization

<Steps>
  <Step title="Enable on the host kernel" icon="terminal">
    Load the hypervisor module with nested support enabled:

    <CodeGroup>
      ```bash title="Intel" theme={null}
      echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
      sudo modprobe -r kvm_intel && sudo modprobe kvm_intel
      ```

      ```bash title="AMD" theme={null}
      echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
      sudo modprobe -r kvm_amd && sudo modprobe kvm_amd
      ```
    </CodeGroup>
  </Step>

  <Step title="Verify nested support is active" icon="circle-check">
    ```bash title="Verify nested virtualization" theme={null}
    cat /sys/module/kvm_intel/parameters/nested   # Should return Y or 1
    cat /sys/module/kvm_amd/parameters/nested     # AMD alternative
    ```

    <Check>Output should be `Y` or `1` confirming nested virtualization is enabled.</Check>
  </Step>

  <Step title="Set CPU mode to host-passthrough" icon="settings">
    Nested VMs require the host CPU feature flags to be visible inside the VM. Set `cpu_mode = host-passthrough` in the Nova configuration, or use `host-model` if cross-node migration is needed.

    ```yaml title="globals.d override for nested virtualization" theme={null}
    nova_cpu_mode: "host-passthrough"
    ```
  </Step>
</Steps>

<Warning>
  **Nested Virtualization Limitations**

  * Performance is significantly reduced compared to first-level VMs due to double emulation overhead
  * Live migration of nested VMs may not be supported depending on the inner hypervisor
  * Not recommended for production workloads — use dedicated bare-metal nodes for performance-sensitive nested environments
  * `host-passthrough` CPU mode restricts live migration to nodes with identical physical CPUs
</Warning>

***

## Performance Tuning

### VHostNet

VHostNet offloads virtio-net packet processing from QEMU user-space to the kernel, significantly reducing CPU overhead for network-intensive workloads.

```bash title="Verify VHostNet is loaded" theme={null}
lsmod | grep vhost_net
```

VHostNet is enabled by default on XOS. No additional configuration is required.

### CPU Pinning

For latency-sensitive workloads, pin instance vCPUs to dedicated physical cores to eliminate CPU scheduler jitter. See the [Advanced Features](/services/compute/advanced-features) guide for CPU pinning configuration.

### Huge Pages

Configure huge page memory backing for memory-intensive or NUMA-sensitive workloads. Huge pages reduce TLB pressure and improve memory throughput. See the [Advanced Features](/services/compute/advanced-features) guide for huge page setup.

***

## Capabilities

<CardGroup>
  <Card title="Advanced Features" icon="wand" href="/services/compute/advanced-features">
    CPU pinning, huge pages, NUMA topology, GPU passthrough, and SR-IOV configuration
  </Card>

  <Card title="Live Migration" icon="arrow-right-left" href="/services/compute/live-migration">
    Configure and execute live migrations between compute nodes
  </Card>

  <Card title="Compute Architecture" icon="layers" href="/services/compute/architecture">
    Understand the full Xloud Compute architecture and service components
  </Card>

  <Card title="Security Hardening" icon="shield" href="/services/compute/security-hardening">
    Hypervisor-level security configuration and CIS compliance hardening
  </Card>
</CardGroup>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="VMs fail to start: hypervisor not available" icon="triangle-alert">
    **Cause**: Hardware virtualization is disabled in BIOS or the hypervisor kernel module is not loaded.

    **Resolution**:

    ```bash title="Check hypervisor module status" theme={null}
    lsmod | grep kvm
    ```

    ```bash title="Load hypervisor modules manually" theme={null}
    sudo modprobe kvm
    sudo modprobe kvm_intel   # or kvm_amd for AMD CPUs
    ```

    If the module fails to load, enable VT-x/AMD-V in the server BIOS and reboot.
  </Accordion>

  <Accordion title="Live migration fails: 'guest CPU doesn't match specification'" icon="arrow-right-left">
    **Cause**: The source and destination compute nodes have incompatible CPU models. This commonly occurs in mixed-CPU clusters when `host-model` or `host-passthrough` is used.

    **Resolution**: Switch to `custom` CPU mode with a common baseline:

    ```yaml title="globals.d fix" theme={null}
    nova_cpu_mode: "custom"
    nova_cpu_model: "Cascadelake-Server-noTSX"
    ```

    Apply with `xavs-ansible deploy -t nova` on all nodes, then retry the migration.
  </Accordion>

  <Accordion title="Poor disk I/O performance on RBD-backed instances" icon="hard-drive">
    **Cause**: qcow2 format images are being used with XSDS distributed storage, adding unnecessary copy-on-write overhead.

    **Resolution**: Use `raw` format for all images on XSDS-backed deployments. Convert existing images:

    ```bash title="Convert qcow2 to raw" theme={null}
    openstack image create \
      --disk-format raw \
      --container-format bare \
      --file <(qemu-img convert -f qcow2 -O raw source.qcow2 /dev/stdout) \
      my-raw-image
    ```
  </Accordion>

  <Accordion title="Nested VMs fail to start inside a guest" icon="boxes">
    **Cause**: The host compute node does not have nested virtualization enabled, or the CPU mode does not expose virtualization feature flags to the guest.

    **Resolution**:

    1. Verify nested support: `cat /sys/module/kvm_intel/parameters/nested`
    2. Confirm the instance is using a flavor with `hw:cpu_mode=host-passthrough` or equivalent
    3. Verify the guest OS can see the virtualization flag: `grep -c vmx /proc/cpuinfo` from inside the VM
  </Accordion>

  <Accordion title="libvirtd not running on compute node" icon="server">
    **Cause**: The `libvirtd` service failed to start or crashed.

    **Resolution**:

    ```bash title="Check libvirtd status" theme={null}
    sudo systemctl status libvirtd
    sudo journalctl -u libvirtd -n 50
    ```

    Common causes include AppArmor policy conflicts and missing QEMU binaries. Review the journal output for the specific error and consult the Xloud support portal.
  </Accordion>
</AccordionGroup>

***

## Heterogeneous Hardware Support

<Info>**Xloud-Developed** — Heterogeneous hardware support is a core capability of XAVS / XPCI.</Info>

Xloud supports mixing different hardware configurations within a single cluster -- no hardware homogeneity required.

<CardGroup>
  <Card title="Mixed Node Types" icon="server">
    Run converged (compute+storage), compute-only, and storage-heavy nodes in the same cluster. Each node contributes its resources to the shared pool.
  </Card>

  <Card title="Mixed CPU Generations" icon="microchip">
    Intel and AMD processors of different generations coexist. CPU feature masking ensures live migration compatibility across generations. See [CPU Feature Masking](/services/compute/advanced-features).
  </Card>

  <Card title="Mixed Storage Media" icon="hard-drive">
    NVMe, SSD, and HDD drives in the same cluster. CRUSH device classes auto-detect media type per device and route data to the correct tier. See [Storage Tiers](/services/storage/storage-tiers).
  </Card>

  <Card title="Varying RAM Capacities" icon="hard-drive">
    Nodes with different RAM sizes -- no configuration needed. The scheduler tracks per-host capacity independently and places instances on hosts with sufficient resources.
  </Card>
</CardGroup>

<Tip>
  Use **Host Aggregates** to group nodes by capability (e.g., GPU hosts, high-memory hosts) and restrict specific flavors to specific hardware groups. See [Scheduling](/services/compute/scheduling).
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Advanced Features" href="/services/compute/advanced-features">
    Configure CPU pinning, huge pages, and GPU passthrough for specialized workloads
  </Card>

  <Card title="Live Migration" href="/services/compute/live-migration">
    Set up and execute live migrations across compute nodes
  </Card>

  <Card title="Compute Admin Guide" href="/services/compute/admin-guide">
    Full administrator reference for the Xloud Compute service
  </Card>

  <Card title="XAVS Product" href="/products/xavs">
    Learn about the full XAVS Advanced Virtualization Suite
  </Card>
</CardGroup>


# Instance Rollback
Source: https://docs.xloud.tech/services/compute/instance-rollback

Restore a running or stopped instance to a previous snapshot in place. The instance UUID, IP addresses, security groups, and metadata are preserved — only disk contents change.

## Overview

Instance Rollback restores a virtual machine to the exact disk state captured by
one of its snapshots, **without** rebuilding the VM, changing its UUID, or
disturbing its network identity. Floating IPs, fixed IPs, ports, security
groups, key pair, flavor, and metadata are all preserved. Only the bytes on
disk change.

<Info>**Xloud-Developed** — In-place instance rollback is built by Xloud and ships with XAVS / XPCI on Xloud Distributed Storage (XSDS) backed clusters. It is the equivalent of VMware vSphere "Revert to Snapshot" and Proxmox "Rollback".</Info>

<Note>
  **Prerequisites**

  * The instance is on **Xloud Distributed Storage (XSDS / Ceph RBD)**. Other
    storage backends are rejected at pre-flight time.
  * The instance is in `Active`, `Shutdown`, or `Error` state and has no other
    operation in progress (no migration, no resize, no other rollback).
  * At least one Instance Snapshot exists for the VM.
  * Your project role permits rollback (`os_compute_api:xloud_rollback:rollback`).
</Note>

***

## What Rollback Preserves vs Replaces

<Tabs>
  <Tab title="Preserved" icon="check">
    | Item                                          | Behaviour                                                                   |
    | --------------------------------------------- | --------------------------------------------------------------------------- |
    | Instance UUID                                 | Same UUID before and after — automation, monitoring, and tags keep working. |
    | Display name                                  | Unchanged.                                                                  |
    | Flavor                                        | Unchanged.                                                                  |
    | Floating IPs                                  | Unchanged.                                                                  |
    | Fixed IPs and ports                           | Unchanged.                                                                  |
    | Security groups                               | Unchanged.                                                                  |
    | SSH key pair                                  | Unchanged.                                                                  |
    | Server metadata, tags, descriptions           | Unchanged.                                                                  |
    | Volume attachments and device names           | Unchanged.                                                                  |
    | Snapshots taken **after** the rollback target | Still listed and still usable.                                              |
  </Tab>

  <Tab title="Replaced" icon="rotate-ccw">
    | Item                                               | Behaviour                         |
    | -------------------------------------------------- | --------------------------------- |
    | Root disk contents                                 | Reverted to the snapshot's bytes. |
    | Attached volume contents (if part of the snapshot) | Reverted to the snapshot's bytes. |
    | Anything written **after** the snapshot was taken  | Lost on rollback.                 |
  </Tab>
</Tabs>

<Warning>
  Rollback overwrites disk contents. Any data, configuration changes, or
  database writes made **after** the target snapshot will be lost on the
  reverted volumes. The rollback dialog requires you to acknowledge this
  before submitting.
</Warning>

***

## Two Differentiators

### Safety Snapshot (automatic)

Before any disk content is changed, the platform creates a per-volume Cinder
snapshot of the **current** state. If the rollback turns out to be the wrong
choice, you can revert again — this time to the safety snapshot — and recover
the pre-rollback state.

<Note>
  Safety snapshots use copy-on-write at the storage layer, so they are
  effectively instant and consume only the storage of changes written after
  they are taken. They are named with the prefix `_xloud_safety_` and tagged
  with metadata so administrators can identify and clean them up later.
</Note>

The safety snapshot is created by default. The dialog exposes a
**Skip Safety Snapshot** checkbox; leaving it unchecked is recommended for any
production VM. Disabling the safety snapshot is only appropriate when you are
absolutely certain you do not need a recovery point — for example, lab
instances with no important state.

If the rollback fails partway through, the safety snapshot is **retained** so
the platform or an administrator can recover the VM to its prior state.

### Roll Back to ANY Snapshot, Not Only the Most Recent

Most cloud platforms only let you revert to the latest snapshot, forcing you
to delete newer snapshots first or to chain rollbacks one at a time. Xloud
Instance Rollback works against any snapshot in the VM's history.

<Tip>
  You can take a snapshot today, take another tomorrow, take a third on the
  weekend, and on Monday roll directly back to the **first** one. The newer
  snapshots are not deleted, are not invalidated, and remain available for
  cloning, exporting, or rolling back to in the future.
</Tip>

This works because the rollback runs at the storage layer rather than going
through the standard volume-revert API path. The full snapshot tree is
preserved on the underlying storage, so any snapshot is a valid target.

***

## Roll Back an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the rollback dialog">
        Navigate to **Compute > Instance Snapshots**. Locate the snapshot you
        want to roll back to. On the snapshot row, open the **More** menu and
        select **Rollback**.

        The dialog title is **Rollback Instance to Snapshot**. The action is
        marked as destructive and the dialog will not close if you click
        outside it — use **Cancel** or the close icon.
      </Step>

      <Step title="Wait for the pre-flight check">
        The platform automatically resolves the source instance and runs a
        pre-flight check. The dialog displays:

        | Field                | What it shows                                                    |
        | -------------------- | ---------------------------------------------------------------- |
        | **Snapshot**         | The snapshot that will be applied.                               |
        | **Target Instance**  | The VM that will be reverted (auto-detected).                    |
        | **Pre-flight Check** | Backend type, number of volumes to revert, warnings, and errors. |
        | **Supported Scope**  | Notes that XSDS is required and the VM will be stopped briefly.  |

        <Note>
          If the pre-flight check fails (for example, the storage backend is not
          XSDS, or another operation is in progress on the VM), the dialog shows
          a clear error and the **Confirm** button stays disabled. Transient
          errors offer a **Retry** button.
        </Note>
      </Step>

      <Step title="Choose post-rollback options">
        | Field                          | Description                                                                                                                                |
        | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
        | **Power State After Rollback** | `Power on` (default) starts the VM after rollback completes. `Power off` leaves the VM stopped so you can inspect the disk before booting. |
        | **Skip Safety Snapshot**       | Leave unchecked (recommended). Tick only if you do not need a recovery point.                                                              |
        | **Irreversible Action**        | Tick to acknowledge the disk content will be replaced. Required to enable Confirm.                                                         |
      </Step>

      <Step title="Confirm and wait">
        Click **Confirm**. The platform performs the rollback synchronously:

        1. Creates a per-volume safety snapshot (unless skipped).
        2. Stops the VM and waits for it to fully release the disk.
        3. Reverts each volume to the chosen snapshot at the storage layer.
        4. Restarts the VM if **Power on** was selected.

        Typical end-to-end duration is **30–60 seconds** for small VMs. Larger
        volumes take longer because the storage layer streams more data. The
        dialog closes when the API returns success.

        <Check>The instance returns to `Active` (or `Shutdown` if you chose Power off) with the same UUID, the same IP, and the disk contents from the chosen snapshot.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Rollback is exposed as a Compute server action. There is no dedicated
    `openstack` subcommand, so use the API directly with a token.

    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="Set variables" theme={null}
    INSTANCE_ID=<your-instance-uuid>
    SNAPSHOT_ID=<glance-image-uuid-of-the-instance-snapshot>
    NOVA_URL=$(openstack endpoint list --service compute --interface public -f value -c URL | head -n1)
    TOKEN=$(openstack token issue -c id -f value)
    ```

    ```bash title="Pre-flight check (dry run)" theme={null}
    curl -sS -X POST \
      -H "X-Auth-Token: $TOKEN" \
      -H "Content-Type: application/json" \
      -d "{\"xloud-rollback-dry-run\": {\"snapshot_id\": \"$SNAPSHOT_ID\"}}" \
      "$NOVA_URL/servers/$INSTANCE_ID/action" | jq .
    ```

    The response includes `can_rollback`, the detected `backend`, the list of
    volumes that will be reverted, and any warnings or errors.

    ```bash title="Execute rollback" theme={null}
    curl -sS -X POST \
      -H "X-Auth-Token: $TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"xloud-rollback\": {
          \"snapshot_id\": \"$SNAPSHOT_ID\",
          \"skip_safety_snapshot\": false,
          \"power_state_after\": \"on\"
        }
      }" \
      "$NOVA_URL/servers/$INSTANCE_ID/action" | jq .
    ```

    The call is synchronous and returns when the rollback is complete:

    ```json theme={null}
    {
      "xloud_rollback": {
        "request_id": "req-...",
        "instance_uuid": "...",
        "snapshot_id": "...",
        "state": "completed",
        "volumes_reverted": [ { "volume_id": "...", "elapsed_seconds": 0.11 } ],
        "safety_snapshot_ids": [ "..." ],
        "duration_seconds": 27.3
      }
    }
    ```

    <Check>The response shows <code>state: completed</code> and includes the safety snapshot UUIDs.</Check>
  </Tab>
</Tabs>

***

## Recover From a Rollback

If the rollback was not what you wanted, you can recover.

<Steps>
  <Step title="Locate the safety snapshot">
    Navigate to **Storage > Volume Snapshots**. Filter by the prefix
    `_xloud_safety_`. Each safety snapshot is named with the rollback request
    ID, the device name, and a timestamp, so you can correlate it with the
    rollback you want to undo.
  </Step>

  <Step title="Roll back to the safety snapshot">
    The safety snapshot is itself a Cinder snapshot of the volume. To use it
    as a rollback target, take an Instance Snapshot of the VM **referencing
    that safety state**, or contact your administrator who can perform the
    same rollback procedure against the safety snapshot.

    <Note>
      Safety snapshots are retained until an administrator cleans them up.
      They consume only the changed bytes (copy-on-write), so keeping them
      around for a recovery window is cheap.
    </Note>
  </Step>
</Steps>

***

## Pre-flight Errors and What They Mean

<AccordionGroup>
  <Accordion title="Storage backend requires Ceph RBD/XSDS">
    The instance has at least one volume on a backend other than XSDS (for
    example, a third-party iSCSI array). Instance Rollback v1 only supports
    XSDS-backed volumes. Migrate the volume to XSDS or use snapshot-based
    cloning to recover state.
  </Accordion>

  <Accordion title="Cannot determine source instance for this snapshot">
    The snapshot's metadata does not contain a usable instance reference and
    the boot volume's attachments could not be walked. This typically means
    the original instance was deleted. Use **Create Instance From Snapshot**
    to launch a new VM from the snapshot instead.
  </Accordion>

  <Accordion title="Another operation is in progress">
    A migration, resize, snapshot, or another rollback is currently running
    on the VM. Wait for it to complete and retry.
  </Accordion>

  <Accordion title="RBD image has active watchers; refusing rollback">
    The VM did not fully release the disk before rollback was attempted. This
    safety check prevents disk corruption from concurrent writers. The
    platform retries on its own; if it persists, contact your administrator
    — a stale libvirt session or a backup agent may be holding the disk.
  </Accordion>

  <Accordion title="Snapshot not found or not active">
    The selected snapshot was deleted, is still uploading, or is in `Error`
    state. Refresh the snapshot list and retry once the snapshot is `Active`.
  </Accordion>
</AccordionGroup>

***

## What Rollback Does Not Do

| Out of scope (v1)             | Why                                                                                                                       |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Non-XSDS backends             | Other backends do not expose the in-place revert primitive that makes the operation safe and instant.                     |
| Hot rollback (running VM)     | The VM is stopped automatically as part of the workflow. The disk cannot be reverted while a hypervisor is writing to it. |
| Mixed backends in one VM      | If any attached volume is not on XSDS, the entire rollback is rejected. No partial rollback.                              |
| Bulk rollback across many VMs | Rollback is per-instance.                                                                                                 |
| Scheduled rollback            | The action runs immediately when triggered.                                                                               |

***

## How It Works Internally

<Note>
  This section is informational. End users do not need to understand it to use
  the feature.
</Note>

For an Xloud Distributed Storage backed volume, a snapshot is a copy-on-write
marker on the underlying object — it does **not** copy data. Rollback is a
single storage-layer call that points the volume's "head" back at the chosen
snapshot. After rollback:

* The volume's database row is unchanged. The block storage service still
  sees `status=in-use, attached`.
* The volume attachment is unchanged.
* Snapshots taken after the rollback target are **still preserved** on the
  underlying storage.
* Only the bytes inside the volume's image have moved back in time.

Before issuing the rollback call, the platform checks that no hypervisor is
still holding the volume open. If any "watcher" is detected, the rollback is
refused — this prevents the corruption case where a writer is still active
during a revert.

For a deeper architecture overview, see [Compute Architecture](/services/compute/architecture).

***

## Next Steps

<CardGroup>
  <Card title="Instance Snapshots" icon="camera" href="/services/compute/snapshots">
    Capture VM state to create rollback points.
  </Card>

  <Card title="Clone Instance" icon="copy" href="/services/compute/clone-instance">
    Create a new VM from an existing one without rolling back.
  </Card>

  <Card title="VM Templates" icon="layers" href="/services/compute/vm-templates">
    Promote a known-good snapshot into a reusable template.
  </Card>
</CardGroup>


# Instance Tagging
Source: https://docs.xloud.tech/services/compute/instance-tagging

Add, edit, and delete user-defined tags on Xloud Compute instances. Use tags to organize instances, drive automation, and filter inventory through the CLI and APIs.

## Overview

Instance tags are short, user-defined labels you attach to a virtual machine.
They are free-form strings you can use to mark environment, owner, application
tier, cost center, lifecycle stage, or any custom taxonomy your team needs.

Tags are stored on the instance record itself, so they travel with the VM
across reboots, snapshots, migrations, and ownership changes.

<iframe title="How to Add, Edit, and Delete Instance Tags on Xloud" />

<Note>
  **Prerequisites**

  * An instance in `Active`, `Paused`, `Suspended`, or `Stopped` state.
  * Permission to update server tags
    (`os_compute_api:os-server-tags:update_all`). This is granted to project
    members by default.
</Note>

***

## Tag Rules

| Rule                       | Limit                                                   |
| -------------------------- | ------------------------------------------------------- |
| Maximum tags per instance  | 50                                                      |
| Maximum characters per tag | 60                                                      |
| Forbidden characters       | `/` (forward slash), `,` (comma)                        |
| Case sensitivity           | Not case sensitive — `Prod` and `prod` are the same tag |
| Duplicates                 | Not allowed (case-insensitive comparison)               |
| Whitespace                 | Allowed inside a tag, trimmed at the edges              |

<Tip>
  Establish a tagging convention before you scale. Decide upfront whether to
  use `env:prod` or `prod`, whether to use `team-platform` or `platform`, and
  publish the convention internally. Consistent tags pay off when filtering
  hundreds of VMs from the CLI or driving Terraform/Ansible from tags.
</Tip>

***

## Where Tags Appear in the Dashboard

| Location                          | Behaviour                                                                                                                |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Compute > Instances** list      | A **Tags** column shows each tag as a colored pill. The column is hideable from the column-picker if you do not need it. |
| **Instance detail page**          | The **Tags Info** card on the BaseDetail tab lists every tag attached to the VM. Empty state shows a dash.               |
| **More menu on the instance row** | Contains **Modify Instance Tags**, the single dialog for all add/edit/delete actions.                                    |

<Note>
  The Dashboard list page does not currently expose a tag-based search filter.
  To filter by tag, use the CLI procedures below — the API supports tag
  filtering and is the recommended path for inventory queries.
</Note>

***

## Add, Edit, and Delete Tags

All three actions — adding a new tag, replacing existing tags, and removing
tags — happen inside a single dialog called **Modify Instance Tags**.
The dialog shows the current tags as pills; you mutate the set and click
**Confirm**, and the platform applies the new set in one operation.

### Add Tags

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Modify Instance Tags dialog">
        Navigate to **Compute > Instances**. On the row of the instance you
        want to tag, open the **More** menu and click **Modify Instance Tags**.
      </Step>

      <Step title="Type a tag and press Enter">
        In the tag input field, type the tag value and press **Enter**. The
        tag becomes a pill in the list. Repeat for each tag you want to add.

        The dialog enforces:

        * Maximum 50 tags per instance.
        * Maximum 60 characters per tag.
        * No forward slash `/` or comma `,`.
        * No duplicate (case-insensitive) tags.

        Invalid tags are rejected immediately with an inline error message.
      </Step>

      <Step title="Save the changes">
        Click **Confirm**. The platform updates the VM's tag set and the
        dialog closes.

        <Check>The new tags appear in the **Tags** column and on the instance detail page within a few seconds.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    The `openstack` CLI offers two patterns. `openstack server set --tag`
    adds a single tag without affecting existing tags. `openstack server set     --tags` replaces the entire tag set in one call.

    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Add a single tag (preserves existing)" theme={null}
      openstack server set --tag production <INSTANCE_ID>
      ```

      ```bash title="Add several tags in one call" theme={null}
      openstack server set \
        --tag production \
        --tag web-tier \
        --tag team-platform \
        <INSTANCE_ID>
      ```

      ```bash title="Replace the entire tag set" theme={null}
      openstack server set --tags "production,web-tier,team-platform" <INSTANCE_ID>
      ```
    </CodeGroup>

    <Check>
      ```bash theme={null}
      openstack server show <INSTANCE_ID> -c tags
      ```

      Output lists every tag attached to the VM.
    </Check>
  </Tab>
</Tabs>

### Edit Tags

Editing a tag means removing the old value and adding the new value — there
is no in-place rename.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open Modify Instance Tags">
        From **Compute > Instances**, open the **More** menu on the row and
        click **Modify Instance Tags**.
      </Step>

      <Step title="Remove the old value">
        Click the **X** on the tag you want to rename. The pill is removed
        from the list. The change is not saved until you click **Confirm**.
      </Step>

      <Step title="Type the new value">
        Type the new tag value and press **Enter**.
      </Step>

      <Step title="Save">
        Click **Confirm**. Both changes — the removal and the addition —
        apply in one update.

        <Check>The instance's **Tags** column shows the new value and no longer shows the old one.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Rename a tag (remove old, add new in one call)" theme={null}
    openstack server unset --tag old-name <INSTANCE_ID>
    openstack server set --tag new-name <INSTANCE_ID>
    ```

    <Tip>
      For atomic edits, replace the whole tag set with `--tags`:

      ```bash theme={null}
      openstack server set --tags "production,web-tier,team-payments" <INSTANCE_ID>
      ```

      Anything not listed is removed; anything new is added.
    </Tip>
  </Tab>
</Tabs>

### Delete Tags

You can remove a single tag or clear every tag from a VM.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open Modify Instance Tags">
        From **Compute > Instances**, open the **More** menu on the row and
        click **Modify Instance Tags**.
      </Step>

      <Step title="Remove tags">
        * To remove a single tag, click the **X** on its pill.
        * To remove every tag, click **X** on each pill until the list is
          empty.
      </Step>

      <Step title="Save">
        Click **Confirm**. The selected tags are deleted from the VM.

        <Check>The instance no longer shows the removed tags.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Remove a single tag" theme={null}
      openstack server unset --tag staging <INSTANCE_ID>
      ```

      ```bash title="Remove several tags in one call" theme={null}
      openstack server unset --tag staging --tag temporary <INSTANCE_ID>
      ```

      ```bash title="Clear ALL tags from an instance" theme={null}
      openstack server set --tags "" <INSTANCE_ID>
      ```
    </CodeGroup>

    <Check>
      ```bash theme={null}
      openstack server show <INSTANCE_ID> -c tags
      ```

      Output shows an empty list.
    </Check>
  </Tab>
</Tabs>

<Warning>
  Tag changes apply immediately. There is no undo. Removing a tag that is
  used by an automation system (Terraform, Ansible inventory, monitoring
  rules) can cause that automation to lose track of the VM. Verify your
  downstream tooling before bulk-removing tags in production.
</Warning>

***

## Key-Value Pairs

Tags in the Dashboard are **plain strings** (one label per pill). Some teams
need true `key=value` pairs — for example, `env=production`, `team=payments`,
`cost-center=R&D-12` — so they can branch automation, dashboards, or RBAC
rules on the *value* of a key rather than scanning a flat list.

Today on Xloud:

| Path                                 | What you get                                                                                                                                                                                                                                                  |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Dashboard (Modify Instance Tags)** | Plain strings only. By convention you can write `env=production` as a single string and split on `=` in your tooling, but the platform stores it as one opaque label.                                                                                         |
| **CLI**                              | True key-value pairs are supported via the `--property` flag, which writes server metadata. Available today.                                                                                                                                                  |
| **Dashboard (XCONNECT v2)**          | A dedicated **Custom Metadata** editor for instances is on the roadmap for XCONNECT v2 — the next-generation Xloud Dashboard. It will surface the same key-value metadata as a first-class form on every instance row, so you will not need the CLI for this. |

<Note>
  Server metadata and instance tags are two different stores under the hood.
  Tags are unordered string labels; metadata is a key-value dictionary. They
  can coexist on the same VM and serve complementary purposes.
</Note>

### Add Key-Value Pairs from the CLI

```bash title="Source credentials" theme={null}
source openrc.sh
```

<CodeGroup>
  ```bash title="Set one or more key=value pairs" theme={null}
  openstack server set \
    --property env=production \
    --property team=payments \
    --property cost-center=R\&D-12 \
    <INSTANCE_ID>
  ```

  ```bash title="Show all key-value pairs on an instance" theme={null}
  openstack server show <INSTANCE_ID> -c properties
  ```

  ```bash title="Update an existing pair (same flag, new value)" theme={null}
  openstack server set --property env=staging <INSTANCE_ID>
  ```

  ```bash title="Delete a single key" theme={null}
  openstack server unset --property env <INSTANCE_ID>
  ```
</CodeGroup>

<Check>
  ```bash theme={null}
  openstack server show <INSTANCE_ID> -c properties
  ```

  Output prints each `key='value'` pair on its own line.
</Check>

<Tip>
  Use **tags** for fast yes/no filtering — "is this VM in the production
  group?". Use **key-value metadata** when the value matters — "which
  environment is this VM in?" or "who owns this VM?". Many teams use both:
  a coarse `production` tag for filtering and `env=production` metadata
  for display.
</Tip>

***

## Where You Can Use Tags

Tags are not exclusive to instances. The Xloud Platform attaches the same
flat-string tag concept to several resource types so you can apply the
same naming convention across the whole cluster.

| Resource                                                           | Tags supported                                           | Where in the Dashboard                            | CLI                                  |
| ------------------------------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------- | ------------------------------------ |
| **Instance** (Compute)                                             | Yes — string tags                                        | Compute > Instances > More > Modify Instance Tags | `openstack server set --tag`         |
| **Image** (Image Service)                                          | Yes — string tags                                        | Compute > Images (CLI is the primary path today)  | `openstack image set --tag`          |
| **Project** (Identity)                                             | Yes — string tags                                        | Identity > Projects > Modify Project Tags         | `openstack project set --tag`        |
| **Network** (Networking)                                           | Yes — string tags                                        | Networking views (CLI is the primary path today)  | `openstack network set --tag`        |
| **Subnet, Router, Port, Floating IP, Security Group** (Networking) | Yes — string tags                                        | CLI is the primary path today                     | `openstack <resource> set --tag`     |
| **Stack** (Orchestration)                                          | Yes — string tags at create time                         | Set in the Heat template's `tags` parameter       | `openstack stack create --tags`      |
| **Flavor** (Compute)                                               | No native tags — use **metadata / extra\_specs** instead | Compute > Flavors > Manage Metadata (admin)       | `openstack flavor set --property`    |
| **Volume** (Block Storage)                                         | No native tags — use **metadata** instead                | Storage > Volumes > Manage Metadata               | `openstack volume set --property`    |
| **Host Aggregate** (Compute, admin)                                | No native tags — use **metadata** instead                | Compute > Host Aggregates > Manage Metadata       | `openstack aggregate set --property` |

<Note>
  Resources marked "use metadata instead" never had a string-tag concept on
  this platform — their key-value metadata is the only labelling mechanism.
  Resources with native tags also support metadata if you need both.
</Note>

***

## Actions You Can Perform Using Tags

Tags become powerful when you use them as a key for filtering, automation,
and reporting. The same tag set drives multiple workflows.

### Filter the Instance Inventory by Tag

The Compute API understands four tag filter parameters. The `openstack` CLI
exposes them as flags on `openstack server list`.

| Flag              | Meaning                                                                              |
| ----------------- | ------------------------------------------------------------------------------------ |
| `--tag X`         | Return instances that have **all** of these tags. Repeat the flag for AND semantics. |
| `--tag-any X`     | Return instances that have **any** of these tags. Repeat the flag for OR semantics.  |
| `--not-tag X`     | Return instances that do **not** have all of these tags.                             |
| `--not-tag-any X` | Return instances that do **not** have any of these tags.                             |

<CodeGroup>
  ```bash title="All production web servers (AND)" theme={null}
  openstack server list --tag production --tag web-tier
  ```

  ```bash title="Any of staging or development (OR)" theme={null}
  openstack server list --tag-any staging --tag-any development
  ```

  ```bash title="Everything except temporary" theme={null}
  openstack server list --not-tag-any temporary
  ```

  ```bash title="Production but NOT in the data-engineering team" theme={null}
  openstack server list --tag production --not-tag team-data-engineering
  ```
</CodeGroup>

<Tip>
  Combine `--tag` and `--not-tag-any` to get clean inventory slices —
  for example "production but not deprecated". The same combination via
  the API drives audit reports and automation triggers.
</Tip>

### Drive Automation From Tags

Most infrastructure tools can read instance tags directly and use them as
the grouping key.

| Tool                                         | Pattern                                                                                                                                                    |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Terraform**                                | `data "openstack_compute_instance_v2"` filters by tag; outputs feed downstream resources.                                                                  |
| **Ansible Dynamic Inventory**                | The `openstack` inventory plugin supports `keyed_groups` from `tags`, so a VM tagged `web-tier` lands in an Ansible group called `web-tier` automatically. |
| **Monitoring (Prometheus + node\_exporter)** | Service discovery picks up `tags` as labels, so a Grafana dashboard or alert rule can scope by `environment="production"`.                                 |
| **Backup scheduling**                        | Tag a VM with `backup-daily` or `backup-weekly`; the backup runner targets the tag.                                                                        |

<Note>
  The Xloud Dashboard does not currently expose a tag filter in its instance
  search bar. For ad-hoc filtering inside the Dashboard, use the column-picker
  to display the **Tags** column and search by name, then visually scan the
  tags. For programmatic inventory queries, use the CLI flags above.
</Note>

### Use Tags as Lifecycle Markers

A common pattern is to tag VMs with their intended lifetime, then run a
periodic job that lists VMs with the relevant tag and acts on them.

| Tag pattern                  | Operator behaviour                                           |
| ---------------------------- | ------------------------------------------------------------ |
| `temporary`, `disposable`    | Automation deletes after N days.                             |
| `do-not-delete`, `protected` | Automation refuses to touch these regardless of other rules. |
| `expires-2026-12-31`         | A scheduled job parses the date and stops the VM on expiry.  |
| `owner:alice@example.com`    | Notifications go to the tagged owner before any maintenance. |

***

## Common Tagging Patterns

| Tag Category    | Example Tags                                            | Purpose                         |
| --------------- | ------------------------------------------------------- | ------------------------------- |
| **Environment** | `production`, `staging`, `development`, `testing`       | Identify deployment stage       |
| **Team**        | `team-platform`, `team-data`, `team-security`           | Track ownership                 |
| **Application** | `web-frontend`, `api-gateway`, `database`, `cache`      | Group by application tier       |
| **Cost Center** | `dept-engineering`, `project-alpha`                     | Budget tracking and chargeback  |
| **Lifecycle**   | `temporary`, `persistent`, `protected`, `do-not-delete` | Cleanup automation              |
| **Compliance**  | `pci-dss`, `hipaa`, `internal`                          | Drive policy and access reviews |
| **Backup**      | `backup-daily`, `backup-weekly`, `no-backup`            | Backup scheduler input          |

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Modify Instance Tags is greyed out">
    The action is only available when the VM is in `Active`, `Paused`,
    `Suspended`, or `Stopped` state. If the instance is `Building`,
    `Error`, `Resizing`, or in any other transient state, wait for it to
    settle and try again.
  </Accordion>

  <Accordion title="Invalid Tag Value error in the dialog">
    The tag contains a forbidden character. Forward slash `/` and comma
    `,` are not allowed because they conflict with the API's list
    separator. Use a hyphen, underscore, or colon instead — for example
    `env-prod` or `env_prod` instead of `env/prod`.
  </Accordion>

  <Accordion title="Duplicate tag name error">
    Tags are case-insensitive. `Production` and `production` are
    considered the same tag. The dialog rejects the second entry. Pick
    a single canonical casing and stick with it.
  </Accordion>

  <Accordion title="Tag changes do not appear in the list view">
    The list view caches results for a few seconds. Refresh the page or
    re-run `openstack server list` to pick up the latest values. If the
    delay persists for more than a minute, contact your administrator —
    the cache layer may need to be cleared.
  </Accordion>

  <Accordion title="Permission denied when modifying tags">
    Your project role does not include
    `os_compute_api:os-server-tags:update_all`. Ask your administrator
    to grant the `member` role on the project that owns the VM.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" icon="server" href="/services/compute/launch-instance">
    Create new instances and tag them at boot time.
  </Card>

  <Card title="Server Groups" icon="layers" href="/services/compute/server-groups">
    Group instances by placement policy.
  </Card>

  <Card title="Instance Snapshots" icon="camera" href="/services/compute/snapshots">
    Snapshot tagged instances for backup and rollback.
  </Card>

  <Card title="Resource Optimizer" icon="sliders" href="/services/optimization">
    Use tags to scope optimization actions to specific workloads.
  </Card>
</CardGroup>


# Create an Instance
Source: https://docs.xloud.tech/services/compute/launch-instance

Create a virtual machine using the Dashboard's 4-step wizard or CLI. Select source, flavor, network, and login credentials.

## Overview

Launching an instance creates a virtual machine on an Xloud Compute host. The Dashboard
provides a 4-step wizard that walks you through base configuration, networking, system
settings, and a confirmation review. Once active, the instance can be accessed via SSH,
console, or RDP depending on the guest OS.

<CardGroup>
  <Card title="Windows VM from ISO" icon="monitor" href="/services/compute/windows-vm-guide">
    ISO boot with VirtIO driver injection, Sysprep, and image creation
  </Card>

  <Card title="Linux VM from ISO" icon="terminal" href="/services/compute/linux-vm-guide">
    Custom Linux installation from ISO with disk setup and image creation
  </Card>
</CardGroup>

<Note>
  **Prerequisites**

  * An active image in the Xloud Image Service (or an existing bootable volume)
  * A flavor appropriate for your workload
  * At least one network available in your project
  * An SSH key pair for Linux instances, or a configured password for Windows
  * A security group with the required inbound rules
</Note>

***

<iframe title="How to Create a Virtual Machine on Xloud" />

## Create an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    The Instance Create wizard has 4 steps: **Base Config**, **Network Config**,
    **System Config**, and **Confirm Config**. An **Instance Count** input and
    real-time **quota display** are shown in the footer throughout all steps.

    <Steps>
      <Step title="Open the Create Instance wizard">
        Navigate to **Compute > Instances** in the sidebar. Click **Create Instance**
        in the top-right corner.
      </Step>

      <Step title="Step 1 — Base Config">
        Configure the boot source, flavor, and storage.

        **Available Zone** (required) — Select the availability zone for host placement.

        <Tip>
          Administrators see an additional **Pin to Host** group that allows selecting
          a specific physical hypervisor (`zone:hostname` format). Regular users see
          only the zone names.
        </Tip>

        **Specification (Flavor)** (required) — Select the virtual hardware profile.
        The flavor table supports filtering by architecture and category:

        | Architecture            | Categories                                                                                  |
        | ----------------------- | ------------------------------------------------------------------------------------------- |
        | X86 Architecture        | General Purpose, Compute Optimized, Memory Optimized, Big Data, Local SSD, High Clock Speed |
        | Heterogeneous Computing | GPU Compute, Visualization Compute                                                          |
        | Bare Metal              | —                                                                                           |
        | ARM Architecture        | —                                                                                           |

        The table shows columns: Name, CPU, Memory, Internal Network Bandwidth,
        Ephemeral Disk (if applicable), and IOPS (if applicable). Filter by Name,
        CPU, or Memory.

        **Start Source** (required) — Select the boot source type:

        | Source                | Description                                                                                    |
        | --------------------- | ---------------------------------------------------------------------------------------------- |
        | **Image**             | Boot from an OS image in the Xloud Image Service                                               |
        | **Instance Snapshot** | Restore from a previously captured instance snapshot                                           |
        | **Bootable Volume**   | Boot from an existing persistent block storage volume (only shown if block storage is enabled) |

        When **Image** is selected, choose the operating system from tabs organized
        by distribution (CentOS, Ubuntu, Fedora, Windows, Debian, CoreOS, Arch, FreeBSD, Others).

        **Boot From Volume** — When source is Image or Instance Snapshot:

        | Option  | Description                                       |
        | ------- | ------------------------------------------------- |
        | **Yes** | Create a new system disk (persistent boot volume) |
        | **No**  | Boot directly from image (ephemeral root disk)    |

        **System Disk** — Shown when Boot From Volume is Yes. Select:

        * **Volume Type** from available storage backends
        * **Size (GiB)** — minimum is determined by the flavor disk size, image minimum
          disk, and image size (whichever is largest)
        * **Delete on Termination** — whether to delete the boot volume when the instance
          is deleted

        **Data Disk** (optional) — Add additional data disks. Click **Add Data Disks** to
        add one or more disks, each with Volume Type, Size, and Delete on Termination
        settings.

        <Note>
          When source is **Bootable Volume**, the instance count is limited to 1 and
          the Data Disk section is shown for additional disks.
        </Note>

        **CD-ROM Source** (optional) — Attach an ISO image or existing volume as a
        CD-ROM device:

        | Option     | Description                                  |
        | ---------- | -------------------------------------------- |
        | **None**   | No CD-ROM attached (default)                 |
        | **Image**  | Select an image to mount as CD-ROM           |
        | **Volume** | Select an existing volume to mount as CD-ROM |
      </Step>

      <Step title="Step 2 — Network Config">
        Configure networking and security.

        **Networks** (required if no ports selected) — Select one or more networks
        from the available list. Networks without subnets are disabled.

        After selecting networks, a **Virtual LAN** section appears for each
        selected network with:

        | Field          | Description                                                 |
        | -------------- | ----------------------------------------------------------- |
        | **Network**    | Auto-populated from selection                               |
        | **Subnet**     | Dropdown of subnets in the selected network                 |
        | **IP Type**    | Automatically Assigned Address or Manually Assigned Address |
        | **IP Address** | Manual IPv4 or IPv6 input (only when IP Type is manual)     |

        <Warning>
          If you specify a manual IP address AND set the instance count to more than 1,
          the wizard will block submission — manual IPs cannot be used with batch creation.
        </Warning>

        **Ports** (required if no networks selected) — Alternatively, select pre-created
        ports (only ports with status `DOWN` are shown). At least one network or port
        must be selected.

        **Security Group** (required when shown) — Select one or more security groups.
        This field is hidden if any selected network or port has port security disabled.

        <Note>
          The security group rules apply to all virtual network interfaces of the instance,
          not just the primary interface.
        </Note>
      </Step>

      <Step title="Step 3 — System Config">
        Configure the instance name, login credentials, and advanced options.

        **Name** (required) — The instance display name. When launching multiple
        instances (count > 1), instances are named `{name}-1`, `{name}-2`, etc.

        **Login Type** (required) — Choose the authentication method:

        | Type         | Description                                                                      |
        | ------------ | -------------------------------------------------------------------------------- |
        | **Keypair**  | Select an existing SSH key pair or create a new one. Disabled for Windows images |
        | **Password** | Set a login username and password. Required for Windows images                   |

        When **Keypair** is selected, choose from the key pair table or click
        **Create Keypair** to generate a new one.

        When **Password** is selected:

        * **Login Name** — Auto-populated from the image's `os_admin_user` property
          if available, otherwise enter manually
        * **Login Password** — Must meet password complexity requirements
        * **Confirm Password** — Must match the login password

        <Warning>
          For Windows images, the Keypair option is automatically disabled.
          Password login is the only option for Windows instances.
        </Warning>

        **Advanced Options** — Click to expand additional settings:

        | Field             | Visibility | Description                                                                                                      |
        | ----------------- | ---------- | ---------------------------------------------------------------------------------------------------------------- |
        | **Physical Node** | Admin only | Smart Scheduling (default) or Manually Specify a hypervisor                                                      |
        | **Server Group**  | All users  | Select an existing server group for affinity/anti-affinity placement                                             |
        | **User Data**     | All users  | Cloud-init script (text area with file upload, ASCII only, max 1000 characters)                                  |
        | **Virtual TPM**   | All users  | Attach a virtual TPM device (requires Xloud KMS). See [vTPM and Secure Boot](/services/compute/vtpm-secure-boot) |
        | **Secure Boot**   | All users  | Require UEFI Secure Boot (requires Xloud KMS). See [vTPM and Secure Boot](/services/compute/vtpm-secure-boot)    |

        <Info>
          **Virtual TPM** and **Secure Boot** both require the **Xloud Key Management**
          service to be enabled on the cluster — both checkboxes are disabled otherwise.
          Full walkthrough: [vTPM and Secure Boot](/services/compute/vtpm-secure-boot).
        </Info>
      </Step>

      <Step title="Step 4 — Confirm Config">
        Review all settings before launching. The confirmation page shows a read-only
        summary organized into three sections:

        * **Base Config** — Source, system disk, data disks, availability zone, flavor, project
        * **Network Config** — Virtual LANs with subnet/IP assignments, selected ports, security groups
        * **System Config** — Instance name, login type, physical node, server group

        Click any section heading to navigate back to that step for corrections.

        The footer shows the **Instance Count** and real-time **quota usage**:

        | Quota                 | Tracked                                                  |
        | --------------------- | -------------------------------------------------------- |
        | Instance              | Count against instance limit                             |
        | CPU                   | vCPUs x count against cores limit                        |
        | Memory (GiB)          | RAM x count against RAM limit                            |
        | Volume                | New volumes against volume limit                         |
        | Volume Capacity (GiB) | Total new volume size against storage limit              |
        | Server Group Member   | Members against group limit (when server group selected) |

        <Warning>
          If any quota would be exceeded, a red badge is displayed and the Confirm
          button is disabled. Free up resources or contact your administrator to
          increase quotas.
        </Warning>

        Click **Confirm** to launch the instance.

        <Check>
          The instance appears in the list with status **Build**, transitioning to
          **Active** within seconds to minutes depending on image size and host
          availability.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Gather required identifiers">
        <CodeGroup>
          ```bash title="List active images" theme={null}
          openstack image list --status active
          ```

          ```bash title="List available flavors" theme={null}
          openstack flavor list
          ```

          ```bash title="List project networks" theme={null}
          openstack network list
          ```

          ```bash title="List key pairs" theme={null}
          openstack keypair list
          ```

          ```bash title="List security groups" theme={null}
          openstack security group list
          ```
        </CodeGroup>
      </Step>

      <Step title="Launch the instance">
        ```bash title="Create an instance from image" theme={null}
        openstack server create \
          --image <IMAGE_ID> \
          --flavor <FLAVOR_NAME> \
          --network <NETWORK_ID> \
          --key-name <KEYPAIR_NAME> \
          --security-group <SECURITY_GROUP> \
          --availability-zone <ZONE_NAME> \
          my-instance
        ```

        <CodeGroup>
          ```bash title="Boot from volume" theme={null}
          openstack server create \
            --volume <VOLUME_ID> \
            --flavor <FLAVOR_NAME> \
            --network <NETWORK_ID> \
            --key-name <KEYPAIR_NAME> \
            my-instance
          ```

          ```bash title="Boot from image with new volume" theme={null}
          openstack server create \
            --image <IMAGE_ID> \
            --boot-from-volume 50 \
            --flavor <FLAVOR_NAME> \
            --network <NETWORK_ID> \
            --key-name <KEYPAIR_NAME> \
            my-instance
          ```

          ```bash title="Launch with user data" theme={null}
          openstack server create \
            --image <IMAGE_ID> \
            --flavor <FLAVOR_NAME> \
            --network <NETWORK_ID> \
            --key-name <KEYPAIR_NAME> \
            --user-data /path/to/init.sh \
            my-instance
          ```

          ```bash title="Launch in a server group" theme={null}
          openstack server create \
            --image <IMAGE_ID> \
            --flavor <FLAVOR_NAME> \
            --network <NETWORK_ID> \
            --key-name <KEYPAIR_NAME> \
            --hint group=<SERVER_GROUP_ID> \
            my-instance
          ```

          ```bash title="Launch multiple instances" theme={null}
          openstack server create \
            --image <IMAGE_ID> \
            --flavor <FLAVOR_NAME> \
            --network <NETWORK_ID> \
            --key-name <KEYPAIR_NAME> \
            --min 3 --max 3 \
            my-instance
          ```
        </CodeGroup>
      </Step>

      <Step title="Verify the instance is active">
        ```bash title="Check instance status" theme={null}
        openstack server show my-instance -c status -c addresses
        ```

        <Check>
          `status` shows `ACTIVE` and `addresses` lists the assigned IP. The instance
          is ready to accept connections.
        </Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Post-Launch Verification

After the instance reaches `ACTIVE` status, confirm it is fully operational:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the instance detail">
        Navigate to **Compute > Instances** and click the instance name. The detail
        page shows 8 tabs: Detail, Volumes, Instance Snapshots, Interfaces, Floating IPs,
        Security Groups, Action Logs, and Logs.
      </Step>

      <Step title="Check console output">
        Click the **Logs** tab to view the serial console output. A successful boot
        shows login prompts or cloud-init completion messages.
      </Step>

      <Step title="Open the console">
        Click the **Console** action (the first row action) to open a VNC console
        in a new browser tab.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Check instance details" theme={null}
    openstack server show my-instance \
      -c status -c addresses -c "OS-EXT-AZ:availability_zone"
    ```

    ```bash title="View console log" theme={null}
    openstack console log show my-instance | tail -30
    ```

    ```bash title="Get VNC console URL" theme={null}
    openstack console url show my-instance
    ```

    <Check>
      Console log shows OS boot completion. Status is `ACTIVE`. IP address is present.
    </Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Security Groups" href="/services/compute/security-groups">
    Create and manage firewall rules to control traffic to your instances
  </Card>

  <Card title="Manage IP Addresses" href="/services/compute/manage-ips">
    Allocate and associate floating IPs for external instance access
  </Card>

  <Card title="Resize an Instance" href="/services/compute/resize-instance">
    Change the flavor of a running instance to adjust vCPU and RAM
  </Card>

  <Card title="Server Groups" href="/services/compute/server-groups">
    Control instance placement with affinity and anti-affinity policies
  </Card>
</CardGroup>


# Live Migration
Source: https://docs.xloud.tech/services/compute/live-migration

Transfer running instances between hypervisor hosts with zero downtime. Covers cold migration, live migration, and block migration.

## Overview

Live migration moves a running instance from one compute host to another without
shutting it down. Cold migration stops the instance, moves it, and restarts it on
the target host. Both operations are administrator-only actions used for planned
maintenance, load balancing, and hardware decommissioning.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator access to the Xloud Dashboard (admin view)
  * Shared storage between source and destination hosts (or block migration enabled)
  * Compatible CPU architecture between hosts
</Note>

***

## Migration Types

| Type             | Instance Status       | Downtime              | Use Case                                        |
| ---------------- | --------------------- | --------------------- | ----------------------------------------------- |
| **Live Migrate** | `Active` or `Paused`  | None (zero downtime)  | Planned maintenance, load balancing             |
| **Cold Migrate** | `Active` or `Shutoff` | Yes (reboot required) | Hardware replacement, when live migration fails |

***

## Live Migrate an Instance

<iframe title="How to Live Migrate a VM on Xloud" />

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Live Migrate dialog">
        Navigate to **Compute > Instances** in the admin sidebar. Click the **More**
        dropdown on the instance row and select **Live Migrate**.

        <Note>
          Live Migrate is available for instances in `Active` or `Paused` status.
          It is not available for bare metal instances. This action appears only
          on the admin page.
        </Note>
      </Step>

      <Step title="Select a destination host (optional)">
        The dialog shows:

        | Field                | Description                                                             |
        | -------------------- | ----------------------------------------------------------------------- |
        | **Instance**         | Instance name (read-only)                                               |
        | **Current Host**     | The compute host currently running the instance (read-only)             |
        | **Destination Host** | Select a target hypervisor, or leave empty for scheduler auto-selection |
        | **Block Migrate**    | Checkbox — enable block migration for instances on local storage        |

        The host table shows available hypervisors. The current host and disabled
        hosts are grayed out. Bare metal hypervisors are filtered from the list.

        <Tip>
          Leave the destination host empty to let the scheduler select the optimal
          target based on available resources. Manually selecting a host is useful
          when you need to place the instance on a specific node.
        </Tip>
      </Step>

      <Step title="Start the migration">
        Click **Confirm**. The instance status changes to `Migrating` during the
        transfer and returns to `Active` on the new host when complete.

        <Check>Instance is `Active` on the new host. Verify with the instance detail page.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Live migrate (scheduler selects host)" theme={null}
      openstack server migrate --live-migration <INSTANCE_ID>
      ```

      ```bash title="Live migrate to specific host" theme={null}
      openstack server migrate --live-migration --host <TARGET_HOST> <INSTANCE_ID>
      ```

      ```bash title="Live migrate with block migration" theme={null}
      openstack server migrate --live-migration --block-migration <INSTANCE_ID>
      ```
    </CodeGroup>

    ```bash title="Monitor migration status" theme={null}
    openstack server migration list --server <INSTANCE_ID>
    ```

    <Check>Migration status shows `completed`. Instance is `Active` on the new host.</Check>
  </Tab>
</Tabs>

***

## Server Group Affinity Is Honored

Live migration respects the **affinity / anti-affinity policy** of any
[server group](/services/compute/server-groups) the instance belongs to. The
scheduler treats migration the same way it treats a launch:

| Policy on the instance's server group | What live migration does                                                                                                                                                      |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **affinity**                          | Picks a destination host that already runs the other instances in the group. If no such host has capacity, the migration fails with `NoValidHost`                             |
| **anti-affinity**                     | Picks a destination host that does **not** already run another instance in the group. If every other host already runs a group member, the migration fails with `NoValidHost` |
| **soft-affinity**                     | Prefers a destination host that already runs other group members; falls back to any available host if none has capacity                                                       |
| **soft-anti-affinity**                | Prefers a destination host with no other group members; falls back to any available host if no separate host is free                                                          |

This holds for both Dashboard-initiated migrations and CLI-initiated migrations,
and for both single-instance Live Migrate and Bulk Live Migrate. If you want to
override the policy for a specific move, remove the instance from its server
group first, migrate, and re-add it (no platform-supported way to bypass the
policy in-place).

<Tip>
  When evacuating a host, use **Bulk Live Migrate** with the source host pre-set as
  the filter. The scheduler picks destinations one at a time and re-evaluates the
  server-group constraint for each move — so anti-affinity remains intact across
  the whole evacuation.
</Tip>

***

## Cold Migrate an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Migrate dialog">
        Navigate to **Compute > Instances** in the admin sidebar. Click the **More**
        dropdown and select **Migrate**.

        <Note>
          Cold Migrate is available for instances in `Active` or `Shutoff` status.
          Not available for bare metal instances. Admin-only action.
        </Note>
      </Step>

      <Step title="Select a destination host (optional)">
        | Field                | Description                                        |
        | -------------------- | -------------------------------------------------- |
        | **Instance**         | Instance name (read-only)                          |
        | **Current Host**     | Current compute host (read-only)                   |
        | **Destination Host** | Select a target, or leave empty for auto-selection |

        Click **Confirm** to start the migration.
      </Step>

      <Step title="Confirm the migration">
        After the cold migration completes, the instance enters `Verify Resize`
        status. You must confirm or revert:

        * **Confirm Resize or Migrate** — accept the migration
        * **Revert Resize or Migrate** — move the instance back

        <Check>Instance returns to `Active` on the new host after confirmation.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Cold migrate" theme={null}
    openstack server migrate <INSTANCE_ID>
    ```

    ```bash title="Confirm the migration" theme={null}
    openstack server migration confirm <INSTANCE_ID>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    Monitor hypervisor resources before and after migration
  </Card>

  <Card title="Availability Zones" href="/services/compute/availability-zones">
    Understand zone boundaries for migration targets
  </Card>

  <Card title="Live vCPU/RAM Scaling" href="/services/compute/live-resize-admin">
    Scale instance resources without migration
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve stuck or failed migrations
  </Card>
</CardGroup>


# Live vCPU & RAM Scaling
Source: https://docs.xloud.tech/services/compute/live-resize

Increase or decrease vCPU and RAM on a running instance with no downtime using the Xloud Dashboard's Resource Adjustment dialog.

## Overview

Live vCPU and RAM scaling lets you adjust the compute resources of a **running instance
without rebooting**. Changes take effect immediately while the instance continues
serving traffic. The Dashboard provides a slider-based dialog for intuitive scaling
within the bounds configured on the instance's flavor.

<Info>**Xloud-Developed** — Bidirectional CPU hotplug (add AND remove vCPUs) and combined memory scaling (balloon + DIMM + virtio-mem) are developed by Xloud and ship with XAVS / XPCI.</Info>

<Note>
  **Prerequisites**

  * Instance must be in `Active` status
  * The instance must not be locked
  * The flavor must have hot-add enabled (`hw:cpu_min` or `hw:mem_min` extra specs) — if you do not see **Adjust Resources** in the actions menu, the flavor does not support live scaling
</Note>

***

## Scale vCPU or RAM

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Resource Adjustment dialog">
        Navigate to **Compute > Instances**. Click the **More** dropdown on
        the instance row, then select **Adjust Resources** under the
        **Configuration Update** group.

        <Note>
          This action only appears for instances in `Active` status whose flavor
          has hot-add enabled. If you do not see it, contact your administrator
          to configure a hot-add-enabled flavor.
        </Note>
      </Step>

      <Step title="Adjust resources with sliders">
        The dialog shows the current state and adjustment controls:

        | Field              | Description                                                                 |
        | ------------------ | --------------------------------------------------------------------------- |
        | **Instance**       | Instance name (read-only)                                                   |
        | **Current Info**   | Current vCPU count and memory allocation (read-only)                        |
        | **vCPUs**          | Slider input — drag or type to set between minimum and maximum vCPUs        |
        | **Memory (GiB)**   | Slider input — adjust between minimum and maximum memory in 0.25 GiB steps  |
        | **Make permanent** | Checkbox — persist the configuration after a soft reboot (default: checked) |

        The dialog fetches the live instance status to determine the current and
        allowed resource ranges. Color-coded hints indicate the scaling method:

        | Color  | Meaning                                             |
        | ------ | --------------------------------------------------- |
        | Green  | virtio-mem — fully adjustable in both directions    |
        | Yellow | No balloon driver detected — limited memory scaling |
        | Blue   | DIMM hotplug information                            |

        <Tip>
          The vCPU slider step aligns to the CPU thread count configured in the
          flavor. For example, with 2 threads per core, vCPUs adjust in steps of 2.
        </Tip>
      </Step>

      <Step title="Apply the change">
        Click **Confirm**. The change takes effect immediately — no reboot required.

        <Warning>
          Scaling **down** RAM while applications are using it may cause
          out-of-memory events inside the guest. Check application memory usage
          before reducing RAM.
        </Warning>
      </Step>

      <Step title="Verify inside the instance">
        SSH into the instance and confirm the new resources:

        ```bash title="Check vCPU count" theme={null}
        lscpu | grep "^CPU(s):"
        ```

        ```bash title="Check available RAM" theme={null}
        free -h
        ```

        <Check>The guest OS shows the updated vCPU count and memory — live scaling applied successfully with zero downtime.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    Live resource adjustment uses the Xloud-developed Nova API extension. The
    `openstack` CLI does not have a native command for this operation.

    <CodeGroup>
      ```bash title="Check current live resource status" theme={null}
      curl -s -H "X-Auth-Token: $TOKEN" \
        "$NOVA_ENDPOINT/v2.1/servers/<INSTANCE_ID>/xloud-status" | python3 -m json.tool
      ```

      ```bash title="Adjust vCPU count" theme={null}
      curl -X POST -H "X-Auth-Token: $TOKEN" \
        -H "Content-Type: application/json" \
        -d '{"current_vcpus": 4}' \
        "$NOVA_ENDPOINT/v2.1/os-xloud-adjust/<INSTANCE_ID>"
      ```

      ```bash title="Adjust memory (in MB)" theme={null}
      curl -X POST -H "X-Auth-Token: $TOKEN" \
        -H "Content-Type: application/json" \
        -d '{"current_memory_mb": 8192}' \
        "$NOVA_ENDPOINT/v2.1/os-xloud-adjust/<INSTANCE_ID>"
      ```

      ```bash title="Adjust both simultaneously" theme={null}
      curl -X POST -H "X-Auth-Token: $TOKEN" \
        -H "Content-Type: application/json" \
        -d '{"current_vcpus": 4, "current_memory_mb": 8192}' \
        "$NOVA_ENDPOINT/v2.1/os-xloud-adjust/<INSTANCE_ID>"
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Comparison: Live Scaling vs. Flavor Resize

|                       | **Live vCPU/RAM Scaling**                      | **Flavor Resize**                    |
| --------------------- | ---------------------------------------------- | ------------------------------------ |
| **Downtime**          | None — instance stays running                  | Reboot required                      |
| **Scope**             | vCPU and RAM only                              | vCPU, RAM, and disk                  |
| **Direction**         | Up or down within flavor envelope              | Up or down (disk: up only)           |
| **Confirmation step** | No — instant                                   | Yes — must confirm or revert         |
| **Dashboard action**  | More > Configuration Update > Adjust Resources | More > Configuration Update > Resize |
| **Use case**          | Responding to live load changes                | Permanent tier change                |

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Adjust Resources not visible in the menu">
    The instance's flavor does not have live scaling enabled. The flavor must have
    `hw:cpu_min` or `hw:mem_min` extra specs set. Contact your administrator to
    configure a hot-add-enabled flavor, or resize to a flavor that supports it.
  </Accordion>

  <Accordion title="Scaling request rejected">
    The requested value is outside the allowed range. The minimum and maximum are
    determined by the flavor's `hw:cpu_min` / `hw:mem_min` (minimum) and the flavor's
    vCPUs / RAM (maximum). Try a value within the slider range.
  </Accordion>

  <Accordion title="Guest OS does not show new vCPUs">
    On older Linux kernels (before 4.15), new CPUs may need to be brought online manually:

    ```bash title="Bring hotplugged CPUs online" theme={null}
    for cpu in /sys/devices/system/cpu/cpu*/online; do echo 1 > $cpu; done
    ```

    Modern kernels (4.15+) bring CPUs online automatically.
  </Accordion>

  <Accordion title="Memory change has no effect">
    The guest OS memory balloon driver may not be loaded. Check inside the guest:

    ```bash title="Check balloon driver" theme={null}
    lsmod | grep virtio_balloon
    ```

    If not loaded: `modprobe virtio_balloon` and add `virtio_balloon` to `/etc/modules`
    for persistence.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Admin: Enable Live Scaling" href="/services/compute/live-resize-admin">
    Configure flavors with hot-add extra specs and resource bounds
  </Card>

  <Card title="Resize Instance" href="/services/compute/resize-instance">
    Standard flavor resize — change vCPU, RAM, and disk with a reboot
  </Card>

  <Card title="Flavors" href="/services/compute/flavors">
    View available flavors and their hot-add capabilities
  </Card>

  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Create an instance with a hot-add-enabled flavor
  </Card>
</CardGroup>


# Live vCPU & RAM Scaling
Source: https://docs.xloud.tech/services/compute/live-resize-admin

Scale instance vCPU and memory in real-time without rebooting. Configure hot-add flavors and manage resource envelopes.

## Overview

Live vCPU and RAM scaling adjusts an instance's compute resources while it is running —
no reboot required. This is distinct from the standard resize operation, which requires
a reboot. Live scaling uses a combination of CPU hotplug, memory balloon, DIMM hotplug,
and virtio-mem to adjust resources within the bounds defined by the instance's flavor.

<Info>
  **Xloud-Developed** — Bidirectional CPU hotplug (add AND remove vCPUs) and
  combined memory scaling (balloon + DIMM + virtio-mem) are developed by Xloud and
  ship with XAVS / XPCI.
</Info>

<Note>
  **Prerequisites**

  * An instance in `Active` status with a hot-add-enabled flavor
  * The flavor must have `hw:cpu_min` or `hw:mem_min` extra specs set
  * The instance must not be locked
</Note>

***

## Adjust Resources (Dashboard)

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Resource Adjustment dialog">
        Navigate to **Compute > Instances**. Click the **More** dropdown on the
        instance row, then select **Adjust Resources** under the **Configuration
        Update** group.

        <Note>
          This action is only available for instances in `Active` status whose
          flavor has hot-add enabled (`hw:cpu_min` or `hw:mem_min` extra specs).
          If the flavor does not support hot-add, this action does not appear.
        </Note>
      </Step>

      <Step title="Adjust vCPU and memory">
        The dialog shows:

        | Field              | Description                                                                 |
        | ------------------ | --------------------------------------------------------------------------- |
        | **Instance**       | Instance name (read-only)                                                   |
        | **Current Info**   | Current vCPU count and memory allocation (read-only)                        |
        | **vCPUs**          | Slider input — adjust between minimum and maximum vCPUs                     |
        | **Memory (GiB)**   | Slider input — adjust between minimum and maximum memory (0.25 GiB steps)   |
        | **Make permanent** | Checkbox — persist the configuration after a soft reboot (default: checked) |

        The vCPU slider step aligns to the CPU thread count configured in the flavor.
        The memory slider uses 0.25 GiB increments.

        The dialog fetches the live instance status to determine current and
        allowed resource ranges. Color-coded hints indicate the scaling method:

        | Color  | Meaning                                          |
        | ------ | ------------------------------------------------ |
        | Green  | virtio-mem — fully adjustable in both directions |
        | Yellow | No balloon driver detected — limited scaling     |
        | Blue   | DIMM hotplug information                         |
      </Step>

      <Step title="Apply the adjustment">
        Click **Confirm**. The resources are adjusted immediately while the
        instance continues running.

        <Check>
          Instance vCPU and memory updated in real-time. Verify from the instance
          detail page or by running `nproc` / `free -h` inside the guest.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="Check current live resource status" theme={null}
    openstack server show <INSTANCE_ID> -c "xloud-status"
    ```

    ```bash title="Adjust vCPU count" theme={null}
    curl -X POST -H "X-Auth-Token: $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"current_vcpus": 4}' \
      "$NOVA_ENDPOINT/v2.1/os-xloud-adjust/<INSTANCE_ID>"
    ```

    ```bash title="Adjust memory" theme={null}
    curl -X POST -H "X-Auth-Token: $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"current_memory_mb": 8192}' \
      "$NOVA_ENDPOINT/v2.1/os-xloud-adjust/<INSTANCE_ID>"
    ```

    <Note>
      Live resource adjustment uses the Xloud-developed Nova API extension.
      The `openstack` CLI does not have a native command for this operation.
    </Note>
  </Tab>
</Tabs>

***

## Configure Hot-Add Flavors

To enable live scaling, create flavors with hot-add parameters:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    When creating a flavor (see [Flavor Management](/services/compute/flavors)),
    enable the **Hot-Add** toggle in the **Step 1 (Params Setting)** section.
    Configure:

    | Field                    | Description                                                      |
    | ------------------------ | ---------------------------------------------------------------- |
    | **Enable Hot-Add**       | Set to Yes                                                       |
    | **Minimum CPU**          | Starting vCPU count at boot (can scale up to flavor's max vCPUs) |
    | **Minimum Memory (GiB)** | Starting memory at boot (can scale up to flavor's max memory)    |

    The instance boots with the minimum resources and can be scaled up to the
    flavor's maximum without rebooting.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a hot-add enabled flavor" theme={null}
    openstack flavor create \
      --vcpus 8 --ram 16384 --disk 0 \
      --public \
      hotadd.large

    openstack flavor set \
      --property hw:cpu_min=2 \
      --property hw:mem_min=4096 \
      hotadd.large
    ```

    This creates a flavor where instances boot with 2 vCPUs / 4 GiB RAM and
    can scale up to 8 vCPUs / 16 GiB RAM while running.
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Resize an Instance" href="/services/compute/resize-instance">
    Change flavors entirely (requires reboot) for capacity beyond hot-add range
  </Card>

  <Card title="Flavors" href="/services/compute/flavors">
    Create and manage flavors with hot-add configuration
  </Card>

  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Launch instances with hot-add enabled flavors
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve live scaling failures and balloon driver issues
  </Card>
</CardGroup>


# Manage IP Addresses
Source: https://docs.xloud.tech/services/compute/manage-ips

Associate and disassociate floating IPs to provide external access to Xloud Compute instances using the Dashboard or CLI.

## Overview

Instances receive fixed IP addresses from their attached networks automatically. To make
an instance reachable from external networks, associate a floating IP address from a
public IP pool. Floating IPs can be moved between instances as needed.

<Note>
  **Prerequisites**

  * An instance with at least one fixed IP address
  * A floating IP pool configured by your administrator
  * An external network with available floating IPs
</Note>

***

## Associate a Floating IP

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Associate Floating IP dialog">
        Navigate to **Compute > Instances**. Click the **More** dropdown on the
        instance row, then select **Associate Floating IP** under the
        **Related Resources** group.

        <Note>
          This action is available when the instance has at least one fixed IP
          that is not already associated with a floating IP. It is not available
          on the admin page or when the instance is in `Error` status.
        </Note>
      </Step>

      <Step title="Select the fixed IP">
        The dialog shows:

        | Field        | Description                                                          |
        | ------------ | -------------------------------------------------------------------- |
        | **Instance** | Current instance name (read-only)                                    |
        | **Fixed IP** | Select the instance port/interface to associate the floating IP with |

        The fixed IP table shows available ports with their network, subnet, and
        current IP information. Ports that already have a floating IP or are on
        unreachable subnets are indicated with reason labels.
      </Step>

      <Step title="Select a floating IP">
        Choose an available floating IP from the pool, or allocate a new one.

        Click **Confirm** to associate the floating IP.

        <Check>The floating IP appears in the instance's IP addresses list.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="List available floating IPs" theme={null}
      openstack floating ip list --status DOWN
      ```

      ```bash title="Create a new floating IP (if none available)" theme={null}
      openstack floating ip create <EXTERNAL_NETWORK>
      ```

      ```bash title="Associate floating IP to instance" theme={null}
      openstack server add floating ip <INSTANCE_ID> <FLOATING_IP>
      ```

      ```bash title="Associate to a specific fixed IP" theme={null}
      openstack server add floating ip \
        --fixed-ip-address <FIXED_IP> \
        <INSTANCE_ID> <FLOATING_IP>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Disassociate a Floating IP

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Disassociate dialog">
        Navigate to **Compute > Instances**. Click the **More** dropdown on the
        instance row, then select **Disassociate Floating Ip** under the
        **Related Resources** group.

        <Note>
          This action is only available when the instance has at least one
          associated floating IP.
        </Note>
      </Step>

      <Step title="Select the floating IP to remove">
        | Field        | Description                                              |
        | ------------ | -------------------------------------------------------- |
        | **Instance** | Current instance name (read-only)                        |
        | **Address**  | Select the floating IP to disassociate from the dropdown |

        Click **Confirm**. The floating IP is released from the instance but
        remains allocated to your project for future use.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Remove floating IP from instance" theme={null}
    openstack server remove floating ip <INSTANCE_ID> <FLOATING_IP>
    ```

    ```bash title="Release floating IP back to pool (optional)" theme={null}
    openstack floating ip delete <FLOATING_IP_ID>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Security Groups" href="/services/compute/security-groups">
    Configure firewall rules to allow traffic on the floating IP
  </Card>

  <Card title="Floating IPs" href="/services/networking/floating-ips">
    Manage floating IP pools and allocations at the network level
  </Card>

  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Create a new instance with network configuration
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve floating IP association failures
  </Card>
</CardGroup>


# Quota Management
Source: https://docs.xloud.tech/services/compute/quotas

Set and enforce per-project resource ceilings in Xloud Compute. Manage instance counts, vCPU allocations, RAM limits, and key pair quotas.

## Overview

Quotas enforce per-project resource limits to prevent over-consumption and ensure fair
resource distribution across tenants. Xloud Compute tracks quotas for instances, vCPUs,
RAM, server groups, key pairs, and other compute resources.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator access for modifying quotas
  * Users can view their own project quota usage in the instance create wizard
</Note>

***

## Default Compute Quotas

| Resource                 | Default Limit | Description                             |
| ------------------------ | ------------- | --------------------------------------- |
| **Instances**            | 10            | Maximum number of instances per project |
| **Cores**                | 20            | Maximum vCPUs across all instances      |
| **RAM**                  | 51200 MB      | Maximum RAM across all instances        |
| **Key Pairs**            | 100           | Maximum SSH key pairs per user          |
| **Server Groups**        | 10            | Maximum server groups per project       |
| **Server Group Members** | 10            | Maximum instances per server group      |

***

## View Quota Usage

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Quota usage is displayed in real-time during instance creation. The
    **Instance Create wizard** (Step 4: Confirm Config) shows a footer badge with
    current usage for: Instances, CPU, Memory, Volumes, Volume Capacity, and
    Server Group Members.

    If any quota would be exceeded, the badge turns red and the submit button
    is disabled.

    <Note>
      The Dashboard does not provide a dedicated quota management page. Quota
      modifications are performed via the CLI or through
      [XDeploy](/deployment) configuration.
    </Note>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Show quota for a project" theme={null}
      openstack quota show <PROJECT_ID>
      ```

      ```bash title="Show current usage" theme={null}
      openstack quota show --usage <PROJECT_ID>
      ```

      ```bash title="Show default quotas" theme={null}
      openstack quota show --default
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Modify Quotas

<Tabs>
  <Tab title="XDeploy" icon="settings">
    <Steps>
      <Step title="Open XDeploy Configuration">
        Open **XDeploy > Configuration > Advance Features**.
      </Step>

      <Step title="Adjust quota defaults">
        Modify the default quota values in the configuration. Save and run
        **Operations > Reconfigure** to apply.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Increase instance quota for a project" theme={null}
      openstack quota set --instances 50 <PROJECT_ID>
      ```

      ```bash title="Increase vCPU quota" theme={null}
      openstack quota set --cores 100 <PROJECT_ID>
      ```

      ```bash title="Increase RAM quota (in MB)" theme={null}
      openstack quota set --ram 102400 <PROJECT_ID>
      ```

      ```bash title="Set unlimited quota (-1)" theme={null}
      openstack quota set --instances -1 <PROJECT_ID>
      ```
    </CodeGroup>

    <Warning>
      Setting a quota to `-1` removes the limit entirely. Use with caution —
      unlimited quotas can lead to resource exhaustion on the cluster.
    </Warning>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    See quota enforcement in action during instance creation
  </Card>

  <Card title="Flavors" href="/services/compute/flavors">
    Manage flavor sizes that consume quota
  </Card>

  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    Monitor cluster-wide resource availability
  </Card>

  <Card title="Storage Quotas" href="/services/storage/quotas">
    Manage block storage volume and capacity quotas
  </Card>
</CardGroup>


# Reboot an Instance
Source: https://docs.xloud.tech/services/compute/reboot-instance

Perform hard or soft reboots on Xloud Compute instances using the Dashboard or CLI. Understand the difference between reboot types and when to use each.

## Overview

Rebooting an instance restarts the guest operating system. Xloud Compute supports
two reboot types: **hard reboot** (equivalent to a power cycle) and **soft reboot**
(sends an ACPI shutdown signal and restarts). Both are available as individual and
batch operations.

<Note>
  **Prerequisites**

  * An instance in `Active` or `Shutoff` status (hard reboot) or `Active` status (soft reboot)
  * The instance must not be locked (unless you are an administrator)
</Note>

***

## Reboot Types

| Type            | Status Required       | Behavior                                                        | Use Case                                  |
| --------------- | --------------------- | --------------------------------------------------------------- | ----------------------------------------- |
| **Hard Reboot** | `Active` or `Shutoff` | Immediate power cycle — equivalent to pressing the reset button | Unresponsive instance, frozen OS          |
| **Soft Reboot** | `Active` only         | Graceful ACPI shutdown signal followed by restart               | Routine restarts, applying config changes |

<Tip>
  Prefer **Soft Reboot** for routine restarts — it allows the OS to flush disk buffers
  and cleanly shut down services. Use **Hard Reboot** only when the instance is
  unresponsive to soft reboot.
</Tip>

<Note>
  Soft Reboot is not available for bare metal instances.
</Note>

***

## Reboot an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Instances">
        Navigate to **Compute > Instances** in the sidebar.
      </Step>

      <Step title="Select the reboot type">
        Click the **More** dropdown on the instance row. Under **Instance Status**,
        select either:

        * **Reboot** — performs a hard reboot
        * **Soft Reboot** — performs a graceful reboot

        Confirm the action in the dialog.

        <Note>
          Both actions are available as batch operations. Select multiple instances
          using checkboxes and choose **Reboot** or **Soft Reboot** from the batch
          actions bar.
        </Note>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Hard reboot" theme={null}
      openstack server reboot --hard <INSTANCE_ID>
      ```

      ```bash title="Soft reboot" theme={null}
      openstack server reboot --soft <INSTANCE_ID>
      ```
    </CodeGroup>

    <Check>Instance returns to `Active` status after reboot completes.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Resize an Instance" href="/services/compute/resize-instance">
    Change the flavor of an instance (requires reboot)
  </Card>

  <Card title="Rescue an Instance" href="/services/compute/rescue-instance">
    Boot from a rescue image to fix a broken OS
  </Card>

  <Card title="Instance Snapshots" href="/services/compute/snapshots">
    Create a snapshot before performing maintenance
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve unresponsive instances and failed reboots
  </Card>
</CardGroup>


# Rescue an Instance
Source: https://docs.xloud.tech/services/compute/rescue-instance

Boot an Xloud Compute instance from a rescue image to recover from OS boot failures, corrupted filesystems, or lost SSH access.

## Overview

Rescue mode boots an instance from a temporary rescue image while preserving the
original root disk as a secondary device. This allows you to diagnose and repair
boot failures, corrupted filesystems, or locked-out SSH configurations without
destroying the instance.

<Note>
  **Prerequisites**

  * An instance in `Active` or `Shutoff` status
  * The instance must not be locked
  * CLI access required — the Rescue action is not available in the Dashboard GUI
</Note>

<Warning>
  The **Rescue** and **Unrescue** actions are **CLI-only** operations. They are not
  available in the Xloud Dashboard. Use the CLI commands below to enter and exit
  rescue mode.
</Warning>

***

## Enter Rescue Mode

<Tabs>
  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Rescue the instance">
        ```bash title="Enter rescue mode (default rescue image)" theme={null}
        openstack server rescue <INSTANCE_ID>
        ```

        ```bash title="Rescue with a specific image" theme={null}
        openstack server rescue \
          --image <RESCUE_IMAGE_ID> \
          <INSTANCE_ID>
        ```

        The instance reboots into rescue mode. The original root disk is attached
        as a secondary device (typically `/dev/vdb`).
      </Step>

      <Step title="Connect and repair">
        SSH into the instance using the rescue image credentials. Mount the
        original root disk and perform repairs:

        ```bash title="Mount the original root disk" theme={null}
        sudo mount /dev/vdb1 /mnt
        ```

        ```bash title="Example: fix SSH config" theme={null}
        sudo vi /mnt/etc/ssh/sshd_config
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Exit Rescue Mode

```bash title="Unrescue the instance" theme={null}
openstack server unrescue <INSTANCE_ID>
```

The instance reboots from its original root disk with the repairs applied.

<Check>Instance returns to `Active` status on its original root disk.</Check>

***

## Next Steps

<CardGroup>
  <Card title="Instance Snapshots" href="/services/compute/snapshots">
    Create a snapshot before rescue as a safety measure
  </Card>

  <Card title="Reboot an Instance" href="/services/compute/reboot-instance">
    Try a reboot before rescue for less severe issues
  </Card>

  <Card title="Console Access" href="/services/compute/console-access">
    Use the VNC console for out-of-band diagnostics
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve boot failures and rescue mode issues
  </Card>
</CardGroup>


# Resize an Instance
Source: https://docs.xloud.tech/services/compute/resize-instance

Change the flavor of an Xloud Compute instance to adjust vCPU, RAM, and disk allocation. Confirm or revert the resize before it is finalized.

## Overview

Resizing an instance changes its flavor — adjusting the vCPU, RAM, and disk allocation
profile. The resize operation stops the instance, moves it to a host that satisfies the
new flavor requirements if necessary, and restarts it. After the resize completes, a
confirmation window allows you to accept or revert to the original flavor.

<Warning>
  Resizing always requires a reboot. Schedule resize operations during a maintenance window
  for production workloads. For zero-downtime vertical scaling on supported instances, use
  [Live vCPU/RAM Scaling](/services/compute/live-resize) — available through the Dashboard
  with no reboot required.
</Warning>

<Note>
  **Prerequisites**

  * An instance in `Active` or `Shutoff` status
  * The instance must not be locked
  * A target flavor with a root disk size greater than or equal to the current root disk
  * Sufficient quota for the new flavor's vCPU and RAM requirements
</Note>

***

## Resize Workflow

| Phase                     | Instance Status | Action Required                     |
| ------------------------- | --------------- | ----------------------------------- |
| **Resize initiated**      | `Resize`        | Automatic — no action required      |
| **Awaiting confirmation** | `Verify Resize` | Confirm or revert within the window |
| **Confirmed**             | `Active`        | Instance running on new flavor      |

***

## Resize an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Resize dialog">
        Navigate to **Compute > Instances**. Click the **More** dropdown on the
        instance row, then select **Resize** under the **Configuration Update** group.

        <Note>
          The Resize action is only available for instances in `Active` or `Shutoff`
          status that are not locked. It is not available on the admin page or for
          bare metal instances.
        </Note>
      </Step>

      <Step title="Select a new flavor">
        The Resize dialog shows:

        | Field              | Description                                        |
        | ------------------ | -------------------------------------------------- |
        | **Instance**       | Current instance name (read-only)                  |
        | **Current Flavor** | Current flavor details — vCPUs and RAM (read-only) |
        | **Flavor**         | Select a new flavor from the table                 |

        The flavor table uses the same architecture and category filters as the
        instance create wizard. Flavors that would exceed your quota are disabled.

        Real-time quota usage is shown for **CPU** and **Memory (GiB)**.
      </Step>

      <Step title="Acknowledge the shutdown">
        Check the **Forced Shutdown** checkbox ("Agree to force shutdown"). This is
        required — the instance will be shut down during the resize operation.
      </Step>

      <Step title="Confirm the resize">
        Click **Confirm**. The instance transitions to `Resize` status.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="Resize an instance" theme={null}
    openstack server resize \
      --flavor <NEW_FLAVOR> \
      <INSTANCE_ID>
    ```

    ```bash title="Check resize status" theme={null}
    openstack server show <INSTANCE_ID> -c status
    ```
  </Tab>
</Tabs>

***

## Confirm or Revert the Resize

After the resize completes, the instance enters `Verify Resize` status. You must
confirm or revert within the confirmation window (default: 24 hours).

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Instances**. The instance shows status `Verify Resize`.
    Click the **More** dropdown and select either:

    * **Confirm Resize or Migrate** — accept the new flavor permanently
    * **Revert Resize or Migrate** — roll back to the original flavor

    <Check>After confirmation, the instance returns to `Active` status on the new flavor.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Confirm the resize" theme={null}
      openstack server resize confirm <INSTANCE_ID>
      ```

      ```bash title="Revert the resize" theme={null}
      openstack server resize revert <INSTANCE_ID>
      ```
    </CodeGroup>

    <Note>
      If you neither confirm nor revert within the confirmation window, the resize
      is automatically confirmed. Contact your administrator to adjust the
      confirmation window duration.
    </Note>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Live vCPU/RAM Scaling" href="/services/compute/live-resize">
    Scale vCPU and memory without rebooting on supported flavors
  </Card>

  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Create a new instance with the desired flavor from the start
  </Card>

  <Card title="Flavors" href="/services/compute/flavors">
    View available flavors and their specifications
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve stuck resize operations and confirmation timeouts
  </Card>
</CardGroup>


# Compute Scheduling
Source: https://docs.xloud.tech/services/compute/scheduling

Configure host aggregates and availability zones in Xloud Compute to control how instances are placed across the cluster.

## Overview

The Xloud Compute scheduler determines which physical host runs each new instance.
Administrators configure the scheduler behavior through host aggregates (logical
groupings of hosts with shared properties) and availability zones (fault domains).
The Dashboard provides management interfaces for both.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator access to the Xloud Dashboard (admin view)
</Note>

***

## Host Aggregates

Host aggregates group compute hosts with shared characteristics (e.g., SSD storage,
GPU hardware, high-memory nodes). The scheduler uses aggregate metadata to match
instances with compatible hosts.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Host Aggregates** in the admin sidebar. The **Host Aggregate**
    tab shows:

    | Column                | Description                             |
    | --------------------- | --------------------------------------- |
    | **Name**              | Aggregate name                          |
    | **Availability Zone** | Zone associated with this aggregate     |
    | **Hosts**             | List of compute hosts in the aggregate  |
    | **Metadata**          | Key-value properties (e.g., `ssd=true`) |
    | **Created At**        | Creation timestamp                      |

    **Available actions**:

    | Action              | Location         | Description                                    |
    | ------------------- | ---------------- | ---------------------------------------------- |
    | **Edit**            | First row action | Change name and availability zone              |
    | **Manage Host**     | More dropdown    | Add or remove compute hosts from the aggregate |
    | **Manage Metadata** | More dropdown    | Set key-value metadata properties              |
    | **Delete**          | More dropdown    | Delete the aggregate                           |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="List host aggregates" theme={null}
      openstack aggregate list
      ```

      ```bash title="Show aggregate details" theme={null}
      openstack aggregate show <AGGREGATE_NAME>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Create a Host Aggregate

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Create Aggregate dialog">
        Navigate to **Compute > Host Aggregates**. Click **Create Host Aggregate**.
      </Step>

      <Step title="Configure the aggregate">
        | Field                            | Type           | Required                     | Description                                |
        | -------------------------------- | -------------- | ---------------------------- | ------------------------------------------ |
        | **Name**                         | Text           | Yes                          | Aggregate name                             |
        | **Create New Availability Zone** | Radio (Yes/No) | Yes                          | Whether to create a new AZ or use existing |
        | **Availability Zone**            | Dropdown       | Required if not creating new | Select existing zone                       |
        | **New Availability Zone**        | Text           | Required if creating new     | Name for the new zone                      |

        Click **Confirm**.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Create aggregate with existing zone" theme={null}
      openstack aggregate create --zone <ZONE_NAME> <AGGREGATE_NAME>
      ```

      ```bash title="Create aggregate with new zone" theme={null}
      openstack aggregate create --zone new-zone <AGGREGATE_NAME>
      ```

      ```bash title="Add hosts to aggregate" theme={null}
      openstack aggregate add host <AGGREGATE_NAME> <HOSTNAME>
      ```

      ```bash title="Set metadata" theme={null}
      openstack aggregate set --property ssd=true <AGGREGATE_NAME>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Manage Hosts in an Aggregate

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click the **More** dropdown on an aggregate row and select **Manage Host**.

    The dialog shows a multi-select table of all compute hosts (service: `nova-compute`).
    Currently assigned hosts are pre-selected. Toggle hosts to add or remove them.

    The table shows: Host, Availability Zone, Admin Status, State, Last Updated.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Add host to aggregate" theme={null}
      openstack aggregate add host <AGGREGATE_NAME> <HOSTNAME>
      ```

      ```bash title="Remove host from aggregate" theme={null}
      openstack aggregate remove host <AGGREGATE_NAME> <HOSTNAME>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Availability Zones

The **Availability Zones** tab on the Host Aggregates page shows a read-only list
of all configured zones. Zones are created through host aggregates — each aggregate
can be associated with one availability zone.

<Note>
  Availability zones are a subset of host aggregates. Create an aggregate with a
  zone name to define a new availability zone. See
  [Availability Zones](/services/compute/availability-zones) for user-facing
  documentation on zone selection during instance launch.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    Monitor hypervisor resource utilization
  </Card>

  <Card title="Availability Zones" href="/services/compute/availability-zones">
    Understand fault domains and zone placement
  </Card>

  <Card title="Live Migration" href="/services/compute/live-migration">
    Move instances between hosts for maintenance
  </Card>

  <Card title="Flavors" href="/services/compute/flavors">
    Create flavors with aggregate-matching extra specs
  </Card>
</CardGroup>


# Security Groups
Source: https://docs.xloud.tech/services/compute/security-groups

Manage security groups on Xloud Compute instances. Add or remove security groups per network interface using the Dashboard or CLI.

## Overview

Security groups act as virtual firewalls that control inbound and outbound network
traffic for instances. Each instance port can have one or more security groups
assigned. Security group rules apply to all traffic on that port.

<Note>
  **Prerequisites**

  * An active instance with at least one network interface
  * Existing security groups with configured rules
  * For creating security groups, see [Network Security Groups](/services/networking/security-groups)
</Note>

***

## Manage Security Groups on an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Manage Security Group dialog">
        Navigate to **Compute > Instances**. Click the **More** dropdown on the
        instance row, then select **Manage Security Group** under the
        **Related Resources** group.

        <Note>
          This action is always available regardless of instance status. No lock
          check is required.
        </Note>
      </Step>

      <Step title="Select a port">
        The dialog shows:

        | Field        | Description                                          |
        | ------------ | ---------------------------------------------------- |
        | **Instance** | Current instance name (read-only)                    |
        | **Port**     | Select the network interface to configure (required) |

        The port table shows all instance interfaces with:

        | Column           | Description                 |
        | ---------------- | --------------------------- |
        | **ID**           | Port identifier             |
        | **Network Name** | Network the port belongs to |
        | **IPv4 Address** | Assigned IPv4 address       |
        | **IPv6 Address** | Assigned IPv6 address       |
        | **MAC Address**  | Hardware address            |
        | **Status**       | Port status                 |

        <Warning>
          Ports with **port security disabled** cannot have security groups
          assigned. These ports are shown as disabled in the selection table.
        </Warning>
      </Step>

      <Step title="Select security groups">
        After selecting a port, the **Security Group** table loads with the port's
        current security groups pre-selected.

        Add or remove security groups by toggling the checkboxes. Multiple
        security groups can be assigned to a single port.

        Click **Confirm** to apply the changes.

        <Check>Security group assignment updated for the selected port.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Add a security group to an instance" theme={null}
      openstack server add security group <INSTANCE_ID> <SECURITY_GROUP>
      ```

      ```bash title="Remove a security group from an instance" theme={null}
      openstack server remove security group <INSTANCE_ID> <SECURITY_GROUP>
      ```

      ```bash title="List security groups on an instance" theme={null}
      openstack server show <INSTANCE_ID> -c security_groups
      ```
    </CodeGroup>

    <Note>
      CLI commands apply security groups to all ports on the instance. For
      per-port security group management, use the `openstack port set` command:

      ```bash title="Set security groups on a specific port" theme={null}
      openstack port set --security-group <SG_ID> <PORT_ID>
      ```
    </Note>
  </Tab>
</Tabs>

***

## View Instance Security Groups

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Instances** and click the instance name. Go to the
    **Security Groups** tab to view all security groups assigned to each interface.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List security groups on an instance" theme={null}
    openstack server show <INSTANCE_ID> -c security_groups
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Network Security Groups" href="/services/networking/security-groups">
    Create security groups and configure inbound/outbound rules
  </Card>

  <Card title="Manage IP Addresses" href="/services/compute/manage-ips">
    Associate floating IPs to make instances reachable externally
  </Card>

  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Assign security groups during instance creation
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve connectivity issues related to security group rules
  </Card>
</CardGroup>


# Compute Security Hardening
Source: https://docs.xloud.tech/services/compute/security-hardening

Harden the Xloud Compute control plane with metadata protection, live migration TLS, and API rate limiting.

## Overview

The Xloud Compute control plane manages hypervisor hosts, instance lifecycle, and
metadata delivery. Securing it against unauthorized access, network eavesdropping, and
API abuse is critical in any production deployment. This guide covers the three primary
hardening areas: metadata service protection, live migration TLS, and API rate limiting.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Danger>
  The following hardening measures must be applied to all production deployments.
  Failure to secure the compute control plane exposes hypervisor hosts, instance
  metadata, and inter-node communication to unauthorized access and data interception.
</Danger>

<Note>
  **Prerequisites**

  * Administrator credentials sourced (`source openrc.sh`)
  * XDeploy access for host-level configuration changes
  * TLS certificates issued by the cluster CA (required for live migration TLS)
</Note>

***

## Metadata Service Protection

The instance metadata service (`169.254.169.254`) is accessible from every running
instance by default. It delivers user data, SSH public keys, and cloud-init configuration.
Unrestricted access to this endpoint is a common attack vector in multi-tenant environments.

<AccordionGroup>
  <Accordion title="Restrict metadata access via security groups" icon="shield">
    Apply security group rules to limit which instances can reach the metadata endpoint.
    In environments where instances do not require cloud-init or credential injection at
    boot, block metadata access entirely.

    ```bash title="Verify instance metadata service status" theme={null}
    openstack compute service list
    ```

    Configure metadata access restrictions through XDeploy under **Compute → Advanced
    Settings → Metadata Security**.
  </Accordion>

  <Accordion title="Enable metadata service authentication" icon="key">
    Metadata service authentication requires instances to present a signed token when
    requesting user data and credentials. This prevents unauthorized metadata reads from
    compromised instances or SSRF attacks.

    Enable authenticated metadata through XDeploy under **Compute → Advanced Settings →
    Metadata Security → Require Authentication**.

    <Warning>
      Enabling metadata authentication requires cloud-init 20.3 or later in the guest OS.
      Instances running older cloud-init versions will fail to retrieve their cloud
      configuration at boot. Verify guest OS compatibility before enabling this setting
      in production.
    </Warning>
  </Accordion>

  <Accordion title="Apply API rate limiting on the metadata endpoint" icon="gauge">
    Configure rate limits on the metadata endpoint to prevent abuse:

    * Maximum 60 requests per minute per instance
    * Temporary block after 3 consecutive rate limit violations within 5 minutes

    Configure endpoint rate limits through XDeploy under **Compute → Security →
    Metadata Rate Limiting**.
  </Accordion>
</AccordionGroup>

***

## Live Migration TLS

By default, live migration transfers instance memory and disk data over the management
network without encryption. Enable TLS for live migration to protect in-flight instance
data from network interception.

<AccordionGroup>
  <Accordion title="Enable live migration TLS" icon="lock">
    Live migration TLS encrypts the migration data channel between source and destination
    hypervisor nodes using mutual TLS authentication.

    ```bash title="Verify current live migration configuration" theme={null}
    openstack compute service list --long
    ```

    Enable TLS through XDeploy under **Compute → Security → Live Migration TLS**.
    This requires all compute nodes to have valid TLS certificates issued by the cluster CA.

    <Warning>
      Enabling live migration TLS requires restarting the Compute Agent on all hypervisor
      nodes. Schedule this change during a maintenance window. Instances remain running
      during the agent restart, but live migrations cannot be initiated until all nodes
      complete the restart.
    </Warning>
  </Accordion>

  <Accordion title="Verify TLS is active after enabling" icon="circle-check">
    After enabling live migration TLS and restarting all compute agents, verify the
    configuration is active:

    ```bash title="Check compute agent configuration on a host" theme={null}
    openstack hypervisor show <hostname>
    ```

    Initiate a test live migration between two hosts and verify in the migration log
    that the connection uses TLS. Inspect the migration network traffic to confirm
    data is encrypted.

    <Check>
      A successful live migration after enabling TLS confirms the configuration is
      working. Any migration failure at this stage typically indicates a certificate
      validation error — verify certificate validity and CA trust chain on all nodes.
    </Check>
  </Accordion>
</AccordionGroup>

***

## API Rate Limiting

The Compute API does not enforce rate limits by default. Without rate limiting, malicious
actors or misconfigured automation can issue thousands of API requests per second,
degrading control plane performance and enabling denial-of-service conditions.

<AccordionGroup>
  <Accordion title="Configure rate limits" icon="gauge">
    Xloud recommends the following rate limits for production deployments:

    | Operation Type                     | Recommended Limit | Scope          |
    | ---------------------------------- | ----------------- | -------------- |
    | Write operations (POST/PUT/DELETE) | 100 per minute    | Per user       |
    | Read operations (GET)              | 1,000 per minute  | Per user       |
    | Admin operations                   | 500 per minute    | Per admin user |

    Apply these limits at the load balancer layer or via the Compute API service
    configuration through XDeploy under **Compute → Security → API Rate Limiting**.
  </Accordion>

  <Accordion title="Enable automatic temporary bans" icon="ban">
    Configure automatic temporary bans for clients that repeatedly exceed rate limits:

    * Clients exceeding the rate limit threshold 3 times within 5 minutes receive a
      temporary 15-minute block
    * All blocked requests return `HTTP 429 Too Many Requests`
    * Block events are logged for security audit review

    Enable automatic bans through XDeploy under **Compute → Security → API Rate
    Limiting → Automatic Blocking**.

    <Tip>
      Review API access logs regularly for patterns that indicate credential stuffing,
      automated instance enumeration, or quota exhaustion attacks. Set up an alert
      in Xloud Monitoring (XIMP) for sustained `429` error rates above 1% of total
      API traffic.
    </Tip>
  </Accordion>
</AccordionGroup>

***

## Security Checklist

Review these items on every production deployment.

| Control                                              | Configured Via                                    | Priority |
| ---------------------------------------------------- | ------------------------------------------------- | -------- |
| Metadata service authentication enabled              | XDeploy → Compute → Metadata Security             | High     |
| Metadata endpoint rate limiting applied              | XDeploy → Compute → Metadata Rate Limiting        | High     |
| Live migration TLS enabled                           | XDeploy → Compute → Security → Live Migration TLS | High     |
| API rate limiting configured                         | XDeploy → Compute → Security → API Rate Limiting  | Medium   |
| Console proxy ports firewalled to admin CIDRs        | Firewall / security group rules                   | Medium   |
| Admin API endpoints restricted to management network | Load balancer / HAProxy ACL                       | High     |

***

## Next Steps

<CardGroup>
  <Card title="Live Migration" href="/services/compute/live-migration">
    Configure and test live migration after enabling TLS on the migration channel.
  </Card>

  <Card title="Advanced Features" href="/services/compute/advanced-features">
    Enable vTPM and UEFI Secure Boot for instance-level hardware security.
  </Card>

  <Card title="Admin Guide" href="/services/compute/admin-guide">
    Return to the Compute Administration Guide index.
  </Card>
</CardGroup>


# Server Groups
Source: https://docs.xloud.tech/services/compute/server-groups

Control instance placement across physical hosts using affinity and anti-affinity policies. Create and manage server groups in the Xloud Dashboard or CLI.

## Overview

Server groups enforce placement policies that determine how a set of instances is
distributed across physical compute hosts. Use server groups to guarantee high availability
through host separation, improve performance through co-location, or meet compliance
requirements for workload isolation.

<Note>
  **Prerequisites**

  * An active Xloud project
  * At least two compute hosts available (required for `anti-affinity` enforcement)
  * Sufficient server group quota
</Note>

***

## Placement Policies

| Policy                 | Behavior                                                         | Failure Mode                                                    |
| ---------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------- |
| **affinity**           | All instances placed on the **same** physical host               | Launch fails if the target host cannot accommodate the instance |
| **anti-affinity**      | Each instance placed on a **different** physical host            | Launch fails if insufficient distinct hosts are available       |
| **soft-affinity**      | Prefer the same host; allow different hosts if required          | Launch succeeds even when co-location is not possible           |
| **soft-anti-affinity** | Prefer different hosts; allow same host if no alternative exists | Launch succeeds even when distinct hosts are unavailable        |

<Tip>
  Use `anti-affinity` for production replicated services (web tiers, application clusters,
  database replicas) where a single host failure must not take down all instances simultaneously.
  Use `soft-anti-affinity` when you want best-effort separation without hard placement failures.
</Tip>

***

## Create a Server Group

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Server Groups">
        Navigate to **Compute > Server Groups** in the sidebar.
        Click **Create Server Group**.
      </Step>

      <Step title="Configure the server group">
        | Field      | Type       | Required | Description                            |
        | ---------- | ---------- | -------- | -------------------------------------- |
        | **Name**   | Text input | Yes      | Human-readable name for the group      |
        | **Policy** | Dropdown   | Yes      | Placement policy — see the table above |

        The dropdown options are:

        * `affinity` — instances on same physical machine (mandatory)
        * `anti-affinity` — instances on different machines (mandatory)
        * `soft-affinity` — same machine preferred (best effort)
        * `soft-anti-affinity` — different machines preferred (best effort)

        The dialog also displays your current **Server Groups** quota usage
        (used / limit). If your quota is exhausted, the submit button is disabled.
      </Step>

      <Step title="Create the group">
        Click **Confirm**. The server group appears in the list.

        <Check>Server group appears in the list with the selected policy.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Create anti-affinity group" theme={null}
      openstack server group create \
        --policy anti-affinity \
        ha-web-tier
      ```

      ```bash title="Create soft-affinity group" theme={null}
      openstack server group create \
        --policy soft-affinity \
        collocated-cache
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## View Server Groups

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Server Groups**. The list shows:

    | Column           | Description                                  |
    | ---------------- | -------------------------------------------- |
    | **ID/Name**      | Group identifier (clickable to view details) |
    | **Member Count** | Number of instances in the group             |
    | **Policy**       | Placement policy                             |

    Filter by **Name** or **Policy** using the search/filter bar.

    Click a group name to view the detail page, which shows the **Members** tab
    listing all instances assigned to this server group.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List server groups" theme={null}
      openstack server group list
      ```

      ```bash title="Show server group details" theme={null}
      openstack server group show <GROUP_ID>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Launch Instances in a Server Group

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    When creating an instance, expand the **Advanced Options** section in **Step 3
    (System Config)** of the wizard. Select your server group from the **Server Group**
    table.

    Alternatively, from the server group detail page, click the **Create Instance**
    action in the **More** dropdown. The server group is pre-selected in the wizard.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Launch instance in a server group" theme={null}
    openstack server create \
      --image <IMAGE_ID> \
      --flavor <FLAVOR> \
      --network <NETWORK> \
      --hint group=<SERVER_GROUP_ID> \
      my-instance
    ```
  </Tab>
</Tabs>

***

## Delete a Server Group

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Server Groups**. Click the **Delete** action (the first
    row action) on the group row. Confirm the deletion in the dialog.

    <Danger>
      Deleting a server group removes the placement constraint. Existing instances
      are not affected, but new launches will no longer enforce the policy.
    </Danger>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a server group" theme={null}
    openstack server group delete <GROUP_ID>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Create instances with server group placement in the 4-step wizard
  </Card>

  <Card title="Availability Zones" href="/services/compute/availability-zones">
    Understand fault domains and zone-level placement
  </Card>

  <Card title="Live Migration" href="/services/compute/live-migration">
    Move instances between hosts while respecting group policies
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve placement failures and group member limit issues
  </Card>
</CardGroup>


# Instance Snapshots
Source: https://docs.xloud.tech/services/compute/snapshots

Create instance snapshots to capture VM state. Use snapshots as templates to launch new instances or restore configurations.

## Overview

An instance snapshot captures the current state of an instance's root disk as an image.
Snapshots can be used to launch new instances with identical configurations, create
backups before maintenance operations, or share instance states across projects.

<Note>
  **Prerequisites**

  * An instance in `Active`, `Shutoff`, or `Suspended` status
  * Sufficient image storage quota for the snapshot
  * Bare metal instances do not support snapshots
</Note>

***

## Create a Snapshot

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Create Snapshot dialog">
        Navigate to **Compute > Instances**. Click the **More** dropdown on the
        instance row, then select **Create Snapshot** under the **Backups & Snapshots**
        group.
      </Step>

      <Step title="Configure the snapshot">
        | Field                      | Description                            |
        | -------------------------- | -------------------------------------- |
        | **Instance**               | Current instance name (read-only)      |
        | **Instance Snapshot Name** | Name for the snapshot image (required) |

        For boot-from-volume instances, a read-only table of attached volumes is
        displayed showing which volumes will be included in the snapshot.

        <Note>
          For boot-from-volume instances, the system checks snapshot quota. If
          insufficient quota is available, the submit button is disabled.
        </Note>
      </Step>

      <Step title="Create the snapshot">
        Click **Confirm**. The snapshot appears in **Compute > Instance Snapshots**
        with status `Saving`, transitioning to `Active` when complete.

        <Check>Snapshot appears in the Instance Snapshots list with status `Active`.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="Create an instance snapshot" theme={null}
    openstack server image create \
      --name my-snapshot \
      <INSTANCE_ID>
    ```

    ```bash title="Check snapshot status" theme={null}
    openstack image show my-snapshot -c status
    ```

    <Check>Snapshot status is `active`.</Check>
  </Tab>
</Tabs>

***

## Manage Instance Snapshots

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Instance Snapshots** in the sidebar. The list shows:

    | Column          | Description                                     |
    | --------------- | ----------------------------------------------- |
    | **ID/Name**     | Snapshot identifier (clickable to view details) |
    | **Description** | Optional description                            |
    | **Disk Format** | Image format (e.g., QCOW2, RAW)                 |
    | **Status**      | Active, Saving, or Error                        |
    | **Created At**  | Creation timestamp                              |

    **Available actions**:

    | Action              | Location              | Description                                                 |
    | ------------------- | --------------------- | ----------------------------------------------------------- |
    | **Edit**            | First row action      | Edit snapshot name and description                          |
    | **Create Instance** | More dropdown         | Launch a new instance from this snapshot (must be `Active`) |
    | **Create Volume**   | More dropdown         | Create a block storage volume from the snapshot             |
    | **Delete**          | More dropdown / batch | Delete the snapshot image                                   |
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List instance snapshots" theme={null}
      openstack image list --property image_type=snapshot
      ```

      ```bash title="Show snapshot details" theme={null}
      openstack image show <SNAPSHOT_ID>
      ```

      ```bash title="Launch instance from snapshot" theme={null}
      openstack server create \
        --image <SNAPSHOT_ID> \
        --flavor <FLAVOR> \
        --network <NETWORK> \
        my-restored-instance
      ```

      ```bash title="Delete a snapshot" theme={null}
      openstack image delete <SNAPSHOT_ID>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Create a new instance from a snapshot
  </Card>

  <Card title="Resize an Instance" href="/services/compute/resize-instance">
    Take a snapshot before resizing as a safety measure
  </Card>

  <Card title="Block Storage Snapshots" href="/services/storage/snapshots">
    Create volume-level snapshots for persistent storage
  </Card>

  <Card title="Troubleshooting" href="/services/compute/troubleshooting">
    Resolve snapshot creation failures
  </Card>
</CardGroup>


# Compute Troubleshooting
Source: https://docs.xloud.tech/services/compute/troubleshooting

Resolve common Xloud Compute issues — instances stuck in BUILD/ERROR, migration failures, scheduling errors, and quota issues.

## Overview

This guide covers the most common operational issues encountered in Xloud Compute
environments. Each section provides the diagnostic commands, root cause analysis,
and resolution steps needed to restore normal operation.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials sourced (`source openrc.sh`)
  * `openstack` CLI installed and configured
  * SSH access to compute nodes for log inspection when needed
</Note>

***

## Common Issues

<AccordionGroup>
  <Accordion title="Instance stuck in BUILD status" icon="clock">
    **Cause**: The scheduler placed the instance on a host but the Compute Agent failed
    to complete provisioning. Common causes include image download failure, networking
    misconfiguration, or storage attachment failure.

    **Diagnosis**:

    ```bash title="Check instance event log" theme={null}
    openstack server event list <instance-id>
    ```

    ```bash title="Identify the target host" theme={null}
    openstack server show <instance-id> \
      -f value -c OS-EXT-SRV-ATTR:host
    ```

    ```bash title="Check Compute Agent logs on the target host (via XDeploy terminal)" theme={null}
    journalctl -u nova-compute --since "1 hour ago" | grep <instance-id>
    ```

    **Common causes and resolutions**:

    | Symptom in Event Log                  | Resolution                                                    |
    | ------------------------------------- | ------------------------------------------------------------- |
    | `Image download failed`               | Verify Xloud Image Service reachability from the compute node |
    | `Quota exceeded on host`              | Check host capacity with `openstack hypervisor show <host>`   |
    | `Network interface allocation failed` | Verify network agent status on the host                       |
    | `Volume attachment failed`            | Check Xloud Block Storage service health                      |

    <Tip>
      If the instance is permanently stuck, force-delete it with
      `openstack server delete --force <instance-id>` and re-launch on a healthy host.
      Verify the target host is `up` and `enabled` before retrying.
    </Tip>
  </Accordion>

  <Accordion title="Instance in ERROR state" icon="circle-x">
    **Cause**: A fatal error occurred during instance creation, a running operation, or
    hypervisor interaction. The fault details are stored in the instance record.

    **Diagnosis**:

    ```bash title="Show error fault details" theme={null}
    openstack server show <instance-id> | grep -A5 fault
    ```

    ```bash title="View full instance event log" theme={null}
    openstack server event list <instance-id>
    ```

    **Resolution**:

    If the error is recoverable (e.g., a temporary network partition that has since
    resolved), attempt to rebuild the instance from its original image:

    ```bash title="Rebuild instance from original image" theme={null}
    openstack server rebuild <instance-id> --image <original-image-id>
    ```

    If the error is caused by a host-level hardware failure, migrate the instance to
    a healthy host before attempting a rebuild. See
    [Live Migration](/services/compute/live-migration) for instructions.

    <Warning>
      Rebuilding an instance replaces the root disk. Any data written to the root
      disk after initial provisioning will be lost. Ensure the instance owner has
      backed up root disk data before issuing a rebuild.
    </Warning>
  </Accordion>

  <Accordion title="Live migration fails" icon="arrow-right-arrow-left">
    **Cause**: CPU compatibility mismatch, insufficient destination capacity, or a
    network timeout during the migration data transfer.

    **Diagnosis**:

    ```bash title="Check migration status and error message" theme={null}
    openstack server migration list --server <instance-id>
    ```

    ```bash title="Show detailed migration information" theme={null}
    openstack server migration show <instance-id> <migration-id>
    ```

    **Common errors and resolutions**:

    | Error Message                                             | Root Cause                                     | Resolution                                                                                                               |
    | --------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
    | `guest CPU doesn't match specification: missing features` | CPU microarchitecture difference between hosts | Configure a common CPU baseline model on all hosts via XDeploy under **Compute → Advanced Settings → CPU Compatibility** |
    | `No valid host found`                                     | Destination host lacks capacity or is disabled | Check destination host capacity with `openstack hypervisor show <host>`; verify host is `enabled` and `up`               |
    | `Connection timeout`                                      | Network disruption on migration network        | Verify network connectivity between compute nodes; check firewall rules on the management interface                      |
    | `Block migration disk copy failed`                        | Insufficient free disk on destination          | Check available disk with `openstack hypervisor show <host>`                                                             |

    See [Live Migration](/services/compute/live-migration) for a full walkthrough of
    the migration procedure.
  </Accordion>

  <Accordion title="No valid host found (scheduling failure)" icon="server">
    **Cause**: All compute hosts were eliminated by the scheduler filter chain — no
    eligible host satisfies the combined instance requirements.

    **Diagnosis**:

    ```bash title="Check cluster-wide capacity" theme={null}
    openstack hypervisor stats show
    ```

    ```bash title="List all hosts with capacity details" theme={null}
    openstack hypervisor list --long
    ```

    **Common causes**:

    | Cause                                        | Resolution                                                                       |
    | -------------------------------------------- | -------------------------------------------------------------------------------- |
    | All hosts at vCPU or RAM capacity            | Scale out the cluster or increase over-commit ratios via XDeploy                 |
    | Availability zone constraint too restrictive | Verify target AZ has active hosts with `openstack availability zone list --long` |
    | Host aggregate metadata mismatch             | Verify flavor extra specs match aggregate metadata keys on target hosts          |
    | Server group anti-affinity exhausted         | Group has used all distinct hosts; scale out or remove anti-affinity constraint  |
    | Flavor requires PCI device not available     | Verify PCI passthrough devices are configured on target hosts                    |

    <Warning>
      If `openstack hypervisor stats show` reports available capacity but scheduling
      still fails, the Placement service inventory may be out of sync with actual host
      state. Trigger a resource reconciliation through XDeploy under **Compute →
      Diagnostics → Reconcile Inventory**.
    </Warning>
  </Accordion>

  <Accordion title="Console connection refused" icon="monitor-x">
    **Cause**: The VNC or SPICE console proxy service is not running, the firewall
    is blocking the console port, or the console token has expired.

    **Diagnosis**:

    ```bash title="Verify console proxy service status" theme={null}
    openstack compute service list | grep consoleauth
    ```

    ```bash title="Check all compute services for degraded state" theme={null}
    openstack compute service list
    ```

    **Resolution**:

    1. Verify ports 6080 (VNC), 6082 (SPICE), and 6083 (serial) are open in your
       firewall rules from the administrator's workstation to the controller node.

    2. If the console proxy service is `down`, restart it through XDeploy under
       **Compute → Services → Console Proxy**.

    3. If the connection is refused immediately after generating a URL, the token
       may have expired. Generate a new console URL:

    ```bash title="Generate a fresh console URL" theme={null}
    openstack console url show --novnc <instance-id>
    ```

    <Note>
      Console tokens expire after a short period. If the browser reports an
      authentication error when accessing the console URL, always generate a new URL
      rather than refreshing the page.
    </Note>

    See [Console Access](/services/compute/console-access) for firewall port requirements
    and proxy configuration details.
  </Accordion>

  <Accordion title="Quota exceeded errors" icon="chart-bar">
    **Cause**: The project has reached its allocation limit for instances, vCPUs,
    or RAM. New instance creation or resize operations are blocked until the quota
    is increased or existing resources are released.

    **Diagnosis**:

    ```bash title="Show current quota usage" theme={null}
    openstack quota show --compute <project-id>
    ```

    ```bash title="List instances consuming quota in the project" theme={null}
    openstack server list \
      --project <project-id> \
      --all-projects \
      --long
    ```

    **Resolution**:

    Option 1 — Increase the project quota:

    ```bash title="Increase quota for instances, vCPUs, and RAM" theme={null}
    openstack quota set \
      --instances 50 \
      --cores 100 \
      --ram 204800 \
      <project-id>
    ```

    Option 2 — Free capacity by removing unused instances:

    Coordinate with the project owner to identify and delete instances that are no
    longer in use. Do not delete instances without explicit confirmation from the
    project owner.

    <Tip>
      Before increasing quotas, verify the cluster has sufficient physical capacity
      with `openstack hypervisor stats show`. See
      [Quota Management](/services/compute/quotas) for quota adjustment procedures.
    </Tip>
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    Monitor and manage hypervisor host health to prevent scheduling failures.
  </Card>

  <Card title="Live Migration" href="/services/compute/live-migration">
    Move instances off degraded hosts before performing maintenance.
  </Card>

  <Card title="Admin Guide" href="/services/compute/admin-guide">
    Return to the Compute Administration Guide index.
  </Card>
</CardGroup>


# Compute User Guide
Source: https://docs.xloud.tech/services/compute/user-guide

Create, configure, and manage virtual machine instances in your Xloud private cloud.

Manage the complete instance lifecycle — launch, resize, reboot, rescue, and advanced configurations.

***

<CardGroup>
  <Card title="Availability Zones" icon="map-pin" href="/services/compute/availability-zones">
    Fault domains and instance placement
  </Card>

  <Card title="Create Instance" icon="play" href="/services/compute/launch-instance">
    Create a VM from image and flavor
  </Card>

  <Card title="Security Groups" icon="shield" href="/services/compute/security-groups">
    Firewall rules for instances
  </Card>

  <Card title="Server Groups" icon="group" href="/services/compute/server-groups">
    Affinity and anti-affinity policies
  </Card>

  <Card title="IP Addresses" icon="globe" href="/services/compute/manage-ips">
    Floating IP allocation and association
  </Card>

  <Card title="Resize Instance" icon="move-vertical" href="/services/compute/resize-instance">
    Change vCPU, RAM, and disk
  </Card>

  <Card title="Reboot Instance" icon="refresh-cw" href="/services/compute/reboot-instance">
    Soft and hard reboot
  </Card>

  <Card title="Rescue Instance" icon="life-buoy" href="/services/compute/rescue-instance">
    Recovery from OS failures
  </Card>

  <Card title="Block Devices" icon="hard-drive" href="/services/compute/block-device-mapping">
    Volumes and disks at launch
  </Card>
</CardGroup>


# VM Templates
Source: https://docs.xloud.tech/services/compute/vm-templates

Create reusable VM templates from instances. Deploy standardized environments with captured disk, flavor, and network configuration.

## Overview

VM Templates let you save a complete instance configuration — disk, flavor, network,
security groups, and key pair — as a reusable template. Deploy templates to create
standardized instances with consistent configurations across your projects.

Templates appear in the dedicated **Compute > VM Templates** page, separate from
regular images and snapshots.

<Info>**Xloud-Developed** — VM Templates are developed by Xloud and ship with XAVS / XPCI.</Info>

<Note>
  **Prerequisites**

  * An active instance to create a template from
  * Sufficient image storage quota
</Note>

***

## What a VM Template Is

A VM Template is a **first-class template object surfaced in the Dashboard**,
implemented on top of the Image Service with a `xloud_template=true` discriminator
and captured config metadata. It is more than a snapshot used as a base image — it
also carries the surrounding launch context.

| Capability                                                                  | VM Template | Snapshot used as base image |
| --------------------------------------------------------------------------- | :---------: | :-------------------------: |
| **Version** field (e.g. `1.0`)                                              |     Yes     |              —              |
| **Category** field (Base OS, Web Server, Database, Application, Custom)     |     Yes     |              —              |
| **Captured flavor** (vCPU / RAM summary shown on the row)                   |     Yes     |              —              |
| **Captured network**, **security groups**, **key pair**                     |     Yes     |              —              |
| **Deploy** action that pre-fills the launch wizard from the captured config |     Yes     |              —              |
| Lives on its own **Compute → VM Templates** page                            |     Yes     |              —              |
| Just disk content for a future boot                                         |      —      |             Yes             |

A snapshot used as a base image gives you the disk only — the next launch needs
flavor, network, security groups, and key pair to be re-chosen by hand. A VM
Template captures the disk **and** the surrounding configuration, and its Deploy
action re-applies all of it in one click.

<Info>
  **Implementation note** — under the hood, a VM Template is stored in the Image
  Service like any other image, marked with `xloud_template=true` plus the captured
  config metadata. Operators can list templates from the CLI with
  `openstack image list --property xloud_template=true`. The same storage backend,
  RBAC privileges, and quotas that apply to images apply to templates — there is
  no separate subsystem to learn.
</Info>

***

## View Templates

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > VM Templates** in the sidebar. The list shows all templates
    in your project.

    | Column            | Description                                           |
    | ----------------- | ----------------------------------------------------- |
    | **Template Name** | Template identifier (clickable to view details)       |
    | **Description**   | Optional template description                         |
    | **Version**       | Template version (e.g., `1.0`)                        |
    | **Category**      | Base OS, Web Server, Database, Application, or Custom |
    | **Flavor**        | Captured flavor name                                  |
    | **Configuration** | vCPUs / RAM summary                                   |
    | **Status**        | Active, Saving, etc.                                  |
    | **Size**          | Template image size                                   |
    | **Created At**    | Creation timestamp                                    |

    Filter by **Name** or **Status**.

    <Note>
      Administrators see an additional **Project ID/Name** column and can view
      templates across all projects.
    </Note>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="List all VM templates" theme={null}
    openstack image list --property xloud_template=true
    ```

    ```bash title="Show template details" theme={null}
    openstack image show <TEMPLATE_ID>
    ```
  </Tab>
</Tabs>

***

## Create a Template

There are two ways to create a template:

### From the VM Templates Page

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Create Template dialog">
        Navigate to **Compute > VM Templates** and click **Create Template**.
      </Step>

      <Step title="Select the source instance">
        | Field                                | Type      | Required | Description                                                      |
        | ------------------------------------ | --------- | -------- | ---------------------------------------------------------------- |
        | **Source Instance**                  | Dropdown  | Yes      | Select an instance (`Active`, `Stopped`, `Shutoff`, or `Paused`) |
        | **Template Name**                    | Text      | Yes      | Name for the template                                            |
        | **Version**                          | Text      | Yes      | Version string (default: `1.0`)                                  |
        | **Category**                         | Dropdown  | No       | Classification (default: `Custom`)                               |
        | **Description**                      | Text area | No       | Optional notes                                                   |
        | **Delete Instance After Conversion** | Checkbox  | No       | Delete the source instance after template creation               |

        **Category options**:

        | Category        | Use Case                                         |
        | --------------- | ------------------------------------------------ |
        | **Base OS**     | Clean OS installations (Ubuntu, CentOS, Windows) |
        | **Web Server**  | Pre-configured web stacks (Nginx, Apache, etc.)  |
        | **Database**    | Database servers (MySQL, PostgreSQL, MongoDB)    |
        | **Application** | Custom application stacks                        |
        | **Custom**      | General-purpose templates (default)              |

        <Warning>
          If **Delete Instance After Conversion** is checked, the source instance
          is permanently deleted after the template is created. This cannot be undone.
        </Warning>
      </Step>

      <Step title="Create the template">
        Click **Confirm**. The template appears in the list with status `Saving`,
        transitioning to `Active` when the snapshot completes.

        <Tip>
          The template captures the full configuration: flavor, network, security
          groups, key pair, and availability zone. When deploying from this template,
          these settings are used as defaults.
        </Tip>

        <Check>Template status is `Active` — ready for deployment.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    Template creation uses the Xloud-developed Nova API extension:

    ```bash title="Create a template from an instance" theme={null}
    curl -X POST -H "X-Auth-Token: $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "xloud-convert-template": {
          "name": "ubuntu-web-template",
          "version": "1.0",
          "category": "web-server",
          "description": "Ubuntu 24.04 with Nginx pre-configured",
          "delete_instance": false
        }
      }' \
      "$NOVA_ENDPOINT/v2.1/servers/<INSTANCE_ID>/action"
    ```
  </Tab>
</Tabs>

### From an Instance (Convert to Template)

Navigate to **Compute > Instances**. Click the **More** dropdown on the instance row
and select **Convert to Template** under the **Clone & Template** group. The form is
the same as above except the source instance is pre-selected.

<Note>
  Convert to Template is available for instances in `Active`, `Stopped`, `Shutoff`,
  or `Paused` status.
</Note>

***

## Template Detail

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a template name to open the detail page. Three info cards are displayed:

    **Template Info**:

    | Field                | Description                                        |
    | -------------------- | -------------------------------------------------- |
    | **Template Name**    | Display name                                       |
    | **Version**          | Version string                                     |
    | **Category**         | Classification (Base OS, Web Server, etc.)         |
    | **Description**      | Template description                               |
    | **Source Instance**  | Instance this template was created from (copyable) |
    | **Created By**       | User who created the template                      |
    | **Template Created** | Creation timestamp                                 |

    **VM Configuration** (captured from the source instance):

    | Field                 | Description          |
    | --------------------- | -------------------- |
    | **Flavor**            | Original flavor name |
    | **vCPUs**             | CPU count            |
    | **RAM (MB)**          | Memory allocation    |
    | **Disk (GB)**         | Root disk size       |
    | **Network**           | Network name         |
    | **Security Groups**   | Comma-separated list |
    | **Key Pair**          | SSH key pair name    |
    | **Availability Zone** | Zone placement       |

    **Image Info** (underlying Glance image):

    | Field           | Description                |
    | --------------- | -------------------------- |
    | **Image ID**    | UUID (copyable)            |
    | **Status**      | Active, Saving, etc.       |
    | **Size**        | Image file size            |
    | **Disk Format** | QCOW2, RAW, etc.           |
    | **Visibility**  | Public, Private, or Shared |
    | **Protected**   | Yes/No                     |
    | **Checksum**    | Image checksum (copyable)  |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show template metadata" theme={null}
    openstack image show <TEMPLATE_ID> -f json | python3 -c "
    import sys, json
    img = json.load(sys.stdin)
    for k, v in sorted(img.items()):
        if k.startswith('xloud_template_'):
            print(f'{k}: {v}')
    "
    ```
  </Tab>
</Tabs>

***

## Deploy a Template

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Select Deploy">
        Navigate to **Compute > VM Templates**. Click the **More** dropdown on the
        template row and select **Deploy**.
      </Step>

      <Step title="Name the new instance">
        Enter a **Name** for the new instance. Click **Confirm**.

        The wizard redirects to the Instance Create page with the template
        pre-selected as the boot source image.

        <Check>New instance created from the template with the captured configuration.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Launch instance from template" theme={null}
    openstack server create \
      --image <TEMPLATE_ID> \
      --flavor <FLAVOR> \
      --network <NETWORK> \
      --key-name <KEYPAIR> \
      my-deployed-instance
    ```

    <Tip>
      Use the flavor, network, and key pair captured in the template metadata
      for consistent deployments:

      ```bash title="Get template's captured config" theme={null}
      openstack image show <TEMPLATE_ID> -c properties -f json | python3 -c "
      import sys, json
      props = json.load(sys.stdin)['properties']
      print(f'Flavor: {props.get(\"xloud_template_flavor_name\", \"N/A\")}')
      print(f'Network: {props.get(\"xloud_template_network_name\", \"N/A\")}')
      print(f'Keypair: {props.get(\"xloud_template_keypair\", \"N/A\")}')
      "
      ```
    </Tip>
  </Tab>
</Tabs>

***

## Edit a Template

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click **Edit** (the first row action) on the template row. You can modify:

    | Field                    | Description                                     |
    | ------------------------ | ----------------------------------------------- |
    | **Name**                 | Template display name                           |
    | **Version**              | Version string                                  |
    | **Category**             | Template classification                         |
    | **Description**          | Template description (max 255 chars)            |
    | **Public**               | Make visible to all projects (admin only)       |
    | **Protected**            | Prevent accidental deletion                     |
    | **Configuration fields** | Flavor, vCPUs, RAM, Disk, Network, Key Pair, AZ |

    Click **Confirm** to save changes.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Update template metadata" theme={null}
    openstack image set \
      --property xloud_template_version="2.0" \
      --property xloud_template_category="web-server" \
      <TEMPLATE_ID>
    ```
  </Tab>
</Tabs>

***

## Delete a Template

Click the **More** dropdown on the template row and select **Delete Template**.
Templates with `Protected` enabled must have protection removed before deletion.

Batch delete is available via checkboxes.

<Danger>
  Deleting a template permanently removes the underlying image. Instances previously
  deployed from this template are not affected, but no new deployments can be made.
</Danger>

***

## Manage Access (Admin Only)

Administrators can share templates with specific projects via the **Manage Access**
action in the More dropdown. This requires the template to have `Shared` visibility.

***

## Template Actions Summary

| Action              | Location              | Description                                               |
| ------------------- | --------------------- | --------------------------------------------------------- |
| **Edit**            | First row action      | Edit template name, version, category, configuration      |
| **Deploy**          | More dropdown         | Launch a new instance from the template                   |
| **Delete**          | More dropdown / batch | Delete the template                                       |
| **Manage Access**   | More dropdown (admin) | Share with specific projects (requires Shared visibility) |
| **Create Template** | Primary action (top)  | Create a new template from an existing instance           |

***

## Next Steps

<CardGroup>
  <Card title="Clone an Instance" href="/services/compute/clone-instance">
    Create an exact copy of a running instance without templates
  </Card>

  <Card title="Instance Snapshots" href="/services/compute/snapshots">
    Create point-in-time snapshots (simpler than templates)
  </Card>

  <Card title="Create an Instance" href="/services/compute/launch-instance">
    Launch instances from the 4-step wizard
  </Card>

  <Card title="Image Management" href="/services/images/upload-image">
    Upload and manage OS images
  </Card>
</CardGroup>


# vTPM and Secure Boot
Source: https://docs.xloud.tech/services/compute/vtpm-secure-boot

Enable Virtual TPM and UEFI Secure Boot on an instance from the Create Instance wizard — for BitLocker, LUKS, measured boot, and Windows 11 workloads.

## Overview

When you launch an instance, you can attach a **Virtual TPM (vTPM)** device and enforce
**UEFI Secure Boot** directly from the launch wizard. These options harden the guest
against bootloader tampering, unlock BitLocker / LUKS disk encryption, and satisfy the
hardware requirements for modern guest operating systems such as Windows 11.

<Warning>
  **Both vTPM and Secure Boot require the Xloud Key Management service (Xloud KMS)** to
  be enabled on the cluster. When Xloud KMS is not available, both checkboxes are
  disabled in the launch wizard with the hint *"Key management service is not enabled.
  Ask your administrator to enable it before using vTPM or Secure Boot."*
</Warning>

<Note>
  **Prerequisites**

  * An active image in the Xloud Image Service (UEFI-capable for Secure Boot)
  * A flavor appropriate for your workload
  * **Xloud KMS** enabled on the cluster
  * For Secure Boot: a guest image that ships signed bootloaders and kernel modules
</Note>

***

## Video Walkthrough

<iframe title="How to Enable vTPM and Secure Boot on Xloud" />

***

## Where to Find These Options

Both settings live in the **Create Instance** wizard at:

**Compute → Instances → Create Instance → Step 3 (System Config) → Advanced Options**

<Steps>
  <Step title="Start Create Instance" icon="server">
    From the Xloud Dashboard, go to **Compute → Instances** and click **Create Instance**.
  </Step>

  <Step title="Complete Base Config and Network Config" icon="network">
    Pick the image or boot source, flavor, and networks in the first two wizard steps.
  </Step>

  <Step title="Open Advanced Options in System Config" icon="settings">
    In **Step 3 — System Config**, scroll down and click **Advanced Options** to expand
    the additional settings panel. **Virtual TPM** and **Secure Boot** appear as
    checkboxes near the bottom of this panel.
  </Step>
</Steps>

<Info>
  If Xloud KMS is disabled, both the **Virtual TPM** and **Secure Boot** checkboxes are
  grayed out. Contact your administrator and point them at the **Key Manager** service
  setup.
</Info>

***

## Virtual TPM (vTPM)

A Virtual TPM emulates a hardware Trusted Platform Module inside the instance. It enables
measured boot, stores disk-encryption keys securely (BitLocker, LUKS with TPM unsealing),
and satisfies TPM-based attestation requirements. The per-instance TPM state is protected
by a key managed by Xloud KMS.

### Enabling vTPM

<Steps>
  <Step title="Tick Virtual TPM" icon="shield-check">
    In the Advanced Options panel, select the **Virtual TPM** checkbox
    (*"Attach a virtual TPM device to this instance"*). Two new fields appear.
  </Step>

  <Step title="Choose a vTPM Version" icon="tag">
    Pick the TPM specification version from the dropdown:

    | Option      | When to use                                                              |
    | ----------- | ------------------------------------------------------------------------ |
    | **TPM 2.0** | Recommended default. Required for Windows 11, modern Linux distributions |
    | **TPM 1.2** | Legacy compatibility only — older operating systems                      |
  </Step>

  <Step title="Choose a vTPM Model" icon="cpu">
    Pick the virtual TPM hardware interface:

    | Option      | When to use                                                                           |
    | ----------- | ------------------------------------------------------------------------------------- |
    | **Auto**    | Default. The platform picks the best interface for the chosen version                 |
    | **tpm-crb** | TPM 2.0 guests (Windows 11, RHEL 9+, Ubuntu 22.04+). Recommended for modern workloads |
    | **tpm-tis** | Legacy TPM 1.2 interface for older operating systems                                  |

    <Tip>Leave the model as **Auto** unless the guest OS requires a specific TPM interface.</Tip>
  </Step>
</Steps>

<Check>
  After the instance boots, verify the vTPM is present inside the guest:

  * **Linux**: `ls /dev/tpm0` and `tpm2_getcap properties-fixed`
  * **Windows**: open **tpm.msc** — the management console should show a TPM manufactured
    by `swtpm` with spec version 2.0
</Check>

### What vTPM Enables

<CardGroup>
  <Card title="BitLocker on Windows" icon="lock">
    Windows 10 / 11 can use the vTPM to seal BitLocker keys, so the disk only decrypts on
    the same virtual machine.
  </Card>

  <Card title="LUKS on Linux" icon="key">
    Linux guests can store LUKS unlock keys in the vTPM with tools like
    `clevis tpm2 bind` — no boot-time passphrase prompt.
  </Card>

  <Card title="Measured Boot" icon="list-checks">
    The vTPM records a cryptographic measurement of the boot chain, enabling remote
    attestation of the guest's boot integrity.
  </Card>

  <Card title="Windows 11 Hardware Requirements" icon="monitor">
    Windows 11 installation requires TPM 2.0 — vTPM satisfies this without any physical
    hardware.
  </Card>
</CardGroup>

***

## UEFI Secure Boot

Secure Boot is a UEFI firmware feature that prevents unsigned bootloaders, kernels, and
drivers from loading — shutting down rootkits and bootkits at the earliest stage of the
boot process. Xloud stores the per-instance Secure Boot variables in Xloud KMS so they
persist across reboots and live migrations.

### Enabling Secure Boot

<Steps>
  <Step title="Tick Secure Boot" icon="shield">
    In the Advanced Options panel, select the **Secure Boot** checkbox
    (*"Require Secure Boot for this instance (UEFI + q35)"*). A **Secure Boot Mode**
    dropdown appears.

    <Info>
      Selecting Secure Boot automatically configures the instance to use **UEFI firmware**
      and the **q35 machine type** — you do not need to set these manually.
    </Info>
  </Step>

  <Step title="Choose a Secure Boot Mode" icon="sliders">
    Pick how strict the enforcement should be:

    | Mode         | Behavior                                                                                                           |
    | ------------ | ------------------------------------------------------------------------------------------------------------------ |
    | **Required** | The instance **will not start** if Secure Boot cannot be activated — for example, if the image is not UEFI-capable |
    | **Optional** | Secure Boot is used when available but falls back gracefully if the image or hypervisor cannot support it          |

    <Tip>Use **Required** for production Windows 11 and hardened Linux images. Use **Optional** while you are validating image compatibility.</Tip>
  </Step>
</Steps>

<Check>
  After boot, confirm Secure Boot is active inside the guest:

  * **Linux**: `mokutil --sb-state` returns `SecureBoot enabled`
  * **Windows**: open **msinfo32** — under System Summary, **Secure Boot State** reads `On`
</Check>

### Image Requirements

For Secure Boot to succeed, the guest image must:

* Be **UEFI-compatible** (not BIOS-only)
* Ship with bootloaders and kernels signed by a Certificate Authority recognized by the
  UEFI firmware (Microsoft UEFI CA for most Windows and Linux distributions)
* Use a recent kernel and `shim` / `grub` version with Secure Boot enforcement enabled

Modern stock images (Windows Server 2019+, Ubuntu 20.04+, RHEL 8+, openSUSE Leap 15+)
meet these requirements by default.

***

## Using vTPM and Secure Boot Together

You can enable both features on the same instance — this is the **recommended configuration**
for Windows 11, hardened Linux workloads, and any guest that uses TPM-backed disk encryption
with measured boot.

<CardGroup>
  <Card title="Windows 11 hardened VM" icon="monitor">
    vTPM 2.0 (tpm-crb) + Secure Boot (Required). BitLocker auto-seals keys to the vTPM
    and Secure Boot blocks unsigned drivers.
  </Card>

  <Card title="Hardened Linux VM" icon="terminal">
    vTPM 2.0 (tpm-crb) + Secure Boot (Required) + LUKS with `clevis tpm2 bind`.
    Full-disk encryption unlocks automatically on the correct hypervisor only.
  </Card>
</CardGroup>

***

## Live Migration Compatibility

Both vTPM state and Secure Boot variables are protected by a key managed by Xloud KMS
and are transferred between hypervisors during **live migration** with no operator
intervention. You can move a vTPM-enabled Windows 11 or hardened Linux VM between
compute hosts without shutting it down and without re-registering the TPM inside the
guest.

<Info>**Xloud-Developed** — Live migration of vTPM-enabled instances with automatic key transfer is developed by Xloud and ships with XAVS / XPCI.</Info>

***

## Confirm Config Summary

When you reach **Step 4 — Confirm Config**, the review pane shows the vTPM and Secure
Boot settings you selected:

| Field           | Example value                                                                            |
| --------------- | ---------------------------------------------------------------------------------------- |
| **Virtual TPM** | `Enabled — TPM 2.0 (auto)` — or `(tpm-crb)` / `(tpm-tis)` if you picked a specific model |
| **Secure Boot** | `Enabled (required, UEFI + q35)` — or `(optional, UEFI + q35)`                           |

Click the **System Config** section heading to go back and adjust any setting before
launching.

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="The vTPM or Secure Boot checkbox is grayed out" icon="circle-x">
    Xloud KMS is not enabled on the cluster. Ask your administrator to enable the Key
    Manager service in XDeploy, then refresh the wizard.
  </Accordion>

  <Accordion title="Instance fails to start with Secure Boot = Required" icon="triangle-alert">
    The image is not UEFI-capable or does not ship signed bootloaders. Either switch the
    mode to **Optional**, or rebuild the image with a UEFI-compatible distribution.
  </Accordion>

  <Accordion title="Windows 11 installer does not detect a TPM" icon="monitor">
    The vTPM was not enabled at launch. You cannot add vTPM to an existing instance —
    create a new instance with vTPM ticked, or migrate the workload.
  </Accordion>

  <Accordion title="BitLocker or LUKS cannot unseal the key after migration" icon="lock">
    Xloud transfers the encrypted vTPM state with the instance. If unsealing fails,
    verify Xloud KMS is healthy on both source and destination hosts and that the
    instance's TPM secret still exists in the key manager.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Create Instance" href="/services/compute/launch-instance">
    Full 4-step wizard walkthrough for launching instances
  </Card>

  <Card title="Security Hardening" href="/services/compute/security-hardening">
    Additional hypervisor-level hardening for sensitive workloads
  </Card>

  <Card title="Key Manager" href="/services/key-manager">
    Xloud KMS setup and secret lifecycle
  </Card>
</CardGroup>


# Dashboard Admin Guide
Source: https://docs.xloud.tech/services/dashboard/admin-guide

Administer the Xloud Dashboard — manage projects, users, flavors, quotas, and platform resources through the Skyline admin view.

## Overview

The Xloud Dashboard admin view provides platform-wide management capabilities.
Administrators can manage projects and users, configure flavors and quotas, monitor
infrastructure health, and administer all cloud services from a single interface.

<Note>
  **Prerequisites**

  * `admin` role in the target domain or globally
  * Access to the Dashboard at `https://connect.<your-domain>`
  * Switch to the **Admin** view using the toggle in the Dashboard header
</Note>

***

## Admin-Only Sections

The following sections are visible only in the Admin view:

<CardGroup>
  <Card title="Projects & Users" icon="users" href="/services/identity/user-guide">
    Create projects, manage users, assign roles, and configure domain settings
  </Card>

  <Card title="Flavors" icon="cpu" href="/services/compute/flavors">
    Create and manage instance flavors with architecture, GPU, NUMA, and hot-add settings
  </Card>

  <Card title="Quotas" icon="chart-pie" href="/services/compute/quotas">
    Set per-project resource limits for instances, vCPUs, RAM, and volumes
  </Card>

  <Card title="Volume Types" icon="hard-drive" href="/services/storage/volume-types-admin">
    Configure storage tiers, QoS specs, and encryption types
  </Card>

  <Card title="Hypervisors" icon="server" href="/services/compute/compute-hosts">
    Monitor compute host resources and manage service availability
  </Card>

  <Card title="Host Aggregates" icon="layers" href="/services/compute/scheduling">
    Organize hosts into groups for scheduling and availability zones
  </Card>

  <Card title="Monitor Center" icon="activity" href="/services/monitoring/user-guide">
    Platform health, service status, logging, and monitoring dashboards
  </Card>

  <Card title="Resource Optimizer" icon="chart-bar" href="/services/optimization/user-guide">
    Automated workload placement with goals, strategies, and action plans
  </Card>
</CardGroup>

***

## Dashboard Configuration

Dashboard configuration is managed through XDeploy, not through the Dashboard
interface itself.

<Tabs>
  <Tab title="XDeploy" icon="settings">
    <Steps>
      <Step title="Configure Dashboard settings">
        Open **XDeploy > Configuration > Advance Features**. The Skyline Dashboard
        can be enabled with:

        * **Enable Skyline Dashboard** toggle
        * **Enable RBAC Management** toggle (when Skyline is enabled)

        Click **Save Configuration**.
      </Step>

      <Step title="Deploy the configuration">
        Navigate to **XDeploy > Operations** and run **Reconfigure** to apply
        Dashboard configuration changes across all nodes.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Dashboard configuration files are located at `/etc/xavs/config/skyline/`.
    The Skyline API server configuration includes:

    * `skyline.yaml` — API server settings, database, secret key
    * `gunicorn.py` — WSGI server configuration
    * `nginx.conf` — Reverse proxy and static file serving

    After editing configuration files, run `xavs-ansible reconfigure -t skyline`
    to apply changes.
  </Tab>
</Tabs>

***

## Session and Authentication

The Skyline Dashboard uses JWT-based session cookies for authentication:

| Setting              | Description                                                 |
| -------------------- | ----------------------------------------------------------- |
| **Session Duration** | Configurable JWT expiration (default: 1 hour)               |
| **Session Cookie**   | `session` cookie containing the JWT with Keystone token     |
| **Authentication**   | Username + password against Keystone Identity service       |
| **RBAC**             | Per-endpoint role-based access control with 30-second cache |

<Note>
  Unlike traditional session stores (Memcached, Redis), the Skyline Dashboard uses
  stateless JWT tokens. No server-side session storage is required.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Dashboard User Guide" icon="book-open" href="/services/dashboard/user-guide">
    End-user workflows for the Console view
  </Card>

  <Card title="Identity Admin Guide" icon="fingerprint" href="/services/identity/admin-guide">
    Configure authentication backends, federation, and policies
  </Card>

  <Card title="Deployment" icon="play" href="/deployment/index">
    Deploy and manage the Dashboard through XDeploy
  </Card>

  <Card title="CLI Setup" icon="terminal" href="/cli-setup">
    Configure CLI access for administrative operations
  </Card>
</CardGroup>


# Manage Flavors
Source: https://docs.xloud.tech/services/dashboard/admin-guide/flavors

Create and manage compute flavors through the Xloud Dashboard admin view. Configure vCPU, RAM, GPU, NUMA, hot-add, and access control.

## Overview

Navigate to **Compute > Flavors** in the admin sidebar to manage the flavor catalog.

***

## Create a Flavor

Click **Create Flavor** to open the 2-step wizard:

| Step | Name                    | Fields                                                                                                                                                                                             |
| ---- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1    | **Params Setting**      | Architecture (X86/Heterogeneous/Bare Metal/ARM), Category, Name, vCPUs, Memory, Hot-Add config, Bandwidth, Ephemeral Disk, IOPS, NUMA, GPU, Compute Optimized (CPU Policy/Thread Policy/Page Size) |
| 2    | **Access Type Setting** | Public (all projects) or Access Control (select specific projects)                                                                                                                                 |

<Note>
  For the complete field-by-field reference with all architecture/category combinations,
  see [Flavor Management](/services/compute/flavors).
</Note>

***

## Flavor Actions

| Action            | Location         | Description                               |
| ----------------- | ---------------- | ----------------------------------------- |
| **Delete**        | First row action | Delete the flavor                         |
| **Manage Access** | More dropdown    | Change which projects can use this flavor |

***

## Flavor Detail

Click a flavor name. Tabs: **Detail** (Base Info, GPU, NUMA, Compute Optimized cards,
Extra Specs JSON) and **Instances** (all instances using this flavor).

***

## Next Steps

<CardGroup>
  <Card title="Flavor Management" href="/services/compute/flavors">
    Complete flavor creation reference
  </Card>

  <Card title="Live Scaling" href="/services/compute/live-resize-admin">
    Configure hot-add flavors for zero-downtime scaling
  </Card>

  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Select flavors during instance creation
  </Card>

  <Card title="Quotas" href="/services/dashboard/admin-guide/quotas">
    Set per-project vCPU and RAM limits
  </Card>
</CardGroup>


# Host Aggregates
Source: https://docs.xloud.tech/services/dashboard/admin-guide/host-aggregates

Organize compute hosts into logical groups for workload placement and availability zones through the Xloud Dashboard.

## Overview

Navigate to **Compute > Host Aggregates** (admin view). The page has two tabs:
**Host Aggregate** (manage groups) and **Availability Zones** (read-only zone list).

***

## Create a Host Aggregate

Click **Create Host Aggregate**. The modal form includes:

| Field                 | Type           | Required            | Description                               |
| --------------------- | -------------- | ------------------- | ----------------------------------------- |
| **Name**              | Text           | Yes                 | Aggregate name                            |
| **Create New AZ**     | Radio (Yes/No) | Yes                 | Whether to create a new availability zone |
| **Availability Zone** | Dropdown       | If not creating new | Select existing zone                      |
| **New AZ Name**       | Text           | If creating new     | Name for the new zone                     |

***

## Manage Aggregates

| Action              | Location         | Description                                                                     |
| ------------------- | ---------------- | ------------------------------------------------------------------------------- |
| **Edit**            | First row action | Change name and availability zone                                               |
| **Manage Host**     | More dropdown    | Add/remove compute hosts (multi-select table showing Host, Zone, Status, State) |
| **Manage Metadata** | More dropdown    | Set key-value metadata for scheduler matching                                   |
| **Delete**          | More dropdown    | Delete the aggregate                                                            |

***

## Availability Zones

The **Availability Zones** tab shows a read-only list of all configured zones.
Zones are created through host aggregates — assign an aggregate to a zone name
to define a new availability zone.

<Note>
  For the complete scheduling and host aggregate reference, see
  [Compute Scheduling](/services/compute/scheduling).
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Compute Scheduling" href="/services/compute/scheduling">
    Complete host aggregate and scheduling reference
  </Card>

  <Card title="Availability Zones" href="/services/compute/availability-zones">
    Zone selection during instance launch
  </Card>

  <Card title="Compute Hosts" href="/services/compute/compute-hosts">
    Monitor hypervisor resources
  </Card>

  <Card title="Flavors" href="/services/dashboard/admin-guide/flavors">
    Create flavors with aggregate-matching extra specs
  </Card>
</CardGroup>


# Manage Images (Admin)
Source: https://docs.xloud.tech/services/dashboard/admin-guide/images

Admin operations for image management — set visibility, manage access, configure metadata, and control the platform image catalog.

## Overview

Navigate to **Compute > Images** in the admin sidebar. The admin view shows all
images across all projects with additional columns (Project ID/Name) and admin-only
actions.

***

## Admin Actions

| Action              | Location         | Description                                                                       |
| ------------------- | ---------------- | --------------------------------------------------------------------------------- |
| **Edit**            | First row action | Edit name, OS details, visibility (Public checkbox), protection, advanced options |
| **Delete**          | More dropdown    | Delete the image                                                                  |
| **Manage Access**   | More dropdown    | Share with specific projects (requires Shared visibility)                         |
| **Manage Metadata** | More dropdown    | Edit custom key-value metadata and system metadata definitions                    |

<Note>
  Admin actions are a completely different set from user actions. The admin More dropdown
  does NOT contain Create Instance or Create Volume — those are user-only actions.
</Note>

***

## Image Visibility

| Visibility  | Who Can See                        | How to Set                                 |
| ----------- | ---------------------------------- | ------------------------------------------ |
| **Private** | Owner project only                 | Default for non-admin uploads              |
| **Public**  | All projects                       | Admin: Edit image > check "Public"         |
| **Shared**  | Owner + explicitly shared projects | Admin: set via CLI, then use Manage Access |

***

## Metadata Management

Navigate to **Global Setting > Metadata Definitions** (admin view) to manage system-wide
metadata schemas that define available properties for images, flavors, and other resources.

<Note>
  For the complete image management reference, see [Upload an Image](/services/images/upload-image)
  and [Image Properties](/services/images/image-properties).
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Upload an Image" href="/services/images/upload-image">
    Full image creation form reference
  </Card>

  <Card title="Image Properties" href="/services/images/image-properties">
    Edit properties and manage metadata
  </Card>

  <Card title="Share Images" href="/services/images/share-images">
    Share images across projects
  </Card>

  <Card title="Image Templates" href="/services/images/image-templates">
    Manage the platform image catalog
  </Card>
</CardGroup>


# Manage Projects & Users
Source: https://docs.xloud.tech/services/dashboard/admin-guide/projects-users

Create and manage projects, users, groups, and role assignments through the Xloud Dashboard admin view.

## Overview

Project and user management is under the **Identity** section in the admin sidebar.
Pages include: **Domains**, **Projects**, **Users**, **User Groups**, **Roles**, and
**RBAC Management**.

<Note>
  The Identity section is only visible in the **Admin** view. Regular Console users
  see a limited Identity section with project and user listing only.
</Note>

***

## Manage Projects

Navigate to **Identity > Projects** (admin view).

**Create a Project**: Click **Create Project**. The full-page form includes Name,
Description, and project settings. After creation, assign users and set quotas.

**Project actions**: Edit (first action), Manage Members, Enable/Disable, Modify
Quotas, Delete (More dropdown).

***

## Manage Users

Navigate to **Identity > Users** (admin view).

**Create a User**: Click **Create User**. The full-page form includes Username,
Email, Password, Project assignment, Role, and Domain.

**User actions**: Edit (first action), Enable/Disable, Change Password, Delete
(More dropdown).

***

## User Groups

Navigate to **Identity > User Groups** (admin view).

Create groups, add members, and assign roles at the group level for bulk access
management.

***

## Domains

Navigate to **Identity > Domains** (admin view).

Domains partition the identity namespace. Most deployments use the `Default` domain.
Multi-domain configurations are used for federated identity and organizational separation.

***

## Detailed Guides

<CardGroup>
  <Card title="Identity User Guide" href="/services/identity/user-guide">
    Complete identity management reference
  </Card>

  <Card title="Quotas" href="/services/dashboard/admin-guide/quotas">
    Set per-project resource limits
  </Card>

  <Card title="Flavors" href="/services/dashboard/admin-guide/flavors">
    Control which flavors projects can access
  </Card>

  <Card title="Dashboard Admin Guide" href="/services/dashboard/admin-guide">
    Return to admin guide overview
  </Card>
</CardGroup>


# Manage Quotas
Source: https://docs.xloud.tech/services/dashboard/admin-guide/quotas

Set and manage per-project resource quotas through the Xloud Dashboard — control instance, vCPU, RAM, storage, and network limits.

## Overview

Quotas enforce per-project resource limits. They are visible to users during resource
creation (real-time quota bars in instance and volume create forms) and managed by
administrators.

***

## View Quota Usage

Users see quota information in two places:

* **Instance Create wizard** (Step 4: Confirm Config) — footer badge shows Instance,
  CPU, Memory, Volume, and Volume Capacity quota usage in real-time
* **Volume Create form** — capacity slider is limited by remaining volume quota

***

## Modify Quotas

<Note>
  The Dashboard does not provide a dedicated standalone quota management page. Quotas
  are modified via the CLI or through [XDeploy](/deployment) configuration.
</Note>

<Tabs>
  <Tab title="XDeploy" icon="settings">
    <Steps>
      <Step title="Open Configuration">
        Navigate to **XDeploy > Configuration > Advance Features** to adjust
        default quota values.
      </Step>

      <Step title="Apply changes">
        Save and run **Operations > Reconfigure**.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Show project quota" theme={null}
      openstack quota show <PROJECT_ID>
      ```

      ```bash title="Show current usage" theme={null}
      openstack quota show --usage <PROJECT_ID>
      ```

      ```bash title="Set instance quota" theme={null}
      openstack quota set --instances 50 <PROJECT_ID>
      ```

      ```bash title="Set vCPU quota" theme={null}
      openstack quota set --cores 100 <PROJECT_ID>
      ```

      ```bash title="Set RAM quota (MB)" theme={null}
      openstack quota set --ram 102400 <PROJECT_ID>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Detailed Guides

<CardGroup>
  <Card title="Compute Quotas" href="/services/compute/quotas">
    Instance, vCPU, and RAM quota management
  </Card>

  <Card title="Storage Quotas" href="/services/storage/quotas">
    Volume count and capacity quotas
  </Card>

  <Card title="Network Quotas" href="/services/networking/quotas">
    Network, router, and floating IP quotas
  </Card>

  <Card title="Deployment" href="/deployment/configuration">
    Configure default quotas through XDeploy
  </Card>
</CardGroup>


# Session Management
Source: https://docs.xloud.tech/services/dashboard/admin-guide/session-storage

Understand how the Xloud Dashboard manages user sessions using JWT tokens. Configure session duration and security settings.

## Overview

The Xloud Dashboard uses JWT (JSON Web Token) based session management. When a user
logs in, the Dashboard obtains a Keystone authentication token and wraps it in a JWT
stored as a browser cookie named `session`. No server-side session storage (Memcached,
Redis, database) is required.

<Note>
  **Prerequisites**

  * Administrator access for session configuration changes
  * XDeploy access for applying configuration changes
</Note>

***

## How Sessions Work

```mermaid theme={null}
sequenceDiagram
    participant Browser
    participant Skyline as Skyline Console
    participant API as Skyline API Server
    participant Keystone as Identity Service

    Browser->>Skyline: Login (username + password)
    Skyline->>API: POST /api/v1/login
    API->>Keystone: Authenticate
    Keystone-->>API: Token
    API-->>Skyline: JWT cookie (session)
    Skyline-->>Browser: Set-Cookie: session=<JWT>
    Browser->>Skyline: Subsequent requests (cookie attached)
    Skyline->>API: Forward with JWT
    API->>API: Validate JWT, extract token
    API->>Keystone: Validate token (cached)
```

***

## Session Properties

| Property             | Description                       | Default                |
| -------------------- | --------------------------------- | ---------------------- |
| **Cookie name**      | `session`                         | Fixed                  |
| **Token format**     | JWT containing Keystone token     | Fixed                  |
| **Session duration** | Follows Keystone token expiration | Typically 1 hour       |
| **Storage**          | Client-side (browser cookie)      | No server-side storage |
| **RBAC cache**       | Per-endpoint permission cache     | 30 seconds TTL         |

<Tip>
  Since sessions are stateless JWT tokens, the Dashboard can be horizontally scaled
  across multiple nodes behind a load balancer without shared session storage.
  Any node can validate any session.
</Tip>

***

## Configuration

<Tabs>
  <Tab title="XDeploy" icon="settings">
    Session behavior is configured through the Skyline API server configuration:

    <Steps>
      <Step title="Open Advanced Configuration">
        Navigate to **XDeploy > Advanced Configuration** and select **skyline-apiserver**
        in the service tree.
      </Step>

      <Step title="Edit session settings">
        In the `skyline.yaml` configuration file, the relevant settings are:

        * **secret\_key** — JWT signing key (auto-generated during deployment)
        * **session\_name** — Cookie name (default: `session`)
        * **token\_expiration** — Inherited from Keystone token settings

        <Warning>
          Changing the `secret_key` invalidates all active sessions. Users will need
          to log in again after the change is applied.
        </Warning>
      </Step>

      <Step title="Apply changes">
        Save the configuration and run **Operations > Reconfigure** to apply.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    The Skyline API server configuration is at `/etc/xavs/config/skyline-apiserver/`.

    ```bash title="View current session configuration" theme={null}
    cat /etc/skyline/skyline.yaml | grep -A 5 session
    ```

    After changes, restart the Skyline API server:

    ```bash title="Apply configuration" theme={null}
    xavs-ansible reconfigure -t skyline
    ```
  </Tab>
</Tabs>

***

## Security Considerations

<AccordionGroup>
  <Accordion title="Cookie security flags">
    The session cookie is set with:

    * **HttpOnly** — not accessible via JavaScript (prevents XSS token theft)
    * **Secure** — only sent over HTTPS connections
    * **SameSite** — prevents CSRF attacks

    These flags are enforced by the Skyline API server and cannot be overridden
    by client-side code.
  </Accordion>

  <Accordion title="Token expiration">
    Session duration is tied to the Keystone token expiration. When the token expires,
    the session cookie becomes invalid and the user is redirected to the login page.
    Configure Keystone token expiration through XDeploy to adjust session duration.
  </Accordion>

  <Accordion title="Multi-node deployments">
    Since sessions are stateless (JWT in cookie), no shared session backend is needed.
    All Dashboard nodes behind HAProxy can validate any session independently using
    the shared `secret_key` configured during deployment.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Dashboard Admin Guide" href="/services/dashboard/admin-guide">
    Return to the admin guide overview
  </Card>

  <Card title="Identity Admin Guide" href="/services/identity/admin-guide">
    Configure token expiration and authentication backends
  </Card>

  <Card title="Deployment" href="/deployment/configuration">
    Configure Dashboard settings through XDeploy
  </Card>

  <Card title="Security" href="/security/index">
    Platform security hardening and compliance
  </Card>
</CardGroup>


# Manage Volumes (Admin)
Source: https://docs.xloud.tech/services/dashboard/admin-guide/volumes

Administer block storage — manage volume types, QoS specs, storage backends, and cross-project volumes.

## Overview

The admin sidebar Storage section includes additional pages not visible to regular users:
**Volume Types** and **Storage Backends**.

***

## Volume Types

Navigate to **Storage > Volume Types** (admin view). The page has two tabs:

**Volume Types tab** — manage storage tiers:

| Action                | Location              | Description                              |
| --------------------- | --------------------- | ---------------------------------------- |
| **Manage QoS Spec**   | First row action      | Associate a QoS spec with this type      |
| **Manage Access**     | More dropdown         | Control which projects can use this type |
| **Create Encryption** | More dropdown         | Add encryption provider (luks)           |
| **Delete Encryption** | More dropdown         | Remove encryption settings               |
| **Edit**              | More dropdown         | Update name and description              |
| **Delete**            | More dropdown / batch | Delete the volume type                   |

**QoS Specs tab** — manage quality of service specifications:

Create QoS specs with consumer (front-end/back-end/all) and bandwidth/IOPS rules.
Associate specs with volume types via the **Manage QoS Spec** action on the Volume Types tab.

***

## Storage Backends

Navigate to **Storage > Storage Backends** (admin view). Read-only list showing all
configured storage pools with protocol, backend name, and host information.

***

## Detailed Guides

<CardGroup>
  <Card title="Volume Types (Admin)" href="/services/storage/volume-types-admin">
    Complete volume type management reference
  </Card>

  <Card title="QoS" href="/services/storage/qos">
    QoS spec creation and association
  </Card>

  <Card title="Encryption" href="/services/storage/encryption">
    Volume encryption configuration
  </Card>

  <Card title="Create a Volume" href="/services/storage/create-volume">
    Volume creation with type selection
  </Card>
</CardGroup>


# Xloud Dashboard
Source: https://docs.xloud.tech/services/dashboard/index

Browser-based management console for all Xloud cloud services — powered by Skyline with integrated monitoring, RBAC, and resource management.

The Xloud Dashboard is a browser-based interface for centralized management of all cloud
resources across Compute, Storage, Networking, Identity, and more. Powered by the Skyline
platform, it provides a modern, high-performance UI with integrated monitoring, role-based
access control, and real-time resource management.

<Note>
  The Dashboard is included in all Xloud products: [XAVS](/products/xavs), [XPCI](/products/xpci),
  and [XHCI](/products/xhci). Access it at `https://connect.<your-domain>` after deployment.
</Note>

***

<p>Guides</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/dashboard/user-guide">
    Navigate the Dashboard, manage resources, and understand the console and admin views.
  </Card>

  <Card title="Admin Guide" icon="shield" href="/services/dashboard/admin-guide">
    Configure the Dashboard, manage projects and users, and administer platform resources.
  </Card>
</CardGroup>

***

<p>Dashboard Layout</p>

The Dashboard is organized into two views — **Console** (project-scoped) and **Admin**
(platform-wide) — accessible via a toggle in the header.

<Tabs>
  <Tab title="Console View (User)" icon="gauge">
    The Console view shows project-scoped resources. Sidebar sections:

    | Section                | Pages                                                                                                                                                  |
    | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
    | **Compute**            | Instances, Instance Snapshots, Flavors, Server Groups, Images, Key Pairs                                                                               |
    | **Storage**            | Volumes, Volume Backups, Volume Snapshots, Object Storage                                                                                              |
    | **Network**            | Networks, Ports, QoS Policies, Routers, Floating IPs, Topology, Load Balancers, Certificates, VPNs, Security Groups, Firewalls, DNS Zones, DNS Reverse |
    | **Orchestration**      | Stacks                                                                                                                                                 |
    | **Share File Storage** | Shares, Share Networks, Share Groups                                                                                                                   |
    | **Key Manager**        | Secrets, Containers                                                                                                                                    |
    | **Identity**           | Projects, Users, User Groups, Roles                                                                                                                    |
  </Tab>

  <Tab title="Admin View" icon="shield">
    The Admin view shows platform-wide resources. Additional sections:

    | Section            | Pages                                                                                                   |
    | ------------------ | ------------------------------------------------------------------------------------------------------- |
    | **Compute**        | + Hypervisors, Host Aggregates, Bare Metal Nodes                                                        |
    | **Storage**        | + Volume Types, Storage Backends                                                                        |
    | **Network**        | + RBAC Policies                                                                                         |
    | **Identity**       | + Domains, RBAC Management                                                                              |
    | **Monitor Center** | Overview, Physical Nodes, Storage Clusters, Services, Monitoring, Logging, Activity Log, Cluster Health |
    | **Optimization**   | Goals, Strategies, Audit Templates, Audits, Action Plans, Actions                                       |
    | **Instance-HA**    | Segments, Hosts, Notifications, VM Moves                                                                |
    | **Global Setting** | System Info, System Config, Metadata Definitions                                                        |
  </Tab>
</Tabs>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Instance Management" icon="server" href="/services/compute/launch-instance">
    4-step creation wizard with real-time quota tracking
  </Card>

  <Card title="Storage Control" icon="hard-drive" href="/services/storage/create-volume">
    Volume creation with storage tier selection
  </Card>

  <Card title="Network Topology" icon="network" href="/services/networking/create-network">
    Visual topology map and full SDN configuration
  </Card>

  <Card title="Image Library" icon="disc" href="/services/images/upload-image">
    Upload images with OS metadata and format support
  </Card>

  <Card title="Identity & Access" icon="fingerprint" href="/services/identity/user-guide">
    Project and user management with RBAC
  </Card>

  <Card title="Monitoring" icon="activity" href="/services/monitoring/user-guide">
    Integrated monitoring, logging, and health dashboards
  </Card>
</CardGroup>

***

<p>Xloud-Developed Features</p>

<Info>
  **Xloud-Developed** — These capabilities are developed by Xloud and ship with XAVS / XPCI.
</Info>

<CardGroup>
  <Card title="Live vCPU/RAM Scaling" icon="sliders" href="/services/compute/live-resize">
    Adjust instance CPU and memory via slider controls — no reboot required
  </Card>

  <Card title="Instance HA" icon="heart-pulse" href="/services/instance-ha/user-guide/how-it-works">
    Failover segments, host monitoring, real-time VM evacuation tracking
  </Card>

  <Card title="Resource Optimizer" icon="chart-bar" href="/services/optimization/user-guide">
    Automated workload placement and resource consolidation
  </Card>

  <Card title="Cluster Health" icon="activity">
    Service health with RabbitMQ, MariaDB/Galera, and service monitoring
  </Card>
</CardGroup>

***

<p>Architecture</p>

```mermaid theme={null}
graph LR
    Browser["Browser (HTTPS)"] --> Skyline["Skyline Console\n(Nginx + React)"]
    Skyline --> SkyAPI["Skyline API Server\n(FastAPI)"]
    SkyAPI --> Keystone["Identity Service"]
    SkyAPI --> Nova["Compute API"]
    SkyAPI --> Cinder["Block Storage"]
    SkyAPI --> Neutron["Networking"]
    SkyAPI --> Glance["Image Service"]
    SkyAPI --> Swift["Object Storage"]
    SkyAPI --> Heat["Orchestration"]
    style Skyline fill:#197560,color:#fff
    style SkyAPI fill:#3F8F7E,color:#fff
```

The Dashboard communicates through the Skyline API server, which proxies requests to
service APIs. All actions enforce the same RBAC rules as CLI and direct API calls.

| Component              | Description                                                     |
| ---------------------- | --------------------------------------------------------------- |
| **Skyline Console**    | React-based single-page application served by Nginx             |
| **Skyline API Server** | FastAPI backend handling authentication, RBAC, and API proxying |
| **Session Management** | JWT-based session cookies with configurable expiration          |
| **RBAC Engine**        | Per-endpoint role-based access control with 30-second cache     |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Identity & Access" icon="fingerprint" href="/services/identity/index">
    Authentication and RBAC for Dashboard login
  </Card>

  <Card title="Compute" icon="server" href="/services/compute">
    Instance lifecycle management
  </Card>

  <Card title="Deployment" icon="play" href="/deployment/index">
    Deploy and configure the Dashboard via XDeploy
  </Card>
</CardGroup>


# Dashboard User Guide
Source: https://docs.xloud.tech/services/dashboard/user-guide

Navigate the Xloud Dashboard to manage instances, volumes, networks, images, and security. Covers the Dashboard layout and key workflows.

## Overview

The Xloud Dashboard provides a browser-based interface for managing all cloud resources
in your project. This guide covers the Dashboard layout, navigation, and links to
detailed workflows for each service.

<Note>
  **Prerequisites**

  * Active account in an Xloud project with `member` role or higher
  * Access to the Dashboard at `https://connect.<your-domain>`
  * Supported browser: Chrome 88+, Firefox 90+, Edge 88+, Safari 14+
</Note>

***

## Dashboard Layout

The Dashboard has three main areas:

| Area             | Description                                                               |
| ---------------- | ------------------------------------------------------------------------- |
| **Header**       | Project selector, Console/Admin view toggle, user menu, notification bell |
| **Sidebar**      | Navigation menu organized by service (Compute, Storage, Network, etc.)    |
| **Content Area** | Resource lists, detail pages, create forms, and wizards                   |

<Tip>
  Switch between the **Console** view (project-scoped operations) and **Admin** view
  (platform-wide administration) using the toggle in the header. The Admin view is
  only visible to users with the `admin` role.
</Tip>

***

## Resource Management

Each service section in the sidebar provides list pages, create forms, and detail views.
The detailed workflows for each service are documented in their respective guides:

<CardGroup>
  <Card title="Instances" icon="server" href="/services/compute/launch-instance">
    Launch, resize, reboot, snapshot, and manage compute instances via the 4-step wizard
  </Card>

  <Card title="Volumes" icon="hard-drive" href="/services/storage/create-volume">
    Create, attach, extend, snapshot, and back up block storage volumes
  </Card>

  <Card title="Networks" icon="network" href="/services/networking/create-network">
    Create networks, subnets, routers, and configure floating IPs
  </Card>

  <Card title="Images" icon="disc" href="/services/images/upload-image">
    Upload OS images, manage snapshots, and share images across projects
  </Card>

  <Card title="Security Groups" icon="shield" href="/services/networking/security-groups">
    Create and manage firewall rules controlling instance traffic
  </Card>

  <Card title="Key Pairs" icon="key" href="/services/compute/launch-instance">
    Manage SSH key pairs for instance authentication
  </Card>

  <Card title="Object Storage" icon="box" href="/services/object-storage/user-guide">
    Browse and manage object storage containers and files
  </Card>

  <Card title="Key Manager" icon="lock" href="/services/key-manager/user-guide">
    Store and manage secrets, certificates, and encryption keys
  </Card>

  <Card title="User Center" icon="circle-user" href="/services/dashboard/user-guide/user-center">
    Manage your profile, change your password, and enable two-factor authentication
  </Card>

  <Card title="Global Search" icon="search" href="/services/dashboard/user-guide/global-search">
    Find any instance, volume, network, image, or other resource from a single search bar — keyboard-driven with Ctrl+K
  </Card>
</CardGroup>

***

## Common Dashboard Patterns

All resource pages in the Dashboard follow consistent interaction patterns:

<AccordionGroup>
  <Accordion title="List pages">
    Every resource type has a list page showing all items in your project. Common features:

    * **Search/filter bar** — narrow results by name, status, or type
    * **Tabs** — switch between views (e.g., Current Project / Public / Shared for images)
    * **Row actions** — the first action is directly visible as a button; additional actions are under the **More** dropdown
    * **Batch actions** — select multiple items with checkboxes, then apply bulk operations
    * **Primary action** — the top-right button (e.g., "Create Instance", "Create Volume")
  </Accordion>

  <Accordion title="Create forms">
    Resources are created via modal dialogs or multi-step wizards:

    * **Modals** — single-page forms for simple resources (networks, security groups, key pairs)
    * **Step wizards** — multi-page forms for complex resources (instances: 4 steps, flavors: 2 steps, bare metal: 3 steps)
    * **Confirm button** — all create forms use "Confirm" as the submit button
    * **Quota display** — real-time quota usage shown in the footer (instances, CPU, memory, volumes)
  </Accordion>

  <Accordion title="Detail pages">
    Click any resource name to open its detail page:

    * **Header** — key resource info (name, status, project, timestamps)
    * **Tabs** — organized sections (Detail, Members, Interfaces, Events, etc.)
    * **Actions** — same row actions available from the list page
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="CLI Setup" icon="terminal" href="/cli-setup">
    Configure the CLI for operations that complement the Dashboard
  </Card>

  <Card title="Dashboard Admin Guide" icon="shield" href="/services/dashboard/admin-guide">
    Platform administration and configuration
  </Card>

  <Card title="Compute User Guide" icon="server" href="/services/compute/user-guide">
    Full compute instance management reference
  </Card>

  <Card title="Networking User Guide" icon="network" href="/services/networking/user-guide">
    Full networking configuration reference
  </Card>
</CardGroup>


# Access & Security
Source: https://docs.xloud.tech/services/dashboard/user-guide/access-security

Manage SSH key pairs, security groups, and application credentials through the Xloud Dashboard.

## Overview

Access and security resources are managed across multiple sidebar sections in the
Dashboard:

| Resource                    | Location                                                  |
| --------------------------- | --------------------------------------------------------- |
| **Key Pairs**               | Compute > Key Pairs                                       |
| **Security Groups**         | Network > Security Groups                                 |
| **Application Credentials** | User profile menu > User Center > Application Credentials |

***

## Key Pairs

Navigate to **Compute > Key Pairs**. Key pairs provide SSH authentication for
Linux instances.

**Create a Key Pair**: Click **Create Key Pair**.

| Field          | Description                                                 |
| -------------- | ----------------------------------------------------------- |
| **Type**       | Create new (generates keypair) or Import (paste public key) |
| **Name**       | Key pair name (required)                                    |
| **Public Key** | Paste your public key (shown when Type is Import)           |

When creating a new key pair, the private key downloads automatically as a `.pem`
file. Store it securely — it cannot be retrieved later.

<Warning>
  Key pair quota is displayed in the create dialog. If exhausted, delete unused
  key pairs before creating new ones.
</Warning>

***

## Security Groups

Navigate to **Network > Security Groups**. Security groups act as virtual firewalls
controlling inbound and outbound traffic per instance port.

**Create a Security Group**: Click **Create Security Group**. Enter a **Name** and
optional **Description**. Default egress rules (allow all outbound IPv4/IPv6) are
created automatically.

**Add Rules**: Click the group name to open the detail page, then click **Add Rule**:

| Field                | Options                           |
| -------------------- | --------------------------------- |
| **Direction**        | Ingress or Egress                 |
| **Ether Type**       | IPv4 or IPv6                      |
| **Protocol**         | TCP, UDP, ICMP, or ANY            |
| **Port Range**       | Min/Max port (TCP/UDP only)       |
| **Remote IP Prefix** | CIDR notation (e.g., `0.0.0.0/0`) |

<Note>
  For the complete security groups reference, see
  [Network Security Groups](/services/networking/security-groups).
</Note>

***

## Application Credentials

Application credentials allow non-interactive API authentication without exposing
your password.

Navigate to the **user profile menu** (top-right avatar) and select **Application
Credentials** under **User Center**.

Click **Create Application Credentials**. Configure name, description, expiration,
roles, and whether the credential is unrestricted.

<Note>
  Application Credentials are accessed via the user profile dropdown in Skyline,
  not through the Identity sidebar section.
</Note>

***

## Detailed Guides

<CardGroup>
  <Card title="Security Groups" href="/services/networking/security-groups">
    Complete security group and rule management
  </Card>

  <Card title="Manage IPs" href="/services/compute/manage-ips">
    Associate floating IPs with instances
  </Card>

  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Select key pairs and security groups during launch
  </Card>

  <Card title="Key Manager" href="/services/key-manager/user-guide">
    Store secrets and certificates
  </Card>
</CardGroup>


# Global Search
Source: https://docs.xloud.tech/services/dashboard/user-guide/global-search

Find any instance, volume, network, image, or other Xloud resource from a single search bar in the Dashboard — fuzzy match, keyboard-driven, with recent searches and quick actions.

## Overview

The Xloud Dashboard ships a **global search bar** that finds resources across
the entire cluster — instances, volumes, networks, images, security groups,
routers, floating IPs, key pairs, flavors, projects, and users — from a single
input. Results show the resource type, name, status, and a direct link to the
detail page. The search is fuzzy, debounced, and keyboard-driven.

<Info>
  Global Search is part of the standard Xloud Dashboard. It is available to any
  signed-in user — results are automatically scoped to projects the user can
  see, and admins see results across every project they have access to.
</Info>

***

## Open Global Search

| Method         | Action                                                                  |
| -------------- | ----------------------------------------------------------------------- |
| **Keyboard**   | Press **Ctrl + K** (Linux / Windows) or **⌘ + K** (macOS) from any page |
| **Header bar** | Click the search icon in the top header of the Dashboard                |

Press **Escape** to close. Both shortcuts toggle — pressing **Ctrl + K** when
the search is already open closes it.

***

## What Global Search Finds

The search hits a single backend endpoint that aggregates results across every
resource type the Dashboard exposes. Results are returned grouped by type:

| Resource type       | Examples of what matches       |
| ------------------- | ------------------------------ |
| **Instances**       | Name, ID, IP address fragments |
| **Volumes**         | Name, ID                       |
| **Networks**        | Name, ID                       |
| **Images**          | Name, ID                       |
| **Security Groups** | Name, ID                       |
| **Routers**         | Name, ID                       |
| **Floating IPs**    | IP address, ID                 |
| **Key Pairs**       | Name                           |
| **Flavors**         | Name, ID                       |
| **Projects**        | Name, ID (admin view)          |
| **Users**           | Name, ID (admin view)          |

Matching is **fuzzy** — a query of `web` matches `web-01`, `web-prod-server`,
and even names where the letters appear in order with gaps. A minimum of
**2 characters** is required before search runs.

<Tip>
  Searches are **debounced by 300 milliseconds**, so the Dashboard waits a
  fraction of a second after you stop typing before firing the query. This
  avoids hammering the backend with a request on every keystroke.
</Tip>

***

## Use Global Search

<Steps>
  <Step title="Open the search panel" icon="search">
    Press **Ctrl + K** (⌘ + K on macOS) anywhere in the Dashboard.
  </Step>

  <Step title="Type at least 2 characters" icon="keyboard">
    The Dashboard waits a moment after you stop typing, then queries the backend
    and groups results by resource type. While the query is running, a spinner
    appears next to the input.
  </Step>

  <Step title="Navigate with the keyboard or mouse" icon="arrow-down">
    Use **↑** and **↓** to move the selection through the result list. The
    currently selected row is highlighted. Press **Enter** to open the resource
    detail page, or click any row directly.

    <Check>The Dashboard navigates to the resource's detail page. The selection is also recorded as a recent search.</Check>
  </Step>

  <Step title="Close the panel" icon="x">
    Press **Escape** or **Ctrl + K** again to close the search.
  </Step>
</Steps>

***

## Result Cards — What's on Each Row

Each result row shows three pieces of context so you can tell the right item
apart from same-named neighbors at a glance:

| Element           | What it tells you                                                                                                                                                                                  |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Resource icon** | Quick visual cue of the type (server, database, cloud, image, etc.)                                                                                                                                |
| **Name**          | The display name of the resource                                                                                                                                                                   |
| **Status tag**    | Colored badge — **green** for active / available / in-use / enabled, **red** for error, **grey** for shutoff / down / disabled, **yellow** for transitional states (creating, deleting, migrating) |
| **Right arrow**   | Indicates the row is clickable and will navigate to the detail page                                                                                                                                |

***

## Recent Searches

The Dashboard keeps your **last 5 selected results** as recent items.

* When the search panel is open and the query is empty, recent items appear
  at the top.
* Click a recent item to jump straight to that resource — no need to retype
  the query.
* Recent searches are stored in your browser's local storage, so they survive
  reloads but are local to that browser and device.

***

## Quick Actions

Below the recent searches (when the query is empty), the search panel also
surfaces a set of **quick actions** — the most common Dashboard operations,
one click away:

| Quick action        | What it opens                                |
| ------------------- | -------------------------------------------- |
| **Launch Instance** | Compute → Instances → Create Instance wizard |
| **Create Volume**   | Storage → Volumes → Create Volume            |
| **Create Network**  | Networking → Networks → Create Network       |
| **Manage Flavors**  | Compute → Flavors (admin)                    |
| **Manage Images**   | Compute → Images                             |
| **Key Pairs**       | Compute → Key Pairs                          |
| **Security Groups** | Networking → Security Groups                 |
| **XAVS Health**     | Monitor Center → XAVS Health (admin)         |

<Tip>
  Quick actions are a faster path than navigating the sidebar — open
  **Ctrl + K**, click the action, done. The keyboard shortcut + click is
  usually two clicks faster than the full sidebar path.
</Tip>

***

## Keyboard Shortcut Reference

| Shortcut                  | Action                                   |
| ------------------------- | ---------------------------------------- |
| **Ctrl + K** or **⌘ + K** | Open or close the search panel           |
| **Escape**                | Close the search panel                   |
| **↑** / **↓**             | Move selection up / down through results |
| **Enter**                 | Open the selected result                 |

***

## Common Tasks

<AccordionGroup>
  <Accordion title="Find an instance by partial name" icon="server">
    Press **Ctrl + K** and type any substring of the name. Fuzzy matching
    means `web` finds `web-01`, `prod-web-server`, and `web-tier-1` together.
    Use the status tag to spot the active one.
  </Accordion>

  <Accordion title="Find a volume by its UUID prefix" icon="hard-drive">
    Type the first 6–8 characters of the UUID. The result row shows the
    full name plus the status tag so you can tell attached volumes apart
    from available ones at a glance.
  </Accordion>

  <Accordion title="Find a floating IP by address" icon="globe">
    Type any segment of the IP — `103.240` returns every floating IP
    starting with that prefix. Click the row to jump to the floating-IP
    detail and its associations.
  </Accordion>

  <Accordion title="Jump straight to a Create action" icon="plus">
    Open **Ctrl + K** with no query — the Quick Actions panel shows
    Launch Instance, Create Volume, Create Network, and more. Click one
    to open the matching create wizard directly.
  </Accordion>

  <Accordion title="Revisit something you opened earlier" icon="clock">
    Open **Ctrl + K** with no query — your last 5 selected results appear
    at the top under Recent. One click jumps you back to the detail
    page.
  </Accordion>
</AccordionGroup>

***

## Limitations and Tips

* **Minimum 2 characters** — single-character queries are ignored to avoid
  flooding the backend with overly broad matches.
* **Scope** — Global Search returns resources you are authorised to see.
  Project-scoped users see only their projects; admin-view users see
  everything.
* **Status tag colors** are derived from the resource's reported state at
  query time. A row that says `creating` will switch to `active` (or
  `error`) once the resource is fully provisioned — re-run the search to
  refresh.
* **Recent searches** are per-browser. Signing in from a new browser or
  clearing local storage resets the list.

***

## Related Topics

<CardGroup>
  <Card title="Manage Instances" href="/services/dashboard/user-guide/instances">
    Every row action on the Instances list — start, stop, console, resize,
    snapshot, delete
  </Card>

  <Card title="User Center" href="/services/dashboard/user-guide/user-center">
    Manage your profile, change your password, and enable 2FA
  </Card>

  <Card title="Activity Log" href="/services/monitoring/user-guide/activity-log">
    Audit every action taken across the Xloud Platform — admin view
  </Card>
</CardGroup>


# Manage Images
Source: https://docs.xloud.tech/services/dashboard/user-guide/images

Upload OS images, manage instance snapshots, and share images between projects through the Xloud Dashboard.

## Overview

Navigate to **Compute > Images** in the sidebar. Images are organized in tabs:
**Current Project Images**, **Public Images**, **Shared Images**, and **All Images**
(admin role required).

***

## Create an Image

Click **Create Image** to open the full-page form:

| Field                            | Description                                                                                   |
| -------------------------------- | --------------------------------------------------------------------------------------------- |
| **Name**                         | Image display name (required)                                                                 |
| **Upload Type**                  | Upload File or File URL                                                                       |
| **Disk Format**                  | RAW, QCOW2, ISO (admin: + AKI, ARI, AMI, VDI, VHD, VMDK)                                      |
| **Container Format**             | Bare or Docker (shown when multiple formats available)                                        |
| **OS / OS Version / OS Admin**   | OS metadata (shown for Bare container format)                                                 |
| **Min System Disk / Min Memory** | Resource requirements (0-500 GiB)                                                             |
| **Visibility**                   | Public/Private/Shared (admin only)                                                            |
| **Usage Type**                   | Common Server, Bare Metal (admin: + Load Balancer, Database, Container, Application Template) |
| **Advanced Options**             | qemu\_guest\_agent, CPU Policy, CPU Thread Policy                                             |

<Note>
  For the complete field reference, see [Upload an Image](/services/images/upload-image).
</Note>

***

## Image Actions

**Edit** is the first row action. Other actions under **More**:

**User view**: Create Instance, Create Ironic Instance, Create Volume, Delete

**Admin view**: Delete, Manage Access (requires Shared visibility), Manage Metadata

***

## Instance Snapshots

Navigate to **Compute > Instance Snapshots** for snapshots captured from running
instances. Actions: Edit, Create Instance, Create Volume, Delete.

***

## Detailed Guides

<CardGroup>
  <Card title="Upload an Image" href="/services/images/upload-image">
    Full create form with all fields
  </Card>

  <Card title="Image Properties" href="/services/images/image-properties">
    Edit properties and manage metadata
  </Card>

  <Card title="Share Images" href="/services/images/share-images">
    Share images across projects via Manage Access
  </Card>

  <Card title="Instance Snapshots" href="/services/compute/snapshots">
    Create snapshots from running instances
  </Card>
</CardGroup>


# Manage Instances
Source: https://docs.xloud.tech/services/dashboard/user-guide/instances

Launch, monitor, scale, snapshot, and control compute instances through the Xloud Dashboard. Covers every action exposed on the Instances page.

## Overview

Navigate to **Compute → Instances** in the sidebar to manage compute instances.

***

## Launch an Instance

Click **Create Instance** to open the 4-step wizard.

| Step | Name               | What you configure                                                                                                                                                       |
| ---- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1    | **Base Config**    | Availability Zone, Flavor (with architecture and category filters), Start Source (Image / Snapshot / Bootable Volume), Boot From Volume, System Disk, Data Disks, CD-ROM |
| 2    | **Network Config** | Networks, Virtual LAN (subnet / IP), Ports, Security Groups                                                                                                              |
| 3    | **System Config**  | Name, Login Type (Keypair / Password), Advanced Options (Physical Node, Server Group, User Data, Virtual TPM, Secure Boot)                                               |
| 4    | **Confirm Config** | Review every setting, instance count, real-time quota display                                                                                                            |

<Note>
  For the complete field-by-field reference, see
  [Launch an Instance](/services/compute/launch-instance).
</Note>

***

## Instance Actions — Every Option Explained

The Instances list exposes one **direct row action** (Console), a **More** dropdown
with five action submenus, and a **batch-action toolbar** at the top of the table
that operates on multi-selected rows.

### Direct row action

| Action      | What it does                                                                                      |
| ----------- | ------------------------------------------------------------------------------------------------- |
| **Console** | Opens an interactive browser console (VNC for graphical guests, serial for headless) in a new tab |

### More → Instance Status

Power and lifecycle actions. Each is a separately gated permission.

| Action          | What it does                                                                                                                     |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| **Start**       | Boots a stopped instance                                                                                                         |
| **Stop**        | Powers off the instance gracefully (ACPI shutdown)                                                                               |
| **Reboot**      | Hard reboot — issues a reset to the running guest                                                                                |
| **Soft Reboot** | Graceful reboot — asks the guest OS to shut down and start again                                                                 |
| **Pause**       | Suspends instance execution in CPU only — RAM stays in place; faster to resume than Suspend                                      |
| **Unpause**     | Resumes a paused instance                                                                                                        |
| **Suspend**     | Saves the instance state to disk and frees its host RAM — slower to resume but releases hypervisor memory                        |
| **Resume**      | Resumes a suspended instance from its saved state                                                                                |
| **Shelve**      | Stops the instance and offloads its image so the hypervisor can reclaim the slot — survives even if the original host is removed |
| **Unshelve**    | Restores a shelved instance, scheduling it onto an available host                                                                |
| **Lock**        | Prevents accidental destructive actions on this instance — required to be unlocked before delete or other state changes          |
| **Unlock**      | Removes the lock                                                                                                                 |

### More → Related Resources

Attach or detach things to a running instance.

| Action                       | What it does                                                                                          |
| ---------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Attach Interface**         | Adds a network interface (port) to the instance — pick an existing port or auto-create from a network |
| **Detach Interface**         | Removes a network interface                                                                           |
| **Attach Volume**            | Attaches an existing block storage volume — choose the volume and the device name                     |
| **Detach Volume**            | Detaches a volume from the instance (volume must be detachable; some boot volumes are not)            |
| **Associate Floating IP**    | Binds a public floating IP to the instance's port                                                     |
| **Disassociate Floating IP** | Releases the public floating IP from the port                                                         |
| **Manage Security Group**    | Add or remove security groups on the instance's ports — controls inbound and outbound traffic rules   |

### More → Backups and Snapshots

| Action              | What it does                                                                                                                       |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| **Create Snapshot** | Captures the current disk state of the instance as a new image — useful as a checkpoint before an upgrade or as a base for cloning |

### More → Clone and Template

| Action                  | What it does                                                                                                                                              |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Clone**               | Creates a new instance from a copy of this instance's disks — copies system and data volumes, preserves the network configuration choice in a single step |
| **Convert to Template** | Captures the running instance's disk and metadata into a reusable VM Template that other users can deploy as a starting point                             |

### More → Configuration Update

| Action                        | What it does                                                                                                                                                           |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Resize**                    | Change the instance's flavor (vCPU / RAM / disk shape). The instance is stopped, migrated to a host with capacity, and restarted on the new flavor                     |
| **Confirm Resize or Migrate** | Confirms a successful resize or migration — frees the original allocation. Required step before further actions                                                        |
| **Revert Resize or Migrate**  | Aborts a resize or migration — rolls back to the original flavor and host                                                                                              |
| **Adjust Resources**          | Live-resize — changes vCPU and RAM (and adds or removes virtual devices) **without rebooting**. See [Live Resize](/services/compute/live-resize)                       |
| **Change Password**           | Resets the guest OS root or administrator password. The guest must have the password reset agent (cloud-init or QEMU guest agent) installed                            |
| **Rebuild Instance**          | Reinstalls the instance from a different image while keeping its IPs, ports, attached volumes, and metadata. Use to refresh a guest OS without losing network identity |

### Other row actions (ungrouped)

| Action                   | What it does                                                                                                          |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------- |
| **Edit Instance**        | Edit the instance name and description                                                                                |
| **Modify Instance Tags** | Add or remove tags on the instance — used by automation, billing, and scheduling. Up to 50 tags per instance          |
| **Delete**               | Soft-delete the instance — the row remains in **Recycle Bin** for the configured retention window before being purged |

### Batch actions (multi-select toolbar)

Select multiple rows with the checkboxes at the left of the table; the batch
toolbar appears above the table.

| Action          | What it does                       |
| --------------- | ---------------------------------- |
| **Start**       | Bulk start every selected instance |
| **Stop**        | Bulk graceful stop                 |
| **Reboot**      | Bulk hard reboot                   |
| **Soft Reboot** | Bulk graceful reboot               |
| **Delete**      | Bulk soft-delete                   |

### Admin-view extras

When viewing instances through the admin view (visible to users with the
`admin` role), additional actions appear:

| Action                | What it does                                                                                         |
| --------------------- | ---------------------------------------------------------------------------------------------------- |
| **Migrate**           | Cold migrate — stops the instance, moves it to a different compute host, restarts                    |
| **Live Migrate**      | Move a running instance to another compute host with no perceptible downtime                         |
| **Bulk Live Migrate** | Live-migrate multiple selected instances in sequence — useful when evacuating a host for maintenance |

***

## Auto-Discovery of Instances Launched Outside the Dashboard

The Xloud Platform's compute layer keeps a single source of truth for every
running virtual machine. Whether an instance was launched through the Xloud
Dashboard, through XCONNECT, or through any external client (`openstack server
create`, Terraform, Ansible, Heat stack, custom script using the SDK), the
instance appears in **Compute → Instances** automatically.

| Where the VM came from                       | Visible in Compute → Instances? |
| -------------------------------------------- | :-----------------------------: |
| Xloud Dashboard — Create Instance wizard     |               Yes               |
| XCONNECT                                     |               Yes               |
| `openstack server create` from the CLI       |               Yes               |
| Terraform `openstack_compute_instance_v2`    |               Yes               |
| Ansible `os_server` module                   |               Yes               |
| Heat orchestration template                  |               Yes               |
| Custom SDK script (Python / Go / JavaScript) |               Yes               |

**How it works** — every compute host added to the cluster through XDEPLOY runs
the Xloud compute agent, which reports every VM running on the hypervisor back
to the platform's central state. The Dashboard reads from that central state, so
it always reflects what is actually running on the cluster — there is no
separate "import" or "register VM" step needed for instances launched outside
the Dashboard.

**Two prerequisites** for auto-discovery:

<Steps>
  <Step title="The compute host is added to the cluster" icon="server">
    The hypervisor running the VM must have been added to the cluster through
    **XDEPLOY → Hosts**. Hosts not registered with the cluster cannot report
    their VMs, and any VMs running on them remain invisible until the host is
    added.
  </Step>

  <Step title="Project visibility applies" icon="users">
    The signed-in user must have access to the project that owns the instance.
    Cross-project visibility requires the admin view, which exposes every
    instance on the cluster.
  </Step>
</Steps>

<Tip>
  This behavior is independent of how the instance was provisioned, what
  toolchain manages it, or which user created it. Even an instance booted from
  a Heat stack months ago by a teammate who has since left appears in
  **Compute → Instances** as long as the host is in the cluster and you have
  project access.
</Tip>

***

## Instance Detail Page

Click any instance name in the list to open the detail page. The detail page
exposes eight tabs:

| Tab                 | What it shows                                                                      |
| ------------------- | ---------------------------------------------------------------------------------- |
| **Detail**          | Identity, project, image, flavor, host, AZ, IPs, security groups, tags, timestamps |
| **Volumes**         | Attached volumes with size, type, device name, and detach action                   |
| **Snapshots**       | Instance snapshots created from this VM                                            |
| **Interfaces**      | Network ports, MAC addresses, fixed IPs, security groups                           |
| **Floating IPs**    | Public IPs bound to the instance's ports                                           |
| **Security Groups** | Effective security group rules applied to the instance                             |
| **Action Logs**     | Audit trail of every action performed on the instance with actor and timestamp     |
| **Logs**            | Console boot log captured by the hypervisor                                        |

Detail-page row actions are the same set as the list row actions — Console as the
direct action, plus the More menu with all submenus described above.

***

## Detailed Guides

<CardGroup>
  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Complete 4-step wizard reference
  </Card>

  <Card title="Resize" href="/services/compute/resize-instance">
    Change flavor with confirm and revert
  </Card>

  <Card title="Live Resize" href="/services/compute/live-resize">
    Adjust vCPU and RAM without reboot
  </Card>

  <Card title="Console Access" href="/services/compute/console-access">
    Browser-based VNC console
  </Card>

  <Card title="Snapshots" href="/services/compute/snapshots">
    Capture and manage instance snapshots
  </Card>

  <Card title="Clone and Template" href="/services/compute/clone-instance">
    Duplicate an instance or save it as a reusable template
  </Card>

  <Card title="Live Migration" href="/services/compute/live-migration">
    Move a running instance to another host with zero downtime
  </Card>

  <Card title="Server Groups" href="/services/compute/server-groups">
    Affinity and anti-affinity placement policies
  </Card>

  <Card title="Instance Tagging" href="/services/compute/instance-tagging">
    Tag instances for organization, billing, and automation
  </Card>
</CardGroup>


# Manage Networks
Source: https://docs.xloud.tech/services/dashboard/user-guide/networks

Create and configure virtual networks, subnets, routers, and floating IPs through the Xloud Dashboard.

## Overview

The **Network** sidebar section provides pages for: Networks, Ports, QoS Policies,
Routers, Floating IPs, Topology, Load Balancers, Certificates, VPNs, Security Groups,
Firewalls, DNS Zones, and DNS Reverse.

***

## Create a Network

Navigate to **Network > Networks** and click **Create Network**. The modal form includes:

| Field                     | Description                                |
| ------------------------- | ------------------------------------------ |
| **Network Name**          | Display name (required)                    |
| **Description**           | Optional notes                             |
| **Available Zone**        | Zone selection                             |
| **MTU**                   | Maximum transmission unit (68-9000)        |
| **Create Subnet**         | Toggle to add a subnet with the network    |
| **Port Security Enabled** | Enable/disable port security (default: on) |

When **Create Subnet** is checked, additional fields appear: Subnet Name, IP Version
(IPv4/IPv6), CIDR, plus advanced options (Gateway, DHCP, DNS, Allocation Pools, Host Routes).

<Note>
  For the complete field reference, see [Create a Network](/services/networking/create-network).
</Note>

***

## Network Topology

Navigate to **Network > Topology** for a visual graph showing all networks, routers,
and instances in your project. Click elements to view details. Use the controls to
toggle instance visibility, and create resources directly from the topology view.

***

## Key Pages

<CardGroup>
  <Card title="Create a Network" href="/services/networking/create-network">
    Full network creation reference
  </Card>

  <Card title="Routers" href="/services/networking/routers">
    Create routers and manage gateways
  </Card>

  <Card title="Floating IPs" href="/services/networking/floating-ips">
    Allocate and associate public IPs
  </Card>

  <Card title="Security Groups" href="/services/networking/security-groups">
    Create firewall rules for instances
  </Card>

  <Card title="VPN" href="/services/networking/vpn">
    IPsec site-to-site VPN tunnels
  </Card>

  <Card title="DNS" href="/services/dns/user-guide">
    Manage DNS zones and records
  </Card>
</CardGroup>


# Object Storage
Source: https://docs.xloud.tech/services/dashboard/user-guide/object-containers

Browse, create, and manage object storage containers and files through the Xloud Dashboard.

## Overview

Navigate to **Storage > Object Storage** in the sidebar to manage containers and
objects. The object storage service provides scalable, replicated storage for
unstructured data like backups, media files, and static assets.

***

## Manage Containers

The container list shows: **Name**, **Size**, **Last Updated**, and a detail popover
with object count, storage policy, and public access URL.

**Create a container**: Click **Create Container**. Enter a **Name** (max 63 characters,
cannot be changed after creation) and toggle **Public Access** if objects should be
accessible via public URL.

**Actions**: **Access** (first action — toggle public access), **Delete** (More dropdown).

***

## Manage Objects

Click a container name to browse its contents. The file browser shows objects with
Name, Size, and Last Updated columns.

**Available actions**:

| Action                 | Description                                 |
| ---------------------- | ------------------------------------------- |
| **Create Folder**      | Create a pseudo-folder (path prefix)        |
| **Upload**             | Upload files (max 1 GiB per file)           |
| **Download**           | Download individual files                   |
| **Edit**               | Replace file content (same filename)        |
| **Rename**             | Rename a file (copies and deletes original) |
| **Copy / Cut / Paste** | Move or copy files between folders          |
| **Delete**             | Delete files or empty folders               |

Use the breadcrumb navigation at the top to navigate folder hierarchy.

<Note>
  For the complete object storage reference, see [Object Storage User Guide](/services/object-storage/user-guide).
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Object Storage User Guide" href="/services/object-storage/user-guide">
    Complete object storage management reference
  </Card>

  <Card title="Block Storage" href="/services/dashboard/user-guide/volumes">
    Persistent volumes for compute instances
  </Card>
</CardGroup>


# User Center
Source: https://docs.xloud.tech/services/dashboard/user-guide/user-center

Manage your Xloud account from the Dashboard — profile, password, two-factor authentication, sign-in activity, and application credentials.

## Overview

User Center is the self-service hub for your own Xloud account. From here you can update
your profile, change your password, enable or manage two-factor authentication (2FA), and
create application credentials for automation. Every page in User Center affects only
your account — nothing here changes cloud resources for other users.

<Note>
  **Prerequisites**

  * An active Xloud account — sign in to the [Xloud Dashboard](/services/dashboard)
  * An authenticator app on your phone (for 2FA): Google Authenticator, Microsoft
    Authenticator, Authy, 1Password, or any standard TOTP app
</Note>

***

## Video Walkthrough

<iframe title="How to Set Up 2FA and User Center on Xloud" />

***

## Open User Center

<Steps>
  <Step title="Open the profile menu" icon="user">
    Click your avatar or name in the **top-right corner** of the Xloud Dashboard.
  </Step>

  <Step title="Pick User Center" icon="circle-user">
    Select **User Center** from the dropdown. The overview page loads with your avatar,
    roles, role/domain/project stats, and quick-edit buttons.
  </Step>
</Steps>

***

## User Center Overview

The landing page shows everything about your account at a glance.

| Section              | What's there                                                                       |
| -------------------- | ---------------------------------------------------------------------------------- |
| **Header**           | Avatar (click to change), display name, email, role tags                           |
| **Stat row**         | Number of Roles, Domains, and Projects you belong to                               |
| **Account Details**  | Username, Email, Phone, Real Name, Job Title, Department, User ID, Current Project |
| **Roles & Security** | My Roles, Domain, Project ID, Account Status                                       |

Two action buttons sit at the bottom of the Account Details card:

* **Edit Profile** → opens Profile Settings
* **Security (2FA)** → opens the Security page

<Tip>
  Click your avatar to upload a custom profile image. It will be used across the entire
  Dashboard.
</Tip>

***

## Edit Profile Settings

The **Profile Settings** page has two tabs for self-service account edits.

### Profile Tab

Fill in your personal information — these fields help teammates identify you in
multi-user projects.

| Field          | Notes                                    |
| -------------- | ---------------------------------------- |
| **First Name** | Up to 64 characters                      |
| **Last Name**  | Up to 64 characters                      |
| **Phone**      | Any standard format, up to 20 characters |
| **Job Title**  | Up to 128 characters                     |
| **Department** | Up to 128 characters                     |

Click **Save** to persist changes.

### Security Tab

The Security tab contains three cards: **Two-Factor Authentication**, **Change Password**,
and **Sign-in activity**.

#### Two-Factor Authentication card

Shows your current 2FA status — **Enabled** or **Not Enabled**. Click **Setup Two-Factor
Authentication** (when not enabled) or **Manage 2FA Settings** (when enabled) to jump to
the dedicated Security page covered [below](#two-factor-authentication-2-fa).

#### Change Password card

<Steps>
  <Step title="Enter your current password" icon="key">
    Type your existing password in the **Current Password** field.
  </Step>

  <Step title="Enter the new password" icon="lock">
    Fill in **New Password** (minimum 8 characters) and **Confirm New Password**. The
    two values must match.
  </Step>

  <Step title="Click Change Password" icon="check">
    Click **Change Password** to apply. You stay signed in on the current device.
  </Step>
</Steps>

#### Sign-in activity card

Lists your current session and up to 10 recent active sessions on your account. Each
entry shows:

| Column         | What it shows                                                                         |
| -------------- | ------------------------------------------------------------------------------------- |
| **IP address** | Where the session signed in from (a green **This session** tag marks the current one) |
| **User agent** | Browser / OS string (trimmed to the first 80 characters)                              |
| **Signed in**  | Relative time since sign-in                                                           |
| **Expires**    | When the session will expire                                                          |

The top of the card shows when your current session started and when the previous sign-in
happened — useful for spotting unexpected sign-ins.

***

## Two-Factor Authentication (2FA)

Two-Factor Authentication adds a second verification step to every sign-in — a rotating
6-digit code from your authenticator app, on top of your password. It is the single
biggest thing you can do to protect your account from stolen-password attacks.

<Info>
  Xloud supports any standard **TOTP** (Time-based One-Time Password) authenticator app:
  Google Authenticator, Microsoft Authenticator, Authy, 1Password, Aegis, and others.
</Info>

### Check 2FA Status

Open **User Center → Security (2FA)**. The page shows one of two states:

| State                             | Meaning                                                                                                                      |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **Not enabled** (blue Info alert) | Your account is password-only. Click **Enable 2FA** to set it up.                                                            |
| **Enabled** (green Success alert) | 2FA is active. The alert shows when it was enabled, when it was last used, and how many unused recovery codes you have left. |

### Enable 2FA

<Steps>
  <Step title="Click Enable 2FA" icon="shield">
    On the **Security** page, click the blue **Enable 2FA** button. The Dashboard
    navigates to a dedicated enrollment page.
  </Step>

  <Step title="Install an authenticator app" icon="smartphone">
    Install an authenticator app on your phone — Google Authenticator, Microsoft
    Authenticator, Authy, 1Password, or Aegis all work.
  </Step>

  <Step title="Scan the QR code" icon="qr-code">
    The page shows a QR code and a **Manual key** below it. Open the authenticator
    app, tap **Add** or **+**, and either scan the QR or paste the manual key.

    The app starts generating fresh 6-digit codes every 30 seconds.
  </Step>

  <Step title="Verify the code" icon="check">
    Type the current 6-digit code from the app into the **6-digit code** field on the
    Dashboard, then click **Verify & Enable**.
  </Step>

  <Step title="Save your recovery codes" icon="download">
    The Dashboard shows **10 one-time recovery codes**. **You will not see these codes
    again.** Click **Download as .txt** or **Copy to clipboard** and store them somewhere
    safe (password manager, printed copy, encrypted note). Each code can be used once
    to sign in if you lose access to your authenticator.

    <Warning>
      If you lose both your authenticator app and your recovery codes, your
      administrator will need to reset 2FA on your account.
    </Warning>

    Click **I've saved them** to close the dialog.

    <Check>The Security page now shows a green "Enabled" alert with the enrollment timestamp.</Check>
  </Step>
</Steps>

### Sign In with 2FA

After enabling 2FA, every sign-in asks for a 6-digit code in addition to your password.
Open your authenticator app, read the current code, and type it in. Codes rotate every
30 seconds — if the current code expires, just wait for the next one.

<Tip>
  If you lose access to your authenticator, click **Use a recovery code** on the sign-in
  screen and enter one of the 10 codes you saved when you enrolled.
</Tip>

<Warning>
  Using a recovery code to sign in **automatically disables 2FA** on your account. The
  Dashboard shows a prominent banner asking you to re-enroll your authenticator as soon
  as possible.
</Warning>

### Regenerate Recovery Codes

If you have used some recovery codes or suspect they have been exposed, you can mint a
fresh set of 10.

<Steps>
  <Step title="Open the Security page" icon="shield-check">
    **User Center → Security (2FA)** (2FA must already be enabled).
  </Step>

  <Step title="Click Regenerate recovery codes" icon="rotate-cw">
    Click **Regenerate recovery codes** on the green status card.
  </Step>

  <Step title="Confirm with a current 6-digit code" icon="key">
    Enter the current code from your authenticator and click **Regenerate**.

    <Warning>
      Any previously saved codes will stop working immediately. Save the new set the
      same way you did at enrollment.
    </Warning>
  </Step>
</Steps>

### Disable 2FA

<Steps>
  <Step title="Click Disable 2FA" icon="shield-off">
    On the Security page, click the red **Disable 2FA** button.
  </Step>

  <Step title="Confirm with a code" icon="key">
    Enter a current **6-digit code** from your authenticator, or click **Use a
    recovery code instead** and enter one of your saved recovery codes.
  </Step>

  <Step title="Click Disable" icon="check">
    Click **Disable**. Future sign-ins use password only until you re-enable 2FA.

    <Warning>
      Disabling 2FA weakens your account's security. Only do this when you are about
      to re-enroll with a new authenticator device.
    </Warning>
  </Step>
</Steps>

***

## Application Credentials

Application credentials are long-lived API tokens for scripts and automation. They can
optionally be scoped to a subset of roles and have an expiry date.

Open **User Center → Application Credentials** to create, view, and revoke your
credentials.

<Card title="Application Credentials Guide" icon="key" href="/services/identity/application-credentials">
  Full walkthrough on creating, using, and revoking application credentials
</Card>

***

## Common Tasks

<AccordionGroup>
  <Accordion title="Set a profile picture" icon="camera">
    Open **User Center**, click your avatar in the header, pick an image, and save. The
    avatar updates across the Dashboard without a sign-out.
  </Accordion>

  <Accordion title="Change your password" icon="lock">
    **User Center → Edit Profile → Security** tab → **Change Password** card. Enter your
    current password, then the new one twice, and click **Change Password**.
  </Accordion>

  <Accordion title="Turn on 2FA for your account" icon="shield">
    **User Center → Security (2FA)** → **Enable 2FA** → scan the QR in your authenticator
    → verify the 6-digit code → download the 10 recovery codes and store them safely.
  </Accordion>

  <Accordion title="Check where you are signed in from" icon="monitor">
    **User Center → Edit Profile → Security** tab → **Sign-in activity** card. Review the
    list of active sessions, their source IP addresses, and last sign-in time. If you
    see a session you do not recognize, change your password immediately.
  </Accordion>

  <Accordion title="Recover access if you lost your phone" icon="alert-triangle">
    At the sign-in prompt, click **Use a recovery code** and enter one of the 10 codes
    you saved during enrollment. This signs you in AND disables 2FA — re-enroll on your
    new phone as soon as possible.

    If you also lost your recovery codes, contact your administrator — only an
    administrator can reset 2FA on your behalf.
  </Accordion>
</AccordionGroup>

***

## Related Topics

<CardGroup>
  <Card title="Multi-Factor Authentication" href="/services/identity/multi-factor-auth">
    Technical reference for TOTP MFA — includes CLI-based enrollment
  </Card>

  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Long-lived API tokens for scripts and CI pipelines
  </Card>

  <Card title="Access & Security" href="/services/dashboard/user-guide/access-security">
    RBAC, keypairs, and other access controls on the Dashboard
  </Card>
</CardGroup>


# Manage Volumes
Source: https://docs.xloud.tech/services/dashboard/user-guide/volumes

Create, attach, extend, snapshot, and back up block storage volumes through the Xloud Dashboard.

## Overview

Block storage is managed under the **Storage** sidebar section with three pages:
**Volumes**, **Volume Backups**, and **Volume Snapshots**.

***

## Create a Volume

Navigate to **Storage > Volumes** and click **Create Volume**. The full-page form includes:

| Field                | Description                                   |
| -------------------- | --------------------------------------------- |
| **Available Zone**   | Availability zone for the volume              |
| **Data Source Type** | Blank Volume, Image, or Snapshot              |
| **Volume Type**      | Storage tier (select from available backends) |
| **Capacity (GiB)**   | Size with slider (quota-limited)              |
| **Name**             | Volume display name                           |
| **Description**      | Optional notes                                |

Footer shows **Count** (create multiple) and real-time **quota usage**.

<Note>
  For the complete field reference, see [Create a Volume](/services/storage/create-volume).
</Note>

***

## Volume Actions

**Edit** is the first (directly visible) row action. Other actions under **More**:

| Group                | Actions                                             |
| -------------------- | --------------------------------------------------- |
| **Data Protection**  | Create Snapshot, Create Backup, Clone Volume        |
| **Instance Related** | Attach, Detach                                      |
| **Capacity & Type**  | Extend Volume, Change Type                          |
| **Transfer**         | Create Transfer, Accept Transfer, Cancel Transfer   |
| —                    | Create Image, Create Instance, Set Bootable, Delete |

***

## Volume Detail

Click a volume name. Three tabs: **Detail** (info + attachments), **Volume Backups** (backups from this volume), **Volume Snapshots** (snapshots of this volume).

***

## Detailed Guides

<CardGroup>
  <Card title="Create a Volume" href="/services/storage/create-volume">
    Full create form reference
  </Card>

  <Card title="Attach/Detach" href="/services/storage/attach-volume">
    Instance attachment workflow
  </Card>

  <Card title="Snapshots" href="/services/storage/snapshots">
    Volume snapshot management
  </Card>

  <Card title="Backups" href="/services/storage/backups">
    Full and incremental backups
  </Card>
</CardGroup>


# XDR Architecture
Source: https://docs.xloud.tech/services/disaster-recovery/admin-guide/architecture

Understand the XDR component topology — control plane, agents, replication pipeline, and site deployment models.

## Overview

XDR operates as a control plane layer over the underlying compute and storage
infrastructure, orchestrating continuous replication and coordinated recovery
across geographically separated sites. Understanding the component architecture
helps administrators size deployments, plan network requirements, and diagnose
failures effectively.

<Note>
  **Prerequisites**

  * Administrator credentials on both primary and DR sites
  * Familiarity with RPO/RTO concepts and distributed storage replication terminology
</Note>

***

## Component Topology

```mermaid theme={null}
graph TD
    subgraph Primary["Primary Site"]
        PA["Compute Instances"]
        PS["XSDS Block / Object Storage"]
        PC["XDR Agent (Primary)"]
        PM["XIMP Agent"]
    end

    subgraph Network["Replication Network"]
        LINK["Dedicated Replication Link\n(TLS-encrypted / Compressed)"]
    end

    subgraph DR["DR Site"]
        DA["Compute Replicas\n(Stopped / Standby)"]
        DS["Storage Replicas\n(Continuous Sync)"]
        DC["XDR Agent (DR)"]
        DM["XIMP Agent"]
    end

    subgraph Control["Control Plane"]
        XDR["XDR Controller\n(Failover Orchestration)"]
        XIMP["XIMP\n(Monitoring & Alerts)"]
    end

    PA --> PC
    PS --> PC
    PC -->|Replication stream| LINK
    LINK -->|Replicated data| DC
    DC --> DA
    DC --> DS
    PM --> XIMP
    DM --> XIMP
    XDR --> PC
    XDR --> DC
    XIMP --> XDR
```

***

## Core Components

| Component               | Role                                                                                                        |
| ----------------------- | ----------------------------------------------------------------------------------------------------------- |
| **XDR Controller**      | Central orchestration service — manages protection plans, tracks recovery state, and triggers failover      |
| **XDR Agent (Primary)** | Runs on primary site nodes; captures change streams from storage and forwards to DR agent                   |
| **XDR Agent (DR)**      | Receives replicated data, applies it to DR-site replicas, and executes the recovery runbook during failover |
| **Replication Network** | Dedicated link carrying encrypted, optionally compressed replication traffic between sites                  |
| **XIMP Integration**    | Provides health visibility for replication lag, RPO adherence, and site availability                        |

***

## Replication Pipeline

Data flows through a multi-stage pipeline from the primary site to the DR site:

<Steps>
  <Step title="Change capture" icon="activity">
    The XDR agent on the primary site intercepts write operations at the storage
    layer, capturing changed data blocks as a continuous stream. For
    application-consistent replication, the agent coordinates with in-guest
    agents to quiesce writes at consistent intervals.
  </Step>

  <Step title="Compression and encryption" icon="lock">
    The change stream is optionally compressed (recommended for WAN links) and
    encrypted using TLS 1.3 before transmission. Compression reduces bandwidth
    consumption by 30–60% for typical mixed workloads.
  </Step>

  <Step title="Transfer" icon="arrow-right">
    The compressed, encrypted stream is transmitted over the replication link
    to the DR-site XDR agent. Bandwidth throttling applies during peak hours
    if configured.
  </Step>

  <Step title="Apply to replicas" icon="hard-drive">
    The DR-site agent writes the received changes to the standby storage replicas.
    Compute replicas remain stopped — the data is current but the instances are
    not running.
  </Step>

  <Step title="Recovery point snapshot" icon="camera">
    At configurable intervals, the XDR agent creates a recovery point — a
    consistent snapshot of the replicated state. Recovery points define the
    available restore targets during failover.
  </Step>
</Steps>

***

## Deployment Models

<Tabs>
  <Tab title="Active-Passive (Standard)" icon="shield">
    The most common deployment model. One site runs production workloads; the
    other site holds warm replicas that activate only during failover.

    | Characteristic                   | Value                                                                   |
    | -------------------------------- | ----------------------------------------------------------------------- |
    | **Sites**                        | 2 (primary + DR)                                                        |
    | **Production traffic**           | Primary site only                                                       |
    | **DR site resource consumption** | Storage cost + agent overhead (no compute billing for stopped replicas) |
    | **Failover time**                | Minutes (RTO depends on workload complexity)                            |
    | **RPO**                          | Seconds to minutes (asynchronous) or zero (synchronous)                 |

    <Tip>
      The DR site requires approximately the same storage capacity as the primary
      site. Compute resources are only consumed during an actual failover or DR test.
    </Tip>
  </Tab>

  <Tab title="Active-Active (Bidirectional)" icon="refresh-cw">
    Both sites run production workloads. XDR replicates in both directions,
    protecting each site from failure of the other.

    | Characteristic         | Value                                                                     |
    | ---------------------- | ------------------------------------------------------------------------- |
    | **Sites**              | 2 (both active)                                                           |
    | **Production traffic** | Both sites simultaneously                                                 |
    | **Complexity**         | Higher — requires split-brain prevention and write conflict resolution    |
    | **Failover time**      | Near-instant (surviving site already running)                             |
    | **Use case**           | Geographically distributed active users; maximum availability requirement |

    <Warning>
      Active-active deployments require careful application design to avoid write
      conflicts. Not all workloads are suitable for bidirectional replication.
      Contact Xloud support before deploying active-active XDR.
    </Warning>
  </Tab>

  <Tab title="Multi-Site Fan-Out" icon="git-branch">
    One primary site replicates to two or more DR sites simultaneously —
    typically used for geographic redundancy or regulatory data residency requirements.

    | Characteristic | Value                                                  |
    | -------------- | ------------------------------------------------------ |
    | **Sites**      | 3 or more (1 primary + N DR)                           |
    | **Bandwidth**  | Multiplied by the number of DR sites                   |
    | **Use case**   | Regulatory requirements for multiple geographic copies |

    <Note>
      Multi-site fan-out doubles or triples replication bandwidth requirements.
      Ensure the primary site uplink can sustain concurrent streams to all DR sites.
    </Note>
  </Tab>
</Tabs>

***

## Control Plane Placement

The XDR controller can be co-located with the primary site or deployed on a
separate management network:

| Placement                     | Consideration                                                            |
| ----------------------------- | ------------------------------------------------------------------------ |
| **Primary site**              | Simpler deployment; controller unavailable if primary site fails         |
| **DR site**                   | Controller survives primary site failure; manages failover independently |
| **Dedicated management host** | Highest availability; additional infrastructure required                 |
| **XDeploy cluster**           | Recommended — integrated with XDeploy's high-availability deployment     |

<Tip>
  Deploy the XDR controller in the XDeploy management cluster. XDeploy runs
  with redundancy across nodes, so the controller remains available even during
  a primary site failure event.
</Tip>

***

## Network Requirements

| Traffic Type        | Direction            | Port         | Protocol |
| ------------------- | -------------------- | ------------ | -------- |
| Replication data    | Primary → DR         | TCP 7000     | TLS      |
| Replication control | Bidirectional        | TCP 7001     | TLS      |
| Agent API           | Controller → Agents  | TCP 7002     | HTTPS    |
| Health checks       | Controller → Primary | Configurable | HTTP/TCP |

All ports must be open in both directions between primary and DR site networks.
Use a dedicated VLAN or MPLS circuit for replication traffic to avoid impacting
production workloads during initial sync or peak change rate periods.

***

## Next Steps

<CardGroup>
  <Card title="Replication Configuration" href="/services/disaster-recovery/admin-guide/replication-config">
    Register sites and configure the replication link
  </Card>

  <Card title="Recovery Plans" href="/services/disaster-recovery/admin-guide/recovery-plans">
    Define resource groups and recovery ordering
  </Card>

  <Card title="Monitoring" href="/services/disaster-recovery/admin-guide/monitoring">
    Configure XIMP alerts for replication health
  </Card>

  <Card title="DR Automation" href="/services/disaster-recovery/admin-guide/dr-automation">
    Set up automatic failover triggers and runbook scripts
  </Card>
</CardGroup>


# DR Compliance
Source: https://docs.xloud.tech/services/disaster-recovery/admin-guide/compliance

Generate audit-ready RPO/RTO compliance reports, DR test audit trails, and failover history from XDR for regulatory and internal governance requirements.

## Overview

XDR generates audit-ready reports documenting replication status, achieved RPO/RTO
values from DR tests, and complete failover history. These reports satisfy common
regulatory requirements (ISO 22301, SOC 2 Type II, HIPAA, PCI-DSS) that mandate
documented DR testing and measurable recovery objectives.

<Note>
  **Prerequisites**

  * Protection plans with at least one completed DR test
  * Administrator access to **Disaster Recovery → Reports**
  * Compliance reporting period defined (typically monthly, quarterly, or annual)
</Note>

***

## Report Types

| Report Type                  | Contents                                                 | Typical Use                                |
| ---------------------------- | -------------------------------------------------------- | ------------------------------------------ |
| **RPO/RTO Compliance**       | Achieved vs configured RPO and RTO per plan per period   | Regulatory submission, internal governance |
| **DR Test History**          | All DR test events — date, plan, measured RTO, pass/fail | Audit evidence of testing cadence          |
| **Failover Audit Log**       | Complete record of all failover and failback events      | Incident investigation, change management  |
| **Replication Status**       | Daily snapshot of replication lag and data currency      | Ongoing compliance monitoring              |
| **Recovery Point Inventory** | Available recovery points per plan at report time        | Data retention compliance                  |

***

## Generating Reports

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Disaster Recovery → Reports** and select the report type:

    <Steps>
      <Step title="Select report type" icon="file-text">
        Select from the report type list. Each report type has configurable parameters
        for the plan name, date range, and output format.
      </Step>

      <Step title="Set parameters" icon="settings">
        | Parameter            | Description                                        |
        | -------------------- | -------------------------------------------------- |
        | **Plan**             | Specific plan or "All Plans"                       |
        | **From / To**        | Reporting period start and end dates               |
        | **Format**           | PDF (for submission) or CSV (for further analysis) |
        | **Include raw data** | Append raw metric data as appendix (PDF only)      |
      </Step>

      <Step title="Generate and download" icon="download">
        Click **Generate Report**. The report is created and available for download
        within seconds for short periods, or minutes for long reporting periods.

        <Check>Report downloaded and shows plan name, period, achieved RPO/RTO values, and compliance status.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      generate and download compliance reports.
    </Info>
  </Tab>
</Tabs>

***

## DR Test Cadence Requirements

Most regulatory frameworks mandate regular DR testing. XDR tracks test history
automatically. Use XIMP to alert when testing cadence slips:

| Regulation        | Typical DR Testing Requirement                                 |
| ----------------- | -------------------------------------------------------------- |
| **ISO 22301**     | Annual minimum; test must cover critical systems               |
| **SOC 2 Type II** | At least annual; evidence of testing required for audit        |
| **HIPAA**         | No explicit cadence; must be "tested and revised" periodically |
| **PCI-DSS**       | Annual test of incident response / DR procedures               |

<Tip>
  Configure a XIMP alert for `xdr_last_test_age_days > 90` to ensure quarterly testing.
  This exceeds all common regulatory minimums and provides early warning to schedule
  tests before the annual window closes. See [Monitoring](/services/disaster-recovery/admin-guide/monitoring)
  for alert rule configuration.
</Tip>

***

## RPO/RTO Compliance Report Contents

The RPO/RTO Compliance report includes the following sections:

<AccordionGroup>
  <Accordion title="Executive summary" icon="file-text">
    One-page summary showing:

    * Protection plans covered
    * Reporting period
    * Overall compliance status (compliant / non-compliant)
    * Number of DR tests performed
    * Number of actual failover events
  </Accordion>

  <Accordion title="Per-plan RPO compliance" icon="clock">
    For each plan:

    * Configured RPO target
    * Average, p95, and maximum replication lag during the period
    * Percentage of time replication lag was within RPO target
    * Number of RPO breach events and total breach duration
  </Accordion>

  <Accordion title="Per-plan RTO compliance" icon="zap">
    From DR test results:

    * Configured RTO target
    * Measured RTO for each DR test performed in the period
    * Pass/fail against configured target
    * Trend (improving, stable, degrading)
  </Accordion>

  <Accordion title="Appendix: raw event data" icon="database">
    Optional detailed data:

    * Replication lag time series
    * All recovery point timestamps
    * Script execution logs from DR tests
    * Failover event timelines
  </Accordion>
</AccordionGroup>

***

## Automated Report Delivery

Schedule compliance reports for automatic generation and delivery. Navigate to
**Disaster Recovery → Reports → Scheduled Reports** and click **Create Schedule**:

| Setting         | Description                                                |
| --------------- | ---------------------------------------------------------- |
| **Report Type** | RPO/RTO Compliance, DR Test History, or Failover Audit Log |
| **Plans**       | Specific plan or all plans                                 |
| **Frequency**   | Monthly, quarterly, or annual                              |
| **Format**      | PDF (for submission) or CSV (for analysis)                 |
| **Recipients**  | Email addresses for automatic delivery                     |

Manage existing schedules from the **Scheduled Reports** list.

***

## Evidence Preservation

Compliance audits may require evidence that DR testing and monitoring occurred
throughout the audit period. XDR retains:

| Data Type               | Retention Period                   | Location                   |
| ----------------------- | ---------------------------------- | -------------------------- |
| DR test reports         | Indefinite (until manual deletion) | XDR controller database    |
| Replication lag metrics | 90 days raw; 1 year downsampled    | XIMP metric store          |
| Failover event logs     | Indefinite                         | XDR controller audit log   |
| Runbook script output   | 90 days                            | XDR controller log archive |

<Warning>
  If your regulatory framework requires longer retention, export reports to an
  external system immediately after generation. XDR does not enforce retention
  beyond the defaults listed above.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Monitoring" href="/services/disaster-recovery/admin-guide/monitoring">
    Configure alerts to maintain compliance proactively
  </Card>

  <Card title="XDR User Guide — DR Testing" href="/services/disaster-recovery/user-guide/test-dr">
    Run DR tests to generate the evidence captured in compliance reports
  </Card>

  <Card title="Security" href="/services/disaster-recovery/admin-guide/security">
    Access control and audit trail for XDR administrative operations
  </Card>

  <Card title="Troubleshooting" href="/services/disaster-recovery/admin-guide/troubleshooting">
    Diagnose replication lag that threatens RPO compliance
  </Card>
</CardGroup>


# DR Automation
Source: https://docs.xloud.tech/services/disaster-recovery/admin-guide/dr-automation

Configure automatic failover triggers, recovery runbook scripts, and replication scheduling to reduce manual intervention during XDR failover events.

## Overview

DR automation reduces manual intervention during failover by orchestrating the
recovery sequence, updating external systems, and validating service health
automatically. Well-tested automation is the difference between a 30-minute RTO
and a 3-hour one.

<Note>
  **Prerequisites**

  * Recovery plans created and in `ACTIVE` replication status
  * Recovery runbook scripts tested against DR test instances before production use
  * Automation hook scripts stored in a version-controlled location accessible from DR site
</Note>

***

## Automatic Failover Triggers

Configure health checks that trigger failover automatically when the primary site
is unreachable, without requiring manual operator intervention.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Disaster Recovery → Recovery Plans → \[Plan] → Automatic Triggers**:

    | Setting                  | Description                                                 |
    | ------------------------ | ----------------------------------------------------------- |
    | **Health Check URL**     | Primary site endpoint to poll (e.g., load balancer VIP)     |
    | **Check Interval**       | How often to poll (default: 30 seconds)                     |
    | **Failure Threshold**    | Consecutive failures before triggering (default: 3)         |
    | **Secondary Check**      | Optional second endpoint — both must fail before triggering |
    | **Notification Channel** | XIMP alert channel to notify when automatic failover begins |

    <Warning>
      Automatic failover bypasses the manual confirmation step. Enable only for
      workloads where downtime cost exceeds the risk of an unnecessary failover.
      Configure the failure threshold high enough to avoid triggering on transient
      network blips.
    </Warning>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      configure automatic failover triggers.
    </Info>
  </Tab>
</Tabs>

***

## Recovery Runbook Scripts

Runbook scripts add application-level coordination to the automated recovery
sequence. Scripts run before or after each resource group recovers.

### Script Environment

Scripts receive the following environment variables from the XDR runtime:

| Variable             | Value                                          |
| -------------------- | ---------------------------------------------- |
| `XDR_PLAN_NAME`      | Name of the recovery plan                      |
| `XDR_SITE`           | Name of the recovery site (DR site)            |
| `XDR_GROUP_NAME`     | Name of the current resource group             |
| `XDR_RESOURCE_IDS`   | Space-separated list of recovered resource IDs |
| `XDR_EVENT_ID`       | Unique identifier for this failover event      |
| `XDR_RECOVERY_POINT` | Timestamp of the recovery point used           |

### Script Requirements

* Must exit 0 for success — any non-zero exit halts recovery and triggers an alert
* Timeout: 300 seconds (configurable per hook)
* Must be idempotent — scripts may be retried after transient failures
* Output is captured in the runbook log — keep output concise and meaningful

### Example Scripts

<AccordionGroup>
  <Accordion title="DNS update after recovery" icon="globe">
    ```bash title="post-recover: update internal DNS" theme={null}
    #!/bin/bash
    # Updates internal DNS records to point to recovered DR instances
    # Runs as: Post-Recover hook on databases group
    # XDR injects environment variables at runtime

    set -euo pipefail

    # XDR_RESOURCE_IDS and XDR_SITE are provided by the XDR runtime
    # Resource IPs are resolved from the recovery metadata
    for id in $XDR_RESOURCE_IDS; do
      # Parse recovery metadata for IP and hostname
      ip=$(echo "$XDR_RECOVERY_METADATA" | jq -r ".resources[\"$id\"].ip")
      hostname=$(echo "$XDR_RECOVERY_METADATA" | jq -r ".resources[\"$id\"].hostname")

      nsupdate -k /etc/xavs/dns.key <<EOF
    server 10.0.1.71
    update delete ${hostname}.internal A
    update add ${hostname}.internal 60 A ${ip}
    send
    EOF

      echo "Updated DNS: ${hostname}.internal -> ${ip}"
    done

    echo "DNS update complete for group: $XDR_GROUP_NAME"
    ```
  </Accordion>

  <Accordion title="Service registry update" icon="list">
    ```bash title="post-recover: update service registry" theme={null}
    #!/bin/bash
    # Registers recovered instances in the internal service registry
    # Runs as: Post-Recover hook on app-servers group
    # XDR injects environment variables at runtime

    set -euo pipefail

    REGISTRY_URL="http://service-registry.internal:8500"

    # XDR_RESOURCE_IDS and XDR_RECOVERY_METADATA are provided by the XDR runtime
    for id in $XDR_RESOURCE_IDS; do
      ip=$(echo "$XDR_RECOVERY_METADATA" | jq -r ".resources[\"$id\"].ip")
      service=$(echo "$XDR_RECOVERY_METADATA" | jq -r ".resources[\"$id\"].tags[\"service-name\"]")

      curl -sf -X PUT "${REGISTRY_URL}/v1/agent/service/register" \
        -H "Content-Type: application/json" \
        -d "{\"ID\": \"${id}\", \"Name\": \"${service}\", \"Address\": \"${ip}\"}"

      echo "Registered: $service at $ip"
    done
    ```
  </Accordion>

  <Accordion title="Pre-failover notification" icon="send">
    ```bash title="pre-failover: notify on-call team" theme={null}
    #!/bin/bash
    # Sends alert to on-call channel before failover begins
    # Runs as: Pre-Failover hook on all groups

    WEBHOOK_URL="https://hooks.<your-domain>/alert"

    curl -sf -X POST "$WEBHOOK_URL" \
      -H "Content-Type: application/json" \
      -d "{
        \"event\": \"DR_FAILOVER_STARTING\",
        \"plan\": \"$XDR_PLAN_NAME\",
        \"site\": \"$XDR_SITE\",
        \"event_id\": \"$XDR_EVENT_ID\",
        \"recovery_point\": \"$XDR_RECOVERY_POINT\"
      }"

    echo "On-call notified for failover event $XDR_EVENT_ID"
    ```
  </Accordion>
</AccordionGroup>

### Attaching Scripts to Resource Groups

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Disaster Recovery → Recovery Plans → \[Plan] → \[Group] → Hooks**
    and click **Add Hook**. Paste the script content or reference a stored script path.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      attach scripts to resource groups.
    </Info>
  </Tab>
</Tabs>

***

## Replication Scheduling

For asynchronous replication, configure replication windows to control when
replication traffic runs and how much bandwidth it consumes.

Navigate to **Disaster Recovery → Recovery Plans → \[Plan] → Replication Schedule**:

| Setting           | Description                                                                       |
| ----------------- | --------------------------------------------------------------------------------- |
| **Mode**          | `Continuous` — replicate in real time; `Scheduled` — replicate in defined windows |
| **Schedule**      | Cron expression for scheduled replication windows                                 |
| **Bandwidth Cap** | Limit throughput during peak production hours                                     |

Configure these settings directly in the replication schedule panel for each recovery plan.

<Warning>
  Scheduled replication increases RPO to the interval between replication windows.
  Use continuous replication for any workload with an RPO shorter than the
  replication interval.
</Warning>

***

## Testing Automation

Always validate runbook scripts with a DR test before relying on them in a real failover.
Navigate to **Disaster Recovery → Protection Plans → \[Plan]** and click **Test Failover**
to exercise runbook scripts in an isolated environment. After the test completes, review
the runbook execution log in **Disaster Recovery → Test Sessions → \[Session] → Runbook Log**.

<Tip>
  After any change to a runbook script, run a DR test immediately to confirm the
  updated script completes successfully. A script that fails silently during a test
  will halt recovery during an actual disaster.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Recovery Plans" href="/services/disaster-recovery/admin-guide/recovery-plans">
    Configure resource groups and health checks
  </Card>

  <Card title="Monitoring" href="/services/disaster-recovery/admin-guide/monitoring">
    Alert on automation failures and replication lag
  </Card>

  <Card title="XDR User Guide — DR Testing" href="/services/disaster-recovery/user-guide/test-dr">
    Run DR tests to validate runbook scripts and measure RTO
  </Card>

  <Card title="Troubleshooting" href="/services/disaster-recovery/admin-guide/troubleshooting">
    Diagnose runbook script failures and unexpected failovers
  </Card>
</CardGroup>


# XDR Monitoring
Source: https://docs.xloud.tech/services/disaster-recovery/admin-guide/monitoring

Integrate XDR with XIMP to monitor replication health, RPO adherence, site availability, and DR readiness across all protection plans.

## Overview

Continuous monitoring of XDR replication is critical — a silent replication failure
discovered only during an actual disaster event can mean data loss far beyond the
configured RPO. Integrate XDR with XIMP to surface replication lag, site health,
and DR readiness metrics before they become incidents.

<Note>
  **Prerequisites**

  * XIMP deployed and agents active on both primary and DR sites
  * XDR controller API accessible from the XIMP collector
  * Protection plans in `ACTIVE` status
</Note>

***

## Key Metrics

| Metric                             | Description                         | Alert Threshold           |
| ---------------------------------- | ----------------------------------- | ------------------------- |
| `xdr_replication_lag_seconds`      | Current replication lag per plan    | > RPO target              |
| `xdr_site_health`                  | Site availability status            | `UNREACHABLE`             |
| `xdr_plan_status`                  | Plan replication state              | Not `ACTIVE`              |
| `xdr_last_test_age_days`           | Days since last DR test for a plan  | > 90 days                 |
| `xdr_recovery_point_count`         | Number of available recovery points | \< minimum configured     |
| `xdr_replication_throughput_bytes` | Replication throughput per link     | N/A (trending)            |
| `xdr_sync_progress_percent`        | Initial sync progress (0–100)       | \< 100 after 48h          |
| `xdr_link_latency_ms`              | Round-trip latency between sites    | > site-specific threshold |

***

## XIMP Dashboard

XDR includes a pre-built XIMP dashboard showing all protection plans and their
current replication status. Navigate to **Monitoring → Dashboards → XDR Overview**.

The dashboard provides:

* Per-plan replication lag (sparkline, 24h history)
* Site health indicator for all registered sites
* RPO compliance percentage per plan (7d rolling)
* Last DR test date and outcome per plan
* Active failover events (if any)

***

## Alert Rules

Configure XIMP alert rules to notify operations teams before replication problems
impact RPO compliance.

<AccordionGroup>
  <Accordion title="Replication lag approaching RPO" icon="clock">
    Alert before lag actually exceeds the RPO target — early warning allows
    investigation before data loss risk is realized.

    Navigate to **Monitoring → Alerting → Alert Rules → Create Rule**:

    ```
    Name: XDR Replication Lag Warning
    Condition: xdr_replication_lag_seconds > (xdr_rpo_target_seconds * 0.75)
    For: 5 minutes
    Severity: Warning
    Channel: ops-alerts
    ```

    ```
    Name: XDR Replication Lag Critical
    Condition: xdr_replication_lag_seconds > xdr_rpo_target_seconds
    For: 2 minutes
    Severity: Critical
    Channel: ops-pagerduty
    ```
  </Accordion>

  <Accordion title="Plan not active" icon="shield">
    Alert immediately if a protection plan transitions out of `ACTIVE` status —
    this means replication has stopped and the DR site data is not being updated.

    ```
    Name: XDR Plan Not Active
    Condition: xdr_plan_status != "ACTIVE"
    For: 1 minute
    Severity: Critical
    Channel: ops-pagerduty
    ```
  </Accordion>

  <Accordion title="Site unreachable" icon="server">
    Alert when a site becomes unreachable — could indicate the primary site has
    failed or network connectivity between sites has been lost.

    ```
    Name: XDR Site Unreachable
    Condition: xdr_site_health == "UNREACHABLE"
    For: 2 minutes
    Severity: Critical
    Channel: ops-pagerduty
    ```
  </Accordion>

  <Accordion title="DR test overdue" icon="flask-conical">
    Alert when a protection plan has not been tested within the configured interval.
    DR plans that are never tested cannot be relied upon during an actual disaster.

    ```
    Name: XDR Test Overdue
    Condition: xdr_last_test_age_days > 90
    For: immediate
    Severity: Warning
    Channel: ops-alerts
    ```

    Navigate to **Monitoring → Alerting → Alert Rules** and create a rule sourcing
    the `xdr_last_test_age_days` metric to ensure tests are not missed.
  </Accordion>
</AccordionGroup>

***

## Diagnostic Views

All diagnostic information is available through the XDR Dashboard:

| Diagnostic                       | Dashboard Location                                                                           |
| -------------------------------- | -------------------------------------------------------------------------------------------- |
| Replication lag across all plans | **Disaster Recovery → Protection Plans** — lag column in the plan list                       |
| Replication health history       | **Disaster Recovery → Protection Plans → \[Plan]** — replication lag sparkline (24h history) |
| Site health status               | **Disaster Recovery → Sites** — health indicator per site                                    |
| Link throughput statistics       | **Disaster Recovery → Sites → Replication Links → \[Link]** — throughput and latency metrics |

***

## Replication Health Thresholds

Use these thresholds when configuring XIMP alert rules:

| Metric                  | Healthy              | Warning               | Critical             |
| ----------------------- | -------------------- | --------------------- | -------------------- |
| Replication lag         | \< 50% of RPO target | 50–100% of RPO target | > RPO target         |
| Plan status             | `ACTIVE`             | `DEGRADED`            | `FAILED` / `STOPPED` |
| Site health             | `CONNECTED`          | `DEGRADED`            | `UNREACHABLE`        |
| Last recovery point age | \< RPO target        | RPO target to 2× RPO  | > 2× RPO target      |
| Sync progress (initial) | Increasing           | Stalled > 30 min      | No progress > 2h     |

***

## Log Collection

XDR agent and controller logs are forwarded to XIMP log analytics automatically
when agents are deployed via XDeploy. Query logs in **Monitoring → Log Explorer**:

| Log Source          | Query Pattern                                      |
| ------------------- | -------------------------------------------------- |
| XDR controller      | `service: xdr-controller`                          |
| XDR agent (primary) | `service: xdr-agent AND site: primary-dc1`         |
| XDR agent (DR)      | `service: xdr-agent AND site: dr-site-a`           |
| Failover events     | `service: xdr-controller AND event_type: failover` |
| Runbook scripts     | `service: xdr-runbook AND plan: prod-database-dr`  |

***

## Next Steps

<CardGroup>
  <Card title="XIMP Admin Guide — Alert Channels" href="/services/monitoring/admin-guide/alert-channels">
    Configure notification channels for XDR alerts
  </Card>

  <Card title="Compliance" href="/services/disaster-recovery/admin-guide/compliance">
    Generate RPO/RTO compliance reports from monitoring history
  </Card>

  <Card title="DR Automation" href="/services/disaster-recovery/admin-guide/dr-automation">
    Configure automatic failover on site health alerts
  </Card>

  <Card title="Troubleshooting" href="/services/disaster-recovery/admin-guide/troubleshooting">
    Diagnose replication lag and plan health issues
  </Card>
</CardGroup>


# Recovery Plans
Source: https://docs.xloud.tech/services/disaster-recovery/admin-guide/recovery-plans

Define ordered resource groups, health check criteria, and automation hooks that govern how XDR recovers workloads during a failover event.

## Overview

Recovery plans define the complete failover procedure — which resources are protected,
in what order they recover, what health checks confirm readiness, and what automation
scripts run at each stage. A well-designed recovery plan is the foundation of a
reliable DR strategy with predictable RTO.

<Note>
  **Prerequisites**

  * Sites registered and replication link verified (see [Replication Configuration](/services/disaster-recovery/admin-guide/replication-config))
  * Administrator credentials on both sites
  * Instances and volumes to protect must exist in the project
</Note>

***

## Creating a Recovery Plan

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open Recovery Plans" icon="shield">
        Navigate to **Disaster Recovery → Recovery Plans → Create Plan**.
      </Step>

      <Step title="Define plan parameters" icon="settings">
        | Field                | Description                                                                |
        | -------------------- | -------------------------------------------------------------------------- |
        | **Plan Name**        | Descriptive label identifying the workload tier (e.g., `prod-database-dr`) |
        | **Primary Site**     | Source site for replication                                                |
        | **DR Site**          | Target site for recovery                                                   |
        | **RPO Target**       | Maximum acceptable data loss (e.g., `5 minutes`)                           |
        | **RTO Target**       | Maximum acceptable recovery time (e.g., `30 minutes`)                      |
        | **Failover Trigger** | `Manual` or `Automatic`                                                    |
        | **Consistency Mode** | `Crash-consistent` or `Application-consistent`                             |
        | **Replication Mode** | `Asynchronous` or `Synchronous`                                            |
      </Step>

      <Step title="Add resource groups" icon="layers">
        Organize protected resources into ordered recovery groups. Resources within
        a group recover in parallel; groups recover sequentially.

        | Group       | Resources                  | Recovery Order                    |
        | ----------- | -------------------------- | --------------------------------- |
        | **Group 1** | Database instances         | 1 — first to recover              |
        | **Group 2** | Application servers        | 2 — start after databases healthy |
        | **Group 3** | Load balancers / frontends | 3 — start after app tier healthy  |

        <Tip>
          Model recovery groups on the actual application dependency chain.
          Starting an application server before its database is ready causes
          service errors and may require manual intervention during a real failover.
        </Tip>
      </Step>

      <Step title="Configure automation hooks" icon="cpu">
        Add pre/post scripts to each resource group:

        | Hook Type         | Trigger                        | Example Use                               |
        | ----------------- | ------------------------------ | ----------------------------------------- |
        | **Pre-Failover**  | Before group starts recovering | Notify on-call; update DNS TTL            |
        | **Post-Recover**  | After group is running         | Run health check; update service registry |
        | **Pre-Failback**  | Before reversing replication   | Drain connections from DR instances       |
        | **Post-Failback** | After primary site is restored | Re-enable scheduled jobs                  |
      </Step>

      <Step title="Set health check criteria" icon="heart-pulse">
        Define what "recovered" means for each resource group:

        * **HTTP health check** — URL and expected response code
        * **TCP port check** — host and port number
        * **Script** — custom validation command (exit 0 = healthy)

        <Note>
          A recovery group advances to the next group only when all health checks
          in the current group pass. This prevents cascading failures where
          dependent services start before their dependencies are ready.
        </Note>
      </Step>

      <Step title="Activate the plan" icon="circle-check">
        Click **Activate**. XDR begins replicating all protected resources to
        the DR site. Initial sync time depends on data volume.

        <Check>Plan status shows `ACTIVE` and initial replication sync progress is visible in the replication dashboard.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      create and configure recovery plans.
    </Info>
  </Tab>
</Tabs>

***

## Managing Existing Plans

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Disaster Recovery → Recovery Plans** to see all plans with
    their current status and replication lag.

    Available actions per plan:

    * **Edit** — update RPO/RTO targets, add/remove resources, modify health checks
    * **Deactivate** — pause replication without deleting the plan
    * **Delete** — permanently remove the plan (stops replication)
    * **Failover** — initiate failover (see [Failover](/services/disaster-recovery/user-guide/failover))
    * **Test Failover** — run an isolated DR test without cutting over production traffic
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      manage existing recovery plans.
    </Info>
  </Tab>
</Tabs>

***

## Consistency Modes

| Mode                       | How It Works                                                                                                | RPO Accuracy                                                  | Overhead                                          |
| -------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------- |
| **Crash-consistent**       | Replicates data as written — like a power failure at the recovery point                                     | May require fsck on recovery; databases may need recovery     | Minimal                                           |
| **Application-consistent** | Coordinates with the XAVS Guest Agent to quiesce writes before snapshot (includes VSS provider for Windows) | Application-clean recovery point; no database recovery needed | XAVS Guest Agent round-trip per snapshot interval |

<Tip>
  Use application-consistent mode for databases and transactional workloads.
  Crash-consistent mode is suitable for stateless compute instances where
  data integrity depends on the application rather than the storage layer.
</Tip>

***

## Recovery Point Retention

XDR retains a configurable number of recovery points, allowing historical
restore targets during failover:

Configure retention settings from **Disaster Recovery → Recovery Plans → \[Plan] → Retention**:

| Retention Setting | Behavior                                                      |
| ----------------- | ------------------------------------------------------------- |
| **Count**         | Number of recovery points to retain (older points are pruned) |
| **Interval**      | Minimum time between recovery points                          |
| **Maximum age**   | Absolute oldest recovery point to retain                      |

<Warning>
  Increasing recovery point retention consumes additional storage on the DR site.
  Each recovery point is an incremental snapshot — for high-change workloads,
  deep retention can accumulate significant storage overhead.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="DR Automation" href="/services/disaster-recovery/admin-guide/dr-automation">
    Configure runbook scripts and automatic failover triggers
  </Card>

  <Card title="Monitoring" href="/services/disaster-recovery/admin-guide/monitoring">
    Monitor plan replication health and RPO adherence
  </Card>

  <Card title="Compliance" href="/services/disaster-recovery/admin-guide/compliance">
    Generate RPO/RTO compliance reports from plan history
  </Card>

  <Card title="XDR User Guide — Protection Plans" href="/services/disaster-recovery/user-guide/protection-plans">
    User-facing protection plan management
  </Card>
</CardGroup>


# Replication Configuration
Source: https://docs.xloud.tech/services/disaster-recovery/admin-guide/replication-config

Register primary and DR sites with the XDR controller, configure replication links, bandwidth policies, and verify cross-site connectivity.

## Overview

Before creating protection plans, register both the primary and DR sites with the
XDR controller and configure the replication link between them. This establishes
the trust relationship and network path that all replication traffic flows through.

<Note>
  **Prerequisites**

  * XDR controller deployed and accessible from both sites
  * Network connectivity open between primary and DR sites on TCP 7000–7002
  * Administrator credentials on both sites
  * XDR agent deployed on both sites via XDeploy
</Note>

***

## Site Registration

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Register the primary site" icon="server">
        Log in to **XDeploy** (`https://connect.<your-domain>`) and navigate to
        **Disaster Recovery → Sites → Register Site**:

        | Field            | Description                                                      |
        | ---------------- | ---------------------------------------------------------------- |
        | **Site Name**    | Unique identifier (e.g., `primary-dc1`)                          |
        | **Role**         | `Primary`                                                        |
        | **API Endpoint** | XDR agent API URL for this site (e.g., `https://10.10.0.1:7002`) |
        | **Auth Token**   | Site authentication token generated during XDR agent deployment  |
        | **Network CIDR** | IP range for this site's compute and storage network             |
        | **Description**  | Optional free-text label (e.g., datacenter name, location)       |
      </Step>

      <Step title="Register the DR site" icon="shield">
        Repeat the registration process for the DR site, selecting role `DR`.
        Provide the DR site's XDR agent endpoint and its authentication token.

        <Check>Both sites appear in the Sites list with status `REGISTERED`.</Check>
      </Step>

      <Step title="Create replication link" icon="link">
        Navigate to **Disaster Recovery → Sites → Replication Links → Create Link**
        and select the primary site as source and DR site as destination.
      </Step>

      <Step title="Configure link settings" icon="settings">
        | Setting             | Recommendation                                                           |
        | ------------------- | ------------------------------------------------------------------------ |
        | **Compression**     | Enable for WAN links — reduces bandwidth 30–60% for typical storage data |
        | **Encryption**      | Always enable — replication traffic crosses network boundaries           |
        | **Bandwidth Limit** | Set to 80% of available link capacity to avoid saturation                |
        | **MTU**             | Match the replication network MTU to avoid fragmentation                 |
        | **QoS Priority**    | Set to high if sharing the link with other traffic types                 |
      </Step>

      <Step title="Verify connectivity" icon="circle-check">
        Click **Test Connectivity** to verify the link is functional in both directions.

        <Check>Connectivity test returns `CONNECTED` with round-trip latency displayed.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      register sites and configure replication links.
    </Info>
  </Tab>
</Tabs>

***

## Bandwidth Management

Replication bandwidth directly affects how quickly the initial sync completes and
how tightly the replication lag tracks the configured RPO. Configure bandwidth
policies to balance replication performance against production workload impact.

<AccordionGroup>
  <Accordion title="Bandwidth limit policies" icon="gauge">
    XDR supports per-link and per-plan bandwidth limits. Per-link limits cap total
    replication throughput on the network connection; per-plan limits allocate
    bandwidth among multiple plans sharing the same link.

    Navigate to **Disaster Recovery → Sites → Replication Links → \[Link] → Bandwidth**:

    | Policy                  | Description                                                   |
    | ----------------------- | ------------------------------------------------------------- |
    | **Hard cap**            | Never exceed this throughput regardless of available capacity |
    | **Peak hours throttle** | Reduce throughput during business hours (cron schedule)       |
    | **Burst allowance**     | Allow brief bursts above the cap to clear backlog             |

    Configure these policies directly in the bandwidth settings panel for each replication link.
  </Accordion>

  <Accordion title="Initial sync sizing" icon="hard-drive">
    The initial sync transfers all protected data to the DR site. Estimate
    completion time before enabling a plan:

    | Data Volume | 100 Mbps Link | 1 Gbps Link |
    | ----------- | ------------- | ----------- |
    | 1 TB        | \~22 hours    | \~2.2 hours |
    | 5 TB        | \~4.5 days    | \~11 hours  |
    | 10 TB       | \~9 days      | \~22 hours  |

    <Tip>
      Schedule initial sync during off-peak hours or temporarily raise the
      bandwidth cap to accelerate it. Once initial sync completes, only
      incremental changes are replicated — bandwidth consumption drops
      significantly.
    </Tip>
  </Accordion>

  <Accordion title="WAN link health" icon="activity">
    Monitor link statistics to detect degradation before it impacts RPO. Navigate to
    **Disaster Recovery → Sites → Replication Links → \[Link]** to view throughput
    and error statistics over time.

    Key indicators of a degraded link:

    * Throughput consistently below configured limit without backlog
    * Retransmit rate above 1% (network packet loss)
    * Round-trip latency increasing over time (congestion)
  </Accordion>
</AccordionGroup>

***

## Replication Modes

| Mode             | RPO                | Overhead                                                            | Use Case                                                                              |
| ---------------- | ------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| **Asynchronous** | Seconds to minutes | Low — primary writes complete without waiting for DR acknowledgment | Sites separated by >10ms RTT; most workloads                                          |
| **Synchronous**  | Zero (RPO = 0)     | High — primary write latency increases by replication RTT           | Databases and financial systems where zero data loss is required; sites under 5ms RTT |

<Warning>
  Synchronous replication adds write latency equal to the round-trip time between
  sites on every write operation. For sites separated by more than 5ms RTT,
  synchronous replication will noticeably degrade application performance.
  Measure your inter-site latency before enabling synchronous mode.
</Warning>

***

## Site Token Management

XDR agents authenticate between sites using site-specific tokens, not user credentials.

Manage site tokens from **Disaster Recovery → Sites → \[Site] → Token Management**:

* **View token status**: The token expiry date and status are displayed for each registered site
* **Rotate token**: Click **Rotate Token** to generate a new authentication token for the selected site
* **Update peer**: After rotating a token, update the peer site with the new token in the peer's site configuration panel

<Note>
  Rotate site tokens at least annually or immediately if a token is suspected
  compromised. Token rotation does not interrupt active replication — the old
  token remains valid for 15 minutes after rotation to allow the update to propagate.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Recovery Plans" href="/services/disaster-recovery/admin-guide/recovery-plans">
    Create ordered recovery groups and automation hooks
  </Card>

  <Card title="DR Automation" href="/services/disaster-recovery/admin-guide/dr-automation">
    Configure automatic failover triggers and runbook scripts
  </Card>

  <Card title="Monitoring" href="/services/disaster-recovery/admin-guide/monitoring">
    Alert on replication lag and link throughput degradation
  </Card>

  <Card title="Troubleshooting" href="/services/disaster-recovery/admin-guide/troubleshooting">
    Diagnose initial sync failures and connectivity issues
  </Card>
</CardGroup>


# XDR Security
Source: https://docs.xloud.tech/services/disaster-recovery/admin-guide/security

Secure XDR deployments with replication encryption, RBAC access control, site token management, and audit logging.

## Overview

XDR security spans three domains: encryption of data in transit across the replication
link, access control over who can initiate potentially destructive failover operations,
and credential isolation between sites using token-based authentication.

<Note>
  **Prerequisites**

  * Administrator credentials with the `dr-admin` role
  * TLS certificates provisioned for site-to-site communication
  * Identity and access management configured in XDeploy
</Note>

***

## Replication Encryption

All replication traffic between primary and DR sites is encrypted in transit.

<AccordionGroup>
  <Accordion title="TLS configuration" icon="lock">
    Configure encryption in **Disaster Recovery → Sites → Replication Links → \[Link] → Security**:

    | Setting                | Recommended Value                                          |
    | ---------------------- | ---------------------------------------------------------- |
    | **Protocol**           | TLS 1.3 minimum (TLS 1.2 allowed for legacy compatibility) |
    | **Certificate source** | Managed by XDeploy (auto-renewed 30 days before expiry)    |
    | **Authentication**     | Mutual TLS — both sites authenticate each other            |
    | **Cipher suites**      | XDeploy defaults (AES-256-GCM, ChaCha20-Poly1305)          |

    Verify the current TLS configuration from **Disaster Recovery → Sites → Replication Links → \[Link] → Security**.
  </Accordion>

  <Accordion title="Certificate management" icon="shield-check">
    XDeploy manages site certificates automatically. Certificates are:

    * Issued per-site at registration time
    * Automatically renewed 30 days before expiry
    * Rotated without interrupting active replication

    To manually trigger certificate renewal, navigate to **Disaster Recovery → Sites → \[Site] → Certificates** and click **Renew Certificate**.

    Monitor certificate expiry in XIMP by creating an alert rule:

    | Setting       | Value                          |
    | ------------- | ------------------------------ |
    | **Condition** | `xdr_cert_days_to_expiry < 30` |
    | **Severity**  | Warning                        |
    | **Channel**   | ops-alerts                     |
  </Accordion>

  <Accordion title="Encryption at rest" icon="hard-drive">
    Replicated data at rest on the DR site is encrypted using the same storage
    encryption policy as the primary site. Configure encryption at rest in XSDS
    at the pool or volume level — XDR inherits the encryption status of the
    source volumes.

    See [XSDS Admin — Security](/services/sds/admin-guide/security) for storage
    encryption configuration.
  </Accordion>
</AccordionGroup>

***

## RBAC Access Control

XDR operations are governed by Xloud identity roles. Failover and failback are
potentially disruptive operations — restrict them to trained personnel.

| Role          | Permissions                                                                                           |
| ------------- | ----------------------------------------------------------------------------------------------------- |
| `dr-viewer`   | View plan status, replication lag, test reports, and site health                                      |
| `dr-operator` | Create and manage protection plans; run DR tests; initiate failover and failback                      |
| `dr-admin`    | Full access including site registration, replication link configuration, and compliance report export |

### Assigning DR Roles

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Identity → Projects → \[Project] → Members** and assign the
    appropriate DR role to each user. DR roles apply at the project level —
    a user must have a DR role in both the primary and DR site projects to
    operate across sites.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Assign dr-operator role to a user" theme={null}
    xloud role add \
      --user operator@example.com \
      --role dr-operator \
      --project prod-project
    ```

    ```bash title="List users with DR roles" theme={null}
    xloud role list \
      --role dr-operator \
      --project prod-project
    ```
  </Tab>
</Tabs>

<Warning>
  Grant `dr-operator` access only to personnel trained in XDR failover procedures.
  An untrained operator initiating an unnecessary failover can cause extended
  service disruption and require a full failback cycle to restore normal operations.
</Warning>

***

## Site Token Management

XDR agents authenticate between sites using site-specific tokens, not user credentials.
This isolates site-to-site authentication from user identity management.

### Token Lifecycle

| Event                | Action Required                                               |
| -------------------- | ------------------------------------------------------------- |
| Initial deployment   | Token generated by XDeploy during site registration           |
| Scheduled rotation   | Rotate via the Dashboard on schedule (recommended: quarterly) |
| Suspected compromise | Rotate immediately; review access logs                        |
| Token expiry         | 90-day expiry by default; configurable at registration        |

Manage tokens from **Disaster Recovery → Sites → \[Site] → Token Management**:

* **View token status**: Displays token expiry date and current validity for all registered sites
* **Rotate token**: Click **Rotate Token** to generate a new authentication token
* **Update peer**: After rotation, navigate to the peer site configuration and enter the new token

<Note>
  Token rotation does not interrupt active replication. The old token remains valid
  for 15 minutes after rotation to allow the update to propagate before the old
  token is invalidated.
</Note>

***

## Audit Logging

XDR records all administrative actions and failover events in an immutable audit log.

| Event Category                     | Logged Information                                                        |
| ---------------------------------- | ------------------------------------------------------------------------- |
| Site registration / deregistration | Admin user, timestamp, site details                                       |
| Plan creation / modification       | Admin user, timestamp, changed fields                                     |
| Failover initiated                 | Operator user, timestamp, trigger type (manual/automatic), recovery point |
| Failback initiated                 | Operator user, timestamp                                                  |
| DR test started / completed        | Operator user, timestamp, outcome                                         |
| Token rotation                     | Admin user, timestamp, affected site                                      |

Navigate to **Disaster Recovery → Reports → Audit Log** to view and export the audit trail:

* Filter by event type (failover, failback, test, site registration, token rotation)
* Set date range for the reporting period
* Export as PDF or CSV for compliance submissions

***

## Network Security

| Requirement                | Implementation                                                                   |
| -------------------------- | -------------------------------------------------------------------------------- |
| Dedicated replication VLAN | Separate replication traffic from production networks                            |
| Firewall rules             | Open only required ports (TCP 7000–7002) between site CIDRs                      |
| No internet exposure       | Replication endpoints should not be reachable from the public internet           |
| VPN / MPLS                 | Use a dedicated circuit rather than the public internet for the replication link |

Run a connectivity audit from **Disaster Recovery → Sites → \[Site] → Test Connectivity**
to verify that only the required replication ports are accessible between sites.

***

## Next Steps

<CardGroup>
  <Card title="Replication Configuration" href="/services/disaster-recovery/admin-guide/replication-config">
    Configure replication link encryption and bandwidth settings
  </Card>

  <Card title="Compliance" href="/services/disaster-recovery/admin-guide/compliance">
    Export audit logs for compliance reporting
  </Card>

  <Card title="XSDS Admin — Security" href="/services/sds/admin-guide/security">
    Configure storage encryption that XDR inherits
  </Card>

  <Card title="Identity & Access Admin Guide" href="/services/identity/admin-guide">
    Manage Xloud RBAC roles and identity federation
  </Card>
</CardGroup>


# XDR Admin Troubleshooting
Source: https://docs.xloud.tech/services/disaster-recovery/admin-guide/troubleshooting

Diagnose and resolve XDR administrator-level issues — initial sync failures, unexpected automatic failovers, runbook script errors, and site connectivity problems.

## Overview

This page covers administrator-level XDR diagnostics — issues requiring access to
site configuration, replication link settings, or runbook script management. For
user-facing issues such as replication lag and DR test access problems, see
[XDR User Guide — Troubleshooting](/services/disaster-recovery/user-guide/troubleshooting).

<Note>
  **Prerequisites**

  * Administrator credentials with the `dr-admin` role
  * Access to both primary and DR site XDeploy instances
  * For replication network issues, coordinate with the network administrator
</Note>

***

## Common Issues

<AccordionGroup>
  <Accordion title="Initial replication sync not completing" icon="clock">
    **Cause**: Insufficient network bandwidth for the data volume being synchronized,
    a firewall rule blocking replication traffic, or insufficient storage quota on the DR site.

    **Diagnosis**: Navigate to **Disaster Recovery → Protection Plans → \[Plan]** and review
    the sync progress percentage. Check link throughput in **Disaster Recovery → Sites → Replication Links → \[Link]**.

    **Resolution**:

    * If throughput is near-zero, verify TCP 7000-7002 is open bidirectionally between sites
    * If throughput is low but non-zero, increase the bandwidth limit or wait
    * If sync stalls at a specific percentage, check DR site storage quota in **Disaster Recovery → Sites → \[DR Site] → Storage**
    * Verify the replication port is accessible by clicking **Test Connectivity** on the site entry
  </Accordion>

  <Accordion title="Automatic failover triggered unexpectedly" icon="zap">
    **Cause**: A transient network issue caused the health check failure threshold
    to be exceeded, triggering an unintended automatic failover.

    **Diagnosis**: Navigate to **Disaster Recovery → Recovery Plans → \[Plan] → Health Check Log**
    and review the timestamps and duration of health check failures over the last 2 hours.

    **Resolution**:

    1. Verify the primary site is actually available before initiating failback
    2. Review the health check log for the timestamps and duration of failures
    3. If the primary site is healthy, initiate failback from **Disaster Recovery → Protection Plans → \[Plan] → Failback**
    4. After failback, increase the failure threshold in **Recovery Plans → \[Plan] → Automatic Triggers** to reduce false positive risk. Consider adding a secondary health check endpoint.
  </Accordion>

  <Accordion title="Recovery runbook script failure" icon="terminal">
    **Cause**: A pre/post recovery script returned a non-zero exit code, halting
    recovery progression for the affected resource group.

    **Diagnosis**: Navigate to **Disaster Recovery → Failover Status → \[Plan] → Runbook Log**
    and review the script output for the failed hook. The log shows the exit code, stdout,
    and stderr for each executed script.

    **Common causes**:

    | Cause                                              | Resolution                                         |
    | -------------------------------------------------- | -------------------------------------------------- |
    | DNS update credentials expired                     | Rotate the DNS key and update the script           |
    | Service registry endpoint unreachable from DR site | Verify network routing from DR to service registry |
    | Script assumes local file paths not on DR site     | Make paths configurable via environment variables  |
    | Script timeout exceeded                            | Increase timeout or optimize the script            |

    After fixing the script, resume the stalled recovery from **Disaster Recovery → Failover Status → \[Plan] → \[Group] → Resume**.
  </Accordion>

  <Accordion title="Site connectivity test fails" icon="network">
    **Cause**: A firewall rule change, routing update, or agent restart caused
    connectivity between sites to be interrupted.

    **Diagnosis**: Navigate to **Disaster Recovery → Sites** and click **Test Connectivity**
    on the affected site entry. Review the site status indicators for all registered sites.

    **Resolution**:

    * Verify firewall rules allow TCP 7000-7002 bidirectionally between site CIDRs
    * Check that XDR agent processes are running on both sites by reviewing the agent
      status indicator in **Disaster Recovery → Sites → \[Site]**
    * If the agent is stopped, restart it via XDeploy on the affected site
    * Check agent logs for TLS certificate errors — certificates may have expired
  </Accordion>

  <Accordion title="Plan status shows DEGRADED" icon="circle-x">
    **Cause**: One or more resources in the plan have fallen behind their RPO target,
    or a resource has been removed from the project while still referenced by the plan.

    **Diagnosis**: Navigate to **Disaster Recovery → Protection Plans → \[Plan]** and
    review the per-resource status breakdown. Resources with issues are highlighted
    with a warning indicator.

    **Resolution**:

    * For lag issues: see [XDR User Guide — Troubleshooting](/services/disaster-recovery/user-guide/troubleshooting)
    * For deleted resources: remove the stale resource reference from the plan by
      selecting the resource in the plan editor and clicking **Remove Resource**
  </Accordion>

  <Accordion title="TLS certificate error on replication link" icon="lock">
    **Cause**: The site authentication certificate has expired, or the certificate
    was not renewed before the previous certificate expired.

    **Diagnosis**: Navigate to **Disaster Recovery → Sites → \[Site] → Certificates** and
    review the certificate expiry date and renewal status for all registered sites.

    **Resolution**:

    1. Click **Renew Certificate** on the affected site to trigger manual renewal
    2. If automatic renewal has been failing, click **Reissue Certificate** to force a
       new certificate to be generated from scratch
    3. After renewal, click **Test Connectivity** on the site entry to confirm the new
       certificate is accepted by the peer site
  </Accordion>
</AccordionGroup>

***

## Diagnostics Reference

All diagnostic operations are performed through the XDR Dashboard:

| Issue                  | Dashboard Location                                                               |
| ---------------------- | -------------------------------------------------------------------------------- |
| Sync not completing    | **Disaster Recovery → Protection Plans → \[Plan]** — sync progress panel         |
| Link throughput low    | **Disaster Recovery → Sites → Replication Links → \[Link]** — throughput metrics |
| Connectivity failure   | **Disaster Recovery → Sites → \[Site]** — Test Connectivity button               |
| Runbook script failure | **Disaster Recovery → Failover Status → \[Plan] → Runbook Log**                  |
| Plan status DEGRADED   | **Disaster Recovery → Protection Plans → \[Plan]** — per-resource status         |
| Cert errors            | **Disaster Recovery → Sites → \[Site] → Certificates**                           |
| Unexpected failover    | **Disaster Recovery → Recovery Plans → \[Plan] → Health Check Log**              |

***

## Log Locations

| Log Source               | Access Method                                                              |
| ------------------------ | -------------------------------------------------------------------------- |
| XDR controller logs      | **Monitoring → Log Explorer** → `service: xdr-controller`                  |
| XDR agent logs (primary) | **Monitoring → Log Explorer** → `service: xdr-agent AND site: primary-dc1` |
| XDR agent logs (DR)      | **Monitoring → Log Explorer** → `service: xdr-agent AND site: dr-site-a`   |
| Failover event timeline  | **Disaster Recovery → Failover Status → \[Plan] → Event Timeline**         |
| Runbook output           | **Disaster Recovery → Failover Status → \[Plan] → Runbook Log**            |

***

## When to Contact Support

Contact [support@xloud.tech](mailto:support@xloud.tech) if:

* Initial sync has made no progress for more than 4 hours despite connectivity being confirmed
* Sites show `CONNECTED` but replication lag continues to increase
* Certificate renewal fails repeatedly and replication has stopped
* A failover event log shows an internal XDR controller error (not a script or connectivity error)
* Failback cannot be initiated after an unexpected automatic failover

***

## Next Steps

<CardGroup>
  <Card title="XDR User Guide — Troubleshooting" href="/services/disaster-recovery/user-guide/troubleshooting">
    User-facing replication lag, stuck failover, and test access issues
  </Card>

  <Card title="Replication Configuration" href="/services/disaster-recovery/admin-guide/replication-config">
    Review and update replication link configuration
  </Card>

  <Card title="DR Automation" href="/services/disaster-recovery/admin-guide/dr-automation">
    Review and test runbook scripts before incidents occur
  </Card>

  <Card title="Monitoring" href="/services/disaster-recovery/admin-guide/monitoring">
    Set up proactive alerts to catch issues before they escalate
  </Card>
</CardGroup>


# Disaster Recovery
Source: https://docs.xloud.tech/services/disaster-recovery/index

Automated failover, replication, and recovery orchestration with XDR — minimize RPO/RTO and ensure business continuity.

Xloud Disaster Recovery (XDR) delivers automated failover, continuous replication, and
orchestrated recovery for workloads running on Xloud infrastructure.

With configurable recovery point and recovery time objectives, XDR helps you restore
critical systems rapidly after any failure — from hardware faults to site-level outages.
Storage costs run up to 70% lower compared to full-copy backup approaches.

<Card title="XDR — Disaster Recovery Solution" icon="external-link" href="https://xloud.tech/disaster-recovery">
  Solution overview, architecture diagrams, and datasheet on xloud.tech
</Card>

***

<p>XDR Documentation</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/disaster-recovery/user-guide">
    Create protection plans, initiate failover and failback, run DR tests, and manage
    recovery points. Step-by-step workflows for both Dashboard and CLI.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/disaster-recovery/admin-guide">
    Configure replication topology, define recovery plans, set automation policies,
    and manage cross-site connectivity for production DR deployments.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/disaster-recovery/cli-reference">
    Manage protection plans, trigger failover and failback, and query replication status
    from the command line.
  </Card>

  <Card title="Software-Defined Storage" icon="database" href="/services/sds/index">
    XSDS provides the distributed storage backend that powers XDR replication.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="RPO & RTO Management" icon="clock" href="/services/disaster-recovery/admin-guide/recovery-plans">
    Define recovery point objectives (how much data you can afford to lose) and recovery
    time objectives (how quickly you must be back online) per workload tier.
  </Card>

  <Card title="Continuous Replication" icon="refresh-cw" href="/services/disaster-recovery/admin-guide/replication-config">
    Asynchronous and synchronous replication modes keep secondary sites up to date,
    minimizing data loss on failover.
  </Card>

  <Card title="Automated Failover" icon="zap" href="/services/disaster-recovery/user-guide/failover">
    Policy-driven failover activation requires minimal operator intervention. Health
    checks continuously validate primary site availability.
  </Card>

  <Card title="DR Automation" icon="cpu" href="/services/disaster-recovery/admin-guide/dr-automation">
    Orchestrated recovery runbooks execute instance restarts, network reconfiguration,
    and service validation in the correct dependency order.
  </Card>

  <Card title="Cost-Efficient Storage" icon="trending-down" href="/services/sds/index">
    Erasure coding and deduplication reduce DR storage footprint by up to 70% compared
    to traditional full-copy backup solutions.
  </Card>

  <Card title="Compliance & Audit" icon="shield-check" href="/services/disaster-recovery/admin-guide/compliance">
    DR test reports, replication status logs, and RPO/RTO compliance dashboards support
    regulatory audit requirements across financial, healthcare, and government sectors.
  </Card>
</CardGroup>

***

<p>Related Services</p>

<CardGroup>
  <Card title="Software-Defined Storage" icon="database" href="/services/sds/index">
    XSDS distributed storage powers the replication and snapshot engine behind XDR
  </Card>

  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Protect virtual machine workloads with instance-level DR protection plans
  </Card>

  <Card title="Infrastructure Monitoring" icon="activity" href="/services/monitoring/index">
    XIMP monitors replication health and triggers alerts on RPO/RTO threshold breaches
  </Card>
</CardGroup>


# Failback
Source: https://docs.xloud.tech/services/disaster-recovery/user-guide/failback

Return XDR-protected workloads to the primary site after it has been restored — reverse replication, synchronize changed data, and execute failback.

## Overview

Failback returns workloads to the primary site after it has been restored following
a failover event. Before initiating failback, confirm the primary site is fully
operational and any data created on the DR site during the failover period has been
synchronized back.

<Note>
  Failback reverses the replication direction — data flows from the DR site back to
  the primary site. The time required depends on the amount of changed data accumulated
  during the failover period. Allow replication to fully synchronize before cutting over.
</Note>

<Note>
  **Prerequisites**

  * Primary site confirmed healthy — all services operational, storage accessible
  * Network connectivity between primary and DR sites restored
  * No active production traffic changes needed until failback is complete
</Note>

***

## Failback Procedure

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Verify primary site is available" icon="server">
        Navigate to **Disaster Recovery → Sites** and confirm the primary site status
        returns to **Healthy**. Run a connectivity test from the DR site if available
        by clicking **Test Connectivity** on the site entry.
      </Step>

      <Step title="Reverse replication" icon="refresh-cw">
        Select the protection plan and click **Reverse Replication**. XDR syncs
        changed data from the DR site back to the primary site.

        Monitor sync progress in the plan status panel. The `replication_lag` field
        shows how much data remains to be transferred.

        <Tip>
          Allow replication to fully synchronize before initiating failback. The
          sync duration depends on how much data changed during the failover period.
          For active production workloads, this may take hours.
        </Tip>
      </Step>

      <Step title="Schedule the failback window" icon="clock">
        Coordinate with application owners and stakeholders to schedule a maintenance
        window for the actual failback cutover. During the cutover:

        * Application connections to the DR site are briefly interrupted
        * Instances stop on the DR site and restart on the primary site

        Typical failback cutover time is 10–30 minutes depending on the number of
        instances and the recovery runbook complexity.
      </Step>

      <Step title="Execute failback" icon="rotate-ccw">
        Once sync is complete and the maintenance window begins, click **Failback**.
        The runbook executes in reverse priority order:

        1. Services stop on the DR site
        2. Final delta sync to primary site
        3. Instances start on the primary site
        4. Health checks validate service availability
      </Step>

      <Step title="Verify and re-protect" icon="circle-check">
        Confirm workloads are running on the primary site. Navigate to **Protection
        Plans** and verify the plan is back in **Active** replication status, now
        protecting the primary site from the DR site.

        <Check>Plan shows primary site as source and replication lag is within RPO target.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above for
      the complete failback procedure.
    </Info>
  </Tab>
</Tabs>

***

## Post-Failback Checklist

After failback completes, restore normal operations:

<Steps>
  <Step title="Update DNS and load balancers" icon="globe">
    Revert DNS records and load balancer configurations back to primary site IP
    addresses. Verify traffic is flowing to the primary site.
  </Step>

  <Step title="Validate application services" icon="circle-check">
    Run application-level health checks against the primary site endpoints. Confirm
    data integrity and service connectivity.
  </Step>

  <Step title="Verify DR protection is active" icon="shield">
    Confirm the protection plan is replicating from the primary site back to the DR site.
    The plan should return to normal `ACTIVE` status with lag within RPO target.
  </Step>

  <Step title="Document the incident" icon="file-text">
    Record the failover and failback timeline, data loss (if any), actual RTO achieved,
    and any issues encountered during the recovery. Update the DR runbook if procedures
    need to be adjusted.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="DR Testing" href="/services/disaster-recovery/user-guide/test-dr">
    Run quarterly DR tests to keep failback procedures current and validated
  </Card>

  <Card title="Protection Plans" href="/services/disaster-recovery/user-guide/protection-plans">
    Review and update protection plans based on incident learnings
  </Card>

  <Card title="Troubleshooting" href="/services/disaster-recovery/user-guide/troubleshooting">
    Diagnose failback synchronization issues
  </Card>

  <Card title="XDR Admin — Compliance" href="/services/disaster-recovery/admin-guide/compliance">
    Generate post-incident RPO/RTO compliance reports (administrator)
  </Card>
</CardGroup>


# Failover
Source: https://docs.xloud.tech/services/disaster-recovery/user-guide/failover

Execute XDR failover to switch protected workloads from the primary site to the DR site after a confirmed primary site failure.

## Overview

Failover switches protected workloads from the primary site to the DR site. Initiate
failover when a primary site failure is confirmed and recovery at the primary site is
not possible within the RTO window.

<Warning>
  Failover is a significant operation. Confirm that the primary site is genuinely
  unavailable before proceeding. An unnecessary failover requires a full failback
  cycle to restore normal operations.
</Warning>

<Note>
  **Prerequisites**

  * An active protection plan in `ACTIVE` replication status
  * Confirmation that the primary site is unavailable — cross-reference with XIMP monitoring
  * DR site confirmed healthy (navigate to **Disaster Recovery → Sites**)
</Note>

***

## Failover Procedure

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Confirm primary site status" icon="activity">
        Navigate to **Project → Disaster Recovery → Sites** and verify the primary site
        health indicator shows **Unreachable** or **Failed**. Cross-reference with the
        XIMP monitoring portal for independent confirmation.

        <Tip>
          Do not rely on a single monitoring source. A network partition may make the
          primary site appear unreachable from the DR site while it is actually still
          operational. Verify from multiple vantage points before proceeding.
        </Tip>
      </Step>

      <Step title="Initiate failover" icon="zap">
        Navigate to **Project → Disaster Recovery → Protection Plans**, select the
        affected plan, and click **Failover**. Confirm the failover dialog.

        | Option                      | Description                                                             |
        | --------------------------- | ----------------------------------------------------------------------- |
        | **Latest Recovery Point**   | Use the most recent replicated snapshot                                 |
        | **Specific Recovery Point** | Select a point-in-time snapshot from the recovery point list            |
        | **Test Mode**               | Bring up workloads in isolation without cutting over production traffic |

        <Danger>
          Selecting **Latest Recovery Point** uses data from the last successful
          replication cycle. Any writes to the primary site since that cycle will be
          lost permanently. Review the current replication lag before confirming.
        </Danger>
      </Step>

      <Step title="Monitor recovery progress" icon="activity">
        The DR Runbook executes automatically in the configured priority order. Track
        progress in **Disaster Recovery → Failover Status**. Each resource shows:

        | Status         | Meaning                                               |
        | -------------- | ----------------------------------------------------- |
        | **Pending**    | Waiting for dependencies to recover first             |
        | **Recovering** | Instance starting on DR site                          |
        | **Validated**  | Recovery script confirmed service is available        |
        | **Failed**     | Recovery step encountered an error — review event log |
      </Step>

      <Step title="Verify workloads on DR site" icon="circle-check">
        Confirm application-level availability by accessing services through the DR
        site endpoints. Update DNS or load balancer configurations to route traffic
        to the DR site.

        <Check>Protected workloads are running on the DR site and serving traffic.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above for
      the complete failover procedure.
    </Info>
  </Tab>
</Tabs>

***

## Post-Failover Checklist

After failover completes, perform these steps:

<Steps>
  <Step title="Validate application services" icon="circle-check">
    Run application-level health checks against the DR site endpoints. Verify
    databases are consistent, application tiers are connected, and external services
    can reach the DR site.
  </Step>

  <Step title="Update DNS and load balancers" icon="globe">
    Route production traffic to DR site IP addresses. Update:

    * External DNS A/CNAME records
    * Load balancer pools and health checks
    * Any hardcoded IP references in application configuration
  </Step>

  <Step title="Notify stakeholders" icon="send">
    Communicate the failover event and DR site endpoints to:

    * Operations and on-call teams
    * Business stakeholders and affected service owners
    * Partners or customers if external connectivity has changed
  </Step>

  <Step title="Begin planning failback" icon="rotate-ccw">
    Once the primary site issue is resolved, plan the failback operation. See
    [Failback](/services/disaster-recovery/user-guide/failback) for the full procedure.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Failback" href="/services/disaster-recovery/user-guide/failback">
    Return workloads to the primary site after it has been restored
  </Card>

  <Card title="Protection Plans" href="/services/disaster-recovery/user-guide/protection-plans">
    Review and update protection plans after the failover event
  </Card>

  <Card title="Troubleshooting" href="/services/disaster-recovery/user-guide/troubleshooting">
    Diagnose failover stuck states and recovery script failures
  </Card>

  <Card title="XDR Admin — DR Automation" href="/services/disaster-recovery/admin-guide/dr-automation">
    Configure automatic failover triggers to reduce response time (administrator)
  </Card>
</CardGroup>


# Protection Plans
Source: https://docs.xloud.tech/services/disaster-recovery/user-guide/protection-plans

Create and activate XDR protection plans that define which workloads are replicated, the replication mode, and your RPO and RTO targets.

## Overview

A protection plan defines which workloads are protected, how they are replicated to
the DR site, and in what order they recover during a failover. Creating a well-structured
plan is the foundation of an effective DR strategy.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * At least one DR site registered and connected (contact your administrator if no DR site is configured). Your administrator can configure this through [XDeploy](/deployment).
  * Instances and volumes you want to protect must exist in the project

  **Key Concepts**

  | Term                  | Definition                                                                        |
  | --------------------- | --------------------------------------------------------------------------------- |
  | **RPO**               | Recovery Point Objective — maximum acceptable data loss measured in time          |
  | **RTO**               | Recovery Time Objective — maximum acceptable recovery time                        |
  | **Replication Mode**  | `Asynchronous` (lower overhead) or `Synchronous` (zero data loss, higher latency) |
  | **Consistency Group** | Resources that must recover together atomically                                   |
</Note>

***

## Creating a Protection Plan

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Protection Plans" icon="shield">
        Log in to the **Xloud Dashboard** (`https://connect.<your-domain>`) and navigate to
        **Project → Disaster Recovery → Protection Plans**. Click **Create Plan**.
      </Step>

      <Step title="Define plan settings" icon="settings">
        | Field                | Description                                       |
        | -------------------- | ------------------------------------------------- |
        | **Plan Name**        | Descriptive label (e.g., `prod-web-tier-dr`)      |
        | **RPO Target**       | Maximum data loss tolerance (e.g., `15 minutes`)  |
        | **RTO Target**       | Maximum acceptable recovery time (e.g., `1 hour`) |
        | **Replication Mode** | `Asynchronous` or `Synchronous`                   |
        | **DR Site**          | Target site where workloads recover               |

        <Tip>
          Synchronous replication guarantees zero data loss (RPO = 0) but adds write
          latency proportional to the round-trip time between sites. Use asynchronous
          mode for sites separated by more than 10ms RTT.
        </Tip>
      </Step>

      <Step title="Add protected resources" icon="plus">
        Click **Add Resource** and select the instances, volumes, or application groups
        to include. For each resource, configure:

        * **Recovery Priority** — order in which this resource starts during failover
        * **Consistency Group** — group resources that must recover together atomically
        * **Pre/Post Scripts** — optional automation hooks for application quiesce and validation
      </Step>

      <Step title="Review and activate" icon="circle-check">
        Review the plan summary and click **Activate**. XDR begins replicating all
        protected resources to the DR site immediately. Initial sync time depends on
        data volume.

        <Check>Plan status shows **Active** and initial replication sync begins.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      create and activate protection plans.
    </Info>
  </Tab>
</Tabs>

***

## Managing Existing Plans

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Project → Disaster Recovery → Protection Plans** to see all plans
    with their current status and replication lag.

    Available actions per plan:

    * **Edit** — update RPO/RTO targets or add resources
    * **Deactivate** — pause replication without deleting the plan
    * **Delete** — permanently remove the plan (stops replication)
    * **Failover** — initiate failover (see [Failover](/services/disaster-recovery/user-guide/failover))
    * **Test Failover** — run a DR test (see [DR Testing](/services/disaster-recovery/user-guide/test-dr))
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      manage existing protection plans.
    </Info>
  </Tab>
</Tabs>

***

## Replication Health

Monitor replication health to ensure your RPO targets are achievable before an actual
disaster event:

| Metric              | Healthy       | Warning                | Action                        |
| ------------------- | ------------- | ---------------------- | ----------------------------- |
| Replication lag     | \< RPO target | Approaching RPO target | Investigate network bandwidth |
| Plan status         | `ACTIVE`      | `DEGRADED`             | Review event log for errors   |
| Last recovery point | \< RPO ago    | > RPO ago              | Check connectivity to DR site |

Monitor replication lag for all plans from the **Disaster Recovery → Protection Plans** list
view, which displays current lag alongside each plan.

<Warning>
  If replication lag consistently exceeds your RPO target, actual data loss on failover
  will exceed the configured target. Escalate to your storage administrator immediately.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Failover" href="/services/disaster-recovery/user-guide/failover">
    Execute failover when a primary site failure is confirmed
  </Card>

  <Card title="DR Testing" href="/services/disaster-recovery/user-guide/test-dr">
    Validate your protection plan without impacting production
  </Card>

  <Card title="XDR Admin — Recovery Plans" href="/services/disaster-recovery/admin-guide/recovery-plans">
    Define ordered resource groups and automation hooks (administrator)
  </Card>

  <Card title="Troubleshooting" href="/services/disaster-recovery/user-guide/troubleshooting">
    Diagnose replication lag and plan activation issues
  </Card>
</CardGroup>


# DR Testing
Source: https://docs.xloud.tech/services/disaster-recovery/user-guide/test-dr

Validate XDR protection plans with isolated DR tests — confirm recovery procedures and measure actual RTO without impacting production workloads.

## Overview

DR tests validate your protection plan without impacting production. Test instances
are brought up in an isolated network environment on the DR site, allowing you to
confirm recovery procedures and measure actual RTO. Xloud recommends running DR tests
at least quarterly.

<Tip>
  Regular DR tests are the only way to verify that your RPO and RTO targets are
  achievable and that recovery runbooks remain accurate as workloads evolve. A
  DR plan that has never been tested is not a reliable DR plan.
</Tip>

<Note>
  **Prerequisites**

  * An active protection plan with replication in `ACTIVE` status
  * DR site is healthy and has sufficient capacity for the test instances
  * Notify the operations team before starting a test — test traffic will appear in monitoring
</Note>

***

## Running a DR Test

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Start a DR test" icon="flask-conical">
        Navigate to **Disaster Recovery → Protection Plans**, select the plan, and click
        **Test Failover**. The test brings up recovered instances in an isolated network
        segment — production traffic is not affected.

        | Option             | Description                                           |
        | ------------------ | ----------------------------------------------------- |
        | **Recovery Point** | Which snapshot to use for the test — usually `latest` |
        | **Test Network**   | Isolated network segment for test instances           |
        | **Test Duration**  | Maximum time before auto-cleanup                      |
      </Step>

      <Step title="Validate recovered workloads" icon="circle-check">
        Access the test instances through the isolated DR test network. Run your
        application validation scripts to confirm the service is operational and
        data is intact.

        | Validation Check     | What to Verify                                    |
        | -------------------- | ------------------------------------------------- |
        | Instance status      | All instances show `ACTIVE` on DR site            |
        | Data integrity       | Application-level health checks pass              |
        | Service connectivity | Intra-app communication works in isolated network |
        | Recovery time        | Actual RTO measured against configured target     |

        <Note>
          Test instances run in an isolated network — they cannot reach production
          services or external systems. This isolation is intentional and prevents
          DR test instances from interfering with production.
        </Note>
      </Step>

      <Step title="End the test" icon="x-circle">
        Click **End Test** to terminate the test instances and release the isolated
        environment. No changes are made to production — the protection plan remains
        active throughout.

        <Check>DR test report generated with measured RTO and pass/fail for each validation check.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Info>
      XDR disaster recovery operations are managed exclusively through the XDR Dashboard.
      CLI access is not available for DR operations. Use the **Dashboard** tab above to
      run DR tests and review test results.
    </Info>
  </Tab>
</Tabs>

***

## Test Report

After each DR test, XDR generates a report including:

| Report Field            | Description                                            |
| ----------------------- | ------------------------------------------------------ |
| **Test Date**           | When the test was executed                             |
| **Recovery Point Used** | Age of data at test time                               |
| **Measured RTO**        | Actual time from test start to all resources validated |
| **Target RTO**          | Configured target for comparison                       |
| **Pass/Fail**           | Whether measured RTO met the configured target         |
| **Resource Status**     | Per-resource recovery outcome                          |
| **Script Results**      | Output from pre/post recovery validation scripts       |

Export DR test reports from **Disaster Recovery → Reports** by selecting the test
report type and choosing PDF or CSV format. See [Compliance](/services/disaster-recovery/admin-guide/compliance)
for scheduled report delivery.

***

## DR Test Schedule

| Frequency             | Trigger              | Scope                                          |
| --------------------- | -------------------- | ---------------------------------------------- |
| Quarterly             | Calendar             | Full protection plan test for all plans        |
| After major changes   | Change event         | Any plan whose workloads changed significantly |
| After site changes    | Infrastructure event | After DR site hardware or network changes      |
| After runbook updates | Policy change        | When pre/post recovery scripts are modified    |

<Warning>
  XIMP can alert when a DR test has not been run within the configured interval.
  Navigate to **Monitoring → Alerting → Alert Rules** and create a rule sourcing
  `xdr_last_test_age_days > 90` to ensure tests are not missed.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Protection Plans" href="/services/disaster-recovery/user-guide/protection-plans">
    Update protection plans based on test results
  </Card>

  <Card title="Failover" href="/services/disaster-recovery/user-guide/failover">
    Full failover procedure when an actual disaster event occurs
  </Card>

  <Card title="XDR Admin — Compliance" href="/services/disaster-recovery/admin-guide/compliance">
    Generate RPO/RTO compliance reports from DR test history (administrator)
  </Card>

  <Card title="Troubleshooting" href="/services/disaster-recovery/user-guide/troubleshooting">
    Diagnose DR test failures and instance access issues
  </Card>
</CardGroup>


# Troubleshooting
Source: https://docs.xloud.tech/services/disaster-recovery/user-guide/troubleshooting

Diagnose common XDR user-facing issues — replication lag exceeding RPO targets, failover stuck states, and DR test instance access problems.

## Overview

This page covers the most common issues encountered when using XDR — from replication
lag that threatens RPO targets, to failover operations stuck on specific resources,
to DR test instances that cannot be reached for validation.

<Note>
  **Prerequisites**

  * An active Xloud account with project access and XDR plan access
  * For site connectivity and replication configuration issues, contact your administrator. Your administrator can configure this through [XDeploy](/deployment).
</Note>

***

## Common Issues

<AccordionGroup>
  <Accordion title="Replication lag exceeding RPO target" icon="clock">
    **Cause**: Network bandwidth between sites is insufficient for the current change
    rate, or the source workload is writing data faster than replication can transfer it.

    **Diagnosis**: Navigate to **Disaster Recovery → Protection Plans → \[Plan]** and
    review the replication lag and throughput metrics displayed in the plan status panel.

    **Resolution**:

    * Increase network bandwidth allocation for replication traffic (contact your administrator). Your administrator can configure this through [XDeploy](/deployment).
    * Switch to a larger replication window that permits more transfer time
    * Review the change rate of protected workloads — peak write periods may cause
      temporary lag spikes that resolve during quieter periods

    <Warning>
      If replication lag consistently exceeds the RPO target, data loss beyond the
      target threshold is possible in a failover scenario. Escalate to your storage
      administrator immediately — do not wait for an actual disaster event.
    </Warning>
  </Accordion>

  <Accordion title="Failover stuck on a specific resource" icon="server">
    **Cause**: A dependency is not yet recovered, a pre/post script failed, or the
    DR site lacks sufficient capacity for the recovering instance.

    **Diagnosis**: Navigate to **Disaster Recovery → Failover Status** and expand
    the stuck resource entry. Review the event log for error messages and timestamps.

    **Common causes and resolutions**:

    | Cause                                           | Resolution                                                   |
    | ----------------------------------------------- | ------------------------------------------------------------ |
    | Pre-recovery script returned non-zero exit code | Review script output in the log; fix the script              |
    | Insufficient quota on DR project                | Check with administrator to increase quota                   |
    | Dependency resource not yet recovered           | Wait for the dependency to complete; check priority ordering |
    | DR site capacity insufficient                   | Contact administrator to add capacity                        |
  </Accordion>

  <Accordion title="DR test instances not accessible" icon="network">
    **Cause**: The isolated test network has no route to the validation host, or security
    group rules block the required ports in the test environment.

    **Resolution**:

    1. Use console access to reach test instances without network:
       Navigate to **Disaster Recovery → Test Sessions → Console**
    2. Verify the test security groups match production configuration within the isolation
       boundary by reviewing the security group assignments in **Disaster Recovery → Test Sessions → \[Instance] → Security Groups**
    3. Confirm the test network allows communication between test instances by reviewing
       the network topology in **Disaster Recovery → Test Sessions → Network**
  </Accordion>

  <Accordion title="Failback synchronization not completing" icon="refresh-cw">
    **Cause**: The reverse replication sync is stalled due to network issues between
    the DR and primary sites, or a large amount of data was written to the DR site
    during the failover period.

    **Diagnosis**: Navigate to **Disaster Recovery → Protection Plans → \[Plan]** and
    review the reverse sync progress and replication lag metrics. Check the replication
    link statistics in **Disaster Recovery → Sites → Replication Links → \[Link]**.

    **Resolution**:

    * Verify network connectivity between DR and primary sites
    * Check that firewall rules permit replication traffic in both directions
    * If sync is making progress but slowly, allow more time — large datasets take
      proportionally longer
    * If throughput is near-zero, check for network path issues or firewall changes
      that occurred during the failover period
  </Accordion>
</AccordionGroup>

***

## Diagnostics Reference

All diagnostic operations are performed through the XDR Dashboard:

| Issue             | Dashboard Location                                                                  |
| ----------------- | ----------------------------------------------------------------------------------- |
| Replication lag   | **Disaster Recovery → Protection Plans → \[Plan]** — replication lag panel          |
| Failover stuck    | **Disaster Recovery → Failover Status → \[Resource]** — event log                   |
| Site connectivity | **Disaster Recovery → Sites → \[Site]** — Test Connectivity button                  |
| Test resources    | **Disaster Recovery → Test Sessions → \[Instance]** — IP and status                 |
| Link throughput   | **Disaster Recovery → Sites → Replication Links → \[Link]** — throughput statistics |

***

## When to Contact Your Administrator

Contact your DR administrator or [support@xloud.tech](mailto:support@xloud.tech) if any of the following persist. Your administrator can configure this through [XDeploy](/deployment).

* Replication lag has exceeded the RPO target for more than 30 minutes
* Failover is stuck and the event log shows an unresolvable error
* Site connectivity tests fail consistently
* Failback synchronization shows zero throughput for more than 10 minutes

***

## Next Steps

<CardGroup>
  <Card title="XDR Admin — Troubleshooting" href="/services/disaster-recovery/admin-guide/troubleshooting">
    Administrator-level DR diagnostics — site registration, replication links
  </Card>

  <Card title="Protection Plans" href="/services/disaster-recovery/user-guide/protection-plans">
    Review and adjust plan configuration based on troubleshooting findings
  </Card>

  <Card title="DR Testing" href="/services/disaster-recovery/user-guide/test-dr">
    Run DR tests after resolving issues to validate recovery still works
  </Card>

  <Card title="Support" href="mailto:support@xloud.tech">
    Contact Xloud support for issues requiring platform-level investigation
  </Card>
</CardGroup>


# DNS Service
Source: https://docs.xloud.tech/services/dns

Manage DNS zones and records for your Xloud private cloud with Xloud DNS — authoritative DNS-as-a-service for zones, record sets, and reverse DNS.

Authoritative DNS management for your private cloud — create zones, manage records, and automate reverse DNS for all your services.

<Card title="Xloud DNS — DNS-as-a-Service" icon="globe" href="https://xloud.tech/xavs">
  Product details and datasheet on xloud.tech
</Card>

***

<p>Xloud DNS Service</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/dns/user-guide">
    Create DNS zones, manage record sets (A, AAAA, CNAME, MX, TXT), configure reverse DNS, and query your DNS infrastructure.
  </Card>

  <Card title="Admin Guide" icon="shield-check" href="/services/dns/admin-guide">
    Configure DNS backend drivers, manage pools, control zone transfers, enforce quotas, and administer the DNS service infrastructure.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/dns/cli-reference">
    `openstack zone` and `openstack recordset` commands for managing DNS from the command line.
  </Card>

  <Card title="Reverse DNS" icon="arrow-left-right">
    Create PTR records for IP-to-hostname resolution supporting compliance, email delivery, and network diagnostics.
  </Card>
</CardGroup>

***

<p>Key Features</p>

<CardGroup>
  <Card title="Authoritative DNS Zones" icon="map-pin" href="/services/dns/create-zone">
    Create and manage authoritative DNS zones for your domains. Full control over SOA, NS, and all resource record types within each zone.
  </Card>

  <Card title="Multi-Record Support" icon="list" href="/services/dns/record-types">
    Native support for A, AAAA, CNAME, MX, NS, PTR, SOA, SRV, SPF, TXT, and CAA record types. Cover every DNS use case from web hosting to email delivery.
  </Card>

  <Card title="Reverse DNS (PTR)" icon="arrow-left-right" href="/services/dns/reverse-dns">
    Automated PTR record management for IPv4 and IPv6 address spaces. Essential for mail server reputation, syslog, and compliance logging.
  </Card>

  <Card title="Zone Transfers" icon="share-2" href="/services/dns/zone-transfers">
    Controlled zone transfer (AXFR/IXFR) support for secondary DNS servers. Delegate authority to external resolvers while maintaining primary control.
  </Card>

  <Card title="TTL Management" icon="clock" href="/services/dns/manage-records">
    Configure Time-to-Live values per record set. Reduce TTLs before planned changes for faster propagation; increase after stabilization to reduce resolver load.
  </Card>

  <Card title="API-Driven Automation" icon="code" href="/services/dns/cli-reference">
    Full REST API for DNS record management. Integrate with deployment pipelines to register service endpoints automatically as infrastructure is provisioned.
  </Card>
</CardGroup>

***

<p>DNS Components</p>

| Component     | Description                                                                         |
| ------------- | ----------------------------------------------------------------------------------- |
| Zone          | An authoritative DNS domain (e.g., `example.com.`) managed by the Xloud DNS service |
| Record Set    | A collection of DNS records sharing the same name and type within a zone            |
| Nameserver    | The authoritative resolvers designated to serve records for a zone                  |
| Pool          | A group of DNS backend servers that handle zone data for a set of zones             |
| PTR Record    | Reverse DNS mapping an IP address to a hostname                                     |
| Zone Transfer | Mechanism for replicating zone data to secondary DNS servers                        |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Load Balancer" icon="combine" href="/services/load-balancer">
    Map DNS names to load balancer VIP addresses for service endpoints
  </Card>

  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Register compute instance hostnames in DNS zones automatically
  </Card>

  <Card title="Xloud Networking" icon="network" href="/services/compute">
    Floating IPs and fixed IPs referenced in DNS A and PTR records
  </Card>

  <Card title="Xloud Identity" icon="fingerprint" href="/cli-setup">
    RBAC policies governing zone and record management permissions
  </Card>

  <Card title="Xloud Key Manager" icon="lock" href="/services/key-manager">
    Store DNSSEC signing keys for zone integrity validation
  </Card>

  <Card title="Xloud Object Storage" icon="box" href="/services/object-storage">
    Archive zone export files and DNS audit logs for compliance
  </Card>
</CardGroup>

***

<p>Getting Started</p>

<CardGroup>
  <Card title="Authentication" icon="key" href="/cli-setup">
    Configure Dashboard access and CLI credentials before working with DNS
  </Card>

  <Card title="User Guide" icon="book-open" href="/services/dns/user-guide">
    Step-by-step instructions for creating your first DNS zone
  </Card>
</CardGroup>


# DNS Admin Guide
Source: https://docs.xloud.tech/services/dns/admin-guide

Administer Xloud DNS infrastructure — configure backend drivers, manage pools, control zone transfers, enforce quotas, and secure the DNS service.

<p>Overview</p>

This guide covers platform-level administration of the Xloud DNS service. Administrators
configure the backend DNS driver, manage server pools that process zone data, control
zone transfer policies, set per-project quotas, and maintain the security posture of
the DNS infrastructure. All operations require administrator privileges.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

<p>Topics in This Guide</p>

<CardGroup>
  <Card title="Architecture" icon="network" href="/services/dns/architecture">
    DNS service topology — API layer, central processing, and backend nameserver pools
  </Card>

  <Card title="Backend Configuration" icon="settings" href="/services/dns/backend-config">
    Configure the backend DNS driver and pool targets through XDeploy
  </Card>

  <Card title="Zone Transfers" icon="share-2" href="/services/dns/zone-transfers">
    Manage AXFR and IXFR zone transfer requests between projects and secondary nameservers
  </Card>

  <Card title="Pool Management" icon="server" href="/services/dns/pool-management">
    Manage nameserver pools, attributes, and geographic distribution
  </Card>

  <Card title="Quotas" icon="gauge" href="/services/dns/quotas">
    Set and manage per-project DNS resource limits
  </Card>

  <Card title="Security" icon="shield" href="/services/dns/security">
    Harden DNS infrastructure — restrict zone transfers, protect apex records, manage DNSSEC keys
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/dns/admin-troubleshooting">
    Diagnose API outages, propagation failures, and nameserver inconsistencies
  </Card>
</CardGroup>

***

<p>Prerequisites</p>

<Note>
  **Required before proceeding**

  * Administrator credentials sourced via `openrc.sh`
  * Access to XDeploy for DNS service configuration
  * Understanding of DNS protocol fundamentals (SOA, NS delegation, AXFR/IXFR)
</Note>

***

<p>Next Steps</p>

<CardGroup>
  <Card title="DNS User Guide" icon="book-open" href="/services/dns/user-guide">
    Step-by-step instructions for managing zones and records
  </Card>

  <Card title="Load Balancer Admin Guide" icon="shield-check" href="/services/load-balancer/admin-guide">
    Administer load balancer infrastructure and quota management
  </Card>

  <Card title="Key Manager" icon="lock" href="/services/key-manager/admin-guide">
    Manage DNSSEC signing keys and TLS certificate secrets
  </Card>

  <Card title="Identity" icon="fingerprint" href="/cli-setup">
    Configure RBAC policies for DNS zone management
  </Card>
</CardGroup>


# DNS Admin Troubleshooting
Source: https://docs.xloud.tech/services/dns/admin-troubleshooting

Diagnose and resolve platform-level Xloud DNS issues — API outages, zone propagation failures, nameserver synchronization inconsistencies, and worker.

## Overview

This guide covers platform-level DNS issues that require administrator access. For
user-facing issues such as record set errors or propagation delays, see the
[DNS Troubleshooting](/services/dns/troubleshooting) guide.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Diagnostic Checklist

Before investigating individual components, run this quick health check:

```bash title="Check DNS service container status" theme={null}
docker ps --filter name=dns
```

```bash title="Verify DNS API is responding" theme={null}
openstack zone list --limit 1
```

```bash title="Check nameserver is authoritative for a zone" theme={null}
dig @<nameserver-ip> example.com. SOA +norecurse
```

***

## Platform Issues

<AccordionGroup>
  <Accordion title="DNS API not responding" icon="server">
    **Cause**: The DNS API service container may be unavailable or crashed.

    **Resolution**:

    ```bash title="Check DNS service containers" theme={null}
    docker ps --filter name=dns
    ```

    ```bash title="Review container logs" theme={null}
    docker logs dns-api --tail 100
    ```

    If the container exited unexpectedly, check for configuration errors in the startup
    log. Restart affected containers:

    ```bash title="Restart DNS API container" theme={null}
    docker restart dns-api
    ```

    If the container fails to start, verify the database connection and configuration
    files deployed by XDeploy.
  </Accordion>

  <Accordion title="Zone changes not propagating to nameservers" icon="refresh-cw">
    **Cause**: The DNS worker may be unable to push updates to backend nameservers.

    **Resolution**:

    1. Verify the worker container is running and connected to the message queue:
       ```bash title="Check DNS worker container" theme={null}
       docker ps --filter name=dns-worker
       docker logs dns-worker --tail 50
       ```
    2. Check for push errors in the worker logs — look for connection refused or timeout
       errors to pool target addresses
    3. Verify network connectivity from the worker host to each nameserver's management port:
       ```bash title="Test connectivity to nameserver target" theme={null}
       curl -s http://<nameserver-ip>:<target-port>/
       ```
    4. If a nameserver is unreachable, temporarily remove it from the pool via XDeploy
       while investigating
  </Accordion>

  <Accordion title="Inconsistent answers from different nameservers" icon="git-branch">
    **Cause**: Zone synchronization lag — one nameserver received the update while another
    did not.

    **Diagnosis**:

    ```bash title="Check zone serial on each nameserver" theme={null}
    dig @<ns1-ip> example.com. SOA +short
    dig @<ns2-ip> example.com. SOA +short
    ```

    SOA serial numbers should match. A lagging nameserver indicates a synchronization
    failure.

    **Resolution**: Check the DNS worker logs for push errors to the specific nameserver's
    target configuration. If the push is failing due to authentication or connectivity,
    correct the target configuration in XDeploy and redeploy.
  </Accordion>

  <Accordion title="DNS worker backlog growing" icon="clock">
    **Cause**: The worker cannot process zone updates as fast as they are being created —
    typically due to a slow or overloaded nameserver backend.

    **Resolution**:

    ```bash title="Check message queue depth" theme={null}
    docker logs dns-central --tail 50 | grep queue
    ```

    * If a specific nameserver backend is slow, investigate its resource usage
    * Consider adding worker replicas via XDeploy to parallelize processing
    * If the backlog grew due to a temporary nameserver outage, it should drain
      automatically once connectivity is restored
  </Accordion>

  <Accordion title="Database connection errors on startup" icon="database">
    **Cause**: The DNS service cannot reach the database, or the database schema
    is not initialized.

    **Resolution**:

    ```bash title="Check DNS API logs for DB errors" theme={null}
    docker logs dns-api --tail 100 | grep -i "error\|database\|connect"
    ```

    Verify the database service is running and that the DNS service configuration
    in XDeploy has the correct connection credentials and host address.
  </Accordion>
</AccordionGroup>

***

## Log Locations

| Component   | Log Command               |
| ----------- | ------------------------- |
| DNS API     | `docker logs dns-api`     |
| DNS Central | `docker logs dns-central` |
| DNS Worker  | `docker logs dns-worker`  |
| DNS MDns    | `docker logs dns-mdns`    |

***

## Next Steps

<CardGroup>
  <Card title="DNS Troubleshooting (User)" href="/services/dns/troubleshooting">
    User-facing DNS issues — zone errors, propagation, CNAME conflicts
  </Card>

  <Card title="Architecture" href="/services/dns/architecture">
    Understand component roles to narrow down failure scope
  </Card>

  <Card title="Backend Configuration" href="/services/dns/backend-config">
    Verify and update pool target configuration
  </Card>

  <Card title="Pool Management" href="/services/dns/pool-management">
    Remove or replace unhealthy nameservers from pools
  </Card>
</CardGroup>


# DNS Service Architecture
Source: https://docs.xloud.tech/services/dns/architecture

Understand the Xloud DNS service topology — API layer, central processing service, DNS worker, and backend nameserver pools with data plane separation.

## Overview

The Xloud DNS service uses a multi-tier architecture separating the API layer, the
central processing service, and the backend DNS server pools. This separation allows the
management plane to scale independently from the data plane that serves resolver queries.

***

## Service Topology

```mermaid theme={null}
graph TD
    User["User / API Client"] --> API["DNS API\n:9001"]
    API --> Central["DNS Central Service\n(zone management)"]
    Central --> DB[("Service Database\n(zones, records)")]
    Central --> MQ["Message Queue"]
    MQ --> Worker["DNS Worker\n(pool manager)"]
    Worker --> Pool1["Pool: Default\nBIND9 / PowerDNS"]
    Worker --> Pool2["Pool: External\n(zone transfers)"]
    Pool1 --> NS1["Nameserver 1"]
    Pool1 --> NS2["Nameserver 2"]
    Pool2 --> Secondary["Secondary DNS\n(AXFR consumer)"]

    subgraph "Management Plane"
        API
        Central
        DB
        MQ
        Worker
    end

    subgraph "Data Plane"
        Pool1
        Pool2
        NS1
        NS2
    end
```

<Info>
  The DNS API and central service operate in the management plane. Nameservers in
  pools operate as data-plane components — they respond to resolver queries directly
  without routing through the API layer.
</Info>

***

## Component Descriptions

| Component            | Role                                                      | Port         |
| -------------------- | --------------------------------------------------------- | ------------ |
| **DNS API**          | REST API for zone and record management                   | 9001         |
| **DNS Central**      | Orchestrates zone lifecycle, writes to the database       | Internal     |
| **DNS Worker**       | Pushes zone data to backend nameserver pools              | Internal     |
| **Message Queue**    | Decouples Central from Workers for async processing       | Internal     |
| **Service Database** | Stores zone metadata, record sets, and pool configuration | Internal     |
| **Nameserver Pool**  | Backend DNS servers that answer resolver queries          | 53 (UDP/TCP) |

***

## Request Flow

### Zone Creation

```mermaid theme={null}
sequenceDiagram
    participant User
    participant API as DNS API
    participant Central as DNS Central
    participant DB as Database
    participant Queue as Message Queue
    participant Worker as DNS Worker
    participant NS as Nameserver Pool

    User->>API: POST /v2/zones {name: "example.com."}
    API->>Central: Create zone request
    Central->>DB: Write zone record (status: PENDING)
    Central->>Queue: Publish zone_create event
    API-->>User: 202 Accepted (zone in PENDING)
    Queue->>Worker: Deliver zone_create event
    Worker->>NS: Push zone to backend
    NS-->>Worker: Acknowledgment
    Worker->>DB: Update status to ACTIVE
```

***

## High Availability

The DNS management plane components (API, Central, Worker) are deployed as containerized
services managed by XDeploy. For production deployments:

* Deploy at least two API containers behind a load balancer
* Run two Worker instances for redundancy — only one processes each event (queue-based)
* Database and message queue are shared, managed services

<Note>
  The data plane (nameservers) operates independently of the management plane. Resolver
  queries continue to be served even if the DNS API or central service is temporarily
  unavailable. Only zone updates and record changes require the management plane.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Backend Configuration" href="/services/dns/backend-config">
    Configure backend DNS drivers and pool targets
  </Card>

  <Card title="Pool Management" href="/services/dns/pool-management">
    Manage nameserver pools and geographic distribution
  </Card>

  <Card title="Security" href="/services/dns/security">
    Harden the DNS service and protect zone data
  </Card>

  <Card title="Troubleshooting" href="/services/dns/admin-troubleshooting">
    Diagnose and resolve platform-level DNS issues
  </Card>
</CardGroup>


# DNS Backend Configuration
Source: https://docs.xloud.tech/services/dns/backend-config

Configure the Xloud DNS backend driver and nameserver pool targets through XDeploy. View current pool configuration, manage zone assignments, and update.

## Overview

The DNS service backend driver determines which DNS server software is used to serve
zone data. The backend is configured through XDeploy and the `xavs-ansible` deployment
tool. This page covers viewing and managing current pool configuration and assigning
zones to specific pools.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials sourced via `openrc.sh`
  * Access to XDeploy for DNS service configuration changes
</Note>

***

## View Current Configuration

<Tabs>
  <Tab title="List DNS pools" icon="list">
    ```bash title="List DNS pools" theme={null}
    openstack zone pool list
    ```

    ```bash title="Show pool detail" theme={null}
    openstack zone pool show <pool-id>
    ```

    Pool detail fields:

    | Field           | Description                                                         |
    | --------------- | ------------------------------------------------------------------- |
    | `name`          | Pool identifier used when assigning zones                           |
    | `description`   | Administrative description                                          |
    | `nameservers`   | List of authoritative nameserver hostnames for this pool            |
    | `targets`       | Backend server endpoints the worker pushes zone data to             |
    | `also_notifies` | Additional servers notified of zone changes (for AXFR)              |
    | `attributes`    | Key-value tags used to match zones to pools via scheduling policies |
  </Tab>

  <Tab title="Verify pool health" icon="heart-pulse">
    Verify that the pool's nameservers are authoritatively answering queries:

    ```bash title="Check zone on each nameserver" theme={null}
    dig @<nameserver-ip> example.com. SOA +norecurse
    ```

    A healthy pool nameserver returns the SOA record with `AUTHORITY: 1`. If the
    nameserver returns `SERVFAIL`, the zone has not synchronized to that nameserver.
  </Tab>
</Tabs>

***

## Assign a Zone to a Pool

By default, new zones are assigned to the default pool based on scheduling policies.
You can override pool assignment:

```bash title="Show zone's current pool assignment" theme={null}
openstack zone show <zone-name> -c masters -c attributes
```

```bash title="Assign a zone to a specific pool" theme={null}
openstack zone set <zone-name> --attributes pool_id:<pool-id>
```

<Warning>
  Reassigning a zone to a different pool triggers a full zone synchronization to the
  new pool's nameservers. During synchronization, queries may return stale data from
  the old pool if resolver caches have not expired.
</Warning>

***

## Pool Configuration Reference

Pool configuration is managed through XDeploy service configuration files and applied
via `xavs-ansible deploy`. The following fields control backend behavior:

| Parameter       | Description                              | Example                 |
| --------------- | ---------------------------------------- | ----------------------- |
| `backend`       | DNS server backend type                  | `bind9`, `pdns4`        |
| `targets`       | Backend RNDC/API endpoints for zone push | `<host>:<port>`         |
| `nameservers`   | Public-facing nameserver FQDNs           | `ns1.example.com.`      |
| `also_notifies` | Hosts to NOTIFY after zone changes       | Secondary server IPs    |
| `attributes`    | Scheduling attributes                    | `service_tier: premium` |

***

## Next Steps

<CardGroup>
  <Card title="Pool Management" href="/services/dns/pool-management">
    Add nameservers, configure attributes, and manage pool health
  </Card>

  <Card title="Zone Transfers" href="/services/dns/zone-transfers">
    Configure AXFR/IXFR zone transfer policies
  </Card>

  <Card title="Architecture" href="/services/dns/architecture">
    Understand the DNS service topology and component roles
  </Card>

  <Card title="Admin Troubleshooting" href="/services/dns/admin-troubleshooting">
    Diagnose backend connectivity and synchronization failures
  </Card>
</CardGroup>


# DNS CLI Reference
Source: https://docs.xloud.tech/services/dns/cli-reference

Complete openstack zone and recordset CLI commands for managing Xloud DNS — zones, records, PTR records, and zone transfers.

## Overview

The `openstack zone` and `openstack recordset` command groups manage authoritative DNS zones and all record types.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python designateclient installed: `pip install python-designateclient`
</Note>

***

## Zones

<CodeGroup>
  ```bash title="List zones" theme={null}
  openstack zone list
  ```

  ```bash title="Create primary zone" theme={null}
  openstack zone create \
    --email admin@example.com \
    --ttl 3600 \
    example.com.
  ```

  ```bash title="Create secondary zone" theme={null}
  openstack zone create \
    --type SECONDARY \
    --masters 192.0.2.1 \
    example.com.
  ```

  ```bash title="Show zone" theme={null}
  openstack zone show example.com.
  ```

  ```bash title="Delete zone" theme={null}
  openstack zone delete example.com.
  ```
</CodeGroup>

***

## Record Sets

<CodeGroup>
  ```bash title="List record sets in zone" theme={null}
  openstack recordset list example.com.
  ```

  ```bash title="Create A record" theme={null}
  openstack recordset create \
    --type A \
    --record 203.0.113.10 \
    example.com. web
  ```

  ```bash title="Create AAAA record" theme={null}
  openstack recordset create \
    --type AAAA \
    --record "2001:db8::1" \
    example.com. web
  ```

  ```bash title="Create CNAME record" theme={null}
  openstack recordset create \
    --type CNAME \
    --record web.example.com. \
    example.com. www
  ```

  ```bash title="Create MX record" theme={null}
  openstack recordset create \
    --type MX \
    --record "10 mail.example.com." \
    example.com. @
  ```

  ```bash title="Create TXT record" theme={null}
  openstack recordset create \
    --type TXT \
    --record '"v=spf1 include:_spf.example.com ~all"' \
    example.com. @
  ```

  ```bash title="Show record set" theme={null}
  openstack recordset show example.com. web
  ```

  ```bash title="Delete record set" theme={null}
  openstack recordset delete example.com. web
  ```
</CodeGroup>

***

## PTR Records (Reverse DNS)

<CodeGroup>
  ```bash title="Set PTR record for floating IP" theme={null}
  openstack ptr record set \
    --description "Web server" \
    --ttl 300 \
    RegionOne:$(openstack floating ip show <fip> -c id -f value) \
    web.example.com.
  ```

  ```bash title="List PTR records" theme={null}
  openstack ptr record list
  ```

  ```bash title="Delete PTR record" theme={null}
  openstack ptr record delete \
    RegionOne:<floating-ip-id>
  ```
</CodeGroup>

***

## Zone Transfers

<CodeGroup>
  ```bash title="Create zone transfer request" theme={null}
  openstack zone transfer request create example.com.
  ```

  ```bash title="Accept zone transfer" theme={null}
  openstack zone transfer accept request \
    --transfer-id <transfer-id> \
    --key <transfer-key>
  ```

  ```bash title="List transfer requests" theme={null}
  openstack zone transfer request list
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Create Zone Guide" href="/services/dns/create-zone">
    Step-by-step guide to creating DNS zones and records
  </Card>

  <Card title="Reverse DNS Guide" href="/services/dns/reverse-dns">
    Configure PTR records for floating IPs
  </Card>
</CardGroup>


# Create a DNS Zone
Source: https://docs.xloud.tech/services/dns/create-zone

Provision authoritative DNS zones in Xloud DNS using the Dashboard or CLI. Configure primary and secondary zone types with TTL, email, and master server.

<iframe title="How to Create DNS Zones and Record Sets on Xloud" />

## Overview

A DNS zone is the authoritative domain boundary that contains all record sets for a
given domain (e.g., `example.com.`). Creating a zone in Xloud DNS registers that domain
with the platform's nameservers and makes it available for record management.

Zones are **project-scoped** — each zone belongs to the project that created it. Users
in other projects cannot see or modify your zones unless access is explicitly shared.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Zone Types

| Type          | Description                                                        | Use Case                                 |
| ------------- | ------------------------------------------------------------------ | ---------------------------------------- |
| **Primary**   | Your authoritative copy — you manage all records directly          | Standard zone management                 |
| **Secondary** | Read-only replica slaved from another DNS server via zone transfer | Disaster recovery, geographic redundancy |

***

## Create a Zone

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to DNS Zones">
        Navigate to
        **Network > DNS Zones**.

        Click **Create Zone** to open the creation dialog.
      </Step>

      <Step title="Enter the zone name">
        Enter the **Name** for your zone. This is the fully qualified domain name (FQDN).

        <Warning>
          The zone name **must end with a trailing dot** (e.g., `example.com.`).
          The API requires more than one label — use `example.com.` rather than
          `example.` for a valid zone.
        </Warning>
      </Step>

      <Step title="Add a description (optional)">
        Enter an optional **Description** to identify this zone in listings
        (e.g., "Production application zone").
      </Step>

      <Step title="Select the zone type">
        Choose the **Type** from the dropdown:

        | Type          | Description                                                           |
        | ------------- | --------------------------------------------------------------------- |
        | **Primary**   | Controlled by Xloud DNS — you manage the records directly             |
        | **Secondary** | Slaved from another DNS server — records are pulled via zone transfer |

        The form fields change dynamically based on your selection.

        <Note>
          Primary is the default and most common type. Select Secondary only when
          replicating an existing zone from an external DNS server.
        </Note>
      </Step>

      <Step title="Configure Primary zone fields">
        For **Primary** zones, these additional fields appear:

        **Email Address** (required) — The administrative contact email for this zone.
        Written into the SOA record. Must be a valid email address.

        **TTL (Time To Live)** (required) — Default TTL in seconds for records in this
        zone. Default value: `3600` (1 hour). Minimum: `0`.

        <Tip>
          Use `3600` for most production zones. Use a lower TTL (e.g., `300`) before
          planned DNS changes to reduce propagation delay, then increase it after the
          change is confirmed stable.
        </Tip>
      </Step>

      <Step title="Configure Secondary zone fields">
        For **Secondary** zones, the Email and TTL fields are hidden and a different
        field appears:

        **Masters** (required) — The IP addresses of the primary DNS servers to slave
        from. Click the add button to add one or more IP addresses.

        | Rule       | Detail                               |
        | ---------- | ------------------------------------ |
        | Minimum    | At least 1 master IP is required     |
        | Format     | Valid IPv4 or IPv6 address           |
        | Duplicates | Not allowed — each IP must be unique |

        <Note>
          The master servers must allow zone transfers (AXFR) to the Xloud DNS
          nameservers. Configure the allow-transfer ACL on your primary DNS server
          before creating the secondary zone.
        </Note>
      </Step>

      <Step title="Create the zone">
        Click **Confirm**. The zone enters **Pending** status during creation and
        transitions to **Active** within a few seconds.

        <Check>Zone appears in the DNS Zones list with status **Active**.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Create the zone">
        <CodeGroup>
          ```bash title="Create primary zone" theme={null}
          openstack zone create \
            --email admin@example.com \
            --ttl 3600 \
            --description "Production application zone" \
            example.com.
          ```

          ```bash title="Create secondary zone" theme={null}
          openstack zone create \
            --type SECONDARY \
            --masters 10.0.1.50 \
            --masters 10.0.1.51 \
            --description "Secondary replica from external DNS" \
            example.com.
          ```
        </CodeGroup>
      </Step>

      <Step title="Verify zone creation">
        ```bash title="Show zone status" theme={null}
        openstack zone show example.com.
        ```

        <Check>Zone status is `ACTIVE`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Zone Lifecycle

```mermaid theme={null}
stateDiagram-v2
    [*] --> PENDING : Create zone
    PENDING --> ACTIVE : Backend provisioned successfully
    PENDING --> ERROR : Backend provisioning failed
    ACTIVE --> PENDING : Zone update (TTL, email)
    ERROR --> PENDING : Retry after conflict resolution
    ACTIVE --> [*] : Zone deleted
```

***

## Edit a Zone

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the edit dialog">
        Navigate to **Network > DNS Zones**. Click the **Edit** action
        on the zone row.
      </Step>

      <Step title="Update editable fields">
        The zone **Name** and **Type** cannot be changed after creation. You can
        update:

        | Zone Type | Editable Fields                 |
        | --------- | ------------------------------- |
        | Primary   | Description, Email Address, TTL |
        | Secondary | Description, Masters            |
      </Step>

      <Step title="Save changes">
        Click **Confirm**. The zone enters **Pending** briefly during the update.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Update zone TTL" theme={null}
      openstack zone set --ttl 7200 example.com.
      ```

      ```bash title="Update zone email" theme={null}
      openstack zone set --email newadmin@example.com example.com.
      ```

      ```bash title="Update zone description" theme={null}
      openstack zone set --description "Updated zone" example.com.
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Delete a Zone

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Select zones to delete">
        Navigate to **Network > DNS Zones**. Select one or more zones
        using the checkboxes and click **Delete** in the batch actions bar.

        Alternatively, click the **More** menu on a zone row and select **Delete**.
        The **More** menu also includes **Create Record Set** — see
        [Manage Records](/services/dns/manage-records) for details.
      </Step>

      <Step title="Confirm deletion">
        Confirm the deletion in the dialog.

        <Danger>
          Deleting a zone removes all record sets within it permanently. DNS resolution
          for the domain will fail immediately. Ensure the zone is no longer in use
          before deletion.
        </Danger>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a zone" theme={null}
    openstack zone delete example.com.
    ```
  </Tab>
</Tabs>

***

## View Zone Details

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a zone name in the list to open the detail page. The **Overview** tab shows:

    | Section                | Fields                                                                            |
    | ---------------------- | --------------------------------------------------------------------------------- |
    | **Summary**            | Name, Description, Type (Primary/Secondary), Status (Active/Pending/Error), Email |
    | **Base Info**          | Action, Serial, TTL, Version                                                      |
    | **Modification Times** | Created At, Updated At, Transferred                                               |
    | **Attributes**         | Zone attributes (JSON)                                                            |
    | **Associations**       | Pool ID, Project ID, Masters (for secondary zones)                                |

    The **Record Sets** tab shows all records in the zone — see
    [Manage Records](/services/dns/manage-records).
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List all zones" theme={null}
      openstack zone list
      ```

      ```bash title="Show zone detail" theme={null}
      openstack zone show example.com.
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Manage Records" href="/services/dns/manage-records">
    Add A, AAAA, CNAME, MX, and TXT records to your zone
  </Card>

  <Card title="Record Types Reference" href="/services/dns/record-types">
    Full reference for every supported DNS record type
  </Card>

  <Card title="Reverse DNS" href="/services/dns/reverse-dns">
    Configure PTR records for your zone's IP addresses
  </Card>

  <Card title="Troubleshooting" href="/services/dns/troubleshooting">
    Resolve zone stuck in Pending or Error status
  </Card>
</CardGroup>


# External DNS Providers
Source: https://docs.xloud.tech/services/dns/external-providers

Configure Xloud DNS as a Service against external authoritative DNS backends — what's actively tested upstream, what's in tree but untested, and how to integrate enterprise DDI platforms.

## Overview

Xloud DNS as a Service uses Designate to manage DNS zones and records on behalf
of tenants. Designate has a pluggable backend architecture: the Designate
control plane handles tenant zones, while a backend driver propagates each
change to the external authoritative DNS server.

When a virtual machine is provisioned, Designate creates the matching A, PTR,
and CNAME records in the tenant's assigned zone. If an external DNS backend is
configured, the records are propagated to the external provider without manual
intervention.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Network connectivity from the Designate control plane nodes to the
    external DNS server's management interface
  * Credentials, TSIG keys, or API tokens for the external DNS provider as
    appropriate
</Note>

***

## Supported Backends

Designate ships several DNS server backend drivers. Two carry **Integrated**
status — they run in continuous CI and are the recommended path for production.
The rest are present in tree but officially **Untested** by upstream, which
means there is no automated regression net; deploy them only after your own
validation against your target version.

| Backend           |           Status           | Integration mechanism                                               |
| ----------------- | :------------------------: | ------------------------------------------------------------------- |
| **BIND9**         | **Integrated** (CI-tested) | RFC 2136 dynamic updates                                            |
| **PowerDNS 4**    | **Integrated** (CI-tested) | REST API (PowerDNS Authoritative Server 4.x or later)               |
| **Infoblox**      |    In tree, **Untested**   | XFR (zone transfer) — refreshed to use the official Infoblox client |
| **Akamai DNS v2** |    In tree, **Untested**   | XFR (zone transfer)                                                 |
| **NS1 DNS**       |    In tree, **Untested**   | XFR (zone transfer)                                                 |
| **DynECT**        |    In tree, **Untested**   | XFR (zone transfer)                                                 |
| **NSD4**          |    In tree, **Untested**   | XFR (zone transfer)                                                 |

Reference: [Designate DNS Server Driver Support Matrix](https://docs.openstack.org/designate/latest/admin/support-matrix.html).

<Warning>
  *Untested* does not mean broken — it means upstream has no automated coverage.
  Validate the target backend in a non-production environment before relying on
  it in customer deployments.
</Warning>

***

## Notes on commonly named providers

### Infoblox

In-tree backend, recently refreshed to use the official Infoblox client.
Officially **Untested** in the upstream support matrix. Suitable for enterprises
that want Infoblox to remain the authoritative DNS / DDI platform — but treat
the integration as customer-validated, not upstream-certified.

### Microsoft DNS / Active Directory DNS

There is no in-tree Designate driver for Microsoft DNS. Two practical paths:

1. **Front Designate with BIND9 or PowerDNS** as the integrated backend, then
   configure a one-way zone-transfer or sync from BIND or PowerDNS into
   Microsoft DNS using Microsoft's own zone-transfer support.
2. **Build a custom Designate backend** that calls the Microsoft DNS PowerShell
   cmdlets through WinRM — see
   [Custom Backend Drivers](#custom-backend-drivers) below.

### BlueCat

BlueCat does **not** ship a Designate backend driver. BlueCat maintains a
separate set of OpenStack integration components — see
[BlueCat OpenStack Drivers](https://github.com/bluecatlabs/bluecat-openstack-drivers) —
that integrate via Neutron IPAM and Nova/Neutron sync agents that route DNS
records into BlueCat Address Manager (BAM). In a BlueCat-integrated
deployment, the platform's DNS records flow through BAM and Designate is
typically not used.

### SolarWinds

SolarWinds is a network and infrastructure **monitoring** suite, not an
authoritative DNS server, so a Designate backend is not applicable. SolarWinds
can receive DNS-related metrics and events from the platform via standard
Prometheus / SNMP / syslog forwarders documented in the
[Monitoring](/services/monitoring) guides.

***

## Architecture

Designate's backend model separates the API and pool management layer from the
DNS protocol layer. Each backend driver handles zone synchronization between
Designate's internal state and the external authoritative server. Tenants
interact only with Designate APIs — the backend synchronization is transparent.

```
Tenant → Designate API → Pool Manager → Backend Driver → External DNS Server
                                             ↕
                                    (zone create / record sync)
```

A **pool** groups one or more DNS backend targets with shared configuration.
Each pool can contain multiple nameservers, and tenants can be assigned to
specific pools based on project or zone type.

***

## Backend Configuration (CI-tested backends)

All backend configuration lives in `designate.conf`. In XAVS deployments, this
file is managed by Ansible — set parameters via XDeploy globals and run
`xavs-ansible deploy --tags designate` to apply changes.

<AccordionGroup>
  <Accordion title="BIND9 — RFC 2136 dynamic updates" icon="globe">
    BIND9 integration uses the standard DNS dynamic update protocol (RFC 2136).
    A TSIG key authenticates updates from Designate to BIND.

    **On the BIND server — generate a TSIG key:**

    ```bash title="Generate TSIG key on the BIND server" theme={null}
    tsig-keygen -a HMAC-SHA256 designate-key
    ```

    Add the output to `/etc/bind/named.conf.keys` and configure the zone to
    accept updates:

    ```bash title="/etc/bind/named.conf — allow dynamic updates" theme={null}
    zone "example.com" {
        type master;
        file "/var/cache/bind/example.com.db";
        allow-update { key designate-key; };
    };
    ```

    **Designate configuration:**

    ```ini title="designate.conf — BIND9 backend" theme={null}
    [backend:bind9]
    host = 10.0.10.5
    port = 53
    tsig_key_name = designate-key
    tsig_key_secret = <base64-encoded-key>
    tsig_key_algorithm = HMAC-SHA256

    [pool:default]
    backends = bind9
    nameservers = ns1.example.com
    also_notifies = 10.0.10.5:53
    ```
  </Accordion>

  <Accordion title="PowerDNS 4 — pdns4 driver" icon="server">
    The `pdns4` driver targets PowerDNS Authoritative Server 4.x or later.

    **PowerDNS server — enable the API in `pdns.conf`:**

    ```ini title="pdns.conf — enable REST API" theme={null}
    api=yes
    api-key=<your-api-key>
    webserver=yes
    webserver-address=0.0.0.0
    webserver-port=8081
    ```

    **Designate configuration:**

    ```ini title="designate.conf — PowerDNS backend" theme={null}
    [backend:pdns4]
    host = 10.0.10.6
    port = 8081
    api_endpoint = http://10.0.10.6:8081
    api_token = <your-api-key>

    [pool:default]
    backends = pdns4
    nameservers = ns1.example.com
    ```
  </Accordion>
</AccordionGroup>

<Tip>
  In-tree backends marked *Untested* (Infoblox, Akamai v2, NS1, DynECT, NSD4)
  follow the same `designate.conf` pattern — refer to the upstream
  [Designate documentation](https://docs.openstack.org/designate/latest/) for
  per-driver configuration keys. Validate every option in a non-production
  environment before relying on these in customer deployments.
</Tip>

***

## Custom Backend Drivers

If your enterprise DNS / DDI provider does not have an in-tree Designate
backend — or you want closer integration with a vendor's REST API rather than
relying on zone-transfer — you can develop a **custom backend driver**.
Designate exposes a stable Python interface for plug-in drivers, so a custom
driver lives outside the Designate codebase and is loaded at runtime.

### When to build a custom driver

* The vendor exposes a modern REST / WAPI / GraphQL management API and you
  want native record creation rather than XFR-based propagation.
* You need richer record types (DDI-managed ranges, vendor-specific
  attributes) that XFR does not carry.
* You want operational hooks (audit logging, change-control workflows,
  approval gates) inside the driver itself.
* The vendor's product (for example, Microsoft DNS, BlueCat) does not have an
  in-tree Designate backend today and you want platform tenants to use their
  Designate APIs unchanged.

### What's involved

A custom driver is a Python class that implements
`designate.backend.base.Backend`, packaged as a setuptools entry point under
the `designate.backend` group. The driver implements zone create / update /
delete and record create / update / delete handlers, and uses the vendor's
SDK or REST API to apply each change to the authoritative DNS provider.

The driver is registered in `designate.conf` exactly like an in-tree backend
(`backends = my-vendor-driver`), and Designate loads it at startup. There are
no upstream-Designate code changes required.

### Maintenance trade-off

A custom driver is **customer-owned** — upstream Designate releases will not
automatically validate against it. The integration must be re-tested at each
Designate upgrade. Plan for a small recurring engineering investment in
exchange for the closer integration.

### Hybrid approach (often the most pragmatic)

When a vendor doesn't fit cleanly into either *XFR-based in-tree backend* or
*custom-driver*, a common middle ground is:

1. Use **BIND9** or **PowerDNS 4** as the Designate-integrated backend
   (full upstream CI coverage).
2. Configure the enterprise DNS / DDI platform (Infoblox, Microsoft DNS,
   BlueCat) to **pull zones from BIND or PowerDNS** via standard zone
   transfer or via the vendor's own sync / gateway product.

This keeps the platform on a CI-tested integration path and uses the vendor's
own tooling for the leg of the path the vendor knows best.

***

## Zone Transfer to External Provider

When a zone is created in Designate, the backend driver provisions it on the
configured external DNS server. Zone transfers can also be initiated manually
for bulk migration of existing zones.

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    <Steps>
      <Step title="Create a zone" icon="plus">
        Navigate to **Network → DNS Zones** and click **Create Zone**.

        Enter the zone name (e.g., `example.com.`), email, TTL, and select the
        pool if multiple pools are configured.
      </Step>

      <Step title="Verify propagation" icon="circle-check">
        After the zone reaches **Active** status, query the external DNS server
        to confirm the zone is visible:

        ```bash title="Query external DNS server for zone" theme={null}
        dig @10.0.10.5 SOA example.com.
        ```
      </Step>

      <Step title="Add records" icon="edit">
        Navigate to the zone and click **Create Record Set**. Records created
        here are automatically propagated to the external backend.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Source your credentials file to authenticate with the Xloud platform:

    ```bash title="Load credentials" theme={null}
    source openrc.sh
    ```

    <Tip>
      Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
    </Tip>

    ```bash title="Create a zone" theme={null}
    openstack zone create \
      --email admin@example.com \
      --ttl 300 \
      example.com.
    ```

    ```bash title="Check zone status" theme={null}
    openstack zone show example.com.
    ```

    ```bash title="Add an A record" theme={null}
    openstack recordset create \
      --records 10.0.1.100 \
      --type A \
      --ttl 300 \
      example.com. \
      web01.example.com.
    ```

    ```bash title="Verify propagation to external DNS" theme={null}
    dig @10.0.10.5 A web01.example.com.
    ```
  </Tab>
</Tabs>

***

## Pool Management

Pools allow routing different zones or tenants to different DNS backends. For
example, internal zones can be handled by BIND9 while external zones use
PowerDNS or a vendor backend.

```ini title="designate.conf — multiple pools" theme={null}
[pools]
names = internal-pool,external-pool

[pool:internal-pool]
backends = bind9
nameservers = ns1.internal.example.com,ns2.internal.example.com
also_notifies = 10.0.10.5:53

[pool:external-pool]
backends = pdns4
nameservers = ns1.example.com,ns2.example.com
```

```bash title="Create a zone in a specific pool" theme={null}
openstack zone create \
  --email admin@example.com \
  --attributes pool_id:<pool-uuid> \
  external.example.com.
```

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Zone stuck in PENDING status" icon="clock">
    **Cause**: Designate cannot reach the external backend, or authentication
    failed.

    **Resolution**:

    * Verify network connectivity from the Designate control node to the
      backend IP and port
    * Check Designate worker logs: `docker logs designate_worker`
    * Confirm credentials (API key, TSIG key, or password) are correct in
      `designate.conf`
  </Accordion>

  <Accordion title="Records not appearing on external DNS server" icon="triangle-alert">
    **Cause**: The zone may be active in Designate but the backend sync failed.

    **Resolution**:

    * Force a zone sync: `openstack zone sync <zone-name>`
    * Check Designate producer and worker logs for backend errors
    * Verify the external DNS server has the zone configured to accept updates
      from the Designate source IP
  </Accordion>

  <Accordion title="BIND9 update refused — TSIG verification failure" icon="lock">
    **Cause**: The TSIG key in `designate.conf` does not match the key
    configured on the BIND9 server.

    **Resolution**:

    * Re-generate and re-sync the TSIG key on both BIND9 and Designate
    * Confirm the key algorithm (`HMAC-SHA256`) matches on both sides
    * Test manually: `nsupdate -k /etc/bind/designate.key`
  </Accordion>

  <Accordion title="Custom driver fails to load at startup" icon="circle-x">
    **Cause**: The driver is not registered as a setuptools entry point under
    the `designate.backend` group, or the Python package is not installed in
    the Designate virtualenv.

    **Resolution**:

    * Verify the entry point: `pip show <your-driver-package>` should list it
      under entry points
    * Confirm the driver class implements every abstract method of
      `designate.backend.base.Backend`
    * Restart `designate-worker` and `designate-producer` after installing or
      updating the driver
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Backend Configuration" href="/services/dns/backend-config">
    Full reference for Designate backend driver options and pool configuration
  </Card>

  <Card title="Pool Management" href="/services/dns/pool-management">
    Manage multiple backend pools for routing zones to different DNS providers
  </Card>

  <Card title="DNS Security" href="/services/dns/security">
    Configure TSIG keys, DNSSEC, and zone access policies
  </Card>

  <Card title="Designate Support Matrix (upstream)" href="https://docs.openstack.org/designate/latest/admin/support-matrix.html">
    Authoritative upstream list of in-tree backends and their CI test status
  </Card>
</CardGroup>


# Manage DNS Record Sets
Source: https://docs.xloud.tech/services/dns/manage-records

Create, update, and delete DNS record sets in Xloud DNS zones. Covers all 12 record types with the exact Dashboard form fields and CLI commands.

## Overview

Record sets are the DNS entries within a zone that map hostnames to values. Each record
set has a type (A, CNAME, MX, etc.), a fully qualified name, a TTL, and one or more
values. Record sets are project-scoped through their parent zone.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  A zone must be in **Active** status before record sets can be created within it. See
  [Create a Zone](/services/dns/create-zone) if you have not yet provisioned a zone.
</Note>

***

## Create a Record Set

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    The Create Record Set form provides dynamic help text and validation based on
    the selected record type.

    <Steps>
      <Step title="Open the zone">
        Navigate to **Network > DNS Zones** and click a zone name to
        open the detail page. Go to the **Record Sets** tab.

        Click **Create Record Set**.
      </Step>

      <Step title="Select the record type">
        Choose the **Type** from the dropdown. This selection controls the
        validation rules and format hints for the Records field.

        | Type      | Description                           |
        | --------- | ------------------------------------- |
        | **A**     | Address Record — maps to IPv4 address |
        | **AAAA**  | IPv6 Address Record                   |
        | **CAA**   | Certificate Authority Authorization   |
        | **CNAME** | Canonical Name (alias)                |
        | **MX**    | Mail Exchange                         |
        | **NS**    | Name Server delegation                |
        | **PTR**   | Pointer Record (reverse DNS)          |
        | **SOA**   | Start Of Authority                    |
        | **SPF**   | Sender Policy Framework               |
        | **SRV**   | Service Locator                       |
        | **SSHFP** | SSH Public Key Fingerprint            |
        | **TXT**   | Text Record                           |

        The default selection is **A**.

        <Note>
          The Type field cannot be changed after creation. If you need a different
          type, delete the record set and create a new one.
        </Note>
      </Step>

      <Step title="Enter the record name">
        Enter the **Name** — the fully qualified hostname for this record.

        <Warning>
          The name **must end with a trailing dot** (e.g., `www.example.com.`).
          Omitting the trailing dot will cause a validation error.
        </Warning>

        The form shows a format example based on the selected type:

        | Type                                            | Name Example                                              |
        | ----------------------------------------------- | --------------------------------------------------------- |
        | A, AAAA, CAA, MX, NS, PTR, SOA, SPF, SSHFP, TXT | `example.com.`                                            |
        | CNAME                                           | `first.example.com.`                                      |
        | SRV                                             | `_sip._tcp.example.com.` (with protocol and service name) |
      </Step>

      <Step title="Add a description (optional)">
        Enter an optional **Description** for this record set.
      </Step>

      <Step title="Set the TTL">
        Enter the **TTL (Time To Live)** in seconds. Default: `3600` (1 hour).

        This controls how long resolvers cache this record before re-querying.
      </Step>

      <Step title="Add record values">
        Click the add button in the **Records** field to add one or more values.
        At least one record value is required.

        The form shows format examples and validation rules based on the selected type:

        | Type      | Format Example                                                       | Validation                                                       |
        | --------- | -------------------------------------------------------------------- | ---------------------------------------------------------------- |
        | **A**     | `192.168.1.1`                                                        | Must be a valid IPv4 address                                     |
        | **AAAA**  | `2001:db8:3333:4444:5555:6666:7777:8888`                             | Must be a valid IPv6 address                                     |
        | **CAA**   | `0 iodef mailto:security@example.com`                                | Flag (0-255), tag (issue/issuewild/iodef), value                 |
        | **CNAME** | `other-example.com`                                                  | Target FQDN                                                      |
        | **MX**    | `10 mail.example.com`                                                | Priority number followed by mail server FQDN                     |
        | **NS**    | `ns1.example.com`                                                    | Nameserver FQDN                                                  |
        | **PTR**   | `1.1.0.192.in-addr.arpa.`                                            | Reverse FQDN                                                     |
        | **SOA**   | `ns1.example.com admin.example.com 2013022001 86400 7200 604800 300` | Primary NS, admin email, serial, refresh, retry, expire, minimum |
        | **SPF**   | `"v=spf1 ipv4=192.1.1.1 include:examplesender.email +all"`           | SPF policy string                                                |
        | **SRV**   | `10 0 5060 server1.example.com.`                                     | Priority, weight, port, target FQDN                              |
        | **SSHFP** | `4 2 123456789abcdef...`                                             | Algorithm, fingerprint type, hex fingerprint                     |
        | **TXT**   | *(no format hint shown)*                                             | Any text string                                                  |

        <Tip>
          Add multiple record values for round-robin distribution (e.g., multiple
          A records for the same hostname). Each value is returned to resolvers
          in rotating order.
        </Tip>
      </Step>

      <Step title="Create the record set">
        Click **Confirm**. The record set enters **Pending** momentarily and then
        transitions to **Active**.

        <Check>Record set appears in the zone's Record Sets tab with status **Active**.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Create A record" theme={null}
      openstack recordset create \
        --type A \
        --record 192.168.1.10 \
        --record 192.168.1.11 \
        --ttl 3600 \
        example.com. www
      ```

      ```bash title="Create AAAA record" theme={null}
      openstack recordset create \
        --type AAAA \
        --record 2001:db8::1 \
        --ttl 3600 \
        example.com. www
      ```

      ```bash title="Create CNAME record" theme={null}
      openstack recordset create \
        --type CNAME \
        --record app.example.com. \
        example.com. api
      ```

      ```bash title="Create MX record" theme={null}
      openstack recordset create \
        --type MX \
        --record "10 mail.example.com." \
        --record "20 mail-backup.example.com." \
        example.com. @
      ```

      ```bash title="Create TXT record (SPF)" theme={null}
      openstack recordset create \
        --type TXT \
        --record '"v=spf1 include:_spf.example.com ~all"' \
        example.com. @
      ```

      ```bash title="Create SRV record" theme={null}
      openstack recordset create \
        --type SRV \
        --record "10 20 5060 sip.example.com." \
        example.com. _sip._tcp
      ```

      ```bash title="Create CAA record" theme={null}
      openstack recordset create \
        --type CAA \
        --record '0 issue "letsencrypt.org"' \
        example.com. @
      ```

      ```bash title="Create NS record (subdomain delegation)" theme={null}
      openstack recordset create \
        --type NS \
        --record "ns1.partner.com." \
        --record "ns2.partner.com." \
        example.com. sub
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## View Record Sets

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="View the record list">
        Navigate to a zone's detail page and go to the **Record Sets** tab. The list shows:

        | Column         | Description                                               |
        | -------------- | --------------------------------------------------------- |
        | **ID/Name**    | Record set FQDN (clickable to view details)               |
        | **Type**       | Record type with description (e.g., "A - Address Record") |
        | **Records**    | All values displayed as individual tags                   |
        | **Status**     | Active, Pending, or Error                                 |
        | **Created At** | Creation timestamp                                        |
      </Step>

      <Step title="Filter records">
        Use the search/filter bar to narrow the list:

        | Filter     | Options                           |
        | ---------- | --------------------------------- |
        | **Name**   | Text search by record name        |
        | **Type**   | Dropdown with all 12 record types |
        | **Status** | Active, Pending, Error            |
      </Step>

      <Step title="View record detail">
        Click a record name to open the detail page showing:

        | Section                | Fields                                             |
        | ---------------------- | -------------------------------------------------- |
        | **Summary**            | Name, Description, Type (with description), Status |
        | **Base Info**          | Action, Records (all values listed), TTL, Version  |
        | **Modification Times** | Created At, Updated At                             |
        | **Associations**       | Zone ID, Zone Name, Project ID                     |
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List all record sets in a zone" theme={null}
      openstack recordset list example.com.
      ```

      ```bash title="Show a specific record set" theme={null}
      openstack recordset show example.com. www
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Update a Record Set

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the update dialog">
        Navigate to the zone's **Record Sets** tab. Click the **Update** action
        on the record row, or open the record detail page and click **Update**.
      </Step>

      <Step title="Edit the record">
        The **Name** and **Type** fields are locked and cannot be changed. You can
        update:

        * **Description**
        * **TTL**
        * **Records** — add, remove, or modify values

        <Note>
          To change a record's type (e.g., A to AAAA), delete the existing record
          set and create a new one with the correct type.
        </Note>
      </Step>

      <Step title="Save the update">
        Click **Confirm**. The record set enters **Pending** briefly during the update.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Update record value" theme={null}
    openstack recordset set \
      --record 192.168.1.20 \
      example.com. www
    ```

    ```bash title="Update TTL only" theme={null}
    openstack recordset set \
      --ttl 300 \
      example.com. www
    ```

    <Tip>
      Use a low TTL (e.g., 300 seconds) before planned IP changes, then increase it after
      the change is confirmed stable. This minimizes propagation delay during migrations.
    </Tip>
  </Tab>
</Tabs>

***

## Delete Record Sets

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Select records to delete">
        Navigate to the zone's **Record Sets** tab. Select one or more records using
        the checkboxes and click **Delete** in the batch actions bar.

        Alternatively, click the **More** menu on a record row and select **Delete**.
      </Step>

      <Step title="Confirm deletion">
        Confirm the deletion in the dialog. The confirmation shows both the record name
        and ID.

        <Warning>
          Deleting a record set is immediate on the server side. Resolvers that have
          cached the record will continue returning the value until their cache TTL expires.
        </Warning>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a record set" theme={null}
    openstack recordset delete example.com. www
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Record Types Reference" href="/services/dns/record-types">
    Detailed reference for every supported record type and its value format
  </Card>

  <Card title="Reverse DNS" href="/services/dns/reverse-dns">
    Configure PTR records for IP-to-hostname resolution
  </Card>

  <Card title="Troubleshooting" href="/services/dns/troubleshooting">
    Resolve propagation delays and CNAME conflicts
  </Card>

  <Card title="Create a Zone" href="/services/dns/create-zone">
    Provision a new authoritative zone for a domain
  </Card>
</CardGroup>


# DNS Pool Management
Source: https://docs.xloud.tech/services/dns/pool-management

Manage Xloud DNS nameserver pools — configure pool attributes for zone scheduling, monitor nameserver health, and add or remove nameservers from pools.

## Overview

DNS pools group nameserver backends that process zone data. Multiple pools support
geographic distribution or tiered service levels. Pool attributes control automatic
zone scheduling — when a zone is created, the DNS service selects a pool based on
attribute matching policies.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Pool Attributes and Zone Scheduling

Pools use key-value attributes to match zones during scheduling. When a zone is created
without an explicit pool assignment, the DNS service selects a pool based on these
attributes.

| Attribute      | Example Values         | Purpose                                    |
| -------------- | ---------------------- | ------------------------------------------ |
| `service_tier` | `standard`, `premium`  | Route zones to capacity-appropriate pools  |
| `region`       | `east`, `west`         | Assign zones to geographically local pools |
| `tenant_type`  | `internal`, `external` | Separate internal and external zones       |

Configure pool attributes through XDeploy service configuration files.

***

## View Pool Information

<CodeGroup>
  ```bash title="List all pools" theme={null}
  openstack zone pool list
  ```

  ```bash title="Show pool detail with nameservers and targets" theme={null}
  openstack zone pool show <pool-id>
  ```

  ```bash title="Check zone's pool assignment" theme={null}
  openstack zone show <zone-name> -c attributes
  ```
</CodeGroup>

***

## Nameserver Health Monitoring

<Tabs>
  <Tab title="Verify authoritative responses" icon="heart-pulse">
    Monitor the health of pool nameservers by verifying they are authoritatively
    answering queries for managed zones:

    ```bash title="Verify nameserver is authoritative" theme={null}
    dig @<nameserver-ip> example.com. SOA +norecurse
    ```

    A healthy nameserver returns the SOA record with `AUTHORITY: 1`. If the nameserver
    returns `SERVFAIL` or does not respond, check the DNS service worker logs for
    synchronization errors.
  </Tab>

  <Tab title="Check SOA serial consistency" icon="refresh-cw">
    All nameservers in a pool should have the same SOA serial number for a given zone:

    ```bash title="Check SOA serial on each nameserver" theme={null}
    dig @<ns1-ip> example.com. SOA +short
    dig @<ns2-ip> example.com. SOA +short
    ```

    Mismatched serials indicate a synchronization lag. Check DNS worker logs for push
    errors to the affected nameserver's target configuration.
  </Tab>
</Tabs>

***

## Adding Nameservers to a Pool

Adding nameservers requires updating the pool configuration through XDeploy. After
adding a nameserver:

<Steps>
  <Step title="Update pool configuration in XDeploy">
    Add the new nameserver to the pool's `nameservers` and `targets` lists in the
    XDeploy DNS service configuration.
  </Step>

  <Step title="Deploy the configuration">
    Apply the updated configuration:

    ```bash title="Deploy DNS service configuration" theme={null}
    xavs-ansible deploy -t dns
    ```
  </Step>

  <Step title="Verify the new nameserver receives zone data">
    ```bash title="Verify zone synchronization on new nameserver" theme={null}
    dig @<new-ns-ip> example.com. SOA +norecurse
    ```

    Wait for the zone data to synchronize before proceeding.
  </Step>

  <Step title="Update external NS delegation records">
    Update the NS delegation records at the domain registrar to include the new
    nameserver. Allow 48 hours for resolver caches to propagate the new NS records globally.
  </Step>
</Steps>

<Warning>
  Do not remove an existing nameserver from a pool until the new nameserver is fully
  synchronized and the old NS records have expired from resolver caches (typically 48 hours).
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Backend Configuration" href="/services/dns/backend-config">
    Configure backend driver targets for each pool
  </Card>

  <Card title="Zone Transfers" href="/services/dns/zone-transfers">
    Manage zone transfer requests to secondary nameservers
  </Card>

  <Card title="Quotas" href="/services/dns/quotas">
    Set per-project DNS resource limits
  </Card>

  <Card title="Admin Troubleshooting" href="/services/dns/admin-troubleshooting">
    Diagnose nameserver synchronization failures
  </Card>
</CardGroup>


# DNS Quotas
Source: https://docs.xloud.tech/services/dns/quotas

Manage per-project DNS resource limits in Xloud DNS — view, set, and reset quotas for zones, record sets, records, and PTR records.

## Overview

DNS quotas prevent individual projects from consuming excessive DNS resources. Default
values are set platform-wide; administrators override them per project. Quota enforcement
is automatic — operations that would exceed a limit return a `413` or `429` error.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Default Quota Reference

| Resource            | Default Limit | Description                                 |
| ------------------- | ------------- | ------------------------------------------- |
| `zones`             | 10            | DNS zones per project                       |
| `zone_records`      | 500           | Total records across all zones in a project |
| `zone_recordsets`   | 500           | Total record sets across all zones          |
| `recordset_records` | 20            | Records within a single record set          |
| `ptr_records`       | 30            | PTR records per project                     |

***

## View Quotas

<Tabs>
  <Tab title="Current project" icon="eye">
    ```bash title="Show quotas for the current project" theme={null}
    openstack quota show --dns
    ```
  </Tab>

  <Tab title="Specific project (admin)" icon="shield">
    ```bash title="Show quotas for a specific project" theme={null}
    openstack quota show <project-id> --dns
    ```
  </Tab>
</Tabs>

***

## Set Quotas

```bash title="Set DNS quotas for a project" theme={null}
openstack quota set \
  --dns-zones 50 \
  --dns-zone-records 2000 \
  --dns-zone-recordsets 2000 \
  --dns-recordset-records 50 \
  <project-id>
```

```bash title="Reset to platform defaults" theme={null}
openstack quota delete <project-id> --dns
```

<Tip>
  Large projects with automated DNS management (e.g., service-discovery-heavy
  environments) may need `zone_recordsets` quotas in the thousands. Review usage
  regularly and right-size allocations based on actual consumption.
</Tip>

***

## Monitor Quota Usage

Check current usage against limits before planning quota changes:

```bash title="Show zone count for a project" theme={null}
openstack zone list --project <project-id> -c name | wc -l
```

```bash title="Count total record sets across all zones" theme={null}
for zone in $(openstack zone list --project <project-id> -f value -c name); do
  openstack recordset list "$zone" -f value -c name
done | wc -l
```

***

## Next Steps

<CardGroup>
  <Card title="Pool Management" href="/services/dns/pool-management">
    Manage nameserver infrastructure that processes zone data
  </Card>

  <Card title="Security" href="/services/dns/security">
    Apply DNS security hardening policies
  </Card>

  <Card title="Admin Troubleshooting" href="/services/dns/admin-troubleshooting">
    Diagnose quota-related errors and service issues
  </Card>

  <Card title="DNS User Guide" href="/services/dns/user-guide">
    User-facing zone and record management
  </Card>
</CardGroup>


# DNS Record Types Reference
Source: https://docs.xloud.tech/services/dns/record-types

Complete reference for the 12 DNS record types supported by Xloud DNS — A, AAAA, CAA, CNAME, MX, NS, PTR, SOA, SPF, SRV, SSHFP, and TXT with format examples.

## Overview

Xloud DNS supports 12 standard DNS record types. Each type serves a specific purpose
in DNS resolution. This reference covers the supported types, their value format,
and usage examples for both Dashboard and CLI.

<Note>
  When creating records in the Dashboard, the form provides format hints and
  validation for each type automatically. See [Manage Records](/services/dns/manage-records)
  for the step-by-step creation workflow.
</Note>

***

## Quick Reference

| Record Type | Full Name                                  | Purpose                                     |
| ----------- | ------------------------------------------ | ------------------------------------------- |
| **A**       | Address Record                             | Maps hostname to IPv4 address               |
| **AAAA**    | IPv6 Address Record                        | Maps hostname to IPv6 address               |
| **CAA**     | Certificate Authority Authorization Record | Controls which CAs can issue certificates   |
| **CNAME**   | Canonical Name Record                      | Hostname alias pointing to another name     |
| **MX**      | Mail Exchange Record                       | Routes email to mail servers                |
| **NS**      | Name Server                                | Delegates a subdomain to other nameservers  |
| **PTR**     | Pointer Record                             | Reverse DNS — maps IP to hostname           |
| **SOA**     | Start Of Authority                         | Zone authority metadata (auto-managed)      |
| **SPF**     | Sender Policy Framework                    | Email sender validation policy              |
| **SRV**     | Service Locator                            | Service discovery with host, port, priority |
| **SSHFP**   | SSH Public Key Fingerprint                 | Publishes SSH host key fingerprints         |
| **TXT**     | Text Record                                | Arbitrary text data                         |

***

## Record Type Details

<AccordionGroup>
  <Accordion title="A — Address Record">
    Maps a hostname to one or more IPv4 addresses. The most common record type.

    | Field               | Value                                        |
    | ------------------- | -------------------------------------------- |
    | **Format**          | IPv4 address                                 |
    | **Example**         | `192.168.1.1`                                |
    | **Validation**      | Must be a valid IPv4 address                 |
    | **Multiple values** | Supported — creates round-robin distribution |

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **A**, enter the FQDN (e.g., `www.example.com.`), and add one or
        more IPv4 addresses in the Records field.
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Create A record with multiple values" theme={null}
        openstack recordset create \
          --type A \
          --record 192.168.1.10 \
          --record 192.168.1.11 \
          --ttl 3600 \
          example.com. www
        ```
      </Tab>
    </Tabs>

    <Tip>
      Multiple A records for the same name create DNS round-robin distribution — each
      resolver query cycles through the values. This is not a replacement for a proper
      load balancer but useful for simple traffic spreading.
    </Tip>
  </Accordion>

  <Accordion title="AAAA — IPv6 Address Record">
    Maps a hostname to an IPv6 address. Identical to A records but uses IPv6 notation.

    | Field          | Value                                    |
    | -------------- | ---------------------------------------- |
    | **Format**     | Full or abbreviated IPv6 address         |
    | **Example**    | `2001:db8:3333:4444:5555:6666:7777:8888` |
    | **Validation** | Must be a valid IPv6 address             |

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **AAAA** and enter one or more IPv6 addresses in the Records field.
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Create AAAA record" theme={null}
        openstack recordset create \
          --type AAAA \
          --record 2001:db8::1 \
          --ttl 3600 \
          example.com. www
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="CAA — Certificate Authority Authorization">
    Restricts which Certificate Authorities can issue TLS certificates for the domain.
    Supported by major CAs to prevent unauthorized certificate issuance.

    | Field       | Value                                 |
    | ----------- | ------------------------------------- |
    | **Format**  | `<flags> <tag> "<value>"`             |
    | **Example** | `0 iodef mailto:security@example.com` |

    **Flags**: `0` = non-critical (CA may issue even if tag is unknown), `128` = critical (CA must not issue if tag is unknown)

    **Tags**:

    | Tag         | Purpose                                                                   |
    | ----------- | ------------------------------------------------------------------------- |
    | `issue`     | Authorizes a CA to issue certificates for this domain                     |
    | `issuewild` | Authorizes a CA to issue wildcard certificates                            |
    | `iodef`     | Reporting URI for policy violations (e.g., `mailto:security@example.com`) |

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **CAA** and enter the flag, tag, and value in the Records field
        (e.g., `0 issue "letsencrypt.org"`).
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Allow only Let's Encrypt" theme={null}
        openstack recordset create \
          --type CAA \
          --record '0 issue "letsencrypt.org"' \
          --record '0 iodef "mailto:security@example.com"' \
          example.com. @
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="CNAME — Canonical Name (Alias)">
    Creates an alias that points one hostname to another. The target must resolve
    to an A or AAAA record.

    | Field            | Value                       |
    | ---------------- | --------------------------- |
    | **Format**       | Fully qualified domain name |
    | **Example**      | `other-example.com`         |
    | **Name example** | `first.example.com.`        |

    <Warning>
      CNAME records cannot coexist with other record types at the same name, and cannot
      be created at the zone apex (`@`). Use an A record for the root domain hostname.
    </Warning>

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **CNAME** and enter the target FQDN in the Records field.
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Create CNAME record" theme={null}
        openstack recordset create \
          --type CNAME \
          --record app.example.com. \
          example.com. api
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="MX — Mail Exchange">
    Specifies the mail servers responsible for receiving email for the domain.
    Multiple MX records with different priorities enable failover.

    | Field       | Value                   |
    | ----------- | ----------------------- |
    | **Format**  | `<priority> <hostname>` |
    | **Example** | `10 mail.example.com`   |

    Lower priority numbers have higher precedence. If the primary fails, resolvers
    try the next lowest priority.

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **MX** and enter each record as `priority hostname.`
        (e.g., `10 mail.example.com.`).
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Create MX records with primary and backup" theme={null}
        openstack recordset create \
          --type MX \
          --record "10 mail.example.com." \
          --record "20 mail-backup.example.com." \
          example.com. @
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="NS — Name Server">
    Delegates a subdomain to a different set of nameservers. Zone apex NS records
    are auto-managed — do not modify them manually.

    | Field       | Value             |
    | ----------- | ----------------- |
    | **Format**  | Nameserver FQDN   |
    | **Example** | `ns1.example.com` |

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **NS** and enter nameserver FQDNs in the Records field.
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Delegate a subdomain" theme={null}
        openstack recordset create \
          --type NS \
          --record "ns1.partner.com." \
          --record "ns2.partner.com." \
          example.com. sub
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="PTR — Pointer Record (Reverse DNS)">
    Maps an IP address back to a hostname. PTR records live in reverse zones
    (`in-addr.arpa.` for IPv4, `ip6.arpa.` for IPv6).

    | Field       | Value                     |
    | ----------- | ------------------------- |
    | **Format**  | Reverse FQDN              |
    | **Example** | `1.1.0.192.in-addr.arpa.` |

    <Note>
      For floating IP reverse DNS, use the dedicated **Reverse DNS** interface
      at **Network > DNS Reverse** instead of creating PTR records
      manually. See [Reverse DNS](/services/dns/reverse-dns).
    </Note>
  </Accordion>

  <Accordion title="SOA — Start Of Authority">
    Defines zone authority parameters. SOA records are auto-managed by the DNS
    service — you typically do not create or modify them directly.

    | Field       | Value                                                                      |
    | ----------- | -------------------------------------------------------------------------- |
    | **Format**  | `<primary-ns> <admin-email> <serial> <refresh> <retry> <expire> <minimum>` |
    | **Example** | `ns1.example.com admin.example.com 2013022001 86400 7200 604800 300`       |

    | Component   | Description                                   |
    | ----------- | --------------------------------------------- |
    | Primary NS  | Primary nameserver for the zone               |
    | Admin email | Contact email (with `.` instead of `@`)       |
    | Serial      | Version number, incremented on each change    |
    | Refresh     | Seconds between slave refresh checks          |
    | Retry       | Seconds between retry after failed refresh    |
    | Expire      | Seconds before slave stops serving stale data |
    | Minimum     | Negative caching TTL                          |
  </Accordion>

  <Accordion title="SPF — Sender Policy Framework">
    Defines which mail servers are authorized to send email for the domain.
    Helps prevent email spoofing.

    | Field       | Value                                                      |
    | ----------- | ---------------------------------------------------------- |
    | **Format**  | SPF policy string                                          |
    | **Example** | `"v=spf1 ipv4=192.1.1.1 include:examplesender.email ~all"` |

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **SPF** and enter the SPF policy string in the Records field.
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Create SPF record" theme={null}
        openstack recordset create \
          --type SPF \
          --record '"v=spf1 include:_spf.example.com ~all"' \
          example.com. @
        ```
      </Tab>
    </Tabs>

    <Tip>
      Modern practice is to publish SPF data as a TXT record rather than the
      dedicated SPF type. Most mail systems check TXT records for SPF. Consider
      creating both for maximum compatibility.
    </Tip>
  </Accordion>

  <Accordion title="SRV — Service Locator">
    Specifies the host and port for a specific service. Used by SIP, XMPP, LDAP,
    and other service-discovery protocols.

    | Field           | Value                                                              |
    | --------------- | ------------------------------------------------------------------ |
    | **Format**      | `<priority> <weight> <port> <target>.`                             |
    | **Example**     | `10 0 5060 server1.example.com.`                                   |
    | **Name format** | `_service._protocol.example.com.` (e.g., `_sip._tcp.example.com.`) |

    | Component | Description                                                          |
    | --------- | -------------------------------------------------------------------- |
    | Priority  | Lower = higher priority (like MX)                                    |
    | Weight    | Load balancing between same-priority targets (higher = more traffic) |
    | Port      | TCP/UDP port number for the service                                  |
    | Target    | FQDN of the server hosting the service                               |

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **SRV**. Enter the name as `_service._protocol.domain.`
        (e.g., `_sip._tcp.example.com.`) and the record as `priority weight port target.`
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Create SIP SRV record" theme={null}
        openstack recordset create \
          --type SRV \
          --record "10 20 5060 sip.example.com." \
          example.com. _sip._tcp
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="SSHFP — SSH Public Key Fingerprint">
    Publishes SSH host key fingerprints in DNS, allowing SSH clients to verify
    host keys via DNSSEC-secured lookups.

    | Field       | Value                                                                  |
    | ----------- | ---------------------------------------------------------------------- |
    | **Format**  | `<algorithm> <fingerprint-type> <hex-fingerprint>`                     |
    | **Example** | `4 2 123456789abcdef67890123456789abcdef67890123456789abcdef123456789` |

    | Algorithm | Key Type |
    | --------- | -------- |
    | 1         | RSA      |
    | 2         | DSA      |
    | 3         | ECDSA    |
    | 4         | Ed25519  |
    | 6         | Ed448    |

    | Fingerprint Type | Hash    |
    | ---------------- | ------- |
    | 1                | SHA-1   |
    | 2                | SHA-256 |

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **SSHFP** and enter the algorithm, fingerprint type, and hex
        fingerprint in the Records field.
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```bash title="Generate and create SSHFP record" theme={null}
        # Generate fingerprints from host keys
        ssh-keygen -r example.com -f /etc/ssh/ssh_host_ed25519_key.pub

        # Create the record
        openstack recordset create \
          --type SSHFP \
          --record "4 2 $(ssh-keygen -l -E sha256 -f /etc/ssh/ssh_host_ed25519_key.pub | awk '{print $2}' | cut -d: -f2- | tr -d ':')" \
          example.com. @
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="TXT — Text Record">
    Stores arbitrary text data. Widely used for domain ownership verification,
    SPF policies, DKIM keys, and DMARC.

    | Field              | Value                                           |
    | ------------------ | ----------------------------------------------- |
    | **Format**         | Any text string                                 |
    | **Dashboard hint** | *(no format hint shown — enter any text value)* |

    <Tabs>
      <Tab title="Dashboard" icon="gauge">
        Select type **TXT** and enter the text value in the Records field.
      </Tab>

      <Tab title="CLI" icon="terminal">
        <CodeGroup>
          ```bash title="Create SPF TXT record" theme={null}
          openstack recordset create \
            --type TXT \
            --record '"v=spf1 include:_spf.example.com ~all"' \
            example.com. @
          ```

          ```bash title="Create DKIM TXT record" theme={null}
          openstack recordset create \
            --type TXT \
            --record '"v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3..."' \
            example.com. mail._domainkey
          ```

          ```bash title="Create domain verification TXT record" theme={null}
          openstack recordset create \
            --type TXT \
            --record '"google-site-verification=abc123..."' \
            example.com. @
          ```
        </CodeGroup>
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Manage Records" href="/services/dns/manage-records">
    Create, update, and delete record sets using the Dashboard and CLI
  </Card>

  <Card title="Reverse DNS" href="/services/dns/reverse-dns">
    Configure PTR records for your zone's IP addresses
  </Card>

  <Card title="Create a Zone" href="/services/dns/create-zone">
    Provision a new authoritative DNS zone
  </Card>

  <Card title="Troubleshooting" href="/services/dns/troubleshooting">
    Resolve record conflicts and propagation issues
  </Card>
</CardGroup>


# Reverse DNS Configuration
Source: https://docs.xloud.tech/services/dns/reverse-dns

Configure PTR records for IPv4 and IPv6 addresses in Xloud DNS using the Dashboard or CLI. Set and unset domain name pointers for mail servers and audit.

## Overview

Reverse DNS maps IP addresses back to hostnames using PTR records. This is distinct from
forward DNS (hostname to IP). Reverse DNS is required by mail servers for spam
reputation checks, improves audit log readability, and is a compliance requirement for
many security frameworks.

The Dashboard provides a dedicated **DNS Reverse** interface for managing PTR records
on your floating IPs without manually creating reverse zones.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## How Reverse DNS Works

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant Resolver
    participant ReverseZone as Reverse Zone (in-addr.arpa.)
    participant DNS as Xloud DNS

    Client->>Resolver: Lookup PTR for 192.168.1.10
    Resolver->>ReverseZone: Query 10.1.168.192.in-addr.arpa.
    ReverseZone->>DNS: Delegate to Xloud nameservers
    DNS-->>Resolver: mail.example.com.
    Resolver-->>Client: mail.example.com.
```

***

## View Reverse DNS Entries

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > DNS Reverse**. The list shows all floating
    IPs in your project with their PTR status.

    | Column              | Description                                           |
    | ------------------- | ----------------------------------------------------- |
    | **Address**         | The IP address (clickable to view details)            |
    | **PTR Domain Name** | The hostname this IP resolves to, or empty if not set |
    | **Status**          | Active, Pending, or Error                             |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List PTR records" theme={null}
    openstack ptr record list
    ```
  </Tab>
</Tabs>

***

## Set a PTR Record

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to DNS Reverse">
        Navigate to **Network > DNS Reverse**. Find the IP address
        you want to configure.

        Click the **More** dropdown in the row actions, then select **Set**.
        The Set action is always available and can be used to create a new PTR
        record or update an existing one.
      </Step>

      <Step title="Enter the domain name">
        In the **Set Domain Name PTR** dialog, enter the **Domain Name** — the
        fully qualified hostname this IP should resolve to.

        | Field                             | Details                                                              |
        | --------------------------------- | -------------------------------------------------------------------- |
        | **Domain Name** (required)        | FQDN ending with a dot (e.g., `smtp.example.com.`)                   |
        | **Description** (optional)        | Notes about this PTR record                                          |
        | **TTL (Time To Live)** (optional) | Cache duration in seconds. Minimum: `0`. Default placeholder: `3600` |

        <Tip>
          The domain name should match an A record in your forward DNS zone. Mismatched
          forward and reverse DNS can cause mail delivery failures and security audit
          warnings.
        </Tip>
      </Step>

      <Step title="Confirm the PTR record">
        Click **Confirm**. The PTR record enters **Pending** briefly and transitions
        to **Active**.

        <Check>The PTR Domain Name column shows your configured hostname with Active status.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Set a PTR record for a floating IP">
        ```bash title="Set PTR record" theme={null}
        openstack ptr record set \
          --description "Mail server reverse DNS" \
          --ttl 3600 \
          <region>:<floating-ip-id> \
          mail.example.com.
        ```

        Replace `<region>` with your region name (e.g., `RegionOne`) and
        `<floating-ip-id>` with the UUID of the floating IP.
      </Step>

      <Step title="Verify the PTR record">
        ```bash title="Show PTR record" theme={null}
        openstack ptr record show <region>:<floating-ip-id>
        ```

        <Check>Output shows the PTR record value as `mail.example.com.` with status `ACTIVE`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Unset a PTR Record

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Find the PTR record">
        Navigate to **Network > DNS Reverse**. Find the IP address
        with the PTR record you want to remove.

        <Note>
          The **Unset** action is only available when a PTR domain name is
          configured and the status is **Active**.
        </Note>
      </Step>

      <Step title="Unset the PTR record">
        Click the **More** dropdown in the row actions, then select **Unset**.
        Confirm the action in the dialog.

        The PTR domain name is removed and the column returns to empty.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete PTR record" theme={null}
    openstack ptr record delete <region>:<floating-ip-id>
    ```
  </Tab>
</Tabs>

***

## View PTR Record Details

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click an IP address in the DNS Reverse list to open the detail page.

    | Field               | Description                 |
    | ------------------- | --------------------------- |
    | **Address**         | The IP address              |
    | **PTR Domain Name** | The configured hostname     |
    | **Description**     | Notes about this PTR record |
    | **ID**              | Unique identifier           |
    | **Time To Live**    | TTL in seconds              |
    | **Status**          | Active, Pending, or Error   |
    | **Action**          | Current action type         |

    From the detail page, you can **Set** (to update) or **Unset** the PTR record
    using the **More** dropdown in the actions menu.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show PTR record details" theme={null}
    openstack ptr record show <region>:<floating-ip-id>
    ```
  </Tab>
</Tabs>

***

## IPv6 Reverse DNS

IPv6 PTR records follow the same process as IPv4. The DNS service automatically
manages the `ip6.arpa.` reverse zones for your allocated IPv6 prefixes.

```bash title="Set PTR record for an IPv6 floating IP" theme={null}
openstack ptr record set \
  <region>:<ipv6-floating-ip-id> \
  ipv6-host.example.com.
```

<Tip>
  IPv6 reverse zones are automatically delegated for your allocated prefix. No manual
  zone creation is required.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Manage Records" href="/services/dns/manage-records">
    Add forward DNS records to complement your PTR configuration
  </Card>

  <Card title="Record Types Reference" href="/services/dns/record-types">
    Reference for all supported DNS record types
  </Card>

  <Card title="Troubleshooting" href="/services/dns/troubleshooting">
    Resolve PTR record failures and zone delegation issues
  </Card>

  <Card title="DNS Admin Guide" href="/services/dns/admin-guide">
    Administer reverse zone pools and nameserver delegation
  </Card>
</CardGroup>


# DNS Security
Source: https://docs.xloud.tech/services/dns/security

Harden Xloud DNS infrastructure — restrict zone transfers, protect zone apex records, manage DNSSEC signing keys, and audit DNS API access.

## Overview

DNS security protects zone integrity, prevents unauthorized data exposure, and maintains
the chain of trust for DNSSEC-signed zones. This guide covers the key hardening areas
for platform administrators.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Hardening Guidelines

<AccordionGroup>
  <Accordion title="Restrict zone transfer recipients" icon="lock">
    Zone transfers expose complete zone data to the recipient. Enforce the principle of
    least privilege:

    * Create transfer requests only for specific target projects — never use open transfers
    * Set short expiration windows on transfer requests (24 hours maximum)
    * Audit accepted transfers monthly:
      ```bash title="Audit zone transfers" theme={null}
      openstack zone transfer accept list --all-projects
      ```
    * Revoke transfer requests immediately after they are no longer needed:
      ```bash title="Delete a transfer request" theme={null}
      openstack zone transfer request delete <transfer-id>
      ```
  </Accordion>

  <Accordion title="Protect zone apex records" icon="shield">
    SOA and NS records at the zone apex define authoritative authority. Unauthorized
    modification redirects queries to attacker-controlled nameservers:

    * Review NS record changes in audit logs after each deployment
    * Restrict zone modification to named service accounts — avoid using personal
      credentials for automated DNS management
    * Enable API rate limiting to prevent bulk zone modification attacks
    * Separate read-only reporter roles from write-capable automation accounts
  </Accordion>

  <Accordion title="DNSSEC key management" icon="key">
    DNSSEC signing protects DNS responses from tampering and spoofing. Signing keys
    are stored in Xloud Key Manager:

    * Store Zone Signing Keys (ZSK) and Key Signing Keys (KSK) as secrets in Key Manager
    * Rotate ZSKs every 90 days; KSKs annually
    * Maintain DS records at the parent zone registrar to complete the chain of trust
    * Test DNSSEC validation after key rotation:
      ```bash title="Validate DNSSEC chain" theme={null}
      dig @<resolver-ip> example.com. A +dnssec
      ```

    <Tip>
      Use Xloud Key Manager's expiration feature to track ZSK and KSK rotation schedules.
      Set expiration dates on signing keys and build a rotation workflow triggered before
      expiry.
    </Tip>
  </Accordion>

  <Accordion title="Audit DNS API access" icon="file-text">
    All DNS API requests are logged. Configure log forwarding to your centralized
    logging platform to retain audit records for:

    * Zone creation and deletion events
    * Record set modifications with before/after values
    * Zone transfer requests and acceptances
    * Quota changes and project assignments

    <Tip>
      Store DNS audit logs in Xloud Object Storage with a minimum 1-year retention policy
      to satisfy compliance requirements for DNS change auditing.
    </Tip>
  </Accordion>

  <Accordion title="Network access controls" icon="network">
    The DNS API should not be exposed to untrusted networks:

    * Bind the DNS API to the internal management network only
    * Configure firewall rules limiting port 9001 access to authorized hosts
    * Apply HAProxy frontend ACLs to restrict source IPs if the API is load-balanced
    * Enable HTTPS on the DNS API endpoint — never manage zones over plain HTTP
  </Accordion>
</AccordionGroup>

***

## Security Checklist

| Control                        | Status           | Notes                                                      |
| ------------------------------ | ---------------- | ---------------------------------------------------------- |
| Zone transfers target-specific | Verify quarterly | Check `openstack zone transfer accept list --all-projects` |
| DNSSEC ZSK rotated             | Every 90 days    | Store keys in Key Manager with expiration                  |
| DNSSEC KSK rotated             | Annually         | Update DS record at registrar after rotation               |
| DNS API over HTTPS             | Always           | Verify HAProxy SSL termination config                      |
| Audit logs forwarded           | Continuous       | 1-year minimum retention                                   |
| API rate limiting enabled      | Platform-wide    | Prevent bulk zone modification attacks                     |

***

## Next Steps

<CardGroup>
  <Card title="Zone Transfers" href="/services/dns/zone-transfers">
    Manage and audit zone transfer requests
  </Card>

  <Card title="Key Manager" href="/services/key-manager/admin-guide">
    Store and rotate DNSSEC signing keys
  </Card>

  <Card title="Quotas" href="/services/dns/quotas">
    Enforce per-project DNS resource limits
  </Card>

  <Card title="Admin Troubleshooting" href="/services/dns/admin-troubleshooting">
    Diagnose security-related DNS service issues
  </Card>
</CardGroup>


# DNS Troubleshooting
Source: https://docs.xloud.tech/services/dns/troubleshooting

Troubleshoot common DNS issues in Xloud DNS — zone errors, record propagation delays, PTR failures, and CNAME conflicts.

## Overview

This guide covers user-facing DNS issues: zones stuck in error states, records not
resolving after creation, PTR record failures, and CNAME conflicts. For platform-level
issues such as service outages or nameserver synchronization failures, see the
[Admin Troubleshooting](/services/dns/admin-troubleshooting) guide.

***

## Common Issues

<AccordionGroup>
  <Accordion title="Zone stuck in PENDING or ERROR status" icon="clock">
    **Cause**: The DNS backend could not create the zone due to a configuration conflict
    or backend connectivity issue.

    **Diagnosis**:

    ```bash title="Show zone detail with action and status" theme={null}
    openstack zone show example.com. \
      -c name -c status -c action
    ```

    **Resolution**:

    * If status is `ERROR`, check whether a zone with the same name already exists under a
      different project. The DNS service prevents duplicate zone names across all projects.
    * Contact your platform administrator if the zone remains in `ERROR` after the
      conflicting zone is removed.
    * If the zone shows `action: CREATE` and `status: PENDING` for more than 60 seconds,
      the DNS worker may be unable to reach the backend nameserver pool.
  </Accordion>

  <Accordion title="Record set not resolving after creation" icon="search">
    **Cause**: DNS propagation is still in progress, or the client resolver is caching a
    negative (NXDOMAIN) response from before the record was created.

    **Diagnosis**:

    ```bash title="Query the authoritative nameserver directly" theme={null}
    dig @<nameserver-ip> www.example.com A
    ```

    **Resolution**:

    * If the authoritative nameserver returns the correct value, the record was created
      successfully. The issue is resolver caching.
    * Flush the local resolver cache:
      ```bash title="Flush resolver cache (Linux)" theme={null}
      systemd-resolve --flush-caches
      ```
    * If the authoritative nameserver also returns NXDOMAIN, verify the record set status:
      ```bash title="Check record set status" theme={null}
      openstack recordset show example.com. www
      ```
      A status of `ERROR` indicates a backend provisioning failure — contact your
      platform administrator.
  </Accordion>

  <Accordion title="PTR record set fails with zone not found" icon="map-pin">
    **Cause**: The reverse zone for the IP range has not been created or delegated to
    the Xloud DNS service.

    **Diagnosis**:

    ```bash title="Show PTR record error" theme={null}
    openstack ptr record show <region>:<floating-ip-id>
    ```

    **Resolution**: Contact your platform administrator to verify that the reverse zone
    for the IP range is provisioned in the DNS service and that nameserver delegation
    is configured correctly for the `in-addr.arpa.` or `ip6.arpa.` parent zone.
  </Accordion>

  <Accordion title="CNAME conflict with existing records" icon="triangle-alert">
    **Cause**: A CNAME cannot coexist with other record types at the same name. If an
    A or MX record already exists at the target name, the CNAME creation fails.

    **Diagnosis**:

    ```bash title="List all records at the target name" theme={null}
    openstack recordset list example.com. | grep <target-name>
    ```

    **Resolution**:

    * Delete any conflicting record sets at the target hostname before creating the CNAME.
    * Note that CNAMEs cannot be created at the zone apex (`@`) — use an A record or
      contact your administrator about ALIAS record support for root domain aliases.
  </Accordion>

  <Accordion title="Record changes not propagating to all clients" icon="refresh-cw">
    **Cause**: Different clients are hitting different resolvers with different cache states,
    or the zone's SOA serial number has not been incremented.

    **Diagnosis**:

    ```bash title="Check SOA serial on the authoritative nameserver" theme={null}
    dig @<nameserver-ip> example.com. SOA
    ```

    Compare this serial number with what recursive resolvers have cached:

    ```bash title="Query a public resolver" theme={null}
    dig @8.8.8.8 example.com. SOA
    ```

    **Resolution**:

    * If serials match, the update has propagated — wait for resolver TTL to expire.
    * If serials differ, the zone update has not yet synchronized to the queried nameserver.
      Wait for the zone's TTL, or contact your administrator to verify nameserver sync.
  </Accordion>

  <Accordion title="Cannot delete a zone — delete returns 400 or 409" icon="trash-2">
    **Cause**: The zone contains record sets that must be removed first, or a transfer
    request for this zone is pending.

    **Resolution**:

    ```bash title="List all record sets in the zone" theme={null}
    openstack recordset list example.com.
    ```

    Delete non-SOA and non-NS records, then retry the zone deletion. If a transfer
    request is pending, cancel it first:

    ```bash title="List and delete transfer requests" theme={null}
    openstack zone transfer request list
    openstack zone transfer request delete <transfer-id>
    ```
  </Accordion>
</AccordionGroup>

***

## Diagnostic Commands

<CodeGroup>
  ```bash title="Show zone status and error detail" theme={null}
  openstack zone show example.com.
  ```

  ```bash title="List record sets in a zone" theme={null}
  openstack recordset list example.com.
  ```

  ```bash title="Check authoritative nameserver response" theme={null}
  dig @<nameserver-ip> www.example.com A +norecurse
  ```

  ```bash title="Validate DNSSEC chain" theme={null}
  dig @<resolver-ip> example.com. A +dnssec
  ```

  ```bash title="Flush local DNS cache" theme={null}
  systemd-resolve --flush-caches
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Admin Troubleshooting" href="/services/dns/admin-troubleshooting">
    Platform-level DNS issues — API outages, worker failures, nameserver sync errors
  </Card>

  <Card title="Create a Zone" href="/services/dns/create-zone">
    Start fresh with a properly configured zone
  </Card>

  <Card title="Manage Records" href="/services/dns/manage-records">
    Add or correct record sets in an existing zone
  </Card>

  <Card title="Reverse DNS" href="/services/dns/reverse-dns">
    Configure PTR records for your IP addresses
  </Card>
</CardGroup>


# DNS User Guide
Source: https://docs.xloud.tech/services/dns/user-guide

Manage DNS zones, record sets, reverse DNS, and record types in Xloud DNS. Learn to create zones, populate records, configure PTR entries, and.

<p>Overview</p>

Xloud DNS provides authoritative DNS management for your private cloud domains. Create
zones, populate them with record sets, and manage reverse DNS for IP-to-hostname
resolution. DNS records integrate with your compute and networking resources to support
service discovery, email routing, and compliance requirements.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

<p>Topics in This Guide</p>

<CardGroup>
  <Card title="Create a Zone" icon="circle-plus" href="/services/dns/create-zone">
    Provision authoritative DNS zones for your domains with TTL and SOA configuration
  </Card>

  <Card title="Manage Records" icon="list" href="/services/dns/manage-records">
    Add, edit, and delete record sets within a zone using the Dashboard or CLI
  </Card>

  <Card title="Record Types Reference" icon="book" href="/services/dns/record-types">
    Full reference for A, AAAA, CNAME, MX, TXT, SRV, NS, CAA, and other record types
  </Card>

  <Card title="Reverse DNS" icon="arrow-left-right" href="/services/dns/reverse-dns">
    Configure PTR records for IPv4 and IPv6 addresses to support mail and audit requirements
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/dns/troubleshooting">
    Resolve zone errors, propagation issues, and CNAME conflicts
  </Card>
</CardGroup>

***

<p>Key Concepts</p>

| Concept        | Description                                                                                                    |
| -------------- | -------------------------------------------------------------------------------------------------------------- |
| **Zone**       | An authoritative DNS domain boundary (e.g., `app.example.com.`). Contains all record sets for that domain      |
| **Record Set** | One or more DNS records sharing the same name and type — e.g., two A records for `www`                         |
| **Nameserver** | The resolver authoritative for the zone. Clients send queries here to resolve names                            |
| **PTR Record** | Reverse mapping — resolves an IP address back to a hostname. Required for mail servers and audit logging       |
| **SOA Record** | Start of Authority — defines zone parameters including serial number, refresh interval, and primary nameserver |
| **TTL**        | Time-to-Live — how long resolvers cache a record before re-querying. Lower values propagate changes faster     |

<Note>
  DNS zones must be properly delegated before public resolvers can use them. For internal
  zones, ensure your clients are configured to query the Xloud DNS service nameservers.
</Note>

***

<p>Next Steps</p>

<CardGroup>
  <Card title="DNS Admin Guide" icon="shield-check" href="/services/dns/admin-guide">
    Configure backend drivers, zone transfers, and quotas as a platform administrator
  </Card>

  <Card title="Load Balancer" icon="combine" href="/services/load-balancer/user-guide">
    Create load balancers and map DNS records to their floating IPs
  </Card>

  <Card title="Key Manager" icon="lock" href="/services/key-manager/user-guide">
    Store DNSSEC signing keys and TLS certificates for your domains
  </Card>

  <Card title="Networking" icon="network" href="/services/networking/user-guide">
    Configure floating IPs and fixed IPs referenced in DNS records
  </Card>
</CardGroup>


# DNS Zone Transfers
Source: https://docs.xloud.tech/services/dns/zone-transfers

Manage AXFR and IXFR zone transfer requests in Xloud DNS. Create, accept, and revoke transfer requests between projects and secondary nameservers.

## Overview

Zone transfers replicate zone data from the Xloud DNS service to secondary nameservers
or other projects. Administrators control which destinations are permitted to perform
zone transfers. Transfer requests use a one-time key mechanism — the requesting admin
generates the request and shares the key with the recipient through a secure channel.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Zone Transfer Workflow

```mermaid theme={null}
sequenceDiagram
    participant Admin as Source Admin
    participant API as DNS API
    participant Recipient as Recipient Project Admin

    Admin->>API: Create transfer request for example.com.
    API-->>Admin: transfer_id + key
    Admin->>Recipient: Share transfer_id and key (secure channel)
    Recipient->>API: Accept transfer with transfer_id + key
    API-->>Recipient: Zone available in recipient project
```

***

## Create a Transfer Request

<Tabs>
  <Tab title="Create and share" icon="plus">
    <Steps>
      <Step title="Create transfer request as admin">
        ```bash title="Create zone transfer request" theme={null}
        openstack zone transfer request create \
          --target-project-id <project-id> \
          --description "Transfer to DR nameserver" \
          example.com.
        ```

        This generates a `key` that the recipient uses to accept the transfer.
      </Step>

      <Step title="Share the transfer key">
        Share the `id` and `key` from the output with the recipient project administrator
        through a secure channel (e.g., encrypted email, secrets manager).

        <Warning>
          Never share transfer keys over unencrypted channels. A compromised key allows
          unauthorized zone transfer.
        </Warning>
      </Step>

      <Step title="Recipient accepts the transfer">
        The recipient project accepts the transfer using the provided credentials:

        ```bash title="Accept zone transfer" theme={null}
        openstack zone transfer accept request \
          --transfer-id <transfer-id> \
          --key <transfer-key>
        ```

        <Check>Zone becomes available in the recipient project.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Manage transfer requests" icon="list">
    <CodeGroup>
      ```bash title="List pending transfer requests" theme={null}
      openstack zone transfer request list
      ```

      ```bash title="Show transfer request detail" theme={null}
      openstack zone transfer request show <transfer-id>
      ```

      ```bash title="Delete a transfer request" theme={null}
      openstack zone transfer request delete <transfer-id>
      ```

      ```bash title="List accepted transfers" theme={null}
      openstack zone transfer accept list
      ```
    </CodeGroup>

    <Tip>
      Delete stale transfer requests that were not accepted within 24 hours to prevent
      unauthorized zone transfers if keys are later compromised.
    </Tip>
  </Tab>
</Tabs>

***

## Security Best Practices

| Practice                     | Description                                                        |
| ---------------------------- | ------------------------------------------------------------------ |
| **Target-specific requests** | Always specify `--target-project-id` — never create open transfers |
| **Short expiration**         | Set 24-hour expiration windows on all transfer requests            |
| **Secure key delivery**      | Deliver transfer keys via encrypted channel only                   |
| **Regular audit**            | Review accepted transfers monthly and revoke unnecessary ones      |

```bash title="Audit all accepted zone transfers (admin)" theme={null}
openstack zone transfer accept list --all-projects
```

***

## Next Steps

<CardGroup>
  <Card title="Pool Management" href="/services/dns/pool-management">
    Manage nameserver pools that receive transferred zone data
  </Card>

  <Card title="Security" href="/services/dns/security">
    Full DNS security hardening guidelines
  </Card>

  <Card title="Backend Configuration" href="/services/dns/backend-config">
    Configure `also_notifies` for AXFR consumer nameservers
  </Card>

  <Card title="Admin Troubleshooting" href="/services/dns/admin-troubleshooting">
    Diagnose zone transfer failures and key errors
  </Card>
</CardGroup>


# Identity Administration
Source: https://docs.xloud.tech/services/identity/admin-guide

Configure authentication backends, federation, LDAP, and access policies for Xloud Identity.

<p>Overview</p>

Xloud Identity administration covers every layer of the authentication and authorization
stack — from the backend that validates credentials to the policies that govern what each
role can do. Use the guides below to configure your deployment, manage domains, secure
token issuance, and troubleshoot platform-level issues.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

<CardGroup>
  <Card title="Architecture" icon="network" href="/services/identity/architecture">
    Service topology, component roles, and data flow through the Identity stack.
  </Card>

  <Card title="Authentication Backends" icon="database" href="/services/identity/auth-backends">
    Configure SQL, LDAP, SAML 2.0, and OpenID Connect authentication drivers.
  </Card>

  <Card title="Domain Management" icon="building" href="/services/identity/domain-management">
    Create and manage organizational domains with independent user namespaces.
  </Card>

  <Card title="Token Configuration" icon="clock" href="/services/identity/token-config">
    Configure Fernet key rotation, token lifetime, and expiration policies.
  </Card>

  <Card title="Service Catalog" icon="list" href="/services/identity/service-catalog">
    Manage endpoint registration for all Xloud services across regions and interfaces.
  </Card>

  <Card title="Federation" icon="link" href="/services/identity/federation">
    Integrate SAML 2.0 and OIDC identity providers for enterprise single sign-on.
  </Card>

  <Card title="Policy Management" icon="shield" href="/services/identity/policy-management">
    Customize service-level policy rules to control which roles can perform each API operation.
  </Card>

  <Card title="Extended RBAC" icon="user-check" href="/services/identity/extended-rbac">
    Fine-grained per-action privileges, custom roles, and tag-conditioned grants — beyond the built-in role set.
  </Card>

  <Card title="Security Hardening" icon="lock" href="/services/identity/security">
    Enforce MFA, rotate Fernet keys, audit role assignments, and apply best practices.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/identity/admin-troubleshooting">
    Resolve token validation failures, LDAP issues, and service catalog misconfigurations.
  </Card>
</CardGroup>

***

<p>Quick Reference</p>

| Task                         | Command                                             |
| ---------------------------- | --------------------------------------------------- |
| Rotate Fernet keys           | `xavs-ansible deploy --tags keystone-fernet-rotate` |
| List all domains             | `openstack domain list`                             |
| List all users               | `openstack user list --domain Default`              |
| List all role assignments    | `openstack role assignment list --names`            |
| Show service endpoints       | `openstack endpoint list`                           |
| Show token expiration config | `openstack --os-cloud admin domain show Default`    |

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Identity User Guide" icon="book-open" href="/services/identity/user-guide">
    Day-to-day operations — projects, users, and application credentials.
  </Card>

  <Card title="Compute Admin Guide" icon="settings" href="/services/compute/admin-guide">
    Configure compute hosts, flavors, quotas, and scheduler policies.
  </Card>
</CardGroup>


# Identity Admin Troubleshooting
Source: https://docs.xloud.tech/services/identity/admin-troubleshooting

Diagnose and resolve platform-level Identity issues — token validation failures, LDAP connectivity, federation errors, and service catalog misconfigurations.

## Overview

This guide covers platform-level Identity issues that require administrator access —
token validation failures across services, LDAP and federation authentication problems,
service catalog misconfigurations, and system-scope permission errors.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  For end-user authentication issues (wrong password, expired token, missing role),
  see the [Identity User Troubleshooting](/services/identity/troubleshooting) guide.
</Note>

***

## Token Validation Failures

<AccordionGroup>
  <Accordion title="Token validation failures across services" icon="circle-x">
    **Cause**: Fernet keys are out of sync between Identity API nodes, or the token has
    expired.

    **Diagnose**: Verify key synchronization across all nodes. All nodes must have
    identical key files in the fernet-keys directory:

    ```bash title="Check key file count and timestamps on all nodes" theme={null}
    ls -la /var/lib/kolla/config_files/fernet-keys/
    ```

    Compare timestamps across nodes. If keys are inconsistent, force a rotation:

    ```bash title="Force key rotation via xavs-ansible" theme={null}
    xavs-ansible deploy --tags keystone-fernet-rotate
    ```

    **Resolution**: After rotation, verify validation works:

    ```bash title="Issue and validate a token" theme={null}
    TOKEN=$(openstack token issue -f value -c id)
    openstack token show $TOKEN
    ```

    <Check>Token validates successfully on all Identity API nodes.</Check>
  </Accordion>

  <Accordion title="All services returning 401 Unauthorized" icon="lock">
    **Cause**: The Identity API is unreachable, or all Identity API nodes are down.

    **Diagnose**:

    ```bash title="Check Identity API container status" theme={null}
    docker ps --filter name=keystone
    ```

    ```bash title="Test Identity API directly" theme={null}
    curl -s https://api.<your-domain>:5000/v3 | python3 -m json.tool
    ```

    **Resolution**: Restart the Identity service via XDeploy if containers are stopped:

    ```bash title="Restart Identity containers" theme={null}
    xavs-ansible deploy --tags keystone
    ```
  </Accordion>
</AccordionGroup>

***

## LDAP Authentication Issues

<AccordionGroup>
  <Accordion title="LDAP users cannot authenticate" icon="network">
    **Cause**: LDAP connection failure, incorrect bind credentials, or user not in the
    configured `user_tree_dn`.

    **Diagnose**: Test the LDAP connection from the Identity API node:

    ```bash title="Test LDAP connectivity" theme={null}
    ldapsearch -x -H ldap://ldap.example.com \
      -D "cn=xloud-svc,dc=example,dc=com" \
      -w "$LDAP_PASSWORD" \
      -b "ou=Users,dc=example,dc=com" \
      "(sAMAccountName=alice)"
    ```

    Confirm the user exists in the expected OU and the bind account has read access.

    **Check Identity API logs**:

    ```bash title="View Identity API logs for LDAP errors" theme={null}
    docker logs keystone --tail 100 | grep -i ldap
    ```
  </Accordion>

  <Accordion title="LDAP users not appearing in user list" icon="users">
    **Cause**: The `user_tree_dn` does not match the OU where users are located, or
    the `user_id_attribute` is set incorrectly for your directory schema.

    **Resolution**: Verify the LDAP configuration matches your directory schema:

    ```bash title="Check keystone LDAP configuration" theme={null}
    docker exec keystone grep -A 30 "\[ldap\]" /etc/keystone/keystone.conf
    ```

    Update the configuration via XDeploy globals and redeploy if attributes are incorrect.
  </Accordion>
</AccordionGroup>

***

## Service Catalog Issues

<AccordionGroup>
  <Accordion title="Service catalog returns wrong endpoint" icon="route">
    **Cause**: An endpoint was registered with an incorrect URL or interface type.

    **Diagnose**:

    ```bash title="List endpoints for a service" theme={null}
    openstack endpoint list --service compute
    ```

    Identify the incorrect endpoint by its ID and update the URL:

    ```bash title="Update endpoint URL" theme={null}
    openstack endpoint set \
      --url https://correct-url:8774/v2.1 \
      <ENDPOINT_ID>
    ```
  </Accordion>

  <Accordion title="CLI resolves to internal endpoint from external network" icon="globe">
    **Cause**: The default endpoint interface is `public` but the CLI is resolving to
    `internal` due to environment variable override.

    **Diagnose**:

    ```bash title="Check current endpoint interface setting" theme={null}
    echo $OS_ENDPOINT_TYPE
    echo $OS_INTERFACE
    ```

    **Resolution**: Unset the override or set it explicitly to `public`:

    ```bash title="Set interface to public" theme={null}
    export OS_INTERFACE=public
    openstack catalog list
    ```
  </Accordion>
</AccordionGroup>

***

## Federation Issues

<AccordionGroup>
  <Accordion title="SAML authentication fails with mapping error" icon="link">
    **Cause**: IdP attributes do not match the mapping rules, or the `remote-id` does not
    match the IdP's entity ID.

    **Diagnose**: Check the Identity API logs for mapping evaluation errors:

    ```bash title="View federation errors in Identity logs" theme={null}
    docker logs keystone --tail 200 | grep -i "federation\|mapping\|saml"
    ```

    **Resolution**: Verify the mapping rules match the attributes your IdP sends.
    Use the mapping walkthrough API to test rules:

    ```bash title="Test mapping rules" theme={null}
    openstack mapping validate \
      --rules mapping-rules.json \
      --properties @test-assertion.json \
      corporate-mapping
    ```
  </Accordion>
</AccordionGroup>

***

## Permission and Scope Errors

<AccordionGroup>
  <Accordion title="403 Forbidden on admin operations" icon="lock">
    **Cause**: The user has the `admin` role in the project but not at the system or domain
    scope required for administrative operations.

    **Resolution**: Grant system-scope admin access:

    ```bash title="Grant system admin role" theme={null}
    openstack role add \
      --user alice \
      --system all \
      admin
    ```

    <Warning>
      System-scope admin grants full control over all domains and projects. Reserve this
      assignment for platform administrators only.
    </Warning>
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Token Configuration" href="/services/identity/token-config">
    Configure Fernet key rotation to prevent token validation failures.
  </Card>

  <Card title="Authentication Backends" href="/services/identity/auth-backends">
    Review LDAP and federation backend configuration options.
  </Card>

  <Card title="Service Catalog" href="/services/identity/service-catalog">
    Manage and correct endpoint registrations in the service catalog.
  </Card>

  <Card title="Security Hardening" href="/services/identity/security">
    Apply security best practices to prevent future authentication issues.
  </Card>
</CardGroup>


# Application Credentials
Source: https://docs.xloud.tech/services/identity/application-credentials

Create and manage scoped application credentials for automation pipelines, CI/CD systems, and service accounts in Xloud Identity.

## Overview

Application credentials allow automation pipelines, CI/CD systems, and service accounts
to authenticate without embedding user passwords. They are scoped to the user's current
project and role assignments, and can be restricted further to a specific subset of roles
or API paths. Unlike user passwords, application credentials have explicit expiry dates
and can be revoked independently.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Warning>
  Application credentials are bound to the creating user. If that user is disabled or
  deleted, all their application credentials are invalidated immediately. For long-lived
  service accounts, create a dedicated service user to own the credentials.
</Warning>

***

## Create an Application Credential

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Application Credentials" icon="terminal">
        Log in as the user who will own the credential. Navigate to
        **User Center > Application Credentials** (via profile dropdown) and click **Create Application Credential**.
      </Step>

      <Step title="Configure the credential" icon="key">
        | Field               | Description                                                    |
        | ------------------- | -------------------------------------------------------------- |
        | **Name**            | Descriptive identifier (e.g., `ci-pipeline-prod`)              |
        | **Secret**          | Leave blank to auto-generate a cryptographically secure secret |
        | **Expiration Date** | Set an expiry for credentials used in short-lived pipelines    |
        | **Roles**           | Restrict to a subset of your role assignments (optional)       |
        | **Access Rules**    | Limit the credential to specific API paths and HTTP methods    |
      </Step>

      <Step title="Save the credential securely" icon="download">
        After creation, the Dashboard displays the credential ID and secret **once**.
        Download the `clouds.yaml` snippet for immediate use.

        <Danger>
          The secret is shown only once and cannot be retrieved again. Store it in a secrets
          manager (such as Xloud Key Management or HashiCorp Vault) immediately after creation.
        </Danger>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the credential" icon="plus">
        ```bash title="Create application credential with expiry" theme={null}
        openstack application credential create \
          --description "CI/CD pipeline credential" \
          --expiration "2026-12-31T00:00:00" \
          ci-pipeline-prod
        ```

        Note the `id` and `secret` values from the output — they are shown only once.
      </Step>

      <Step title="Create credential with restricted roles" icon="shield">
        ```bash title="Create credential restricted to reader role only" theme={null}
        openstack application credential create \
          --description "Read-only monitoring credential" \
          --role reader \
          --expiration "2026-12-31T00:00:00" \
          monitoring-readonly
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Authenticate with Application Credentials

Application credentials replace user passwords in the `clouds.yaml` configuration file.

<Steps>
  <Step title="Add to clouds.yaml" icon="file-code">
    Add the following to your `~/.config/openstack/clouds.yaml`:

    ```yaml title="~/.config/openstack/clouds.yaml" theme={null}
    clouds:
      xloud-ci:
        auth:
          auth_url: https://api.<your-domain>:5000/v3
          application_credential_id: "<CREDENTIAL_ID>"
          application_credential_secret: "<CREDENTIAL_SECRET>"
        auth_type: v3applicationcredential
        region_name: RegionOne
    ```
  </Step>

  <Step title="Test the credential" icon="circle-check">
    ```bash title="Verify authentication with the credential" theme={null}
    openstack --os-cloud xloud-ci token issue
    ```

    <Check>A token is issued — the credential is valid and functional.</Check>
  </Step>
</Steps>

***

## Access Rules

Access rules restrict a credential to specific API operations, providing fine-grained
control beyond role-level permissions.

```bash title="Create credential with access rules" theme={null}
openstack application credential create \
  --description "Image upload only" \
  --access-rules '[
    {"path": "/v2/images", "method": "POST", "service": "image"},
    {"path": "/v2/images/**", "method": "PUT", "service": "image"}
  ]' \
  image-uploader
```

| Field     | Description                                                       |
| --------- | ----------------------------------------------------------------- |
| `path`    | API path pattern (supports `**` wildcard)                         |
| `method`  | HTTP method: `GET`, `POST`, `PUT`, `DELETE`, `PATCH`              |
| `service` | Service type: `compute`, `image`, `identity`, `volume`, `network` |

***

## Manage Existing Credentials

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **User Center > Application Credentials** (via profile dropdown) to view all credentials owned by
    the current user. Delete expired or unused credentials to reduce attack surface.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List application credentials" theme={null}
      openstack application credential list
      ```

      ```bash title="Show credential details (without secret)" theme={null}
      openstack application credential show ci-pipeline-prod
      ```

      ```bash title="Delete a credential" theme={null}
      openstack application credential delete ci-pipeline-prod
      ```
    </CodeGroup>
  </Tab>
</Tabs>

<Tip>
  Rotate application credentials before their expiration date. Create the replacement
  credential first, update all consumers, then delete the old credential. This zero-downtime
  rotation pattern avoids pipeline interruptions.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Users" href="/services/identity/users">
    Manage user accounts that own application credentials.
  </Card>

  <Card title="Multi-Factor Authentication" href="/services/identity/multi-factor-auth">
    Add TOTP-based two-factor authentication to user accounts.
  </Card>

  <Card title="Identity Admin Guide" href="/services/identity/admin-guide">
    Configure token policies and security hardening for your Identity deployment.
  </Card>

  <Card title="Troubleshooting" href="/services/identity/troubleshooting">
    Resolve credential rejection and authentication failure issues.
  </Card>
</CardGroup>


# Identity Service Architecture
Source: https://docs.xloud.tech/services/identity/architecture

Understand the Xloud Identity service topology, component roles, and the authentication data flow across a distributed deployment.

## Overview

Xloud Identity runs as a distributed service with API endpoints fronted by HAProxy. The
token validation path is on every service's critical path — all Xloud services call the
Identity API to validate incoming requests. Understanding the architecture is essential
for sizing, high availability planning, and troubleshooting.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Service Topology

```mermaid theme={null}
graph TD
    U([User / Dashboard / CLI]) --> HAP[HAProxy<br/>:5000]
    HAP --> I1[Identity API<br/>Node 1]
    HAP --> I2[Identity API<br/>Node 2]
    I1 --> DB[(MariaDB<br/>Identity DB)]
    I2 --> DB
    I1 --> LDAP[LDAP / AD<br/>optional]
    I1 --> FED[SAML / OIDC<br/>Federation optional]
    I1 --> FER[Fernet Key<br/>Repository]
    NOVA[Compute API] --> HAP
    CINDER[Storage API] --> HAP
    IMG[Image API] --> HAP
    style HAP fill:#197560,color:#fff
    style I1 fill:#3F8F7E,color:#fff
    style I2 fill:#3F8F7E,color:#fff
    style DB fill:#145C4C,color:#fff
```

***

## Component Reference

| Component             | Port    | Description                                                         |
| --------------------- | ------- | ------------------------------------------------------------------- |
| Identity API (v3)     | 5000    | Authentication, token issuance, and service catalog                 |
| HAProxy               | 5000    | Load balances API requests across all Identity API nodes            |
| MariaDB               | 3306    | Persistent storage for users, projects, roles, domains, and catalog |
| Fernet Key Repository | —       | Symmetric keys for stateless token signing and encryption           |
| LDAP (optional)       | 389/636 | External user directory for enterprise AD/LDAP integration          |
| Federation (optional) | —       | SAML 2.0 or OIDC IdP integration                                    |

***

## Authentication Flow

```mermaid theme={null}
sequenceDiagram
    participant U as User / CLI
    participant H as HAProxy :5000
    participant I as Identity API
    participant DB as MariaDB
    participant F as Fernet Keys
    U->>H: POST /v3/auth/tokens (credentials + scope)
    H->>I: Forward request
    I->>DB: Validate user + role assignments
    DB-->>I: User record + assignments
    I->>F: Encrypt token with active key
    F-->>I: Encrypted Fernet token
    I-->>H: X-Subject-Token + service catalog
    H-->>U: Token response
    U->>+Other Service: API Request + X-Auth-Token
    Other Service->>I: GET /v3/auth/tokens (validate)
    I->>F: Decrypt token
    F-->>I: Token payload
    I-->>Other Service: Valid + roles
    Other Service-->>U: API Response
```

***

## High Availability Considerations

<AccordionGroup>
  <Accordion title="Stateless token validation" icon="key">
    Fernet tokens are stateless — no database lookup is needed to validate them. The
    Identity API decrypts the token locally using the Fernet key repository. This means
    token validation scales horizontally without database pressure, and any Identity
    API node can validate any token as long as all nodes share the same Fernet key set.
  </Accordion>

  <Accordion title="Fernet key synchronization" icon="rotate-cw">
    All Identity API nodes must have identical Fernet key sets. XDeploy manages
    key distribution automatically during rotation. If nodes become out of sync,
    tokens signed by one node cannot be validated by another.

    Verify key consistency:

    ```bash title="Check key file count on all nodes" theme={null}
    ls -1 /var/lib/kolla/config_files/fernet-keys/ | wc -l
    ```

    All nodes must report the same count and file timestamps.
  </Accordion>

  <Accordion title="Database redundancy" icon="database">
    The Identity database is a MariaDB Galera cluster in a multi-node deployment.
    Identity writes (user creation, role assignments) are replicated synchronously
    across all MariaDB nodes. HAProxy in front of MariaDB distributes read operations.
  </Accordion>
</AccordionGroup>

***

## Deployment Footprint

<Tree>
  <Tree.Folder name="Identity API container">
    <Tree.Folder name="/etc/keystone">
      <Tree.File name="keystone.conf" />

      <Tree.File name="policy.yaml" />
    </Tree.Folder>

    <Tree.Folder name="/etc/keystone/fernet-keys">
      <Tree.File name="0 (staged key)" />

      <Tree.File name="1 (primary key)" />

      <Tree.File name="2 (secondary key)" />
    </Tree.Folder>
  </Tree.Folder>
</Tree>

***

## Next Steps

<CardGroup>
  <Card title="Authentication Backends" href="/services/identity/auth-backends">
    Configure SQL, LDAP, and federation authentication drivers.
  </Card>

  <Card title="Token Configuration" href="/services/identity/token-config">
    Set Fernet key rotation schedules and token lifetime policies.
  </Card>

  <Card title="Security Hardening" href="/services/identity/security">
    Enforce MFA, audit assignments, and apply hardening best practices.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/identity/admin-troubleshooting">
    Diagnose token validation failures and service communication issues.
  </Card>
</CardGroup>


# Authentication Backends
Source: https://docs.xloud.tech/services/identity/auth-backends

Configure SQL, LDAP, SAML 2.0, and OpenID Connect authentication drivers for Xloud Identity.

## Overview

Xloud Identity supports multiple authentication drivers that can be combined within the
same deployment. Each domain can use a different backend, allowing you to integrate
enterprise LDAP directories or federated identity providers alongside local SQL accounts.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Backend Comparison

| Backend      | Use Case                                                                                      | Configuration                                      |
| ------------ | --------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| **SQL**      | Default. Local users stored in MariaDB. Zero external dependencies.                           | Built-in; no additional config required.           |
| **LDAP**     | Enterprise directory integration. Users and groups sourced from Active Directory or OpenLDAP. | Configured per-domain via XDeploy globals.         |
| **SAML 2.0** | SSO with corporate IdPs (Okta, Azure AD, ADFS).                                               | Requires `mod_shib` and federation mapping rules.  |
| **OIDC**     | Modern SSO via OAuth 2.0 / OpenID Connect providers.                                          | Requires `mod_auth_openidc` and attribute mapping. |

***

## SQL Backend (Default)

The SQL backend is active by default and requires no additional configuration.
All user accounts created through the Dashboard or CLI are stored in MariaDB.

```bash title="Verify the SQL backend is active" theme={null}
openstack --os-cloud admin domain show Default -f json | grep -i driver
```

<Tip>
  The SQL backend is appropriate for most deployments. Use LDAP or federation only
  when integrating with an existing enterprise directory.
</Tip>

***

## LDAP Integration

LDAP integration sources users and groups from an external directory. Xloud Identity
connects in read-only mode — user creation and password changes must happen in the
directory, not in Xloud.

<Steps>
  <Step title="Configure LDAP in XDeploy globals" icon="settings">
    Set the following in your deployment globals via XDeploy:

    ```yaml title="LDAP configuration in deployment globals" theme={null}
    keystone_ldap:
      url: ldap://ldap.example.com
      user: cn=xloud-svc,dc=example,dc=com
      password: "{{ ldap_bind_password }}"
      suffix: dc=example,dc=com
      user_tree_dn: ou=Users,dc=example,dc=com
      group_tree_dn: ou=Groups,dc=example,dc=com
      user_id_attribute: sAMAccountName
      user_name_attribute: sAMAccountName
      user_mail_attribute: mail
      group_id_attribute: cn
      group_name_attribute: cn
      group_member_attribute: member
    ```
  </Step>

  <Step title="Deploy the configuration" icon="upload">
    ```bash title="Apply LDAP configuration" theme={null}
    xavs-ansible deploy --tags keystone
    ```
  </Step>

  <Step title="Verify LDAP connectivity" icon="circle-check">
    Test the LDAP connection from the Identity API node:

    ```bash title="Test LDAP connectivity" theme={null}
    ldapsearch -x -H ldap://ldap.example.com \
      -D "cn=xloud-svc,dc=example,dc=com" \
      -w "$LDAP_PASSWORD" \
      -b "ou=Users,dc=example,dc=com" \
      "(sAMAccountName=alice)"
    ```

    <Check>User record is returned — LDAP is reachable and the bind account has read access.</Check>
  </Step>
</Steps>

<Warning>
  LDAP integration is read-only. User management (password resets, account creation)
  must be performed in the directory, not through the Xloud Dashboard or CLI.
</Warning>

***

## SAML 2.0 Federation

SAML 2.0 federation enables SSO with corporate identity providers. Users authenticate
at the IdP and receive Xloud tokens without a local password.

<Steps>
  <Step title="Configure the identity provider in your IdP" icon="building">
    Register Xloud as a service provider in your IdP. Provide the Xloud SAML metadata URL:

    ```
    https://api.<your-domain>:5000/v3/OS-FEDERATION/identity_providers/<IDP_ID>/protocols/saml2/auth
    ```
  </Step>

  <Step title="Register the IdP in Xloud" icon="plus">
    ```bash title="Create identity provider" theme={null}
    openstack identity provider create \
      --remote-id https://idp.example.com/sso/saml \
      corporate-idp
    ```
  </Step>

  <Step title="Create attribute mapping" icon="route">
    Define how IdP attributes map to Xloud groups and projects:

    ```bash title="Create mapping rules" theme={null}
    openstack mapping create \
      --rules mapping-rules.json \
      corporate-mapping
    ```

    Example mapping rules:

    ```json title="mapping-rules.json" theme={null}
    [
      {
        "local": [
          {"user": {"name": "{0}"}},
          {"group": {"id": "<XLOUD_GROUP_ID>"}}
        ],
        "remote": [
          {"type": "ADFS_LOGIN"},
          {"type": "memberOf", "any_one_of": ["CN=xloud-users,OU=Groups,DC=example,DC=com"]}
        ]
      }
    ]
    ```
  </Step>

  <Step title="Create the federation protocol" icon="link">
    ```bash title="Link IdP, mapping, and protocol" theme={null}
    openstack federation protocol create saml2 \
      --identity-provider corporate-idp \
      --mapping corporate-mapping
    ```

    <Check>Federation protocol is created. IdP users can now authenticate via SAML SSO.</Check>
  </Step>
</Steps>

***

## OpenID Connect

OIDC federation uses OAuth 2.0 bearer tokens from a compatible provider (Google, Azure AD,
Okta, Keycloak).

```bash title="Create OIDC identity provider" theme={null}
openstack identity provider create \
  --remote-id https://accounts.google.com \
  google-oidc
```

```bash title="Create OIDC mapping" theme={null}
openstack mapping create \
  --rules oidc-mapping-rules.json \
  google-mapping
```

```bash title="Create OIDC federation protocol" theme={null}
openstack federation protocol create openid \
  --identity-provider google-oidc \
  --mapping google-mapping
```

***

## Next Steps

<CardGroup>
  <Card title="Domain Management" href="/services/identity/domain-management">
    Assign different authentication backends to different domains.
  </Card>

  <Card title="Federation" href="/services/identity/federation">
    Advanced federation configuration — mapping rules and attribute assertions.
  </Card>

  <Card title="Security Hardening" href="/services/identity/security">
    Secure your authentication backends with encryption and access controls.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/identity/admin-troubleshooting">
    Debug LDAP connectivity and federation authentication issues.
  </Card>
</CardGroup>


# Identity & Access CLI Reference
Source: https://docs.xloud.tech/services/identity/cli-reference

Complete openstack CLI commands for managing projects, users, roles, groups, domains, application credentials, and tokens in Xloud Identity.

## Overview

The `openstack` identity commands manage projects, users, roles, groups, domains, application credentials, and authentication tokens. Admin-scoped commands require the `admin` role.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Admin role required for user, project, role, and domain management
  * Source your `openrc.sh` before running admin commands
</Note>

***

## Projects

<CodeGroup>
  ```bash title="List projects" theme={null}
  openstack project list
  openstack project list --domain Default
  ```

  ```bash title="Create project" theme={null}
  openstack project create \
    --domain Default \
    --description "Production workloads" \
    my-project
  ```

  ```bash title="Show project" theme={null}
  openstack project show my-project
  ```

  ```bash title="Rename project" theme={null}
  openstack project set my-project --name new-project-name
  ```

  ```bash title="Enable / disable project" theme={null}
  openstack project set --enable my-project
  openstack project set --disable my-project
  ```

  ```bash title="Delete project" theme={null}
  openstack project delete my-project
  ```
</CodeGroup>

***

## Users

<CodeGroup>
  ```bash title="List users" theme={null}
  openstack user list
  openstack user list --domain Default
  ```

  ```bash title="Create user" theme={null}
  openstack user create \
    --domain Default \
    --password-prompt \
    --email user@example.com \
    john.doe
  ```

  ```bash title="Create user with project" theme={null}
  openstack user create \
    --project my-project \
    --password PASSWORD \
    john.doe
  ```

  ```bash title="Show user" theme={null}
  openstack user show john.doe
  ```

  ```bash title="Update user name and email" theme={null}
  openstack user set john.doe \
    --name john.smith \
    --email john.smith@example.com
  ```

  ```bash title="Set password" theme={null}
  openstack user set --password-prompt john.doe
  ```

  ```bash title="Enable / disable user" theme={null}
  openstack user set --enable john.doe
  openstack user set --disable john.doe
  ```

  ```bash title="Delete user" theme={null}
  openstack user delete john.doe
  ```
</CodeGroup>

<Warning>
  Before deleting a user account, remove all role assignments for that user. A user with active role assignments cannot be deleted.
</Warning>

***

## Roles

<CodeGroup>
  ```bash title="List all roles" theme={null}
  openstack role list
  ```

  ```bash title="Show role details" theme={null}
  openstack role show member
  ```

  ```bash title="Create a custom role" theme={null}
  openstack role create network-operator
  ```

  ```bash title="Create a domain-scoped role" theme={null}
  openstack role create --domain my-domain billing-reader
  ```

  ```bash title="Delete a role" theme={null}
  openstack role delete network-operator
  ```
</CodeGroup>

***

## Role Assignments

<CodeGroup>
  ```bash title="Assign role to user on a project" theme={null}
  openstack role add \
    --user john.doe \
    --project my-project \
    member
  ```

  ```bash title="Assign role to user at domain scope" theme={null}
  openstack role add \
    --user john.doe \
    --domain my-domain \
    member
  ```

  ```bash title="Assign role to a group on a project" theme={null}
  openstack role add \
    --group operators \
    --project my-project \
    member
  ```

  ```bash title="List all role assignments (with names)" theme={null}
  openstack role assignment list --names
  ```

  ```bash title="List assignments for a specific user" theme={null}
  openstack role assignment list \
    --user john.doe \
    --names
  ```

  ```bash title="List assignments on a project" theme={null}
  openstack role assignment list \
    --project my-project \
    --names
  ```

  ```bash title="Remove role from user on a project" theme={null}
  openstack role remove \
    --user john.doe \
    --project my-project \
    member
  ```

  ```bash title="Remove group role assignment" theme={null}
  openstack role remove \
    --group operators \
    --project my-project \
    member
  ```
</CodeGroup>

***

## Implied Roles (Role Hierarchies)

Implied roles let a "prior" role automatically grant an "implied" role. Assignment is one-directional — prior → implied only.

<CodeGroup>
  ```bash title="Create an implied role rule" theme={null}
  openstack implied role create admin --implied-role member
  ```

  ```bash title="List all implied role rules" theme={null}
  openstack implied role list
  ```

  ```bash title="Delete an implied role rule" theme={null}
  openstack implied role delete admin --implied-role member
  ```
</CodeGroup>

***

## Domains

<CodeGroup>
  ```bash title="List domains" theme={null}
  openstack domain list
  ```

  ```bash title="Create domain" theme={null}
  openstack domain create \
    --description "Engineering department" \
    engineering
  ```

  ```bash title="Show domain" theme={null}
  openstack domain show engineering
  ```

  ```bash title="Enable / disable domain" theme={null}
  openstack domain set --enable engineering
  openstack domain set --disable engineering
  ```

  ```bash title="Delete domain" theme={null}
  openstack domain delete engineering
  ```
</CodeGroup>

***

## Groups

<CodeGroup>
  ```bash title="List groups" theme={null}
  openstack group list
  ```

  ```bash title="Create group" theme={null}
  openstack group create \
    --domain Default \
    --description "Cloud operators" \
    operators
  ```

  ```bash title="Add user to group" theme={null}
  openstack group add user operators john.doe
  ```

  ```bash title="Check group membership" theme={null}
  openstack group contains user operators john.doe
  ```

  ```bash title="List users in group" theme={null}
  openstack group list --user john.doe
  ```

  ```bash title="Remove user from group" theme={null}
  openstack group remove user operators john.doe
  ```

  ```bash title="Delete group" theme={null}
  openstack group delete operators
  ```
</CodeGroup>

***

## Application Credentials

<CodeGroup>
  ```bash title="List application credentials" theme={null}
  openstack application credential list
  ```

  ```bash title="Create application credential" theme={null}
  openstack application credential create \
    --role member \
    --description "CI/CD pipeline credential" \
    ci-pipeline
  ```

  ```bash title="Create with expiry" theme={null}
  openstack application credential create \
    --role member \
    --expiration "2026-12-31T00:00:00" \
    temp-credential
  ```

  ```bash title="Create with restricted access rules" theme={null}
  openstack application credential create \
    --role member \
    --access-rules '[{"service": "compute", "method": "GET", "path": "/v2.1/servers"}]' \
    readonly-compute
  ```

  ```bash title="Show application credential" theme={null}
  openstack application credential show ci-pipeline
  ```

  ```bash title="Delete application credential" theme={null}
  openstack application credential delete ci-pipeline
  ```
</CodeGroup>

***

## Tokens

<CodeGroup>
  ```bash title="Issue a token" theme={null}
  openstack token issue
  ```

  ```bash title="Issue a token (project-scoped)" theme={null}
  openstack token issue \
    --os-project-name my-project \
    --os-domain-name Default
  ```

  ```bash title="Revoke a token" theme={null}
  openstack token revoke <token-id>
  ```
</CodeGroup>

***

## Service Catalog & Endpoints

<CodeGroup>
  ```bash title="List all endpoints" theme={null}
  openstack endpoint list
  ```

  ```bash title="List public endpoints only" theme={null}
  openstack endpoint list --interface public
  ```

  ```bash title="Show endpoint details" theme={null}
  openstack endpoint show <endpoint-id>
  ```

  ```bash title="List registered services" theme={null}
  openstack service list
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Roles & Role Assignments Guide" href="/services/identity/roles">
    Create custom roles, build role hierarchies, and manage role assignments
  </Card>

  <Card title="Application Credentials Guide" href="/services/identity/application-credentials">
    Create and manage non-interactive credentials for automation and CI/CD
  </Card>

  <Card title="Projects Guide" href="/services/identity/projects">
    Manage projects, quotas, and membership
  </Card>

  <Card title="Policy Management" href="/services/identity/policy-management">
    Define per-service policy rules for custom roles
  </Card>
</CardGroup>


# Domain Management
Source: https://docs.xloud.tech/services/identity/domain-management

Create and manage organizational domains with independent user namespaces and authentication backends in Xloud Identity.

## Overview

Domains provide administrative isolation between organizations, business units, or customers.
Each domain has its own user namespace, and users in one domain cannot see users in another.
A single domain can be configured with its own authentication backend (SQL, LDAP, or
federation), making domains the fundamental multi-tenancy boundary in Xloud Identity.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Domain Concepts

| Concept            | Description                                                                                                |
| ------------------ | ---------------------------------------------------------------------------------------------------------- |
| **Default domain** | Created automatically during deployment. Contains all initial admin users and projects. Cannot be deleted. |
| **Custom domain**  | Administrator-created domain for a business unit, customer, or organizational boundary.                    |
| **Domain admin**   | A user with the `admin` role scoped to the domain. Can manage users and projects within that domain only.  |
| **Domain backend** | Each domain can use a different authentication driver — one domain uses SQL, another uses LDAP.            |

***

## Create a Domain

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Domains" icon="building">
        Log in with admin credentials. Navigate to **Identity > Domains** (admin view) and click
        **Create Domain**.
      </Step>

      <Step title="Configure the domain" icon="settings">
        | Field           | Description                            |
        | --------------- | -------------------------------------- |
        | **Name**        | Unique identifier for the domain       |
        | **Description** | Purpose or owner of the domain         |
        | **Enabled**     | Toggle on to allow user authentication |
      </Step>

      <Step title="Confirm creation" icon="circle-check">
        Click **Confirm**.
        <Check>The domain appears in **Identity > Domains** (admin view) with status Enabled.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a domain" theme={null}
    openstack domain create \
      --description "Customer A organization" \
      customer-a
    ```

    ```bash title="List all domains" theme={null}
    openstack domain list
    ```

    ```bash title="Show domain details" theme={null}
    openstack domain show customer-a
    ```
  </Tab>
</Tabs>

***

## Assign Domain Administrators

As a domain administrator, you can manage users, projects, and groups within your domain without
platform-level admin access.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    On the domain row, click the **More** dropdown and select **Manage User**. Add a user
    and assign the `admin` role to grant domain-level administration privileges.
    <Check>The domain administrator can now manage users and projects within that domain.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a domain administrator user" theme={null}
    openstack user create \
      --domain customer-a \
      --password-prompt \
      customer-a-admin
    ```

    ```bash title="Grant domain admin role" theme={null}
    openstack role add \
      --domain customer-a \
      --user customer-a-admin \
      admin
    ```

    ```bash title="Verify the domain admin assignment" theme={null}
    openstack role assignment list \
      --user customer-a-admin \
      --domain customer-a \
      --names
    ```

    <Check>The admin role assignment is visible for the domain scope.</Check>
  </Tab>
</Tabs>

***

## Disable and Delete Domains

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Identity > Domains** (admin view), open the domain, and click **Edit**.
    Toggle **Enabled** off to disable the domain. Disabled domains block all authentication
    for every user in that domain.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Disable a domain (blocks all authentication for domain users)" theme={null}
    openstack domain set --disable customer-a
    ```

    ```bash title="Delete a domain" theme={null}
    openstack domain delete customer-a
    ```

    <Warning>
      Disabling a domain immediately blocks all authentication for every user in that
      domain. All running instances and active sessions are unaffected until their
      tokens expire. Deleting a domain permanently removes all users, projects, and
      resources within it — this action cannot be undone.
    </Warning>
  </Tab>
</Tabs>

***

## Per-Domain Authentication Backends

Each domain can be assigned its own authentication driver. This enables a deployment
where the Default domain uses SQL while a `corporate` domain uses LDAP:

```yaml title="XDeploy globals: per-domain LDAP backend" theme={null}
keystone_domain_config:
  corporate:
    identity:
      driver: ldap
    ldap:
      url: ldap://ldap.corp.example.com
      user_tree_dn: ou=Users,dc=corp,dc=example,dc=com
      user_id_attribute: sAMAccountName
```

Deploy after configuring:

```bash title="Apply domain configuration" theme={null}
xavs-ansible deploy --tags keystone
```

***

## Next Steps

<CardGroup>
  <Card title="Authentication Backends" href="/services/identity/auth-backends">
    Configure LDAP and federation backends for domain authentication.
  </Card>

  <Card title="Service Catalog" href="/services/identity/service-catalog">
    Manage endpoint registration across regions for all Xloud services.
  </Card>

  <Card title="Policy Management" href="/services/identity/policy-management">
    Customize RBAC policies for domain-scoped administrative operations.
  </Card>

  <Card title="Security Hardening" href="/services/identity/security">
    Apply security best practices for domain isolation and access controls.
  </Card>
</CardGroup>


# Extended RBAC
Source: https://docs.xloud.tech/services/identity/extended-rbac

Fine-grained role-based access control for Xloud Platform — privileges, custom roles, tag conditions, and per-action enforcement on every Dashboard action.

## Overview

**Xloud Extended RBAC** is the per-action access control layer of the Xloud Platform.
It builds on the platform's existing identity and policy stack (Xloud Identity tokens,
project scoping, service-level role policy) and adds a **privilege catalog** so cloud
operators can construct custom roles that grant or deny **specific Dashboard actions**.
Each grant can optionally be scoped to a tag condition — for example, *"this role can
act on instances tagged `env=staging` only"*.

<Info>
  **Xloud-Developed** — Extended RBAC is developed by Xloud and ships as part of the
  Xloud Platform identity layer. It is an addition to (not a replacement for) the
  standard role and policy stack.
</Info>

<Note>
  **Prerequisites**

  * An administrator account on the Xloud Dashboard with the `rbac_admin` privilege
    (or the built-in `admin` role)
  * The Xloud Platform identity layer enabled (default on every cluster)
  * The RBAC gateway enabled in **XDEPLOY → Configuration** for cluster-wide
    enforcement on every direct platform service API call
</Note>

***

## Video Walkthrough

<iframe title="How to Edit Permissions on a Custom Role with Extended RBAC on Xloud" />

***

## What Fine-Grained RBAC Lets You Express

Extended RBAC was designed to express the access patterns customers expect from
mature private cloud platforms. Concrete examples:

| Statement                                                                            |                      Expressible                      |
| ------------------------------------------------------------------------------------ | :---------------------------------------------------: |
| Role *VM Power User* can power on / off VMs but cannot resize CPU or RAM             |                          Yes                          |
| Role *DBA* can act on VMs in project `prod` but only with tag `team=db`              |                          Yes                          |
| User Alice has role *Network Admin* on projects `prod` and `stage` but not `sandbox` |             Yes (via Identity assignment)             |
| Role *Storage Operator* can attach volumes but not create or delete them             |                          Yes                          |
| Read-only auditor sees read-only data even when calling aggregator endpoints         |                          Yes                          |
| Every change to a permission is auditable with who / when / old / new                |                          Yes                          |
| Revoking a role takes effect within 5 seconds across every node                      |                          Yes                          |
| The `admin` role has zero privileges on project X if explicitly denied               | No — admin remains a deliberate platform bypass in v1 |

***

## Levels of Access Control

Extended RBAC is one layer in a five-layer access stack:

| Level                               | What it controls                                                  | Configuration surface                                                                      |
| ----------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| **1. Authentication**               | Who can sign in (password, TOTP MFA, federation)                  | Xloud Identity, [User Center](/services/dashboard/user-guide/user-center)                  |
| **2. Project membership and roles** | Project membership and Identity-side roles                        | [Identity → Roles](/services/identity/roles), [Identity → Users](/services/identity/users) |
| **3. Service-level policy**         | Per-service `policy.yaml` rules                                   | [Policy Management](/services/identity/policy-management)                                  |
| **4. Extended RBAC privileges**     | Per-Dashboard-action allow / deny rules with optional tag scoping | This page                                                                                  |
| **5. Audit + invalidation**         | Role and grant changes recorded; cross-node cache stays coherent  | RBAC audit log + invalidation poller                                                       |

A typical production setup uses all five.

***

## Enabling the RBAC Gateway

Extended RBAC's management surface is always active — privileges, custom roles, tag
conditions, and the audit log are available from the Dashboard out of the box. The
**RBAC gateway** is the additional switch that decides whether Extended RBAC
enforces every direct platform service API call cluster-wide, or only the calls
that go through the Xloud Dashboard's extension APIs.

The gateway is enabled through **XDEPLOY** — there is no other supported
configuration path.

<Steps>
  <Step title="Open XDEPLOY → Configuration" icon="settings">
    From XDEPLOY, navigate to **Configuration**.
  </Step>

  <Step title="Enable the RBAC gateway" icon="toggle-right">
    Locate the **Enable RBAC Authorization Gateway** toggle and switch it on. Save
    the configuration.
  </Step>

  <Step title="Run reconfigure" icon="play">
    From XDEPLOY → Operations, run a **reconfigure** for the Dashboard service.
    The reverse proxy is regenerated with the gateway active and reloaded
    without downtime.
  </Step>

  <Step title="Verify" icon="check">
    Open the Dashboard. From this point forward, every direct platform service API
    call passes through Extended RBAC's enforce layer before reaching the
    backend service.

    <Check>The cluster now enforces Extended RBAC on every direct API call.</Check>
  </Step>
</Steps>

<Tip>
  Rolling back is symmetrical — flip the toggle off and reconfigure. No database
  changes are needed; privilege grants and audit history persist regardless of
  whether the gateway is on.
</Tip>

***

## What You Can Control Per Service

Extended RBAC privileges map one-to-one to the **action menus the Dashboard
exposes**. The lists below are the actual GUI actions available per service —
every entry is independently grantable to a custom role, and most can be
tag-conditioned.

<AccordionGroup>
  <Accordion title="Compute — Instances" icon="server">
    **Primary action**: Create Instance.

    **Row actions** (organized into submenus in the More dropdown):

    * **Instance Status**: Start, Stop, Lock, Unlock, Reboot, Soft Reboot, Suspend,
      Resume, Pause, Unpause, Shelve, Unshelve
    * **Related Resources**: Attach Interface, Detach Interface, Attach Volume,
      Detach Volume, Associate Floating IP, Disassociate Floating IP, Manage
      Security Group
    * **Backups and Snapshots**: Create Snapshot
    * **Clone and Template**: Clone, Convert to Template
    * **Configuration Update**: Resize, Confirm Resize or Migrate, Revert Resize or
      Migrate, Adjust Resources (live resize — vCPU / RAM / device hot-add), Change
      Password, Rebuild Instance
    * **Other row actions**: Console, Edit Instance, Modify Instance Tags, Delete
    * **Batch actions**: Start, Stop, Reboot, Soft Reboot, Delete
    * **Admin-view extras**: Migrate, Live Migrate, Bulk Live Migrate

    Each item is independently grantable. Console, Create Instance, and Delete are
    three separate privileges — they are not bundled.

    **Script execution endpoints** — Xloud has no general "exec arbitrary command
    on a running VM" endpoint. Script execution flows through two distinct Compute
    API endpoints, each gated by its own privilege:

    | Endpoint                                                 | What it does                                                            | Compute privilege                                                                                                  |
    | -------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
    | `POST /v2.1/servers` (with `user_data`)                  | Inject cloud-init / Bash / PowerShell at first boot                     | `os_compute_api:servers:create` plus a separate User Data attachment privilege when fine-grained gating is enabled |
    | `POST /v2.1/servers/{id}/action` (with `changePassword`) | Reset the guest OS root or administrator password on a running instance | `os_compute_api:os-admin-password`                                                                                 |

    Implications: a role can be granted Create Instance while User Data attachment
    is independently allowed or denied; a role with Start / Stop / Reboot / Resize
    does not automatically get `changePassword`. Both privileges can be
    tag-conditioned (for example, *allow `changePassword` on `env=staging`
    instances only*).
  </Accordion>

  <Accordion title="Compute — Snapshots, Images, Flavors, Keypairs, Server Groups, Templates" icon="layers">
    * **Instance Snapshots**: Edit, Browse Files (file-level recovery), Rollback,
      Create Instance, Create Volume, Delete
    * **Images**: Create, Edit, Browse Files, Create Instance, Create Volume, Delete;
      admin extras — Manage Access, Manage Metadata
    * **Flavors**: Create, Manage Access, Delete (read-only catalog access is itself
      a separate privilege)
    * **Keypairs**: Create, Delete
    * **Server Groups**: Create, Delete, Create Instance into the group
    * **VM Templates**: Create Template, Edit, Deploy, Delete; admin — Manage Access
    * **Hypervisors** (admin): Host Management
    * **Bare Metal Nodes** (admin): Create, Manage State, Edit, Power On, Power Off,
      Inspect, Set Maintenance, Clear Maintenance, Set Boot Device, Create Port,
      Create Port Group, Delete
  </Accordion>

  <Accordion title="Storage — Volumes, Snapshots, Backups, Volume Types" icon="hard-drive">
    **Volumes**:

    * Primary actions: Create, Accept Volume Transfer
    * Row first action: Edit
    * Submenu **Data Protection**: Create Snapshot, Create Backup, Create Image,
      Clone Volume, Restore
    * Submenu **Instance Related**: Bootable, Create Instance, Attach, Detach
    * Submenu **Capacity and Type**: Extend Volume, Change Type
    * Other row actions: Create Transfer, Cancel Transfer, Delete
    * Admin extras: Update Status, Migrate, Live Retype

    **Snapshots**: Create, Edit, Restore, Create Volume, Delete

    **Backups**: Create, Edit, Browse Files (file-level recovery), Create Volume,
    Delete

    **Volume Types** (admin): Create, Manage QoS, Set Provisioning, Manage Access,
    Create Encryption, Delete Encryption, Edit, Delete
  </Accordion>

  <Accordion title="Network — Networks, Routers, Floating IPs, Security Groups, Ports, Load Balancers, DNS" icon="network">
    * **Networks**: Create Network, Edit, Create Subnet, Delete
    * **Routers**: Create, Edit, Connect Subnet, Disconnect Subnet, Set Gateway,
      Close Gateway, Enable SNAT, Disable SNAT, Delete
    * **Floating IPs**: Allocate, Edit, Associate, Disassociate, Create Port
      Forwarding, Release
    * **Security Groups**: Create, Edit, Create Rule, Delete
    * **Ports**: Create, Edit, Attach Instance, Associate FIP, Disassociate FIP,
      Detach, Modify QoS, Manage Security Group, Delete
    * **Load Balancers**: Create, Edit, Associate FIP, Disassociate FIP, Delete
    * **DNS Zones**: Create, Update, Create Records, Delete
  </Accordion>

  <Accordion title="Identity — Projects, Users, Roles, Domains, User Groups" icon="users">
    * **Projects**: Create, Edit, Delete, Manage Quota, Manage User, Manage User
      Group, Enable, Forbidden, Modify Tags
    * **Users**: Create, Edit, System Role, Set Default Project, Password, Reset MFA
      (admin escape hatch), Enable, Forbidden, Delete
    * **Roles**: Create, Edit, Delete
    * **Domains**: Create, Edit, Enable, Forbidden, Delete
    * **User Groups**: Create, Edit, Manage User, Delete
  </Accordion>

  <Accordion title="Key Manager — Secrets, Containers, Certificates" icon="lock">
    * **Secrets**: Create, Delete
    * **Containers**: Create, Delete
    * **Certificates** (under Network → Certificate): Create, Delete
  </Accordion>

  <Accordion title="Orchestration — Stacks" icon="code">
    * **Stacks**: Create, Edit, Abandon, Delete
  </Accordion>

  <Accordion title="Optimization (Resource Optimizer)" icon="rotate-cw">
    * **Audits**: Create, Delete
    * **Audit Templates**: Create, Delete
    * **Action Plans**: Start (execute the plan), Delete

    Goals and Strategies are read-only catalogues with no actions to gate.
  </Accordion>

  <Accordion title="Instance HA — Segments, Hosts, Notifications" icon="shield">
    * **Segments**: Create, Update, Add Host, Delete
    * **Hosts**: Update, Delete
    * **Notifications**: Create, Delete
  </Accordion>

  <Accordion title="Container Infrastructure — Clusters, Cluster Templates" icon="cpu">
    * **Clusters**: Create, Delete, Get Cluster Config, Show Certificate, Sign
      Certificate, Resize, Upgrade, Launch Dashboard
    * **Cluster Templates**: Create, Edit, Create Cluster, Delete
  </Accordion>
</AccordionGroup>

<Info>
  Some Xloud-developed actions (Clone, Convert to Template, Adjust Resources, Browse
  Files, Rollback, Reset MFA, Sign Certificate, Bulk Live Migrate) are gated **only**
  by Extended RBAC. For these actions Extended RBAC is the sole enforcement layer,
  even when the cluster gateway is off.
</Info>

***

## Manage RBAC from the Dashboard

Extended RBAC is managed from three pages under **Identity** in the admin view of
the Dashboard. The pages have separate responsibilities — managing roles is
intentionally distinct from editing what a role can do, and from reviewing what
changed.

### Roles — `Identity → Roles`

The existing Roles list is enhanced with Extended RBAC awareness. It remains the
single place where roles are created, deleted, and assigned to users.

| Column / action            | What it does                                                                                                                                                  |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Type** column            | Tags each role as **Custom** (blue) or **System** (grey). Custom roles are editable on the RBAC Permissions page; System roles are read-only                  |
| **Create Role** (primary)  | Opens the create-role wizard. The role is created in Xloud Identity AND a matching Custom-role record is created in Extended RBAC — there is no separate step |
| **Edit** (row)             | Edit name and description                                                                                                                                     |
| **Delete** (row)           | Removes the role from both Xloud Identity and Extended RBAC in one transaction. Disabled for System roles                                                     |
| **Edit Permissions** (row) | Jumps to the RBAC Permissions page pre-filled for that role. Disabled for System roles with a tooltip *"System roles cannot be edited"*                       |
| **Manage Users**           | Existing flow — assign or remove the role on a per-user basis                                                                                                 |

### RBAC Permissions — `Identity → RBAC Permissions`

The dedicated permissions editor — its single responsibility is *"given a custom
role, edit what privileges it has."* No role create / delete / assignment lives
here.

<Steps>
  <Step title="Pick a Custom role" icon="user-check">
    The role selector at the top of the page lists every Custom role. System roles
    do not appear because they cannot be edited.
  </Step>

  <Step title="Toggle privileges in the matrix" icon="grid">
    Privileges are grouped by service (Compute, Storage, Network, Identity, Key
    Manager, Orchestration, Optimization, Instance HA, Container Infrastructure)
    and by category within each service (for example, **Instance Lifecycle**,
    **Instance Actions**, **Backups and Snapshots**). Each row has:

    * A checkbox to grant or revoke the privilege
    * A hover tooltip showing the privilege description, prerequisite chain, and
      any associated service-level policy rule
    * A condition chip if a tag condition is attached to this grant
  </Step>

  <Step title="Add a tag condition" icon="tag">
    Click **+ Add Tag Condition** on any granted privilege to scope it. The tag
    condition modal lets you pick a tag key, an operator
    (`equals`, `not_equals`, `contains`, `starts_with`, `any_of`), and a value.
    Conditions are reusable — one *"env=prod"* condition can be referenced by many
    grants.
  </Step>

  <Step title="Preview Effective Rules" icon="flask-conical">
    Before saving, click **Preview Effective Rules** to simulate what a real user
    holding this role would be allowed to do against representative resources. The
    simulator runs the same enforcement engine the live request path uses.
  </Step>

  <Step title="Save" icon="check">
    On Save, the backend validates that prerequisite chains are intact. If a
    privilege you toggled requires a prerequisite that is not yet granted, a
    **Missing Prerequisites** modal lists them and offers to enable them
    automatically. The grant batch is written atomically with an audit log entry
    and a cross-node invalidation event — every Dashboard and apiserver node sees
    the change within 5 seconds.

    <Check>Your changes are live within 5 seconds across the entire cluster.</Check>
  </Step>
</Steps>

### RBAC Audit Log — `Identity → RBAC Audit Log`

Every change made through Extended RBAC is recorded in an append-only log,
queryable from the Audit Log page.

| Filter          | Purpose                                                                                                         |
| --------------- | --------------------------------------------------------------------------------------------------------------- |
| **Actor**       | Who made the change (user dropdown)                                                                             |
| **Target Role** | Which role was modified                                                                                         |
| **Action Type** | `role.create`, `role.delete`, `role.grant`, `role.revoke`, `priv.add`, `priv.remove`, `condition.set`, `bypass` |
| **Time Range**  | Scope to a window                                                                                               |

Each row shows actor, action, target, source IP, request ID, and an old → new diff.
Expand a row to see the full JSON diff.

<Tip>
  The Audit Log page is also reachable as **View History** from any Custom row on
  the Roles page and as **Audit Log** from the header of the RBAC Permissions page.
</Tip>

***

## Examples — Common Role Designs

These are example role designs you can build using the privilege matrix and tag
conditions described above.

<AccordionGroup>
  <Accordion title="VM Power User — power on / off but cannot resize or edit" icon="power">
    Grant `instance.start`, `instance.stop`, `instance.reboot`, `instance.suspend`,
    `instance.resume`, `instance.pause`, and `instance.unpause`. Withhold
    `instance.resize`, `instance.adjust_resources` (live resize), and
    `instance.edit`. Operators keep workloads alive without changing their shape.
  </Accordion>

  <Accordion title="Project DBA — act only on tagged database VMs" icon="database">
    A role scoped to project `prod` that grants instance management privileges only
    when the target VM carries the tag `team=db`. The user holds zero privileges on
    instances tagged `team=web` or `team=api`.
  </Accordion>

  <Accordion title="Storage Operator — attach but not create or delete" icon="hard-drive">
    Grant `volume.attach`, `volume.detach`, `volume.read`. Withhold
    `volume.create` and `volume.delete`. The operator wires existing storage but
    cannot add or remove volumes.
  </Accordion>

  <Accordion title="Read-only auditor — list and view across services" icon="eye">
    Grant only `*.list` and `*.read` privileges. Even when the user calls
    aggregator endpoints, Extended RBAC filters the response rows so the auditor
    sees only resources their tag conditions permit.
  </Accordion>

  <Accordion title="Decommission role — delete only" icon="trash">
    Grant `instance.delete` and `volume.delete`, nothing else. Useful for
    end-of-life automation.
  </Accordion>
</AccordionGroup>

<Note>
  The `admin` role remains a deliberate platform bypass. Patterns of the form
  *"admin has zero privileges on project X if denied"* are not expressible — the
  only ways to constrain admin are `disable_admin_bypass` (cluster-wide) or
  `require_system_scope_for_admin` (force system-scope tokens for admin actions).
</Note>

***

## Tag Conditions

Tag conditions let one privilege grant target only resources matching a tag —
without creating multiple roles. They are evaluated against the resource being
acted on (instance tags, volume metadata, network tags, etc.).

| Operator      | Example      | Matches                           |
| ------------- | ------------ | --------------------------------- |
| `equals`      | `env=prod`   | tag exactly equals `prod`         |
| `not_equals`  | `env=prod`   | tag is anything other than `prod` |
| `contains`    | `prod`       | tag contains the substring `prod` |
| `starts_with` | `web-`       | tag starts with `web-`            |
| `any_of`      | `[web, api]` | tag matches any value in the list |

Build conditions in the **+ Add Tag Condition** modal on the RBAC Permissions
page. Conditions are reusable across roles — one *"env=prod"* condition can be
referenced by many grants.

***

## Enforcement Model

When Extended RBAC evaluates whether a request is allowed:

<Steps>
  <Step title="Anonymous check" icon="user-x">
    Requests with no user context are denied.
  </Step>

  <Step title="Admin bypass" icon="key">
    The built-in `admin` role passes through unless `disable_admin_bypass` is set.
    An emergency override is available for incident response.
  </Step>

  <Step title="Privilege lookup" icon="search">
    Resolve the request to a privilege code. Unknown URLs fall through to allow
    (compatibility mode) by default; switch to fail-closed in strict deployments.
  </Step>

  <Step title="Prerequisite walk" icon="link">
    Walk the prerequisite chain (max depth 8). A grant with a tag condition does
    not satisfy an unconditional prerequisite — this prevents bypass.
  </Step>

  <Step title="Rule resolution" icon="scale">
    Collect every grant the caller's roles produce, sort by priority descending,
    and return the first rule whose tag condition matches the target.
  </Step>

  <Step title="Audit" icon="clipboard-list">
    Role and grant changes are recorded unconditionally. Allow / deny decisions
    are recorded when audit-deny logging is enabled.
  </Step>
</Steps>

***

## Cross-Node Invalidation

Multi-node deployments stay coherent through a transactional invalidation log.
Every role mutation, privilege grant, or condition change is appended to the log
in the same database transaction as the underlying change. Other nodes poll the
log every couple of seconds and refresh their local caches when they see a new
event — privilege changes propagate to every node within \~5 seconds without
restarting any service.

***

## Best Practices

<AccordionGroup>
  <Accordion title="Start with built-in roles, add Custom roles only for exceptions" icon="layers">
    The built-in roles cover most cases. Reach for a Custom role when you have a
    specific exception — *"this team needs to resize but not delete"* — rather than
    rebuilding from scratch.
  </Accordion>

  <Accordion title="Use tag conditions for environment isolation" icon="tag">
    Tag every resource with `env=prod`, `env=staging`, or `env=dev`, and create
    one Custom role per privilege scope, scoped to the relevant tag. Avoids
    combinatorial role explosion.
  </Accordion>

  <Accordion title="Preview effective rules before you save" icon="flask-conical">
    Use **Preview Effective Rules** on the RBAC Permissions page to simulate a
    role change against representative users and resources before applying it.
  </Accordion>

  <Accordion title="Keep admin bypass on for break-glass" icon="alert-triangle">
    Disabling admin bypass globally is rarely worth the operational risk. Audit
    the bypass log and tighten via `require_system_scope_for_admin` if your
    compliance framework requires project-scoped admin separation.
  </Accordion>

  <Accordion title="Forward audit log to long-term storage" icon="archive">
    The audit log retention defaults to one year. Forward to your SIEM or
    long-term object storage for longer compliance windows.
  </Accordion>
</AccordionGroup>

***

## Related Topics

<CardGroup>
  <Card title="Roles" href="/services/identity/roles">
    Manage Xloud Identity roles — create, assign, implied-roles
  </Card>

  <Card title="Policy Management" href="/services/identity/policy-management">
    Service-level policy YAML hardening
  </Card>

  <Card title="Multi-Factor Auth" href="/services/identity/multi-factor-auth">
    Strengthen sign-in with TOTP MFA
  </Card>
</CardGroup>


# Identity Federation
Source: https://docs.xloud.tech/services/identity/federation

Configure SAML 2.0 and OpenID Connect federation for enterprise single sign-on with Xloud Identity.

## Overview

Federation allows enterprise users to authenticate with Xloud using their existing
corporate identity provider (IdP) — no separate Xloud password required. Xloud Identity
supports SAML 2.0 and OpenID Connect (OIDC) protocols. Users authenticate at the IdP
and receive Xloud tokens mapped from their IdP attributes, inheriting project membership
and roles through attribute mapping rules.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Federation Architecture

```mermaid theme={null}
sequenceDiagram
    participant U as User (Browser/CLI)
    participant I as Xloud Identity
    participant IDP as Corporate IdP (SAML/OIDC)
    U->>I: Request token (federation protocol)
    I-->>U: Redirect to IdP
    U->>IDP: Authenticate (AD password, MFA)
    IDP-->>U: SAML assertion / OIDC token
    U->>I: Submit assertion/token
    I->>I: Apply mapping rules
    I-->>U: Xloud token + service catalog
```

***

## SAML 2.0 Setup

<Steps>
  <Step title="Register Xloud as SP in your IdP" icon="building">
    Provide your IdP with the Xloud SAML SP metadata URL:

    ```
    https://api.<your-domain>:5000/v3/OS-FEDERATION/identity_providers/<IDP_ID>/protocols/saml2/auth
    ```

    Configure the IdP to send the following SAML attributes:

    * `ADFS_LOGIN` or `mail` — the user's login name
    * `memberOf` — group membership for role mapping
  </Step>

  <Step title="Register the IdP in Xloud" icon="plus">
    ```bash title="Create identity provider" theme={null}
    openstack identity provider create \
      --remote-id https://idp.example.com/sso/saml \
      --description "Corporate Active Directory Federation" \
      corporate-idp
    ```
  </Step>

  <Step title="Create attribute mapping rules" icon="route">
    Mapping rules translate IdP attributes into Xloud group memberships:

    ```json title="mapping-rules.json" theme={null}
    [
      {
        "local": [
          {"user": {"name": "{0}", "domain": {"name": "Default"}}},
          {"group": {"id": "<XLOUD_GROUP_ID>"}}
        ],
        "remote": [
          {"type": "ADFS_LOGIN"},
          {
            "type": "memberOf",
            "any_one_of": ["CN=cloud-users,OU=Groups,DC=example,DC=com"]
          }
        ]
      }
    ]
    ```

    ```bash title="Upload mapping rules" theme={null}
    openstack mapping create \
      --rules mapping-rules.json \
      corporate-mapping
    ```
  </Step>

  <Step title="Create the federation protocol" icon="link">
    ```bash title="Link IdP, mapping, and SAML protocol" theme={null}
    openstack federation protocol create saml2 \
      --identity-provider corporate-idp \
      --mapping corporate-mapping
    ```

    <Check>Federation protocol is active. Test by authenticating via the SSO URL.</Check>
  </Step>
</Steps>

***

## OpenID Connect Setup

<Steps>
  <Step title="Register Xloud as OIDC client in your IdP" icon="building">
    Register a new application in your OIDC provider (Keycloak, Azure AD, Okta):

    * **Redirect URI**: `https://api.<your-domain>:5000/v3/OS-FEDERATION/identity_providers/<IDP_ID>/protocols/openid/auth/callback`
    * **Grant type**: Authorization Code
    * **Scopes**: `openid`, `profile`, `email`, `groups`
  </Step>

  <Step title="Register the OIDC IdP in Xloud" icon="plus">
    ```bash title="Create OIDC identity provider" theme={null}
    openstack identity provider create \
      --remote-id https://accounts.google.com \
      --description "Google Workspace SSO" \
      google-oidc
    ```
  </Step>

  <Step title="Create OIDC mapping rules" icon="route">
    ```json title="oidc-mapping-rules.json" theme={null}
    [
      {
        "local": [
          {"user": {"name": "{0}"}},
          {"group": {"id": "<XLOUD_GROUP_ID>"}}
        ],
        "remote": [
          {"type": "email"},
          {"type": "groups", "any_one_of": ["xloud-admins@example.com"]}
        ]
      }
    ]
    ```

    ```bash title="Create OIDC mapping" theme={null}
    openstack mapping create \
      --rules oidc-mapping-rules.json \
      google-mapping
    ```
  </Step>

  <Step title="Create the OIDC protocol" icon="link">
    ```bash title="Create OIDC federation protocol" theme={null}
    openstack federation protocol create openid \
      --identity-provider google-oidc \
      --mapping google-mapping
    ```
  </Step>
</Steps>

***

## Mapping Rule Reference

| Mapping Field       | Description                                                            |
| ------------------- | ---------------------------------------------------------------------- |
| `local.user.name`   | Maps to the Xloud username for the federated session                   |
| `local.group.id`    | Assigns the user to an Xloud group (inherits group's role assignments) |
| `remote.type`       | The IdP attribute name to match                                        |
| `remote.any_one_of` | User must belong to at least one of these values                       |
| `remote.not_any_of` | User must not belong to any of these values                            |

***

## Next Steps

<CardGroup>
  <Card title="Authentication Backends" href="/services/identity/auth-backends">
    Compare federation with LDAP and SQL backend options.
  </Card>

  <Card title="Domain Management" href="/services/identity/domain-management">
    Assign federation backends to specific organizational domains.
  </Card>

  <Card title="Security Hardening" href="/services/identity/security">
    Secure federation endpoints and enforce MFA for federated sessions.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/identity/admin-troubleshooting">
    Debug SAML assertion errors and OIDC token mapping failures.
  </Card>
</CardGroup>


# Identity & Access
Source: https://docs.xloud.tech/services/identity/index

Authentication, authorization, and access management for Xloud Cloud Platform.

<p>Overview</p>

Xloud Identity is the authentication and authorization backbone of the Xloud Cloud Platform.
Every API request, Dashboard login, and CLI command is validated against Xloud Identity before
any resource operation proceeds. It manages the complete access control lifecycle — from
issuing scoped tokens to enforcing fine-grained role-based policies across every service.

<Note>
  **Prerequisites**

  * An active Xloud account with admin or project-member privileges
  * Access to the **Xloud Dashboard** (`https://connect.<your-domain>`) or `openstack` CLI
  * For administration tasks: XDeploy access and admin credentials
</Note>

***

<p>What Xloud Identity Provides</p>

<CardGroup>
  <Card title="Authentication" icon="key" href="/services/identity/auth-backends">
    Token-based authentication with configurable backends — local SQL, LDAP, and federated
    identity providers.
  </Card>

  <Card title="Authorization" icon="shield" href="/services/identity/policy-management">
    Role-based access control (RBAC) with fine-grained policy rules governing every
    service operation across all projects.
  </Card>

  <Card title="Multi-Domain Tenancy" icon="building" href="/services/identity/domain-management">
    Hierarchical domain and project structure supporting full organizational separation
    across teams, departments, and customers.
  </Card>

  <Card title="Federation" icon="link" href="/services/identity/federation">
    Single sign-on integration with SAML 2.0 and OpenID Connect identity providers for
    enterprise directory integration.
  </Card>

  <Card title="Application Credentials" icon="terminal" href="/services/identity/application-credentials">
    Non-interactive, scoped credentials for automation pipelines, CI/CD, and service
    accounts — without exposing user passwords.
  </Card>

  <Card title="Service Catalog" icon="list" href="/services/identity/service-catalog">
    Centralized registry of all Xloud service endpoints, enabling clients to discover
    the correct API address for each region and interface.
  </Card>
</CardGroup>

***

<p>Core Concepts</p>

| Concept                    | Description                                                                                                                             |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Domain**                 | Top-level administrative boundary. Contains projects, users, and groups. The `Default` domain is created during deployment.             |
| **Project**                | Resource namespace for billing, quotas, and access isolation. All cloud resources belong to a project.                                  |
| **User**                   | A human or service account identity. Users authenticate and receive tokens scoped to a project or domain.                               |
| **Role**                   | Named set of permissions. Roles are assigned to users or groups within a project or domain.                                             |
| **Token**                  | A time-limited bearer credential issued after successful authentication. Tokens encode the scope (project/domain) and role assignments. |
| **Group**                  | A collection of users. Role assignments on a group propagate to all members.                                                            |
| **Application Credential** | A delegated credential bound to a user's roles, used for non-interactive automation without password exposure.                          |

***

<p>How Authentication Works</p>

```mermaid theme={null}
sequenceDiagram
    participant U as User / Service
    participant I as Xloud Identity
    participant S as Xloud Service (e.g. Compute)
    U->>I: POST /v3/auth/tokens (credentials + scope)
    I-->>U: X-Subject-Token + catalog
    U->>S: API Request + X-Auth-Token header
    S->>I: GET /v3/auth/tokens (token validation)
    I-->>S: Token valid + roles
    S-->>U: API Response
```

Every token carries a <Tooltip>scope</Tooltip> and a set of role assignments. Services validate the token on every request and enforce the platform's RBAC policies before executing any operation.

***

<p>Guides</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/identity/user-guide">
    Manage projects, users, roles, application credentials, and multi-factor authentication
    from the Dashboard and CLI.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/identity/admin-guide">
    Configure authentication backends, domains, token policies, federation, and security
    hardening for production deployments.
  </Card>

  <Card title="Authentication & CLI" icon="terminal" href="/services/identity/cli-reference">
    Source credentials, configure the `openstack` CLI, and authenticate to the Xloud
    Dashboard.
  </Card>

  <Card title="Compute Service" icon="server" href="/services/compute">
    Learn how Xloud Identity tokens authorize access to compute resources and instances.
  </Card>
</CardGroup>


# Multi-Factor Authentication
Source: https://docs.xloud.tech/services/identity/multi-factor-auth

Enable and manage TOTP-based multi-factor authentication for Xloud Identity user accounts.

## Overview

Xloud Identity supports TOTP-based (Time-based One-Time Password) multi-factor authentication
for user accounts. Enabling MFA adds a second verification step — a rotating 6-digit code
generated by an authenticator app — beyond the user's password. MFA significantly reduces
the risk of account compromise from credential theft.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  **Requirements**

  * A compatible authenticator application: Google Authenticator, Authy, 1Password, or any
    RFC 6238-compliant TOTP app
  * The user account must be active and have a valid password before enrolling MFA
</Note>

***

## Video Walkthrough

<iframe title="How to Set Up 2FA and User Center on Xloud" />

***

## Enroll a TOTP Device

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    The Dashboard has a self-service enrollment flow in **User Center → Security (2FA)**.
    Open the profile menu in the top-right, pick **Security (2FA)**, click
    **Enable 2FA**, scan the QR with any authenticator app, verify the 6-digit code,
    and save the recovery codes.

    For the full step-by-step walkthrough, see the
    [User Center guide](/services/dashboard/user-guide/user-center).
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Generate a TOTP seed" icon="plus">
        ```bash title="Create TOTP credential for the current user" theme={null}
        openstack credential create \
          --type totp \
          --user $(openstack token issue -f value -c user_id) \
          "{\"seed\": \"$(python3 -c 'import base64,os; print(base64.b32encode(os.urandom(20)).decode())')\"}"
        ```

        Note the `seed` value from the output.
      </Step>

      <Step title="Register seed in authenticator app" icon="smartphone">
        Import the base32-encoded seed into your authenticator app manually (use the
        "Enter setup key" option). The app begins generating 6-digit TOTP codes.
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Authenticate with MFA

Once MFA is enabled, every login requires the TOTP code in addition to the password.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    On the login page, enter your username and password as usual. A second prompt appears
    requesting the TOTP code from your authenticator app. Enter the current 6-digit code
    and click **Sign In**.

    <Warning>
      TOTP codes are valid for 30 seconds. If you enter an expired code, wait for the next
      code to appear in your authenticator app and try again.
    </Warning>
  </Tab>

  <Tab title="CLI" icon="terminal">
    When MFA is enabled, standard token issuance fails. Use the multi-factor auth method:

    ```bash title="Authenticate with password + TOTP" theme={null}
    openstack token issue \
      --os-auth-type v3multifactor \
      --os-auth-methods password,totp \
      --os-passcode <6-DIGIT-CODE>
    ```

    For automation with application credentials, MFA is **not** required — application
    credentials bypass the MFA requirement by design.
  </Tab>
</Tabs>

***

## Remove MFA Enrollment

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Open **User Center → Security (2FA)**, click **Disable 2FA**, and confirm with a
    current 6-digit code from your authenticator (or a recovery code if you have
    lost the authenticator). Full walkthrough in the
    [User Center guide](/services/dashboard/user-guide/user-center).

    <Warning>
      Removing MFA reduces account security — only do so when you are about to
      re-enroll with a new authenticator device.
    </Warning>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List TOTP credentials for current user" theme={null}
    openstack credential list \
      --type totp \
      --user $(openstack token issue -f value -c user_id)
    ```

    ```bash title="Delete TOTP credential" theme={null}
    openstack credential delete <CREDENTIAL_ID>
    ```
  </Tab>
</Tabs>

***

## MFA Best Practices

<AccordionGroup>
  <Accordion title="Require MFA for administrative accounts" icon="shield">
    All accounts with the `admin` role should have MFA enforced. Platform administrators
    can configure an MFA enforcement policy via the Identity service to block admin token
    issuance without a valid TOTP factor. See the
    [Identity Admin Guide](/services/identity/admin-guide) for policy configuration.
  </Accordion>

  <Accordion title="Use application credentials for automation" icon="terminal">
    Automation pipelines should never depend on interactive MFA. Use
    [application credentials](/services/identity/application-credentials) for CI/CD systems
    and service accounts — these bypass MFA by design and provide explicit expiry and
    access rule controls.
  </Accordion>

  <Accordion title="Plan for device loss" icon="life-buoy">
    Establish a recovery procedure before users lose access to their authenticator device:

    * Store backup codes in a password manager at enrollment time
    * Designate an administrator contact who can reset MFA enrollments
    * Document the reset process in your team's runbook
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Generate automation credentials that bypass MFA for CI/CD pipelines.
  </Card>

  <Card title="Users" href="/services/identity/users">
    Manage the user accounts on which MFA is enrolled.
  </Card>

  <Card title="Identity Admin Guide — Security" href="/services/identity/security">
    Configure MFA enforcement policies and security hardening for the platform.
  </Card>

  <Card title="Troubleshooting" href="/services/identity/troubleshooting">
    Resolve MFA authentication failures and device enrollment issues.
  </Card>
</CardGroup>


# Policy Management
Source: https://docs.xloud.tech/services/identity/policy-management

Customize RBAC policies to control which roles can perform each API operation across Xloud services. Create custom roles and map them to service-level.

## Overview

Xloud services enforce role-based access control through **policy files**. Each service has a `policy.yaml` (or `policy.json`) file that maps API operations to role requirements. The default policies follow the principle of least privilege — `admin` gets full access, `member` gets project-scoped CRUD, and `reader` gets read-only access. This guide covers viewing default policies, creating custom roles, writing per-service overrides, and applying changes safely.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Default Policy Overview

Xloud ships with three built-in roles and a set of default policies enforced across all services:

| Role     | Typical Policy Permissions                                 |
| -------- | ---------------------------------------------------------- |
| `admin`  | Full access to all APIs including administrative endpoints |
| `member` | Project-scoped create, read, update, delete operations     |
| `reader` | Project-scoped read-only access                            |

The `admin` role implies `member`, which implies `reader` — so each role inherits the permissions of those below it.

<Info>
  Policy files use **oslo.policy** syntax. Rules can reference roles, project context, object ownership, and system scope. You only need to define overrides — unspecified rules use the service's compiled-in defaults.
</Info>

***

## Policy File Locations

Each Xloud service reads its policy file from inside its container. Overrides are placed in `/etc/xavs/<service>/` on the host and deployed by XDeploy:

| Service                     | Policy Override Path              | Container Path               |
| --------------------------- | --------------------------------- | ---------------------------- |
| **Compute** (Nova)          | `/etc/xavs/nova/policy.yaml`      | `/etc/nova/policy.yaml`      |
| **Block Storage** (Cinder)  | `/etc/xavs/cinder/policy.yaml`    | `/etc/cinder/policy.yaml`    |
| **Networking** (Neutron)    | `/etc/xavs/neutron/policy.yaml`   | `/etc/neutron/policy.yaml`   |
| **Image** (Glance)          | `/etc/xavs/glance/policy.yaml`    | `/etc/glance/policy.yaml`    |
| **Identity** (Keystone)     | `/etc/xavs/keystone/policy.yaml`  | `/etc/keystone/policy.yaml`  |
| **Orchestration** (Heat)    | `/etc/xavs/heat/policy.yaml`      | `/etc/heat/policy.yaml`      |
| **Object Storage** (Swift)  | `/etc/xavs/swift/policy.yaml`     | `/etc/swift/policy.yaml`     |
| **DNS** (Designate)         | `/etc/xavs/designate/policy.yaml` | `/etc/designate/policy.yaml` |
| **Key Manager** (Barbican)  | `/etc/xavs/barbican/policy.yaml`  | `/etc/barbican/policy.yaml`  |
| **Load Balancer** (Octavia) | `/etc/xavs/octavia/policy.yaml`   | `/etc/octavia/policy.yaml`   |

<Tip>
  Start with an empty override file and add only the rules you need to change. All other rules fall back to the service's built-in defaults.
</Tip>

***

## Custom Roles and Per-Service Policies

Custom roles are names you create in Keystone. They have **no permissions by default** — you must explicitly grant them permissions in each service's policy file.

### Workflow

<Steps>
  <Step title="Create the custom role in Keystone" icon="plus">
    ```bash title="Create a custom role" theme={null}
    openstack role create network-operator
    ```

    ```bash title="Verify" theme={null}
    openstack role show network-operator
    ```
  </Step>

  <Step title="Find the policy rule names for the operations you want to allow" icon="search">
    View the current built-in policy rules for the target service:

    ```bash title="Show Nova's current effective policies" theme={null}
    docker exec nova_api oslopolicy-policy-generator \
      --namespace nova \
      --output-file /tmp/nova-effective-policy.yaml
    docker exec nova_api cat /tmp/nova-effective-policy.yaml | grep "flavor"
    ```

    ```bash title="Show Neutron's current effective policies" theme={null}
    docker exec neutron_server oslopolicy-policy-generator \
      --namespace neutron \
      --output-file /tmp/neutron-policy.yaml
    ```
  </Step>

  <Step title="Write the policy override file" icon="file-code">
    Create `/etc/xavs/<service>/policy.yaml` with only the rules you want to change:

    ```yaml title="/etc/xavs/nova/policy.yaml — allow network-operator to list servers" theme={null}
    "os_compute_api:servers:index": "role:network-operator or role:member"
    "os_compute_api:servers:detail": "role:network-operator or role:member"
    ```

    ```yaml title="/etc/xavs/neutron/policy.yaml — allow network-operator to manage ports" theme={null}
    "create_port": "role:network-operator or role:admin"
    "update_port": "role:network-operator or role:admin"
    "delete_port": "role:network-operator or role:admin"
    "get_port": "role:network-operator or role:member"
    ```
  </Step>

  <Step title="Apply the policy changes via XDeploy" icon="upload">
    ```bash title="Redeploy the affected service" theme={null}
    xavs-ansible deploy --tags nova
    xavs-ansible deploy --tags neutron
    ```

    <Check>The new policy is active. Test with a user holding the custom role to verify enforcement.</Check>
  </Step>

  <Step title="Assign the custom role to users" icon="user-plus">
    ```bash title="Assign the custom role to a user on a project" theme={null}
    openstack role add \
      --user alice \
      --project ops-project \
      network-operator
    ```
  </Step>
</Steps>

***

## Policy Syntax Reference

<AccordionGroup>
  <Accordion title="Role-based rules" icon="shield">
    ```yaml title="Common policy rule patterns" theme={null}
    # Allow only admin role
    "rule_name": "role:admin"

    # Allow a custom role or admin
    "rule_name": "role:network-operator or role:admin"

    # Allow member and admin roles
    "rule_name": "role:member or role:admin"

    # Allow any authenticated user
    "rule_name": "@"

    # Deny everyone (disable an operation)
    "rule_name": "!"
    ```
  </Accordion>

  <Accordion title="Project-scoped rules" icon="folder">
    ```yaml title="Project ownership rules" theme={null}
    # Allow only the resource owner or admin
    "rule_name": "project_id:%(project_id)s or role:admin"

    # Allow only within the same project
    "rule_name": "project_id:%(target.project.id)s"
    ```
  </Accordion>

  <Accordion title="System-scope rules" icon="globe">
    ```yaml title="System admin rules" theme={null}
    # Require system-scope admin (cross-project operations)
    "rule_name": "role:admin and system_scope:all"
    ```
  </Accordion>

  <Accordion title="Combining rules with aliases" icon="code">
    Define reusable aliases at the top of the policy file to avoid repetition:

    ```yaml title="Policy with reusable aliases" theme={null}
    # Define aliases
    "is_admin": "role:admin"
    "is_operator": "role:network-operator or role:admin"
    "is_member": "role:member or role:admin"

    # Use aliases in rules
    "get_network": "rule:is_member"
    "create_network": "rule:is_operator"
    "delete_network": "rule:is_admin"
    ```
  </Accordion>
</AccordionGroup>

***

## Per-Service Policy Examples

<Tabs>
  <Tab title="Compute (Nova)" icon="server">
    ```yaml title="/etc/xavs/nova/policy.yaml" theme={null}
    # Restrict flavor creation and deletion to admins only
    "os_compute_api:os-flavor-manage:create": "role:admin"
    "os_compute_api:os-flavor-manage:delete": "role:admin"

    # Allow a custom 'compute-operator' role to resize instances
    "os_compute_api:servers:resize": "role:compute-operator or role:admin"

    # Restrict live migration to admins
    "os_compute_api:os-migrate-server:migrate_live": "role:admin"

    # Allow readers to list hypervisors (normally admin-only)
    "os_compute_api:os-hypervisors:list": "role:reader or role:admin"
    ```
  </Tab>

  <Tab title="Networking (Neutron)" icon="network">
    ```yaml title="/etc/xavs/neutron/policy.yaml" theme={null}
    # Allow a 'network-operator' role to manage routers
    "create_router": "role:network-operator or role:admin"
    "update_router": "role:network-operator or role:admin"
    "delete_router": "role:network-operator or role:admin"

    # Allow QoS policy creation for network admins
    "create_policy": "role:network-operator or role:admin"
    "update_policy": "role:network-operator or role:admin"

    # Restrict floating IP allocation to members and above
    "create_floatingip": "role:member or role:admin"
    ```
  </Tab>

  <Tab title="Block Storage (Cinder)" icon="hard-drive">
    ```yaml title="/etc/xavs/cinder/policy.yaml" theme={null}
    # Restrict volume type creation to admins
    "volume_extension:types_manage": "role:admin"

    # Allow a 'storage-operator' to manage volume backups
    "backup:create": "role:storage-operator or role:member"
    "backup:delete": "role:storage-operator or role:admin"

    # Allow members to set volume metadata
    "volume:update_volume_metadata": "role:member or role:admin"
    ```
  </Tab>

  <Tab title="Image (Glance)" icon="image">
    ```yaml title="/etc/xavs/glance/policy.yaml" theme={null}
    # Restrict image deletion to image owner or admin
    "delete_image": "rule:image_owner or role:admin"

    # Allow a custom 'image-publisher' role to publicize images
    "publicize_image": "role:image-publisher or role:admin"

    # Restrict community image creation to members+
    "communitize_image": "role:member or role:admin"
    ```
  </Tab>

  <Tab title="Identity (Keystone)" icon="fingerprint">
    ```yaml title="/etc/xavs/keystone/policy.yaml" theme={null}
    # Allow a 'domain-admin' role to manage users within a domain
    "identity:list_users": "rule:cloud_admin or rule:domain_admin"
    "identity:create_user": "rule:cloud_admin or rule:domain_admin"
    "identity:update_user": "rule:cloud_admin or rule:domain_admin"

    # Restrict role assignment to cloud admins only
    "identity:create_grant": "role:admin and system_scope:all"
    ```
  </Tab>
</Tabs>

***

## Discover Policy Rule Names

Before writing an override, find the exact rule name used by the service:

<CodeGroup>
  ```bash title="List Nova policy rules (from inside container)" theme={null}
  docker exec nova_api oslopolicy-list-redundant --namespace nova
  ```

  ```bash title="Generate full effective policy (Nova)" theme={null}
  docker exec nova_api oslopolicy-policy-generator \
    --namespace nova 2>/dev/null | grep -A1 "flavor"
  ```

  ```bash title="Show Neutron policy rules" theme={null}
  docker exec neutron_server \
    cat /etc/neutron/policy.yaml 2>/dev/null || \
    docker exec neutron_server \
    oslopolicy-policy-generator --namespace neutron 2>/dev/null | head -100
  ```

  ```bash title="Check current policy file for a service" theme={null}
  cat /etc/xavs/nova/policy.yaml 2>/dev/null || echo "No override — using defaults"
  ```
</CodeGroup>

***

## Audit Policy Enforcement

Regularly review and verify policy configurations across all services:

<CodeGroup>
  ```bash title="List all policy overrides for a service" theme={null}
  cat /etc/xavs/nova/policy.yaml
  cat /etc/xavs/neutron/policy.yaml
  ```

  ```bash title="Test policy as a non-admin user" theme={null}
  # Source a non-admin openrc, then:
  openstack server list
  openstack image list
  openstack network list
  ```

  ```bash title="List all role assignments (full audit)" theme={null}
  openstack role assignment list --names
  ```
</CodeGroup>

<Warning>
  Overly permissive policies are a leading cause of privilege escalation incidents. Test policy changes in a staging environment before applying to production. Always restrict — never relax — default policies without documented justification.
</Warning>

<Tip>
  Maintain a change log for all policy modifications. Document the business justification, date of change, and approver for each policy override. This record is essential for compliance audits.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Roles & Role Assignments" href="/services/identity/roles">
    Create custom roles and assign them to users on projects and domains
  </Card>

  <Card title="Security Hardening" href="/services/identity/security">
    Apply the full security hardening checklist including policy auditing best practices
  </Card>

  <Card title="Domain Management" href="/services/identity/domain-management">
    Manage the domain and project hierarchy that policies operate within
  </Card>

  <Card title="Admin Troubleshooting" href="/services/identity/admin-troubleshooting">
    Resolve 403 Forbidden errors caused by policy misconfigurations
  </Card>
</CardGroup>


# Manage Projects
Source: https://docs.xloud.tech/services/identity/projects

Create and manage resource namespaces, add members, and assign roles to teams in Xloud Identity.

## Overview

Projects are the fundamental resource namespace in Xloud. Every instance, volume, network,
and image belongs to a project. Projects allow you to isolate resources between teams,
enforce independent quotas, and apply fine-grained role-based access control. This guide
covers creating projects, adding members, assigning roles, and managing project lifecycle.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Create a Project

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Identity" icon="fingerprint">
        Navigate to
        **Identity > Projects** (admin view).
      </Step>

      <Step title="Create a project" icon="plus">
        Click **Create Project** and fill in the form:

        | Field           | Description                                      |
        | --------------- | ------------------------------------------------ |
        | **Name**        | Unique identifier within the domain              |
        | **Description** | Optional — describes the project's purpose       |
        | **Enabled**     | Toggle on to allow resource creation immediately |

        <Tip>
          Use a consistent naming convention such as `team-environment` (e.g., `backend-prod`,
          `frontend-staging`) to keep the project list scannable as it grows.
        </Tip>
      </Step>

      <Step title="Confirm creation" icon="circle-check">
        Click **Confirm**. The project appears in the list immediately.
        <Check>The new project is visible in **Identity > Projects** (admin view) with status Enabled.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the project" icon="plus">
        ```bash title="Create project" theme={null}
        openstack project create \
          --domain Default \
          --description "Backend production environment" \
          backend-prod
        ```
      </Step>

      <Step title="List projects" icon="list">
        ```bash title="List all projects" theme={null}
        openstack project list
        ```

        ```bash title="Show project details" theme={null}
        openstack project show backend-prod
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Manage Project Members

Add users to a project and assign the appropriate role to control what each user can do.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open project members" icon="users">
        Navigate to **Identity > Projects** (admin view). On the target project row, click the
        **More** dropdown and select **Manage User**.
      </Step>

      <Step title="Add a member" icon="user-plus">
        Click **Add Member**, select a user from the dropdown, and select a role:

        | Role     | What the user can do                                                           |
        | -------- | ------------------------------------------------------------------------------ |
        | `admin`  | Full management rights — create, modify, delete all resources and manage users |
        | `member` | Standard access — create and manage resources within the project               |
        | `reader` | Read-only — view resources but cannot create or modify anything                |

        <Tip>
          Changes take effect on the user's next token request. Existing tokens retain the
          old permissions until they expire.
        </Tip>
      </Step>

      <Step title="Confirm membership" icon="circle-check">
        <Check>The user appears in the members list with the assigned role.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Grant a role" icon="shield">
        ```bash title="Grant role to user in project" theme={null}
        openstack role add \
          --project backend-prod \
          --user alice \
          member
        ```

        <Note>No output is returned on success — the absence of an error confirms the assignment.</Note>
      </Step>

      <Step title="Verify role assignment" icon="search">
        ```bash title="List role assignments for user" theme={null}
        openstack role assignment list \
          --user alice \
          --project backend-prod \
          --names
        ```

        <Check>The role assignment appears in the output with the correct project and user names.</Check>
      </Step>

      <Step title="Remove a role" icon="user-minus">
        ```bash title="Revoke role from user in project" theme={null}
        openstack role remove \
          --project backend-prod \
          --user alice \
          member
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Update and Disable Projects

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Open the project in **Identity > Projects** (admin view) and click **Edit** to modify the
    name, description, or enabled state. Disabling a project immediately prevents new
    resource creation but does not delete existing resources.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Rename a project" theme={null}
      openstack project set --name new-name backend-prod
      ```

      ```bash title="Disable a project" theme={null}
      openstack project set --disable backend-prod
      ```

      ```bash title="Re-enable a project" theme={null}
      openstack project set --enable backend-prod
      ```

      ```bash title="Delete a project permanently" theme={null}
      openstack project delete backend-prod
      ```
    </CodeGroup>

    <Warning>
      Deleting a project does not automatically delete the resources within it. Clean up
      instances, volumes, and networks before deleting to avoid orphaned resources that
      continue consuming quota.
    </Warning>
  </Tab>
</Tabs>

***

## Project Quotas

Quotas cap the total resources a project can consume. Administrators set quotas per project
to prevent any single team from exhausting shared infrastructure.

```bash title="View current project quotas" theme={null}
openstack quota show backend-prod
```

```bash title="Check actual usage against quotas" theme={null}
openstack quota show --usage backend-prod
```

<Note>
  Quota changes are applied by your platform administrator. Contact your admin or refer
  to the [Identity Admin Guide](/services/identity/admin-guide) to modify project quotas.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Users" href="/services/identity/users">
    Create user accounts and assign them to projects with appropriate roles.
  </Card>

  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Generate non-interactive credentials scoped to project roles for automation pipelines.
  </Card>

  <Card title="Compute User Guide" href="/services/compute/user-guide">
    Launch and manage compute instances within your project.
  </Card>

  <Card title="Identity Admin Guide" href="/services/identity/admin-guide">
    Configure domain management, token policies, and authentication backends.
  </Card>
</CardGroup>


# Roles & Role Assignments
Source: https://docs.xloud.tech/services/identity/roles

Create custom roles, assign them to users and groups on projects or domains, and build role hierarchies with implied roles in Xloud Identity.

## Overview

<iframe title="How to Manage RBAC Permissions on Xloud Dashboard" />

Roles are the foundation of Xloud's role-based access control (RBAC) system. A role grants a set of permissions and is always assigned in the context of a **user + project** (or **user + domain**) pair. Xloud ships three built-in roles — `admin`, `member`, and `reader` — and supports creating custom roles to match your organizational requirements.

<Note>
  **Prerequisites**

  * CLI authenticated with admin credentials — see [CLI Setup](/cli-setup)
  * Dashboard: logged in as an admin user
  * To create or modify roles you need the `admin` role on the system scope
</Note>

***

## Built-in Roles

Xloud ships three standard roles that cascade via implied role inheritance:

| Role     | Permissions                                        | Typical Assignment                    |
| -------- | -------------------------------------------------- | ------------------------------------- |
| `admin`  | Full API access including administrative endpoints | Cloud operators, service accounts     |
| `member` | Project-scoped create, read, update, delete        | Regular project members               |
| `reader` | Project-scoped read-only access                    | Auditors, monitoring service accounts |

<Info>
  **Implied role hierarchy**: `admin` implies `member`, and `member` implies `reader`. Assigning `admin` automatically grants all three sets of permissions. You do not need to assign all three roles separately.
</Info>

***

## List and View Roles

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Identity > Roles** (admin view) to see all available roles, their IDs, and whether they are domain-scoped.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List all roles" theme={null}
      openstack role list
      ```

      ```bash title="Show role details" theme={null}
      openstack role show member
      ```

      ```bash title="List all role assignments (with names)" theme={null}
      openstack role assignment list --names
      ```

      ```bash title="List assignments for a specific user" theme={null}
      openstack role assignment list \
        --user john.doe \
        --names
      ```

      ```bash title="List assignments on a specific project" theme={null}
      openstack role assignment list \
        --project my-project \
        --names
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Create a Custom Role

Custom roles let you create named access tiers beyond the three built-in roles. The role itself is just a name — its effective permissions are defined in each service's `policy.yaml` file.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open Roles" icon="list">
        Navigate to **Identity > Roles** (admin view) and click **Create Role**.
      </Step>

      <Step title="Name the role" icon="tag">
        Enter a descriptive name such as `network-operator` or `image-uploader`. Role names are case-sensitive.

        <Tip>
          Use lowercase, hyphen-separated names (e.g., `billing-reader`, `vpc-admin`) to stay consistent with the built-in role naming convention.
        </Tip>
      </Step>

      <Step title="Save and assign" icon="check">
        Click **Confirm**. The new role is created and available to assign to users. Its permissions are still `{}` (no extra access) until you add policy rules in each service.

        <Check>The role appears in **Identity > Roles** (admin view) with a new UUID.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a custom role" theme={null}
    openstack role create network-operator
    ```

    ```bash title="Create a domain-scoped role" theme={null}
    openstack role create --domain my-domain billing-reader
    ```

    ```bash title="Verify the role was created" theme={null}
    openstack role show network-operator
    ```

    <Tip>
      After creating the role, define its permissions in the relevant service policy files. See [Policy Management](/services/identity/policy-management) for how to map role names to API operations.
    </Tip>
  </Tab>
</Tabs>

***

## Assign Roles

Role assignments always connect a **principal** (user or group) to a **scope** (project or domain).

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open project members" icon="folder">
        Navigate to **Identity > Projects** (admin view), select the project, and click the **Manage User** action (More dropdown).
      </Step>

      <Step title="Add the user" icon="user-plus">
        Click **Add Member**, search for the user, select the desired role, and click **Add**.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Assign role to a user on a project" theme={null}
      openstack role add \
        --user john.doe \
        --project my-project \
        member
      ```

      ```bash title="Assign admin role on a project" theme={null}
      openstack role add \
        --user alice \
        --project ops-project \
        admin
      ```

      ```bash title="Assign role to a user at domain scope" theme={null}
      openstack role add \
        --user john.doe \
        --domain my-domain \
        member
      ```

      ```bash title="Assign role to a group on a project" theme={null}
      openstack role add \
        --group operators \
        --project my-project \
        member
      ```

      ```bash title="Verify the assignment" theme={null}
      openstack role assignment list \
        --user john.doe \
        --project my-project \
        --names
      ```
    </CodeGroup>
  </Tab>
</Tabs>

<Warning>
  A user with **no role on a project** cannot access that project at all — not even read-only. Always assign at least the `reader` role for users who need visibility into a project.
</Warning>

***

## Remove Role Assignments

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Identity > Projects** (admin view) > **Manage User** action, find the user, and click **Remove Role**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Remove a role from a user on a project" theme={null}
      openstack role remove \
        --user john.doe \
        --project my-project \
        member
      ```

      ```bash title="Remove group role assignment" theme={null}
      openstack role remove \
        --group operators \
        --project my-project \
        member
      ```

      ```bash title="Verify removal" theme={null}
      openstack role assignment list \
        --user john.doe \
        --project my-project \
        --names
      ```
    </CodeGroup>
  </Tab>
</Tabs>

<Warning>
  Before deleting a user account, remove all of their role assignments. A user cannot be deleted while they hold active role assignments on projects or domains.
</Warning>

***

## Implied Roles (Role Hierarchies)

Implied roles (also called role inference rules) let you build role hierarchies where assigning one role automatically grants another. This keeps assignments simple while enabling fine-grained permission structures.

<Info>
  **Direction matters**: Role inference is one-directional. If `admin` implies `member`, assigning `admin` grants both — but assigning `member` does **not** grant `admin`.
</Info>

<CodeGroup>
  ```bash title="Create an implied role rule (admin implies member)" theme={null}
  openstack implied role create admin --implied-role member
  ```

  ```bash title="List all implied role rules" theme={null}
  openstack implied role list
  ```

  ```bash title="Delete an implied role rule" theme={null}
  openstack implied role delete admin --implied-role member
  ```
</CodeGroup>

### Use Case: Fine-Grained Service Roles

You can create service-specific sub-roles and have the `member` role imply all of them:

```bash title="Create service-specific sub-roles" theme={null}
openstack role create compute-member
openstack role create network-member
openstack role create volume-member
```

```bash title="Have member imply all service sub-roles" theme={null}
openstack implied role create member --implied-role compute-member
openstack implied role create member --implied-role network-member
openstack implied role create member --implied-role volume-member
```

Now assigning `member` to a user automatically grants compute, network, and volume access — while you can still assign individual sub-roles for more granular control.

***

## Delete a Role

<Warning>
  Deleting a role removes it from the system but does **not** clean up existing role assignments. Any user previously assigned the deleted role loses that access silently. Audit assignments before deleting.
</Warning>

```bash title="List assignments before deleting (sanity check)" theme={null}
openstack role assignment list --names | grep my-custom-role
```

```bash title="Delete the role" theme={null}
openstack role delete my-custom-role
```

***

## Custom Roles and Service Policies

A custom role is just a name until you define what it can do in each service's policy file. Xloud uses `policy.yaml` (or `policy.json`) files in each service container to map role names to allowed API operations.

See [Policy Management](/services/identity/policy-management) for the full guide on creating per-service policy overrides for your custom roles.

**Quick example** — restrict Compute flavor creation to a custom `compute-admin` role:

```yaml title="/etc/xavs/nova/policy.yaml" theme={null}
"os_compute_api:os-flavor-manage:create": "role:compute-admin"
```

***

## Next Steps

<CardGroup>
  <Card title="Policy Management" href="/services/identity/policy-management">
    Define what each role can do in Nova, Glance, Cinder, and other services via policy.yaml overrides
  </Card>

  <Card title="Projects" href="/services/identity/projects">
    Create projects and manage the project hierarchy that roles are assigned within
  </Card>

  <Card title="Users" href="/services/identity/users">
    Create users and manage their project memberships
  </Card>

  <Card title="Domain Management" href="/services/identity/domain-management">
    Organize projects and users into domains with independent namespaces
  </Card>
</CardGroup>


# Identity Security Hardening
Source: https://docs.xloud.tech/services/identity/security

Enforce MFA, rotate Fernet keys, audit role assignments, and apply security best practices for Xloud Identity.

## Overview

Xloud Identity is the authentication and authorization backbone for the entire platform.
Hardening this service reduces the blast radius of credential compromise, limits lateral
movement across projects, and ensures audit trails are maintained. This guide covers the
complete security hardening checklist for production Identity deployments.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Security Hardening Checklist

<CardGroup>
  <Card title="Enforce MFA for Admins" icon="shield">
    Require multi-factor authentication for all accounts with the `admin` role. Configure
    an MFA enforcement rule via the Identity service policy to block admin token issuance
    without a valid TOTP factor.
  </Card>

  <Card title="Rotate Fernet Keys" icon="rotate-cw">
    Schedule automated key rotation every 24 hours. XDeploy includes a cron-based rotation
    playbook that synchronizes keys across all Identity API nodes simultaneously.
  </Card>

  <Card title="Minimize Token Lifetime" icon="clock">
    Set `keystone_token_expiration` to 3600 seconds (1 hour) or less. Use application
    credentials with explicit expiry dates for automation pipelines instead of long-lived
    user tokens.
  </Card>

  <Card title="Audit Role Assignments" icon="search">
    Review role assignments quarterly. Remove the `admin` role from any account that no
    longer requires elevated access. Export full audit reports regularly.
  </Card>
</CardGroup>

***

## MFA Enforcement Policy

Enforce MFA for all accounts with the `admin` role by configuring an auth rules policy:

```yaml title="/etc/xavs/keystone/policy.yaml — enforce MFA for admin token issuance" theme={null}
"identity:get_auth_token": "rule:admin_required and (rule:mfa_enabled or not role:admin)"
```

After applying, admin accounts without MFA enrolled cannot issue tokens:

```bash title="Apply MFA enforcement policy" theme={null}
xavs-ansible deploy --tags keystone
```

<Warning>
  Enforce MFA in a staged rollout. Ensure all admin accounts have enrolled a TOTP
  device before applying the policy — otherwise admin accounts will be locked out.
</Warning>

***

## Fernet Key Security

<Steps>
  <Step title="Verify key file permissions" icon="lock">
    Fernet keys must be readable only by the Identity service user:

    ```bash title="Check key file permissions" theme={null}
    ls -la /var/lib/kolla/config_files/fernet-keys/
    ```

    Expected: `600 keystone:keystone` for all key files.
  </Step>

  <Step title="Configure automated rotation" icon="rotate-cw">
    ```yaml title="XDeploy globals: automated Fernet rotation" theme={null}
    keystone_fernet_key_rotation: "0 */24 * * *"
    keystone_fernet_max_active_keys: 3
    ```

    Deploy to activate:

    ```bash title="Apply rotation configuration" theme={null}
    xavs-ansible deploy --tags keystone
    ```
  </Step>

  <Step title="Verify rotation is running" icon="circle-check">
    ```bash title="Check rotation cron job" theme={null}
    docker exec keystone crontab -l
    ```

    <Check>Cron job is scheduled and shows the configured rotation interval.</Check>
  </Step>
</Steps>

***

## Role Assignment Auditing

Run quarterly access reviews to identify over-provisioned accounts:

```bash title="Export all role assignments" theme={null}
openstack role assignment list --names \
  -f csv > role-assignments-$(date +%Y%m%d).csv
```

```bash title="Find all admin role assignments" theme={null}
openstack role assignment list \
  --role admin \
  --names
```

```bash title="Find users with admin role in multiple projects" theme={null}
openstack role assignment list \
  --role admin \
  --names | grep -v "system"
```

<Tip>
  Use the `reader` role for monitoring and dashboard accounts — it provides the
  visibility they need without write access. Reserve `admin` for accounts that
  genuinely require resource management capabilities.
</Tip>

***

## Network-Level Controls

<AccordionGroup>
  <Accordion title="Restrict Identity API access" icon="network">
    The Identity API public endpoint (port 5000) should be accessible only from:

    * Internal cluster networks
    * VPN or bastion hosts for administrative access
    * Dashboard and CLI clients via HAProxy

    Configure HAProxy ACLs to block direct public access to the admin interface.
  </Accordion>

  <Accordion title="TLS configuration" icon="shield">
    Ensure all Identity API endpoints use TLS with certificates from a trusted CA:

    ```yaml title="XDeploy globals: TLS configuration" theme={null}
    kolla_enable_tls_external: "yes"
    kolla_external_tls_cert: /etc/xavs/certs/external.crt
    kolla_external_tls_key: /etc/xavs/certs/external.key
    ```
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Token Configuration" href="/services/identity/token-config">
    Configure Fernet key rotation schedules and token lifetime policies.
  </Card>

  <Card title="Policy Management" href="/services/identity/policy-management">
    Customize RBAC policies and implement least-privilege access controls.
  </Card>

  <Card title="Multi-Factor Authentication" href="/services/identity/multi-factor-auth">
    Enable TOTP enrollment for user accounts.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/identity/admin-troubleshooting">
    Diagnose security-related authentication and authorization failures.
  </Card>
</CardGroup>


# Service Catalog
Source: https://docs.xloud.tech/services/identity/service-catalog

Manage endpoint registration for all Xloud services across regions and interface types in Xloud Identity.

## Overview

The service catalog registers every Xloud service endpoint so clients can discover the
correct API URL for each service and region. It is included in every authentication token
response, enabling the `openstack` CLI and Dashboard to route requests correctly without
hard-coded URLs. Each catalog entry includes the service type, endpoint interface
(public, internal, admin), region, and URL.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Catalog Structure

| Concept       | Description                                                                                          |
| ------------- | ---------------------------------------------------------------------------------------------------- |
| **Service**   | A registered Xloud service (compute, image, block-storage, network, etc.)                            |
| **Endpoint**  | A URL for a specific service, interface type, and region combination                                 |
| **Interface** | `public` (external clients), `internal` (service-to-service), or `admin` (administrative operations) |
| **Region**    | Logical grouping for geographically or administratively separate deployments                         |

***

## View the Service Catalog

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Global Setting > System Info** to view all registered services, and
    **Global Setting > System Info** to view and manage endpoint URLs for each service.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all registered services" theme={null}
    openstack service list
    ```

    ```bash title="List all endpoints" theme={null}
    openstack endpoint list
    ```

    ```bash title="List endpoints for a specific service" theme={null}
    openstack endpoint list --service compute
    ```

    ```bash title="Show the current token's service catalog" theme={null}
    openstack catalog list
    ```
  </Tab>
</Tabs>

***

## Manage Endpoints

<Tabs>
  <Tab title="Create endpoint" icon="plus">
    ```bash title="Create a public endpoint for Compute" theme={null}
    openstack endpoint create \
      --region RegionOne \
      compute public https://api.<your-domain>:8774/v2.1
    ```

    ```bash title="Create an internal endpoint" theme={null}
    openstack endpoint create \
      --region RegionOne \
      compute internal http://10.0.1.71:8774/v2.1
    ```
  </Tab>

  <Tab title="Update endpoint" icon="pencil">
    ```bash title="Update an existing endpoint URL" theme={null}
    openstack endpoint set \
      --url https://api.<your-domain>:8774/v2.1 \
      <ENDPOINT_ID>
    ```

    ```bash title="Disable an endpoint" theme={null}
    openstack endpoint set --disable <ENDPOINT_ID>
    ```
  </Tab>

  <Tab title="Delete endpoint" icon="trash">
    ```bash title="Delete an endpoint" theme={null}
    openstack endpoint delete <ENDPOINT_ID>
    ```

    <Warning>
      Deleting an endpoint makes the associated service undiscoverable via the catalog.
      Clients that rely on catalog discovery will fail to resolve the service URL.
      Always ensure a replacement endpoint is registered before removing an existing one.
    </Warning>
  </Tab>
</Tabs>

***

## Standard Service Registrations

The following services are registered during XDeploy deployment. Verify they are present
after a fresh deployment or endpoint migration:

| Service Type    | Public Port | Description                             |
| --------------- | ----------- | --------------------------------------- |
| `identity`      | 5000        | Authentication and authorization        |
| `compute`       | 8774        | Virtual machine management              |
| `image`         | 9292        | Image and snapshot management           |
| `block-storage` | 8776        | Persistent block volumes                |
| `network`       | 9696        | Virtual networking                      |
| `object-store`  | 8080        | Object storage (if enabled)             |
| `load-balancer` | 9876        | Load balancing (if enabled)             |
| `dns`           | 9001        | DNS as a Service (if enabled)           |
| `key-manager`   | 9311        | Secrets and key management (if enabled) |

***

## Verify Catalog After Endpoint Changes

After modifying endpoints, verify that clients can resolve the updated URLs:

```bash title="Test catalog resolution" theme={null}
openstack --os-cloud admin catalog show compute
```

```bash title="Verify CLI uses the correct endpoint" theme={null}
openstack server list -v 2>&1 | grep "GET http"
```

***

## Next Steps

<CardGroup>
  <Card title="Domain Management" href="/services/identity/domain-management">
    Manage the domains that own the users and projects accessing these services.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/identity/admin-troubleshooting">
    Resolve service catalog misconfigurations and endpoint routing errors.
  </Card>

  <Card title="Authentication Backends" href="/services/identity/auth-backends">
    Configure the authentication drivers that issue tokens containing the service catalog.
  </Card>

  <Card title="Architecture" href="/services/identity/architecture">
    Understand how the service catalog fits into the authentication flow.
  </Card>
</CardGroup>


# Token Configuration
Source: https://docs.xloud.tech/services/identity/token-config

Configure Fernet key rotation, token lifetime, and expiration policies for Xloud Identity.

## Overview

Xloud Identity uses Fernet tokens by default — stateless, symmetric-key-encrypted tokens
that do not require a database lookup on every validation request. This guide covers
token format selection, Fernet key rotation procedures, and lifetime configuration.
Proper token configuration balances security (short lifetimes) with operational convenience
(longer windows for automation pipelines).

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Token Format Reference

| Token Type | Storage          | Validation                             | Use Case                                         |
| ---------- | ---------------- | -------------------------------------- | ------------------------------------------------ |
| **Fernet** | None (stateless) | Decrypted locally using key repository | Default. Recommended for all deployments.        |
| **JWT**    | None (stateless) | Verified locally via public key        | Alternative to Fernet with standard JWT tooling. |

<Tip>
  Fernet is the recommended format for all deployments. JWT tokens are useful when
  you need to validate tokens outside of Xloud (e.g., in a sidecar proxy or API gateway).
</Tip>

***

## Fernet Key Rotation

Fernet uses a key repository with three key roles:

| Key Role      | Position | Description                                                      |
| ------------- | -------- | ---------------------------------------------------------------- |
| **Primary**   | `1`      | Used to sign all new tokens                                      |
| **Secondary** | `2+`     | Used to validate tokens signed by previous primary keys          |
| **Staged**    | `0`      | Pre-positioned key that will become the next primary on rotation |

<Steps>
  <Step title="Configure rotation schedule" icon="clock">
    XDeploy manages Fernet key rotation automatically via a scheduled cron job.
    Configure the rotation interval in your deployment globals:

    ```yaml title="Fernet key rotation schedule" theme={null}
    keystone_fernet_key_rotation: "0 */24 * * *"  # Every 24 hours
    keystone_fernet_max_active_keys: 3             # Primary + 1 secondary + staged
    ```
  </Step>

  <Step title="Rotate keys manually" icon="rotate-cw">
    To trigger an immediate rotation outside the scheduled window:

    ```bash title="Rotate Fernet keys" theme={null}
    keystone-manage fernet_rotate \
      --keystone-user keystone \
      --keystone-group keystone
    ```

    After rotation, XDeploy synchronizes the new key set to all Identity API nodes.
  </Step>

  <Step title="Verify key synchronization" icon="circle-check">
    All nodes must have identical key files:

    ```bash title="Check key file timestamps on all nodes" theme={null}
    ls -la /var/lib/kolla/config_files/fernet-keys/
    ```

    <Check>All nodes report the same key files with matching timestamps.</Check>
  </Step>
</Steps>

<Warning>
  Rotate keys on all Identity API nodes simultaneously. Keys not in sync across
  nodes cause token validation failures. XDeploy's rotation playbook handles
  synchronization automatically.
</Warning>

***

## Token Lifetime Configuration

Token lifetime is configured in XDeploy globals. Shorter lifetimes improve security
but increase re-authentication overhead for users and automation pipelines.

```yaml title="Token lifetime settings" theme={null}
keystone_token_expiration: 3600        # Default token lifetime in seconds (1 hour)
keystone_allow_expired_window: 172800  # Allow expired tokens for re-issue (48 hours)
```

| Parameter                       | Default | Recommended       | Notes                                              |
| ------------------------------- | ------- | ----------------- | -------------------------------------------------- |
| `keystone_token_expiration`     | 3600    | 3600 (1 hour)     | Reduce for high-security environments              |
| `keystone_allow_expired_window` | 172800  | 172800 (48 hours) | Allows token re-issuance without password re-entry |

<Note>
  For long-running automation jobs, use [application credentials](/services/identity/application-credentials)
  rather than increasing token lifetime. Application credentials can be scoped, restricted,
  and rotated independently of user accounts.
</Note>

***

## Verify Token Configuration

```bash title="Issue a token and inspect its expiry" theme={null}
openstack token issue -f json | python3 -c "
import json, sys, datetime
t = json.load(sys.stdin)
expires = datetime.datetime.fromisoformat(t['expires'].replace('Z', '+00:00'))
now = datetime.datetime.now(datetime.timezone.utc)
print(f'Expires: {t[\"expires\"]}')
print(f'Valid for: {(expires - now).seconds // 60} minutes')
"
```

***

## Next Steps

<CardGroup>
  <Card title="Security Hardening" href="/services/identity/security">
    Enforce MFA requirements and audit token usage patterns.
  </Card>

  <Card title="Architecture" href="/services/identity/architecture">
    Understand how Fernet keys flow through the distributed Identity service.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/identity/admin-troubleshooting">
    Diagnose token validation failures caused by key synchronization issues.
  </Card>

  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Create long-lived automation credentials as an alternative to extended token lifetimes.
  </Card>
</CardGroup>


# Identity Troubleshooting
Source: https://docs.xloud.tech/services/identity/troubleshooting

Diagnose and resolve authentication failures, permission errors, token scope issues, and application credential problems in Xloud Identity.

## Overview

This guide covers the most common issues encountered when working with Xloud Identity —
including authentication failures, permission errors, token scope mismatches, and
application credential rejections. Each section provides diagnostic steps and resolution
commands you can run immediately.

<Note>
  For platform-level issues such as LDAP connectivity failures or Fernet key sync errors,
  refer to the [Identity Admin Guide — Troubleshooting](/services/identity/admin-troubleshooting).
</Note>

***

## Authentication Failures

<AccordionGroup>
  <Accordion title="Authentication failed — invalid credentials" icon="lock">
    **Cause**: Incorrect password, disabled user account, or expired token.

    **Diagnose**:

    ```bash title="Verify user status" theme={null}
    openstack user show alice
    ```

    Confirm `enabled: True`. If the account is disabled, contact your administrator to
    re-enable it.

    **Resolution**:

    * For an expired token, re-authenticate by sourcing your credentials file:
      ```bash title="Re-authenticate" theme={null}
      source openrc.sh
      openstack token issue
      ```
    * For a forgotten password, ask your administrator to reset it:
      ```bash title="Admin: reset user password" theme={null}
      openstack user set --password-prompt alice
      ```
  </Accordion>

  <Accordion title="MFA code rejected" icon="smartphone">
    **Cause**: TOTP code has expired (codes are valid for 30 seconds), or the device
    clock is not synchronized.

    **Resolution**:

    * Wait for the next code to appear in your authenticator app (new code every 30 seconds)
    * Ensure your device clock is set to the correct UTC time (time drift causes TOTP failures)
    * If codes consistently fail, re-enroll your MFA device under
      MFA re-enrollment via CLI (`openstack credential delete` then re-create)
  </Accordion>

  <Accordion title="Application credential rejected" icon="terminal">
    **Cause**: The credential may have expired, the owning user may be disabled, or the
    credential was deleted.

    **Diagnose**:

    ```bash title="List active credentials" theme={null}
    openstack application credential list
    ```

    Verify the credential exists and check its expiration date:

    ```bash title="Show credential details" theme={null}
    openstack application credential show ci-pipeline-prod
    ```

    **Resolution**: If expired, create a new credential and update your pipeline
    configuration. If the owning user is disabled, re-enable the user or create a
    new credential under an active service user account.
  </Accordion>
</AccordionGroup>

***

## Permission Errors

<AccordionGroup>
  <Accordion title="Permission denied on resource operation" icon="shield-off">
    **Cause**: The current user lacks the required role in the target project.

    **Diagnose**:

    ```bash title="Check role assignments for user in project" theme={null}
    openstack role assignment list \
      --user alice \
      --project backend-prod \
      --names
    ```

    **Resolution**: If no assignment exists, request the appropriate role from your
    project administrator:

    ```bash title="Admin: grant member role" theme={null}
    openstack role add \
      --project backend-prod \
      --user alice \
      member
    ```
  </Accordion>

  <Accordion title="403 Forbidden on administrative operations" icon="lock">
    **Cause**: The user has the `admin` role in a project but not at the system or domain
    scope required for platform-level administrative operations.

    **Diagnose**: Check if the operation requires system-scope admin access.
    API operations on domains, users across domains, and service configurations typically
    require system-scope admin.

    **Resolution**: Grant system-scope admin access (requires an existing system admin):

    ```bash title="Grant system admin role" theme={null}
    openstack role add \
      --user alice \
      --system all \
      admin
    ```

    <Warning>
      System-scope admin grants full control over all domains and projects. Reserve this
      assignment for platform administrators only. Use project-scoped admin for day-to-day
      project management.
    </Warning>
  </Accordion>
</AccordionGroup>

***

## Token Scope Issues

<AccordionGroup>
  <Accordion title="Token scope error — resource not visible" icon="eye-off">
    **Cause**: The token was issued against a different project than where the resource lives.

    **Diagnose**:

    ```bash title="Inspect current token scope" theme={null}
    openstack token issue -f json
    ```

    Check the `project` field. If it does not match the project containing your resource,
    re-authenticate with the correct project scope.

    **Resolution**:

    ```bash title="Re-authenticate with correct project scope" theme={null}
    export OS_PROJECT_NAME=backend-prod
    source openrc.sh
    openstack server list
    ```
  </Accordion>

  <Accordion title="Token expired mid-session" icon="clock">
    **Cause**: Default token lifetime is 1 hour. Long-running CLI sessions or scripts
    may encounter expired tokens.

    **Resolution**: Re-authenticate before executing long operations:

    ```bash title="Refresh token" theme={null}
    source openrc.sh
    ```

    For automation scripts, use [application credentials](/services/identity/application-credentials)
    which auto-renew through the `v3applicationcredential` auth type.
  </Accordion>
</AccordionGroup>

***

## Account Management Issues

<AccordionGroup>
  <Accordion title="User cannot log in — account not visible" icon="user-x">
    **Cause**: The user may belong to a different domain, or the account is disabled.

    **Diagnose**:

    ```bash title="Search user across all domains" theme={null}
    openstack user list --long | grep alice
    ```

    ```bash title="Show user status" theme={null}
    openstack user show --domain Default alice
    ```

    **Resolution**: Ensure the user is in the correct domain. If disabled, re-enable:

    ```bash title="Re-enable user" theme={null}
    openstack user set --enable alice
    ```
  </Accordion>

  <Accordion title="Project quota exceeded" icon="gauge">
    **Cause**: The project has reached its resource quota limit for the requested
    resource type (instances, volumes, networks, etc.).

    **Diagnose**:

    ```bash title="Check quota usage" theme={null}
    openstack quota show --usage backend-prod
    ```

    Look for fields where `used` equals or exceeds `limit`.

    **Resolution**: Contact your platform administrator to increase the quota, or delete
    unused resources to free space within the existing quota.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Identity Admin Guide — Troubleshooting" href="/services/identity/admin-troubleshooting">
    Platform-level diagnostics — LDAP connectivity, Fernet key sync, and service catalog issues.
  </Card>

  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Create robust non-interactive credentials for automation that avoid session expiry issues.
  </Card>

  <Card title="Users" href="/services/identity/users">
    Manage user accounts, passwords, and role assignments.
  </Card>

  <Card title="Multi-Factor Authentication" href="/services/identity/multi-factor-auth">
    Resolve MFA enrollment and TOTP code validation issues.
  </Card>
</CardGroup>


# Identity User Guide
Source: https://docs.xloud.tech/services/identity/user-guide

Manage projects, users, roles, and authentication credentials in Xloud Cloud Platform.

<p>Overview</p>

Xloud Identity lets you control who can access your cloud environment and what they are permitted
to do. Use the guides below to manage projects, users, application credentials, and
multi-factor authentication from the Dashboard or CLI.

<CardGroup>
  <Card title="Projects" icon="folder" href="/services/identity/projects">
    Create and manage resource namespaces, add team members, and assign roles within each project.
  </Card>

  <Card title="Users" icon="user" href="/services/identity/users">
    Create user accounts, set passwords, assign roles, and manage user lifecycle operations.
  </Card>

  <Card title="Application Credentials" icon="terminal" href="/services/identity/application-credentials">
    Generate scoped credentials for CI/CD pipelines, automation, and service accounts without embedding passwords.
  </Card>

  <Card title="Multi-Factor Authentication" icon="shield" href="/services/identity/multi-factor-auth">
    Enable TOTP-based two-factor authentication for enhanced account security.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/identity/troubleshooting">
    Resolve authentication failures, permission errors, and token scope issues.
  </Card>
</CardGroup>

***

<p>Key Concepts</p>

<AccordionGroup>
  <Accordion title="Domains, Projects, and Users" icon="building">
    | Concept             | Scope           | Description                                                                                                           |
    | ------------------- | --------------- | --------------------------------------------------------------------------------------------------------------------- |
    | **Domain**          | Top-level       | Administrative boundary. Separates organizations, business units, or customers.                                       |
    | **Project**         | Within a domain | Resource namespace for quotas, billing, and access control. All instances, volumes, and networks belong to a project. |
    | **User**            | Within a domain | An identity (human or service) that authenticates and receives tokens scoped to a project.                            |
    | **Group**           | Within a domain | A collection of users. Role assignments on a group apply to all members.                                              |
    | **Role**            | Assignment      | Named permission set. Common roles: `admin`, `member`, `reader`.                                                      |
    | **Token**           | Session         | A scoped, time-limited bearer credential issued after successful authentication.                                      |
    | **Service Catalog** | Token payload   | Lists every Xloud service endpoint available to the authenticated user in the current scope.                          |
  </Accordion>

  <Accordion title="Role Hierarchy" icon="shield">
    Xloud Identity ships with three built-in roles:

    | Role     | Capability                                                                                                          |
    | -------- | ------------------------------------------------------------------------------------------------------------------- |
    | `admin`  | Full management rights within the assigned scope (project or domain). Can create, modify, and delete all resources. |
    | `member` | Standard user. Can create and manage resources within the project. Cannot manage users or quotas.                   |
    | `reader` | Read-only access. Cannot create or modify any resource. Suitable for monitoring and audit use cases.                |

    <Tip>
      Assign the least-privileged role that satisfies the user's requirement. Use `reader`
      for dashboards, `member` for developers, and `admin` only for project administrators.
    </Tip>
  </Accordion>
</AccordionGroup>

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Identity Admin Guide" icon="settings" href="/services/identity/admin-guide">
    Configure LDAP, federation, token policies, and security hardening for your Xloud Identity deployment.
  </Card>

  <Card title="Authentication & CLI" icon="terminal" href="/cli-setup">
    Source credentials and configure the `openstack` CLI for your environment.
  </Card>
</CardGroup>


# Manage Users
Source: https://docs.xloud.tech/services/identity/users

Create, configure, and manage user accounts and role assignments in Xloud Identity.

## Overview

User accounts in Xloud Identity represent individual humans or service identities that
authenticate against the platform. Each user belongs to a domain, can be a member of
multiple projects with different roles, and can hold application credentials for
non-interactive access. This guide covers creating users, assigning roles, and managing
the full user lifecycle.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Create a User

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open User Management" icon="users">
        Navigate to
        **Identity > Users** (admin view). Click **Create User**.
      </Step>

      <Step title="Configure the user" icon="user">
        | Field               | Description                                         |
        | ------------------- | --------------------------------------------------- |
        | **Username**        | Login identifier. Must be unique within the domain. |
        | **Email**           | Used for password reset and notifications.          |
        | **Password**        | Initial password. Communicate securely to the user. |
        | **Primary Project** | Default project context on login.                   |
        | **Enabled**         | Must be toggled on for the user to authenticate.    |
      </Step>

      <Step title="Confirm creation">
        Click **Confirm**. The new account appears immediately in the user list.
        <Check>The user can now authenticate using their credentials.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the user" icon="user-plus">
        ```bash title="Create user with password prompt" theme={null}
        openstack user create \
          --domain Default \
          --project backend-prod \
          --password-prompt \
          --email alice@example.com \
          alice
        ```

        <Tip>
          The `--password-prompt` flag avoids exposing the password in shell history. The
          CLI will interactively prompt for the password securely.
        </Tip>
      </Step>

      <Step title="Verify user creation" icon="search">
        ```bash title="Show user details" theme={null}
        openstack user show alice
        ```

        <Check>The output shows `enabled: True` — the user is active and can authenticate.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Assign Roles to Users

Roles determine what a user can do within a project. Assign the minimum role necessary
for the user's responsibilities.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Identity > Projects** (admin view). On the target project row, click the
    **More** dropdown and select **Manage User**. Select users and assign roles.

    | Role     | Capability                                                        |
    | -------- | ----------------------------------------------------------------- |
    | `admin`  | Full project administration — manage resources, users, and quotas |
    | `member` | Standard access — create and manage resources within the project  |
    | `reader` | Read-only — suitable for monitoring, auditing, and dashboards     |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Assign member role to user in project" theme={null}
    openstack role add \
      --project backend-prod \
      --user alice \
      member
    ```

    ```bash title="List all role assignments for a user" theme={null}
    openstack role assignment list \
      --user alice \
      --names
    ```

    ```bash title="Grant reader access to another project" theme={null}
    openstack role add \
      --project monitoring \
      --user alice \
      reader
    ```

    <Note>
      A user can hold different roles in different projects simultaneously. The token scope
      determines which role is active for each API request.
    </Note>
  </Tab>
</Tabs>

***

## Update User Accounts

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Open a user in **Identity > Users** (admin view) and click **Edit** to modify their email,
    primary project, or enabled state. Use **Change Password** to set a new password.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Change user email" theme={null}
      openstack user set --email newemail@example.com alice
      ```

      ```bash title="Reset user password" theme={null}
      openstack user set --password-prompt alice
      ```

      ```bash title="Disable user (preserves resource ownership)" theme={null}
      openstack user set --disable alice
      ```

      ```bash title="Re-enable a disabled user" theme={null}
      openstack user set --enable alice
      ```

      ```bash title="Delete user permanently" theme={null}
      openstack user delete alice
      ```
    </CodeGroup>

    <Warning>
      Deleting a user does not delete resources they own. Orphaned instances, volumes, and
      networks must be reassigned or cleaned up manually before removing the account.
    </Warning>
  </Tab>
</Tabs>

***

## List and Audit Users

Regularly review active user accounts and role assignments as part of access governance.

```bash title="List all users in the Default domain" theme={null}
openstack user list --domain Default
```

```bash title="List all users with their enabled status" theme={null}
openstack user list -c Name -c Enabled
```

```bash title="Audit all role assignments across all projects" theme={null}
openstack role assignment list --names
```

<Tip>
  Run quarterly access reviews using `openstack role assignment list --names` to identify
  accounts with elevated roles that may no longer be required.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Projects" href="/services/identity/projects">
    Create projects and manage team membership with role assignments.
  </Card>

  <Card title="Application Credentials" href="/services/identity/application-credentials">
    Create non-interactive credentials for automation pipelines and CI/CD systems.
  </Card>

  <Card title="Multi-Factor Authentication" href="/services/identity/multi-factor-auth">
    Enable TOTP-based two-factor authentication for enhanced user account security.
  </Card>

  <Card title="Troubleshooting" href="/services/identity/troubleshooting">
    Resolve authentication failures, permission errors, and token scope issues.
  </Card>
</CardGroup>


# Image Service Administration
Source: https://docs.xloud.tech/services/images/admin-guide

Configure image storage backends, metadata schemas, caching, and access policies for Xloud Image Service.

<p>Overview</p>

The Xloud Image Service administration guide covers every operational concern for running
the image service in production — from selecting and configuring the storage backend to
securing the image catalog, defining metadata schemas, and tuning the image cache for
optimal instance launch performance.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

<CardGroup>
  <Card title="Architecture" icon="network" href="/services/images/architecture">
    Service topology, storage backend options, and data flow for the Image Service.
  </Card>

  <Card title="Storage Backends" icon="database" href="/services/images/storage-backends">
    Configure Xloud Distributed Storage (RBD), file store, or object storage as the image backend.
  </Card>

  <Card title="Image Import" icon="cloud-download" href="/services/images/image-import">
    Use the web-download and chunked upload import APIs for large-scale image ingestion.
  </Card>

  <Card title="Metadata Definitions" icon="tag" href="/services/images/metadata">
    Define structured metadata namespaces for consistent image property schemas.
  </Card>

  <Card title="Image Cache" icon="zap" href="/services/images/image-cache">
    Configure per-node image caching to reduce instance launch times.
  </Card>

  <Card title="Quotas" icon="gauge" href="/services/images/quotas">
    Enforce per-project image count and storage size limits.
  </Card>

  <Card title="Security" icon="shield" href="/services/images/image-security">
    Configure image signing, property protections, and public image access controls.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/images/admin-troubleshooting">
    Diagnose storage backend failures, API errors, and cache performance issues.
  </Card>
</CardGroup>

***

<p>Quick Reference</p>

| Task                             | Command                                          |
| -------------------------------- | ------------------------------------------------ |
| Check Image API container status | `docker ps --filter name=glance`                 |
| View Image API logs              | `docker logs glance_api --tail 100`              |
| List all images (admin)          | `openstack image list --all-projects`            |
| Show image quota for project     | `openstack quota show --project <name>`          |
| Verify RBD pool exists           | `ceph osd pool ls \| grep images`                |
| Trigger cache pre-fetch          | `docker exec glance_api glance-cache-prefetcher` |

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Image User Guide" icon="book-open" href="/services/images/user-guide">
    Day-to-day operations — uploading images, snapshots, and sharing.
  </Card>

  <Card title="Identity Admin Guide" icon="fingerprint" href="/services/identity/admin-guide">
    Manage authentication backends and access policies governing image operations.
  </Card>
</CardGroup>


# Image Service Admin Troubleshooting
Source: https://docs.xloud.tech/services/images/admin-troubleshooting

Diagnose and resolve Image Service API failures, storage backend connectivity issues, and cache performance problems.

## Overview

This guide covers platform-level Image Service issues that require administrator access —
storage backend connectivity, API container failures, cache performance, and upload
size limit issues.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  For user-facing issues such as upload progress, shared image visibility, or launch
  failures, see the [Image User Troubleshooting](/services/images/troubleshooting) guide.
</Note>

***

## API Failures

<AccordionGroup>
  <Accordion title="Image API not responding" icon="server">
    **Cause**: The Image API container is stopped or the service has crashed.

    **Diagnose**:

    ```bash title="Check Image API container status" theme={null}
    docker ps --filter name=glance
    ```

    If the container is not running:

    ```bash title="View container exit logs" theme={null}
    docker logs glance_api --tail 50
    ```

    **Resolution**: Restart via XDeploy:

    ```bash title="Restart Image API" theme={null}
    xavs-ansible deploy --tags glance
    ```
  </Accordion>

  <Accordion title="Upload fails with 413 Request Entity Too Large" icon="upload">
    **Cause**: HAProxy is enforcing an upload size limit smaller than the image file.

    **Resolution**: Adjust the timeout and body size settings in XDeploy globals:

    ```yaml title="Increase HAProxy upload limits" theme={null}
    haproxy_client_body_timeout: 300s
    haproxy_http_request_timeout: 600s
    ```

    Apply:

    ```bash title="Apply HAProxy configuration" theme={null}
    xavs-ansible deploy --tags haproxy
    ```
  </Accordion>
</AccordionGroup>

***

## Storage Backend Issues

<AccordionGroup>
  <Accordion title="Image status remains 'queued' after upload" icon="clock">
    **Cause**: The Image API cannot reach the storage backend — RBD cluster unreachable,
    wrong keyring, or Swift authentication failed.

    **Diagnose**:

    ```bash title="Check Image API logs for storage errors" theme={null}
    docker logs glance_api --tail 100 | grep -i "error\|exception\|ceph\|rbd"
    ```

    For RBD backend:

    ```bash title="Verify RBD pool exists" theme={null}
    ceph osd pool ls | grep images
    ```

    ```bash title="Test glance keyring access" theme={null}
    rbd --keyring /etc/ceph/ceph.client.glance.keyring \
      --id glance ls images
    ```
  </Accordion>

  <Accordion title="Compute cannot download image during instance launch" icon="server">
    **Cause**: The compute node cannot reach the Image API, or the image data is corrupt.

    **Diagnose**: Test connectivity from the compute node:

    ```bash title="Test image API reachability from compute node" theme={null}
    curl -H "X-Auth-Token: $OS_AUTH_TOKEN" \
      https://api.<your-domain>:9292/v2/images/<IMAGE_ID>
    ```

    For corrupt images, verify the checksum:

    ```bash title="Verify image checksum" theme={null}
    openstack image show <IMAGE_ID> -c checksum
    md5sum /path/to/original-image.qcow2
    ```

    If checksums differ, re-upload the image.
  </Accordion>
</AccordionGroup>

***

## Cache Issues

<AccordionGroup>
  <Accordion title="Image cache not reducing launch times" icon="zap">
    **Cause**: Cache is not enabled, the compute node has insufficient local disk, or
    the pre-fetcher has not yet run.

    **Diagnose**: Verify cache is enabled and check the cache directory:

    ```bash title="Check cache directory on Image API node" theme={null}
    docker exec glance_api ls -lh /var/lib/glance/image-cache/
    ```

    Images appear in the cache directory after the first instance launch from each image.
    Trigger the pre-fetcher manually:

    ```bash title="Trigger cache pre-fetch" theme={null}
    docker exec glance_api glance-cache-prefetcher
    ```
  </Accordion>

  <Accordion title="Cache directory full — disk pressure" icon="hard-drive">
    **Cause**: The cache has grown beyond the configured `glance_cache_max_size`.

    **Resolution**: Clear old cached entries:

    ```bash title="Clear stale cache entries" theme={null}
    docker exec glance_api glance-cache-manage delete-all-cached-images
    ```

    Then increase the cache size limit in XDeploy globals and redeploy:

    ```yaml title="Increase cache size limit" theme={null}
    glance_cache_max_size: 21474836480  # 20 GB
    ```
  </Accordion>
</AccordionGroup>

***

## Service Log Reference

| Component        | Log command                                              |
| ---------------- | -------------------------------------------------------- |
| Image API        | `docker logs glance_api --tail 100`                      |
| HAProxy          | `docker logs haproxy --tail 100 \| grep 9292`            |
| Image API config | `docker exec glance_api cat /etc/glance/glance-api.conf` |

***

## Next Steps

<CardGroup>
  <Card title="Storage Backends" href="/services/images/storage-backends">
    Review backend configuration to prevent connectivity failures.
  </Card>

  <Card title="Image Cache" href="/services/images/image-cache">
    Tune cache size and pre-fetch settings for optimal performance.
  </Card>

  <Card title="Security" href="/services/images/image-security">
    Review security configuration after resolving access-related issues.
  </Card>

  <Card title="Architecture" href="/services/images/architecture">
    Understand component relationships to identify the source of failures.
  </Card>
</CardGroup>


# Image Service Architecture
Source: https://docs.xloud.tech/services/images/architecture

Understand the Xloud Image Service topology, storage backend options, and the upload and download data flow.

## Overview

The Xloud Image Service consists of an API tier, a registry database, and a pluggable
storage backend. Upload and download traffic flows through the API, while metadata queries
are handled by the registry backed by MariaDB. Understanding the architecture is essential
for sizing deployments, selecting the right storage backend, and troubleshooting
performance issues.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Service Topology

```mermaid theme={null}
graph TD
    U([User / Dashboard / Compute]) --> HAP[HAProxy<br/>:9292]
    HAP --> G1[Image API<br/>Node 1]
    HAP --> G2[Image API<br/>Node 2]
    G1 --> REG[(Registry DB<br/>MariaDB)]
    G2 --> REG
    G1 --> STORE[Storage Backend]
    G2 --> STORE
    STORE --> FILE[File Store<br/>/var/lib/glance/images]
    STORE --> RBD[Xloud Distributed Storage<br/>RBD pool]
    STORE --> SWIFT[Xloud Object Storage<br/>Swift container]
    G1 --> CACHE[Image Cache<br/>optional per-node]
    style HAP fill:#197560,color:#fff
    style G1 fill:#3F8F7E,color:#fff
    style G2 fill:#3F8F7E,color:#fff
    style REG fill:#145C4C,color:#fff
    style STORE fill:#3F8F7E,color:#fff
```

***

## Component Reference

| Component    | Port     | Description                                               |
| ------------ | -------- | --------------------------------------------------------- |
| Image API    | 9292     | REST API for image CRUD operations and data streaming     |
| HAProxy      | 9292     | Load balances requests across all Image API nodes         |
| Registry     | Internal | Stores image metadata in MariaDB                          |
| File Store   | —        | Local filesystem storage (single-node or NFS)             |
| RBD Store    | —        | Xloud Distributed Storage (Ceph RBD) — recommended for HA |
| Object Store | —        | Xloud Object Storage (Swift)                              |
| Image Cache  | —        | Optional local cache of frequently used image data        |

***

## Upload Data Flow

```mermaid theme={null}
sequenceDiagram
    participant U as User / CLI
    participant H as HAProxy :9292
    participant A as Image API
    participant DB as MariaDB
    participant S as Storage Backend
    U->>H: POST /v2/images (metadata)
    H->>A: Create image record
    A->>DB: INSERT image record (status=queued)
    DB-->>A: Image ID
    A-->>U: Image ID
    U->>H: PUT /v2/images/{id}/file (binary data)
    H->>A: Stream upload
    A->>DB: UPDATE status=saving
    A->>S: Write image data
    S-->>A: Write confirmed
    A->>DB: UPDATE status=active, checksum
    A-->>U: 204 No Content
```

***

## Storage Backend Selection

| Backend            | HA               | Performance | Use Case                           |
| ------------------ | ---------------- | ----------- | ---------------------------------- |
| **RBD (Ceph)**     | Yes              | Highest     | Production HA deployments          |
| **File Store**     | Single-node only | Good        | Single-node dev/test or NFS-backed |
| **Object Storage** | Yes              | Moderate    | When Swift is already deployed     |

<Tip>
  For production deployments with Xloud Distributed Storage, use RBD as both the image
  and volume backend. This enables zero-copy RBD cloning — instances launch in seconds
  regardless of image size.
</Tip>

***

## High Availability Considerations

<AccordionGroup>
  <Accordion title="Multiple API nodes" icon="server">
    Deploy two or more Image API nodes for redundancy. HAProxy distributes upload and
    download requests across all healthy nodes. All nodes must share the same storage
    backend — RBD or Swift — to ensure images uploaded to one node are readable from
    another.
  </Accordion>

  <Accordion title="File store limitations" icon="circle-x">
    The file store writes to a local directory. In a multi-node Image API deployment,
    all nodes must mount the same NFS directory. If NFS is unavailable, all image
    operations fail. Use RBD for true HA without NFS dependencies.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Storage Backends" href="/services/images/storage-backends">
    Configure RBD, file store, or Swift as the image storage backend.
  </Card>

  <Card title="Image Cache" href="/services/images/image-cache">
    Enable per-node caching to accelerate instance launch times.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/images/admin-troubleshooting">
    Diagnose backend connectivity and API-level failures.
  </Card>

  <Card title="Security" href="/services/images/image-security">
    Configure image signing and property protections.
  </Card>
</CardGroup>


# Image Service CLI Reference
Source: https://docs.xloud.tech/services/images/cli-reference

Complete openstack image CLI commands for managing Xloud Images — upload, download, share, set properties, and manage image metadata.

## Overview

The `openstack image` command group manages the full lifecycle of VM images — uploading, downloading, sharing across projects, and managing properties.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python glanceclient installed: `pip install python-glanceclient`
</Note>

***

## List and Inspect

<CodeGroup>
  ```bash title="List images" theme={null}
  openstack image list
  openstack image list --status active
  openstack image list --public
  openstack image list --private
  openstack image list --shared
  ```

  ```bash title="Show image details" theme={null}
  openstack image show <image>
  ```

  ```bash title="Show image properties" theme={null}
  openstack image show <image> --format json | jq .properties
  ```
</CodeGroup>

***

## Upload Images

<CodeGroup>
  ```bash title="Upload QCOW2 image" theme={null}
  openstack image create \
    --container-format bare \
    --disk-format qcow2 \
    --file ubuntu-22.04.qcow2 \
    --public \
    Ubuntu-22.04
  ```

  ```bash title="Upload RAW image" theme={null}
  openstack image create \
    --container-format bare \
    --disk-format raw \
    --file ubuntu-22.04.raw \
    Ubuntu-22.04-raw
  ```

  ```bash title="Upload with minimum requirements" theme={null}
  openstack image create \
    --container-format bare \
    --disk-format qcow2 \
    --file image.qcow2 \
    --min-ram 1024 \
    --min-disk 10 \
    my-image
  ```
</CodeGroup>

***

## Download Images

```bash title="Download image to file" theme={null}
openstack image save --file ubuntu-22.04.qcow2 <image>
```

***

## Set Properties

<CodeGroup>
  ```bash title="Set image visibility" theme={null}
  openstack image set --public <image>
  openstack image set --private <image>
  openstack image set --shared <image>
  openstack image set --community <image>
  ```

  ```bash title="Set OS and boot properties" theme={null}
  openstack image set \
    --property os_type=linux \
    --property os_distro=ubuntu \
    --property hw_firmware_type=uefi \
    <image>
  ```

  ```bash title="Enable Secure Boot" theme={null}
  openstack image set \
    --property hw_firmware_type=uefi \
    --property os_secure_boot=required \
    <image>
  ```

  ```bash title="Remove a property" theme={null}
  openstack image unset --property hw_firmware_type <image>
  ```
</CodeGroup>

***

## Share Images

<CodeGroup>
  ```bash title="Share image with a project" theme={null}
  openstack image add project <image> <project-id>
  ```

  ```bash title="Accept shared image (recipient project)" theme={null}
  openstack image set --accept <image>
  ```

  ```bash title="List projects image is shared with" theme={null}
  openstack image member list <image>
  ```

  ```bash title="Unshare image" theme={null}
  openstack image remove project <image> <project-id>
  ```
</CodeGroup>

***

## Delete

```bash title="Delete image" theme={null}
openstack image delete <image>
```

```bash title="Delete multiple images" theme={null}
openstack image delete <image1> <image2>
```

***

## Next Steps

<CardGroup>
  <Card title="Upload Image Guide" href="/services/images/upload-image">
    Detailed guide for preparing and uploading images
  </Card>

  <Card title="Compute CLI" href="/services/compute/cli-reference">
    Launch instances from uploaded images
  </Card>
</CardGroup>


# Convert Image Formats
Source: https://docs.xloud.tech/services/images/convert-formats

Convert virtual machine images between QCOW2, RAW, VMDK, VHD, and VDI formats using qemu-img before uploading to Xloud.

## Overview

Images obtained from VMware, VirtualBox, Hyper-V, or other virtualization platforms use
formats that differ from what Xloud Compute expects. The `qemu-img` tool converts between
all major disk image formats quickly and without data loss. QCOW2 is the recommended
format for Xloud — it provides compression, snapshot support, and efficient sparse storage.

<Note>
  **Prerequisites**

  * A Linux workstation with `qemu-utils` installed
  * The source image file in its original format
  * Free disk space: RAW conversions require space equal to the full virtual disk size
</Note>

***

## Supported Formats

| Format         | Extension       | Description                       | Use Case                                                     |
| -------------- | --------------- | --------------------------------- | ------------------------------------------------------------ |
| **QCOW2**      | `.qcow2`        | Compressed, sparse, copy-on-write | Recommended for Xloud — best balance of size and performance |
| **RAW**        | `.img`, `.raw`  | Uncompressed, full disk image     | Maximum I/O performance; large file size                     |
| **VMDK**       | `.vmdk`         | VMware native format              | Import from VMware ESXi or Workstation                       |
| **VHD / VHDX** | `.vhd`, `.vhdx` | Hyper-V / Azure native format     | Import from Hyper-V or Microsoft Azure                       |
| **VDI**        | `.vdi`          | VirtualBox native format          | Import from Oracle VirtualBox                                |

<Warning>
  Converting to RAW format creates a file equal to the full virtual disk size, not just
  the used space. A 50 GB virtual disk produces a 50 GB RAW file even if only 5 GB is
  used. Ensure sufficient disk space before converting.
</Warning>

***

## Install qemu-img

<Tabs>
  <Tab title="Ubuntu / Debian" icon="server">
    ```bash title="Install qemu-utils" theme={null}
    apt-get update && apt-get install -y qemu-utils
    ```
  </Tab>

  <Tab title="CentOS / AlmaLinux / Rocky Linux" icon="server">
    ```bash title="Install qemu-img" theme={null}
    dnf install -y qemu-img
    ```
  </Tab>
</Tabs>

***

## Check Image Information

Before converting, inspect the source image to confirm its format, virtual size, and
backing chain:

```bash title="Inspect image metadata" theme={null}
qemu-img info source-image.vmdk
```

Expected output:

```text title="Example qemu-img info output" theme={null}
image: source-image.vmdk
file format: vmdk
virtual size: 20 GiB (21474836480 bytes)
disk size: 4.2 GiB
cluster_size: 65536
```

<Tip>
  The `disk size` shows actual data written; `virtual size` shows the size instances
  see. QCOW2 preserves this sparse allocation — RAW expands to the full virtual size.
</Tip>

***

## Common Conversions

<Tabs>
  <Tab title="VMDK to QCOW2" icon="terminal">
    Convert a VMware disk image to QCOW2:

    <Steps>
      <Step title="Convert the image" icon="repeat">
        ```bash title="Convert VMDK to QCOW2" theme={null}
        qemu-img convert \
          -f vmdk \
          -O qcow2 \
          source-image.vmdk \
          output-image.qcow2
        ```

        The `-f` flag specifies the source format; `-O` specifies the output format.
      </Step>

      <Step title="Verify the output" icon="circle-check">
        ```bash title="Verify converted image" theme={null}
        qemu-img info output-image.qcow2
        ```

        <Check>Output shows `file format: qcow2` and virtual size matches the source.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="VHD to QCOW2" icon="terminal">
    Convert a Hyper-V or Azure VHD image to QCOW2:

    <Steps>
      <Step title="Convert the image" icon="repeat">
        ```bash title="Convert VHD to QCOW2" theme={null}
        qemu-img convert \
          -f vpc \
          -O qcow2 \
          source-image.vhd \
          output-image.qcow2
        ```

        <Note>VHD uses the `vpc` format identifier in `qemu-img`. VHDX uses `vhdx`.</Note>
      </Step>

      <Step title="Verify the output" icon="circle-check">
        ```bash title="Verify converted image" theme={null}
        qemu-img info output-image.qcow2
        ```

        <Check>Output shows `file format: qcow2`.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="RAW to QCOW2" icon="terminal">
    Compress a RAW image into QCOW2 to reduce storage footprint:

    <Steps>
      <Step title="Convert with compression" icon="repeat">
        ```bash title="Convert RAW to QCOW2 with compression" theme={null}
        qemu-img convert \
          -f raw \
          -O qcow2 \
          -c \
          source-image.raw \
          output-image.qcow2
        ```

        The `-c` flag enables compression. Compressed QCOW2 images can be significantly
        smaller than their RAW equivalents.
      </Step>

      <Step title="Compare sizes" icon="chart-bar">
        ```bash title="Compare source and output sizes" theme={null}
        ls -lh source-image.raw output-image.qcow2
        ```

        <Check>QCOW2 output is typically 30–70% smaller than the RAW source for OS images.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="QCOW2 to RAW" icon="terminal">
    Convert to RAW for maximum I/O performance or compatibility testing:

    <Steps>
      <Step title="Check available disk space first" icon="hard-drive">
        ```bash title="Check free space vs. virtual image size" theme={null}
        df -h .
        qemu-img info source-image.qcow2 | grep "virtual size"
        ```

        Ensure free space exceeds the virtual size of the image.
      </Step>

      <Step title="Convert to RAW" icon="repeat">
        ```bash title="Convert QCOW2 to RAW" theme={null}
        qemu-img convert \
          -f qcow2 \
          -O raw \
          source-image.qcow2 \
          output-image.raw
        ```
      </Step>

      <Step title="Verify the output" icon="circle-check">
        ```bash title="Verify the RAW image" theme={null}
        qemu-img info output-image.raw
        ```

        <Check>Output shows `file format: raw` and file size equals the virtual disk size.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Validation

After conversion, perform a consistency check before uploading to Xloud:

```bash title="Check image integrity" theme={null}
qemu-img check output-image.qcow2
```

Expected output:

```text title="Clean image output" theme={null}
No errors were found on the image.
```

<Warning>
  If `qemu-img check` reports errors, the conversion may have encountered a corrupt
  source image. Re-obtain the source and retry the conversion. Do not upload a
  corrupted image — instances launched from it will fail unpredictably.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Upload an Image" href="/services/images/upload-image">
    Upload the converted QCOW2 image to the Xloud Image Service.
  </Card>

  <Card title="Image Requirements" href="/services/images/image-requirements">
    Verify the image meets all Xloud Compute compatibility requirements.
  </Card>

  <Card title="Modify Images" href="/services/images/modify-images">
    Customize images with additional packages and configuration before uploading.
  </Card>

  <Card title="Image Formats" href="/services/images/image-formats">
    Compare all supported image formats and choose the right one for your workload.
  </Card>
</CardGroup>


# Create Instance Snapshots
Source: https://docs.xloud.tech/services/images/create-snapshot

Capture running or stopped instances as reusable image snapshots for backup, cloning, and golden image workflows.

## Overview

Instance snapshots capture the entire disk state of a running or stopped instance as a new
image registered in the Xloud Image Service. Snapshots are used for backups, creating
golden images for new instance launches, and cloning instances across projects. The
snapshot is stored in the same format as the original boot image and can be launched
on any compute host.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Warning>
  Snapshotting a running instance captures disk state while the instance is live. For
  application consistency, quiesce the guest OS (flush writes, stop services, or pause the
  database) before initiating the snapshot to avoid capturing a partially-written state.
</Warning>

***

## Create a Snapshot

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Locate the instance" icon="monitor">
        Navigate to **Compute > Instances**. Find the instance you want to snapshot.
      </Step>

      <Step title="Initiate the snapshot" icon="camera">
        In the instance row, open the **Actions** dropdown and select **Create Snapshot**.
      </Step>

      <Step title="Name the snapshot" icon="tag">
        Enter a descriptive snapshot name using a consistent convention, e.g.:
        `<instance-name>-<date>` (e.g., `web-server-2026-03-18`)

        Click **Create Snapshot**. The instance continues running during snapshot creation.
      </Step>

      <Step title="Monitor completion" icon="activity">
        Navigate to **Compute > Images** to see the snapshot in progress.

        <Check>
          The snapshot appears with status **Active** when complete. It is immediately
          available for launching new instances.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the snapshot" icon="camera">
        ```bash title="Create instance snapshot" theme={null}
        openstack server image create \
          --name web-server-2026-03-18 \
          my-instance
        ```

        The command returns the new image ID. The snapshot is created asynchronously.
      </Step>

      <Step title="Wait for completion" icon="clock">
        ```bash title="Poll snapshot status" theme={null}
        openstack image show web-server-2026-03-18 -c status
        ```

        <Check>`status` shows `active` — the snapshot is ready to launch new instances.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Snapshot for Consistency

For production workloads, follow this pre-snapshot checklist to ensure data consistency:

<Steps>
  <Step title="Prepare the guest OS" icon="settings">
    SSH into the instance and run:

    ```bash title="Flush filesystem buffers (Linux)" theme={null}
    sync
    ```

    For database servers, flush and lock tables before snapshotting:

    ```bash title="MySQL/MariaDB: flush and lock" theme={null}
    mysql -u root -e "FLUSH TABLES WITH READ LOCK;"
    ```
  </Step>

  <Step title="Create the snapshot" icon="camera">
    ```bash title="Create snapshot immediately after flush" theme={null}
    openstack server image create \
      --name db-primary-2026-03-18 \
      db-primary
    ```
  </Step>

  <Step title="Release locks after snapshot initiates" icon="unlock">
    Once the snapshot command returns (the snapshot is queued in the backend), release any locks:

    ```bash title="MySQL/MariaDB: release lock" theme={null}
    mysql -u root -e "UNLOCK TABLES;"
    ```

    <Tip>
      The snapshot operation itself is fast (metadata only at initiation). The actual data
      copy happens in the background — you can release locks as soon as the CLI command returns.
    </Tip>
  </Step>
</Steps>

***

## Manage Snapshots

Snapshots are regular images stored in the Image Service. Manage them using the same
image commands:

<CodeGroup>
  ```bash title="List snapshots" theme={null}
  openstack image list \
    --property image_type=snapshot
  ```

  ```bash title="Delete a snapshot" theme={null}
  openstack image delete web-server-2026-03-18
  ```

  ```bash title="Set snapshot visibility to shared" theme={null}
  openstack image set --shared web-server-2026-03-18
  ```
</CodeGroup>

***

## Launch from Snapshot

After a snapshot is active, launch new instances directly from it:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    When creating a new instance, in **Step 1 (Base Config)**, select **Instance Snapshot**
    as the Start Source. The snapshot appears in the selection table.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Launch instance from snapshot" theme={null}
    openstack server create \
      --image web-server-2026-03-18 \
      --flavor m1.medium \
      --network private \
      --key-name my-keypair \
      restored-web-server
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Image Properties" href="/services/images/image-properties">
    Add metadata and hardware requirements to your snapshots.
  </Card>

  <Card title="Share Images" href="/services/images/share-images">
    Share snapshots with other projects in your organization.
  </Card>

  <Card title="Upload an Image" href="/services/images/upload-image">
    Upload external OS images alongside your captured snapshots.
  </Card>

  <Card title="Troubleshooting" href="/services/images/troubleshooting">
    Resolve snapshot creation failures and status issues.
  </Card>
</CardGroup>


# Get Cloud Images
Source: https://docs.xloud.tech/services/images/get-images

Download pre-built cloud images for popular Linux distributions and Windows Server for use with Xloud Compute.

## Overview

Pre-built cloud images are official distribution releases packaged for virtual machine
deployment. Each image includes cloud-init for first-boot customization, SSH server
access, and storage drivers optimized for virtual environments. Use these images as the
starting point for new instances rather than installing an operating system from scratch.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Tip>
  Start with CirrOS for initial environment testing. It is a minimal test image that
  launches in seconds and consumes almost no resources.
</Tip>

***

## Available Images

<Note>
  All images listed below require [cloud-init](https://cloud-init.io) to function
  correctly in Xloud. Images without cloud-init will not receive SSH keys, hostname
  assignments, or network configuration on launch.
</Note>

| Distribution            | Download URL                                                                                             | Format | Default User                |
| ----------------------- | -------------------------------------------------------------------------------------------------------- | ------ | --------------------------- |
| **Ubuntu 24.04 LTS**    | [cloud-images.ubuntu.com](https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img) | QCOW2  | `ubuntu`                    |
| **Ubuntu 22.04 LTS**    | [cloud-images.ubuntu.com](https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img) | QCOW2  | `ubuntu`                    |
| **CentOS Stream 9**     | [cloud.centos.org](https://cloud.centos.org/centos/9-stream/x86_64/images/)                              | QCOW2  | `cloud-user`                |
| **AlmaLinux 9**         | [repo.almalinux.org](https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/)                        | QCOW2  | `almalinux`                 |
| **Rocky Linux 9**       | [dl.rockylinux.org](https://dl.rockylinux.org/pub/rocky/9/images/x86_64/)                                | QCOW2  | `rocky`                     |
| **Debian 12**           | [cloud.debian.org](https://cloud.debian.org/images/cloud/bookworm/latest/)                               | QCOW2  | `debian`                    |
| **Fedora 41**           | [fedoraproject.org](https://fedoraproject.org/cloud/download)                                            | QCOW2  | `fedora`                    |
| **CirrOS 0.6.2**        | [download.cirros-cloud.net](https://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img)        | QCOW2  | `cirros` / pass: `gocubsgo` |
| **Windows Server 2022** | [microsoft.com/evalcenter](https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2022)      | VHD    | `Administrator`             |

<Warning>
  Windows Server evaluation images expire after 180 days. For production use, supply a
  licensed Windows image prepared with Cloudbase-Init installed.
</Warning>

***

## Download and Upload an Image

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Images" icon="disc">
        Navigate to
        **Compute > Images**. Click **Create Image**.
      </Step>

      <Step title="Configure the image record">
        | Field                     | Recommended Value                                                     |
        | ------------------------- | --------------------------------------------------------------------- |
        | **Name**                  | `ubuntu-24.04-lts`                                                    |
        | **Upload Type**           | **File URL** (for web download) or **Upload File** (for local upload) |
        | **File URL**              | Paste the download URL from the table above                           |
        | **Disk Format**           | `QCOW2`                                                               |
        | **OS**                    | Select the matching distribution (e.g., Ubuntu)                       |
        | **OS Version**            | `24.04`                                                               |
        | **OS Admin**              | `root` (Linux) or `Administrator` (Windows)                           |
        | **Min System Disk (GiB)** | `10` for most Linux images; `40` for Windows                          |
        | **Min Memory (GiB)**      | `1` for Linux; `2` for Windows                                        |

        See [Upload an Image](/services/images/upload-image) for the complete field reference.
      </Step>

      <Step title="Create the image">
        Click **Confirm**. The image enters `Saving` status while the file is
        fetched and stored. Large images may take several minutes.

        <Check>
          The status transitions to **Active** when storage is complete. The image
          is available immediately for launching instances.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Download the image locally" icon="download">
        ```bash title="Download Ubuntu 24.04 LTS cloud image" theme={null}
        curl -L -o ubuntu-24.04-lts.img \
          https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
        ```

        Verify the download completed and matches the published checksum:

        ```bash title="Verify checksum (SHA256)" theme={null}
        sha256sum ubuntu-24.04-lts.img
        ```
      </Step>

      <Step title="Upload to the image service" icon="upload">
        ```bash title="Upload image with metadata" theme={null}
        openstack image create \
          --disk-format qcow2 \
          --container-format bare \
          --file ubuntu-24.04-lts.img \
          --min-disk 10 \
          --min-ram 512 \
          --property os_type=linux \
          --property os_distro=ubuntu \
          --property os_version=24.04 \
          ubuntu-24.04-lts
        ```

        Note the image `id` from the output for verification.
      </Step>

      <Step title="Verify the image is active" icon="circle-check">
        ```bash title="Check image status" theme={null}
        openstack image show ubuntu-24.04-lts -c status -c size -c disk_format
        ```

        <Check>`status` shows `active` — the image is ready for use with instances.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Validation

Confirm the uploaded image is registered and ready before launching instances.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Images**. Locate the image by name and verify:

    * **Status**: Active (green indicator)
    * **Format**: Matches the selected disk format
    * **Size**: Matches the expected download size

    <Check>Image shows **Active** status and is selectable when launching a new instance.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List images and filter by name" theme={null}
    openstack image list --name ubuntu-24.04-lts -c Name -c Status -c "Disk Format"
    ```

    Expected output:

    ```text title="Expected output" theme={null}
    +------------------+--------+-------------+
    | Name             | Status | Disk Format |
    +------------------+--------+-------------+
    | ubuntu-24.04-lts | active | qcow2       |
    +------------------+--------+-------------+
    ```

    <Check>Status `active` confirms the image is stored and ready for compute use.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Image Requirements" href="/services/images/image-requirements">
    Understand what makes an image compatible with Xloud Compute.
  </Card>

  <Card title="Upload an Image" href="/services/images/upload-image">
    Upload images from local files or import via web URL.
  </Card>

  <Card title="Convert Formats" href="/services/images/convert-formats">
    Convert VMDK, VHD, or RAW images to QCOW2 before uploading.
  </Card>

  <Card title="Image Properties" href="/services/images/image-properties">
    Set OS metadata, hardware requirements, and scheduler hints.
  </Card>
</CardGroup>


# Image Cache
Source: https://docs.xloud.tech/services/images/image-cache

Configure the Xloud Image Service cache to reduce instance launch times by storing frequently-used images locally on compute nodes.

## Overview

The image cache stores a local copy of frequently-used images on each compute node,
eliminating repeated downloads from the central image service at instance launch time.
Enabling the cache is most impactful when the same image is used to launch many instances
on the same compute node — typical in auto-scaling and cluster workloads.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Cache Configuration

Configure cache settings via XDeploy globals before deploying:

| Setting                          | Default | Description                                     |
| -------------------------------- | ------- | ----------------------------------------------- |
| `glance_enable_image_cache`      | `no`    | Enable the image cache                          |
| `glance_cache_max_size`          | 10 GB   | Maximum total size of the cache directory       |
| `glance_cache_staleness_seconds` | 86400   | How long a cached entry remains fresh (seconds) |
| `glance_cache_prefetcher_delay`  | 300     | Seconds between pre-fetch sweep runs            |

```yaml title="Image cache configuration in XDeploy globals" theme={null}
glance_enable_image_cache: "yes"
glance_cache_max_size: 10737418240      # 10 GB in bytes
glance_cache_staleness_seconds: 86400   # 24 hours
glance_cache_prefetcher_delay: 300      # 5 minutes
```

Deploy after configuring:

```bash title="Apply image cache configuration" theme={null}
xavs-ansible deploy --tags glance
```

***

## How the Cache Works

```mermaid theme={null}
sequenceDiagram
    participant C as Compute Node
    participant CA as Image Cache
    participant G as Image Service
    participant S as Storage Backend
    C->>CA: Request image data (launch)
    alt Cache hit
        CA-->>C: Return cached data (fast)
    else Cache miss
        CA->>G: Forward request
        G->>S: Fetch from backend
        S-->>G: Image data
        G-->>CA: Image data
        CA->>CA: Store in cache directory
        CA-->>C: Return image data
    end
```

***

## Verify Cache Status

```bash title="Check cache directory on Image API node" theme={null}
docker exec glance_api ls -lh /var/lib/glance/image-cache/
```

```bash title="View cached image entries" theme={null}
docker exec glance_api glance-cache-manage list-cached
```

```bash title="View pending pre-fetch queue" theme={null}
docker exec glance_api glance-cache-manage list-queued
```

***

## Manual Cache Management

<Tabs>
  <Tab title="Pre-fetch an image" icon="download">
    Queue an image for pre-caching before its first use:

    ```bash title="Queue image for pre-fetch" theme={null}
    docker exec glance_api glance-cache-manage queue-image <IMAGE_ID>
    ```

    The pre-fetcher picks it up on the next sweep interval.
  </Tab>

  <Tab title="Clear the cache" icon="trash">
    Remove all cached entries to free disk space:

    ```bash title="Clear the image cache" theme={null}
    docker exec glance_api glance-cache-manage delete-all-cached-images
    ```

    <Warning>
      Clearing the cache causes the next instance launch from each image to download
      from the storage backend. This temporarily increases launch times.
    </Warning>
  </Tab>

  <Tab title="Trigger pre-fetch manually" icon="play">
    Run the pre-fetcher outside its scheduled interval:

    ```bash title="Trigger cache pre-fetch" theme={null}
    docker exec glance_api glance-cache-prefetcher
    ```
  </Tab>
</Tabs>

***

## When to Use the Cache

<AccordionGroup>
  <Accordion title="Best case: Many instances from same image" icon="zap">
    The cache provides the greatest benefit when the same image is repeatedly used on
    the same compute node — typical patterns include:

    * Auto-scaling groups launching many identical instances
    * Development clusters where all developers use the same base image
    * CI/CD pipelines that launch many ephemeral test instances

    After the first launch (cache miss), all subsequent launches are served from the
    local cache — typically 10-50x faster than fetching from the storage backend.
  </Accordion>

  <Accordion title="Limited benefit: RBD-to-RBD cloning" icon="database">
    When both the Image Service and Block Storage use Xloud Distributed Storage (RBD),
    instance launches use zero-copy RBD clones. This already achieves near-instantaneous
    boot times regardless of image size — the image cache provides minimal additional
    benefit in this configuration.
  </Accordion>
</AccordionGroup>

<Tip>
  Enable the image cache on compute nodes with SSD-backed local storage. Storing the
  cache on slow HDDs may actually increase launch latency compared to fetching from
  a fast Ceph cluster.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Storage Backends" href="/services/images/storage-backends">
    Configure the primary storage backend that feeds the cache.
  </Card>

  <Card title="Architecture" href="/services/images/architecture">
    Understand where the cache fits in the overall Image Service topology.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/images/admin-troubleshooting">
    Diagnose cache effectiveness and storage issues.
  </Card>

  <Card title="Quotas" href="/services/images/quotas">
    Control storage consumption to ensure cache space is not exhausted.
  </Card>
</CardGroup>


# Image Formats
Source: https://docs.xloud.tech/services/images/image-formats

Compare QCOW2, RAW, VHD, and VMDK image formats and choose the right one for your Xloud workloads.

## Overview

The Xloud Image Service supports multiple virtual disk formats. Selecting the right format
affects launch performance, storage efficiency, and compatibility with your source
environment. This guide compares supported formats, explains conversion workflows, and
provides recommendations for common use cases.

<Note>
  **Prerequisites**

  * Basic familiarity with virtual disk concepts
  * `qemu-img` installed locally for format conversion (optional)
</Note>

***

## Format Comparison

| Format         | Extension       | Copy-on-Write     | Compression | Best For                                                   |
| -------------- | --------------- | ----------------- | ----------- | ---------------------------------------------------------- |
| **QCOW2**      | `.qcow2`        | Yes               | Yes         | Default. General-purpose workloads, fast snapshot support. |
| **RAW**        | `.img`, `.raw`  | No (storage-side) | No          | Maximum performance on Ceph-backed storage.                |
| **VHD / VHDX** | `.vhd`, `.vhdx` | Fixed/Dynamic     | —           | Images migrated from Hyper-V.                              |
| **VMDK**       | `.vmdk`         | Sparse/Flat       | —           | Images migrated from VMware.                               |

***

## Format Details

<AccordionGroup>
  <Accordion title="QCOW2 — Recommended" icon="layers">
    QCOW2 (QEMU Copy-On-Write v2) is the default and recommended format for Xloud.
    It supports thin provisioning (only stores data that has been written), internal
    snapshots, compression, and optional AES encryption.

    | Property      | Value                                                            |
    | ------------- | ---------------------------------------------------------------- |
    | Extension     | `.qcow2`                                                         |
    | Copy-on-write | Yes                                                              |
    | Compression   | Yes                                                              |
    | Maximum size  | No limit                                                         |
    | Best for      | General-purpose workloads, fast launch times, snapshot workflows |

    Create a new empty QCOW2 image:

    ```bash title="Create empty 20GB QCOW2 image" theme={null}
    qemu-img create -f qcow2 my-disk.qcow2 20G
    ```
  </Accordion>

  <Accordion title="RAW — Maximum Performance" icon="zap">
    RAW images are unformatted disk dumps. There is zero overhead from the format layer.
    When Xloud Distributed Storage (Ceph RBD) is the image and volume backend, RBD handles
    copy-on-write natively — making RAW the optimal format.

    | Property      | Value                                                  |
    | ------------- | ------------------------------------------------------ |
    | Extension     | `.img` or `.raw`                                       |
    | Copy-on-write | No (handled by storage backend)                        |
    | Compression   | No                                                     |
    | Best for      | Performance-sensitive workloads on Ceph-backed storage |

    <Tip>
      On Ceph-backed deployments, instance launches from RAW images use zero-copy RBD clones
      — resulting in near-instantaneous boot times regardless of image size.
    </Tip>
  </Accordion>

  <Accordion title="VHD / VHDX — Hyper-V Migration" icon="arrow-right-left">
    VHD and VHDX are the native disk formats for Microsoft Hyper-V. Import them directly
    into Xloud when migrating workloads from a Hyper-V environment.

    | Format | Source Platform | Notes                              |
    | ------ | --------------- | ---------------------------------- |
    | VHD    | Hyper-V (Gen 1) | Fixed and dynamic VHDs supported   |
    | VHDX   | Hyper-V (Gen 2) | Preferred format for Hyper-V 2012+ |

    <Tip>
      Convert VHD/VHDX to QCOW2 after import for optimal performance and Xloud snapshot support.
    </Tip>
  </Accordion>

  <Accordion title="VMDK — VMware Migration" icon="server">
    VMDK is the native disk format for VMware ESXi and Workstation. Import VMDKs directly
    when migrating virtual machines from a VMware environment.

    | Format      | Notes                                           |
    | ----------- | ----------------------------------------------- |
    | Sparse VMDK | Thin-provisioned. Import directly.              |
    | Flat VMDK   | Pre-allocated. Larger file, no format overhead. |

    <Tip>
      Convert VMDKs to QCOW2 after import to enable Xloud snapshot and resize operations.
    </Tip>
  </Accordion>
</AccordionGroup>

***

## Convert Between Formats

Use `qemu-img` to convert images locally before upload:

<CodeGroup>
  ```bash title="VHD to QCOW2" theme={null}
  qemu-img convert -f vpc -O qcow2 windows-server.vhd windows-server.qcow2
  ```

  ```bash title="VMDK to QCOW2" theme={null}
  qemu-img convert -f vmdk -O qcow2 vm-disk.vmdk vm-disk.qcow2
  ```

  ```bash title="RAW to QCOW2" theme={null}
  qemu-img convert -f raw -O qcow2 disk.img disk.qcow2
  ```

  ```bash title="QCOW2 to RAW" theme={null}
  qemu-img convert -f qcow2 -O raw ubuntu.qcow2 ubuntu.img
  ```
</CodeGroup>

Verify the converted image:

```bash title="Verify converted image" theme={null}
qemu-img info disk.qcow2
```

***

## Next Steps

<CardGroup>
  <Card title="Upload an Image" href="/services/images/upload-image">
    Upload your converted image to the Xloud Image Service.
  </Card>

  <Card title="Image Properties" href="/services/images/image-properties">
    Set hardware properties matching your image format and source platform.
  </Card>

  <Card title="Image Admin Guide — Storage Backends" href="/services/images/storage-backends">
    Configure the storage backend where images are stored after upload.
  </Card>

  <Card title="Troubleshooting" href="/services/images/troubleshooting">
    Resolve format-related launch failures and image upload errors.
  </Card>
</CardGroup>


# Image Import API
Source: https://docs.xloud.tech/services/images/image-import

Use the web-download and chunked upload import methods for large-scale or automated image ingestion in Xloud.

## Overview

The Image Import API provides advanced upload mechanisms beyond the basic PUT endpoint.
Web download allows server-side fetching directly from a URL — no local staging needed.
Chunked upload supports resumable large-image transfers. Both methods are available to
users, but administrators configure the allowed import methods and staging area at the
platform level.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Allowed Import Methods

Configure which import methods are available to users via XDeploy globals:

```yaml title="Enable import methods" theme={null}
glance_enabled_import_methods: "web-download,glance-direct"
```

| Method          | Description                                       | Best For                                       |
| --------------- | ------------------------------------------------- | ---------------------------------------------- |
| `web-download`  | Image Service fetches data from a URL server-side | Importing from public cloud image registries   |
| `glance-direct` | Client uploads in chunks via staging area         | Large files (>5 GB) requiring resumable upload |
| `copy-image`    | Copy image between stores                         | Multi-store deployments                        |

***

## Web Download Import

The web download method imports images directly from a public URL. The Image Service
downloads the file server-side — no local disk space required on the client.

<Steps>
  <Step title="Create the image record" icon="plus">
    ```bash title="Create image record for web-download" theme={null}
    openstack image create \
      --disk-format qcow2 \
      --container-format bare \
      --import-method web-download \
      ubuntu-24.04-noble
    ```

    Note the image `id` from the output.
  </Step>

  <Step title="Trigger the import" icon="cloud-download">
    ```bash title="Import from Ubuntu cloud images" theme={null}
    openstack image import \
      --method web-download \
      --uri https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img \
      <IMAGE_ID>
    ```
  </Step>

  <Step title="Monitor import progress" icon="activity">
    ```bash title="Poll import status" theme={null}
    watch -n 5 "openstack image show <IMAGE_ID> -c status"
    ```

    <Check>Status transitions from `importing` to `active` when complete.</Check>
  </Step>
</Steps>

***

## Chunked Upload (Glance Direct)

Chunked upload allows large images to be uploaded in parts, with the ability to resume
if the connection is interrupted.

<Steps>
  <Step title="Create the image record" icon="plus">
    ```bash title="Create image record" theme={null}
    openstack image create \
      --disk-format qcow2 \
      --container-format bare \
      large-windows-image
    ```

    Note the image `id`.
  </Step>

  <Step title="Stage data to the import queue" icon="upload">
    Stage image data via the v2 import API. The staging area is temporary and managed
    by the Image Service:

    ```bash title="Stage image data" theme={null}
    TOKEN=$(openstack token issue -f value -c id)

    curl -X PUT \
      -H "X-Auth-Token: $TOKEN" \
      -H "Content-Type: application/octet-stream" \
      --data-binary @/path/to/large-image.qcow2 \
      "https://api.<your-domain>:9292/v2/images/<IMAGE_ID>/stage"
    ```
  </Step>

  <Step title="Trigger import from staging" icon="circle-check">
    ```bash title="Import from staging area" theme={null}
    openstack image import \
      --method glance-direct \
      <IMAGE_ID>
    ```

    <Check>Image transitions to `active` after import completes from the staging area.</Check>
  </Step>
</Steps>

***

## Staging Area Configuration

The staging area is a temporary directory used by the glance-direct import method.
Configure its location and size limits:

```yaml title="Staging area configuration" theme={null}
glance_image_import_plugins: "image_conversion,inject_image_metadata"
glance_staging_store_uri: "file:///var/lib/glance/staging"
```

<Tip>
  The staging area should be on fast local storage (SSD) with sufficient capacity
  for your largest expected image. Staging data is deleted after successful import.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Storage Backends" href="/services/images/storage-backends">
    Configure where imported images are stored after the import completes.
  </Card>

  <Card title="Metadata" href="/services/images/metadata">
    Define metadata namespaces for consistent property schemas on imported images.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/images/admin-troubleshooting">
    Resolve web-download timeouts and staging area failures.
  </Card>

  <Card title="Upload an Image" href="/services/images/upload-image">
    Standard upload workflow for user-facing image ingestion.
  </Card>
</CardGroup>


# Image Properties
Source: https://docs.xloud.tech/services/images/image-properties

Set hardware requirements, OS metadata, and scheduler hints on Xloud images to control instance launch behavior.

## Overview

Image properties are key-value metadata attached to images that influence how the compute
service launches instances from that image. Properties communicate the OS type to the
hypervisor, set minimum hardware requirements, configure boot firmware, and define the
virtual hardware model. Well-configured properties improve compatibility and prevent
launch failures from under-resourced flavors.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Essential Properties Reference

| Property Key       | Example Value | Purpose                                                          |
| ------------------ | ------------- | ---------------------------------------------------------------- |
| `os_type`          | `linux`       | Informs the hypervisor of the guest OS family                    |
| `os_distro`        | `ubuntu`      | OS distribution for XAVS Guest Agent configuration               |
| `os_version`       | `24.04`       | OS version string                                                |
| `hw_machine_type`  | `q35`         | Sets the emulated chipset (q35 for UEFI, i440fx for legacy BIOS) |
| `hw_firmware_type` | `uefi`        | Boot firmware — `uefi` or `bios`                                 |
| `hw_disk_bus`      | `virtio`      | Disk controller — `virtio` (best performance) or `ide`           |
| `hw_scsi_model`    | `virtio-scsi` | SCSI controller model when `hw_disk_bus=scsi`                    |
| `hw_vif_model`     | `virtio`      | Network interface model                                          |
| `hw_tpm_model`     | `tpm-crb`     | Enable virtual TPM device (requires platform support)            |

***

## Set Properties

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open image details" icon="disc">
        Navigate to **Compute > Images** and click the image name.
      </Step>

      <Step title="Edit basic properties">
        Click **Edit** (the first row action) to open the Edit Image dialog. You can
        modify:

        | Field                     | Type                  | Description                                  |
        | ------------------------- | --------------------- | -------------------------------------------- |
        | **Name**                  | Text                  | Image display name                           |
        | **OS**                    | Dropdown              | Operating system distribution                |
        | **OS Version**            | Text                  | Version string                               |
        | **OS Admin**              | Text                  | Default admin username                       |
        | **Min System Disk (GiB)** | Number                | Minimum root disk (if block storage enabled) |
        | **Min Memory (GiB)**      | Number                | Minimum RAM                                  |
        | **Public**                | Checkbox (admin only) | Toggle public visibility                     |
        | **Protected**             | Checkbox              | Prevent accidental deletion                  |
        | **Description**           | Text area             | Optional notes                               |

        Click **Advanced Options** to edit qemu\_guest\_agent, CPU Policy, and CPU
        Thread Policy.

        Click **Confirm** to save.

        <Note>
          The Edit form does NOT include metadata key-value editing. To manage
          custom metadata properties, administrators use the **Manage Metadata**
          action from the More dropdown (admin view only).
        </Note>
      </Step>

      <Step title="Manage custom metadata (admin only)">
        In the admin view, click the **More** dropdown on the image row and select
        **Manage Metadata**. This opens a dialog where you can add, edit, or remove
        custom key-value metadata properties and system metadata definitions.

        <Check>Properties are saved and immediately active for new instance launches.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Set multiple properties at once" icon="tag">
        ```bash title="Set OS and hardware properties" theme={null}
        openstack image set \
          --property os_type=linux \
          --property os_distro=ubuntu \
          --property os_version=24.04 \
          --property hw_firmware_type=uefi \
          --property hw_machine_type=q35 \
          --property hw_disk_bus=virtio \
          ubuntu-24.04-lts
        ```
      </Step>

      <Step title="Verify properties were applied" icon="search">
        ```bash title="Show image properties" theme={null}
        openstack image show ubuntu-24.04-lts -c properties
        ```

        <Check>All configured properties appear in the output.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Hardware Configuration Examples

<AccordionGroup>
  <Accordion title="Modern Linux with UEFI boot" icon="settings">
    ```bash title="Ubuntu 24.04 / RHEL 9 UEFI configuration" theme={null}
    openstack image set \
      --property hw_firmware_type=uefi \
      --property hw_machine_type=q35 \
      --property hw_disk_bus=virtio \
      --property hw_vif_model=virtio \
      --property os_type=linux \
      ubuntu-24.04-lts
    ```
  </Accordion>

  <Accordion title="Windows Server with VirtIO drivers" icon="app-window">
    ```bash title="Windows Server 2022 configuration" theme={null}
    openstack image set \
      --property hw_firmware_type=uefi \
      --property hw_machine_type=q35 \
      --property hw_disk_bus=virtio \
      --property hw_vif_model=virtio \
      --property os_type=windows \
      --property os_distro=windows \
      --property os_version=2022 \
      windows-server-2022
    ```

    <Tip>
      Windows instances require VirtIO drivers installed in the guest before setting
      `hw_disk_bus=virtio`. Without drivers, Windows will fail to boot from a VirtIO disk.
    </Tip>
  </Accordion>

  <Accordion title="Legacy BIOS compatibility" icon="microchip">
    ```bash title="Legacy BIOS configuration for older OS images" theme={null}
    openstack image set \
      --property hw_firmware_type=bios \
      --property hw_machine_type=i440fx \
      --property hw_disk_bus=ide \
      legacy-centos7
    ```
  </Accordion>
</AccordionGroup>

***

## Image Metadata & Tags

Beyond hardware properties, images support two additional metadata mechanisms: **arbitrary key-value properties** for any custom annotation, and **tags** — simple string labels for filtering and searching across the image catalog.

### Arbitrary Properties

Any `--property key=value` pair is valid. Properties not recognized by the hypervisor are stored and returned but have no behavioral effect. Use them for operational metadata:

```bash title="Set operational metadata properties" theme={null}
openstack image set \
  --property os_lifecycle=lts \
  --property os_support_until=2029-04-30 \
  --property xloud_base_image=true \
  --property os_security_patch_date=2026-03-01 \
  --property data_classification=public \
  --property compliance=pci-dss \
  ubuntu-24.04-lts
```

| Custom Property Key      | Example Value                  | Purpose                            |
| ------------------------ | ------------------------------ | ---------------------------------- |
| `os_lifecycle`           | `lts`, `standard`, `eol`       | OS support status                  |
| `os_support_until`       | `2029-04-30`                   | End-of-support date for filtering  |
| `xloud_base_image`       | `true`                         | Marks Xloud-maintained base images |
| `os_security_patch_date` | `2026-03-01`                   | Last security patch applied        |
| `data_classification`    | `public`, `confidential`       | Data handling tag                  |
| `compliance`             | `pci-dss`, `hipaa`, `iso27001` | Regulatory compliance designation  |
| `build_version`          | `20260301-001`                 | CI/CD build identifier             |
| `min_kernel_version`     | `5.15`                         | Minimum kernel required by the OS  |

### Image Tags

Tags are plain string labels — no key, just a value. They enable fast catalog filtering in the Dashboard and CLI without requiring exact property key lookups.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Edit the image" icon="tag">
        Navigate to **Compute > Images**, click **Edit** (the first row action).
      </Step>

      <Step title="Add tags" icon="plus">
        In the **Edit Image** dialog, locate the **Tags** field. Type a tag and press **Enter** to add it. Add multiple tags.

        Useful tags: `ubuntu`, `lts`, `production-ready`, `gpu-compatible`, `windows`, `pci-dss`, `baseline-2026`, `no-cloud-init`
      </Step>

      <Step title="Save" icon="circle-check">
        Click **Confirm**. Tags are immediately searchable in the image list.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Add tags to an image" theme={null}
    openstack image set \
      --tag ubuntu \
      --tag lts \
      --tag production-ready \
      ubuntu-24.04-lts
    ```

    ```bash title="Remove a tag" theme={null}
    openstack image unset \
      --tag production-ready \
      ubuntu-24.04-lts
    ```

    ```bash title="List all images with a specific tag" theme={null}
    openstack image list --tag lts
    ```

    ```bash title="Filter by multiple tags" theme={null}
    openstack image list --tag lts --tag production-ready
    ```

    ```bash title="Show all tags on an image" theme={null}
    openstack image show ubuntu-24.04-lts -c tags
    ```
  </Tab>
</Tabs>

### Tags vs Properties

| Feature              | Tags                                | Properties                                  |
| -------------------- | ----------------------------------- | ------------------------------------------- |
| **Format**           | Simple strings (no key)             | Key-value pairs                             |
| **Use for**          | Catalog filtering, search, grouping | Hypervisor hints, operational metadata      |
| **Filtering**        | `openstack image list --tag <tag>`  | `openstack image list --property key=value` |
| **Hypervisor-aware** | Never                               | Some keys are (`hw_*`, `os_*`)              |

***

## Remove Properties

```bash title="Remove a specific property" theme={null}
openstack image unset \
  --property hw_tpm_model \
  ubuntu-24.04-lts
```

***

## Next Steps

<CardGroup>
  <Card title="Image Formats" href="/services/images/image-formats">
    Choose the right disk format to match your property configuration.
  </Card>

  <Card title="Upload an Image" href="/services/images/upload-image">
    Upload images and set properties at upload time using CLI flags.
  </Card>

  <Card title="Image Admin Guide — Metadata" href="/services/images/metadata">
    Define structured metadata namespaces for consistent property schemas.
  </Card>

  <Card title="Troubleshooting" href="/services/images/troubleshooting">
    Resolve launch failures caused by incorrect image property values.
  </Card>
</CardGroup>


# Image Requirements
Source: https://docs.xloud.tech/services/images/image-requirements

Requirements and compatibility checklist for virtual machine images to work correctly with Xloud Compute.

## Overview

Xloud Compute relies on image-side configuration to inject SSH keys, assign hostnames,
configure network interfaces, and deliver user data at instance launch. Images that do
not meet these requirements will fail to boot correctly, be unreachable over SSH, or
require manual reconfiguration after launch.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Tip>
  Test compatibility using a CirrOS image first. CirrOS meets all requirements and
  launches in seconds, making it ideal for validating your environment before importing
  production images.
</Tip>

***

## Required Components

| Component                     | Purpose                                                          | Notes                                                            |
| ----------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------- |
| **cloud-init**                | Injects SSH keys, sets hostname, runs user-data scripts          | Version 22.4 or later recommended                                |
| **SSH server**                | Enables key-based remote access                                  | `openssh-server`; password auth should be disabled               |
| **VirtIO drivers**            | Paravirtualized disk and network drivers for optimal performance | Built into Linux 2.6.25+; Windows requires separate installation |
| **Serial console**            | Enables emergency console access via the Dashboard               | Kernel parameter `console=ttyS0`                                 |
| **DHCP client**               | Assigns IP address on first boot                                 | Must use DHCP — static IPs break multi-project networking        |
| **Growroot / cloud-growpart** | Expands root partition to fill allocated disk on launch          | Required when flavor disk is larger than image base size         |

<Warning>
  Images missing cloud-init will not receive SSH keys at launch. The instance boots
  but remains inaccessible unless a password was baked into the image.
</Warning>

***

## Cloud-Init Configuration

Cloud-init runs during the first boot of every new instance and handles:

* SSH public key injection from the keypair selected at launch
* Hostname assignment and `/etc/hosts` update
* User-data script execution (shell scripts, Ansible, etc.)
* Network interface configuration via DHCP

### Install cloud-init

<Tabs>
  <Tab title="Ubuntu / Debian" icon="server">
    <Steps>
      <Step title="Install cloud-init and cloud-utils" icon="download">
        ```bash title="Install on Ubuntu or Debian" theme={null}
        apt-get update && apt-get install -y cloud-init cloud-utils cloud-initramfs-growroot
        ```
      </Step>

      <Step title="Clean the cloud-init state" icon="trash-2">
        Remove any cached instance data before capturing the image:

        ```bash title="Reset cloud-init state" theme={null}
        cloud-init clean --logs --seed
        ```

        <Check>The instance data directory `/var/lib/cloud/` is cleared and ready for re-initialization.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CentOS / AlmaLinux / Rocky Linux" icon="server">
    <Steps>
      <Step title="Install cloud-init" icon="download">
        ```bash title="Install on RHEL-compatible distributions" theme={null}
        dnf install -y cloud-init cloud-utils-growpart gdisk
        ```
      </Step>

      <Step title="Enable cloud-init services" icon="zap">
        ```bash title="Enable required systemd services" theme={null}
        systemctl enable cloud-init-local cloud-init cloud-config cloud-final
        ```
      </Step>

      <Step title="Clean the cloud-init state" icon="trash-2">
        ```bash title="Reset cloud-init state" theme={null}
        cloud-init clean --logs --seed
        ```

        <Check>Cloud-init services enabled and state cleared.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## SSH Access

Xloud injects the selected keypair's public key into the instance's authorized keys file
via cloud-init. Ensure the image is configured correctly for key-based authentication.

<Steps>
  <Step title="Disable password authentication" icon="lock">
    Edit `/etc/ssh/sshd_config` and set:

    ```text title="/etc/ssh/sshd_config" theme={null}
    PasswordAuthentication no
    PubkeyAuthentication yes
    ```

    This prevents brute-force attacks on instances with public IP addresses.
  </Step>

  <Step title="Remove pre-seeded SSH host keys" icon="key">
    Host keys are regenerated on first boot by cloud-init. Remove existing keys so
    every instance gets unique host keys:

    ```bash title="Remove SSH host keys" theme={null}
    rm -f /etc/ssh/ssh_host_*
    ```
  </Step>

  <Step title="Verify cloud-init SSH module is enabled" icon="circle-check">
    Confirm `ssh` is listed in the `cc_ssh` cloud-init module:

    ```bash title="Check cloud-init configuration" theme={null}
    grep -A 5 'cc_ssh' /etc/cloud/cloud.cfg
    ```

    <Check>SSH host keys are regenerated and the instance keypair is injected on first boot.</Check>
  </Step>
</Steps>

***

## Disk Partitioning

<AccordionGroup>
  <Accordion title="Use a single root partition" icon="hard-drive">
    Xloud Compute resizes the root disk to the flavor's allocated size at launch. A single
    root partition (`/`) is the simplest layout and works reliably with cloud-growpart.

    Avoid:

    * LVM (Logical Volume Manager) — growpart does not automatically extend LVM logical volumes
    * Separate `/boot` partitions unless required by the OS
    * Swap partitions — use swap files instead, which cloud-init can create dynamically
  </Accordion>

  <Accordion title="GPT vs MBR partition table" icon="table">
    Both GPT and MBR partition tables are supported. GPT is recommended for disks over
    2 TB. When using GPT, ensure a BIOS boot partition is present for GRUB compatibility
    with non-UEFI instances.
  </Accordion>

  <Accordion title="Filesystem recommendations" icon="folder">
    | Filesystem | Recommendation                                                         |
    | ---------- | ---------------------------------------------------------------------- |
    | `ext4`     | Recommended for all Linux images — reliable growroot support           |
    | `xfs`      | Supported — requires `xfs_growfs` (included in `cloud-utils-growpart`) |
    | `btrfs`    | Not recommended — growroot support varies across distributions         |
    | `ntfs`     | Required for Windows — use Cloudbase-Init for Windows images           |
  </Accordion>
</AccordionGroup>

***

## Network Configuration

<Note>
  Images must be configured to obtain IP addresses via DHCP. Xloud assigns instance IP
  addresses through its DHCP service. Static IP configuration in the image causes network
  conflicts and prevents proper instance addressing.
</Note>

<Steps>
  <Step title="Configure DHCP on all interfaces" icon="network">
    Ensure the primary network interface is set to DHCP. For cloud-init-managed images,
    `cloud-init` handles network configuration automatically via the metadata service.

    Remove any persistent interface rules that bind MAC addresses to interface names:

    ```bash title="Remove persistent network rules (Debian/Ubuntu)" theme={null}
    rm -f /etc/udev/rules.d/70-persistent-net.rules
    ```
  </Step>

  <Step title="Disable network manager caching" icon="wifi">
    On images using NetworkManager, clear cached connection profiles:

    ```bash title="Remove NetworkManager connection cache" theme={null}
    rm -f /etc/NetworkManager/system-connections/*
    ```
  </Step>
</Steps>

***

## Console Logging

Serial console output enables troubleshooting when SSH is unavailable. Enable it by
adding the serial console to the kernel command line.

<Tabs>
  <Tab title="GRUB 2 (most Linux distributions)" icon="terminal">
    <Steps>
      <Step title="Edit GRUB default configuration" icon="edit">
        Edit `/etc/default/grub` and add the serial console parameters:

        ```text title="/etc/default/grub" theme={null}
        GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
        ```
      </Step>

      <Step title="Regenerate GRUB configuration" icon="refresh-cw">
        ```bash title="Regenerate GRUB (Debian/Ubuntu)" theme={null}
        update-grub
        ```

        ```bash title="Regenerate GRUB (RHEL-compatible)" theme={null}
        grub2-mkconfig -o /boot/grub2/grub.cfg
        ```

        <Check>Console output streams to both the local display and the serial port.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Validation

Verify your image meets all requirements before uploading to Xloud.

<AccordionGroup>
  <Accordion title="Pre-upload checklist" icon="list-checks">
    Run this checklist on the image before capturing or uploading:

    * [ ] `cloud-init --version` returns version 22.4 or later
    * [ ] `cloud-init clean` has been run to clear cached state
    * [ ] SSH host keys removed from `/etc/ssh/ssh_host_*`
    * [ ] `PasswordAuthentication no` set in sshd\_config
    * [ ] Root partition uses `ext4` or `xfs`
    * [ ] Serial console enabled in GRUB
    * [ ] Network interface configured for DHCP
    * [ ] No static IP addresses in network configuration files
    * [ ] Persistent network interface rules removed
  </Accordion>

  <Accordion title="Post-launch validation" icon="circle-check">
    After launching an instance from the image, verify:

    ```bash title="Check cloud-init completed successfully" theme={null}
    cloud-init status --long
    ```

    Expected output:

    ```text title="Expected result" theme={null}
    status: done
    ```

    Also confirm SSH key injection:

    ```bash title="Check injected SSH keys" theme={null}
    cat ~/.ssh/authorized_keys
    ```
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Get Cloud Images" href="/services/images/get-images">
    Download pre-built images for Ubuntu, AlmaLinux, Rocky Linux, and more.
  </Card>

  <Card title="Modify Images" href="/services/images/modify-images">
    Customize existing images using virt-customize and guestfish.
  </Card>

  <Card title="Convert Formats" href="/services/images/convert-formats">
    Convert VMDK, VHD, or RAW images to the recommended QCOW2 format.
  </Card>

  <Card title="Upload an Image" href="/services/images/upload-image">
    Upload a compatible image to the Xloud Image Service.
  </Card>
</CardGroup>


# Image Service Security
Source: https://docs.xloud.tech/services/images/image-security

Configure image signing, property protections, and public image access controls for Xloud Image Service.

## Overview

The Xloud Image Service security configuration covers three areas: image signature
verification (preventing tampered images from being launched), property protections
(preventing unauthorized modification of critical metadata fields), and public image
access controls (restricting who can publish images to the global catalog).

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Image Signing and Verification

Xloud Image Service supports image signature verification using certificates stored in
Xloud Key Management. When enabled, the compute service verifies the image signature
before launching an instance, preventing tampered images from being used.

<Steps>
  <Step title="Generate a signing key and certificate" icon="key">
    Store the certificate in Xloud Key Management:

    ```bash title="Create a certificate container in Key Management" theme={null}
    openstack secret store \
      --name image-signing-cert \
      --payload-content-type "application/pkix-cert" \
      --payload "$(cat signing-cert.pem | base64)"
    ```

    Note the `Secret href` — this is the `<CERT_UUID>`.
  </Step>

  <Step title="Upload image with signature" icon="shield-check">
    ```bash title="Upload image with signature metadata" theme={null}
    openstack image create \
      --disk-format qcow2 \
      --container-format bare \
      --file ubuntu-24.04.qcow2 \
      --property img_signature="<BASE64_SIGNATURE>" \
      --property img_signature_certificate_uuid="<CERT_UUID>" \
      --property img_signature_hash_method="SHA-256" \
      --property img_signature_key_type="RSA-PSS" \
      ubuntu-24.04-signed
    ```
  </Step>

  <Step title="Enable verification in Compute" icon="settings">
    Configure the compute service to verify image signatures before launching:

    ```yaml title="Compute service: enable image signature verification" theme={null}
    nova_verify_glance_signatures: "true"
    ```

    Deploy after configuring:

    ```bash title="Apply compute configuration" theme={null}
    xavs-ansible deploy --tags nova
    ```

    <Check>
      The compute service now rejects instances launched from images with invalid or
      missing signatures when verification is enforced.
    </Check>
  </Step>
</Steps>

<Tip>
  Enable signature verification enforcement via the compute service policy to ensure
  only signed images from your approved certificate authority can be launched.
</Tip>

***

## Property Protections

Property protections prevent unauthorized users from modifying sensitive image
properties — such as signature fields or hardware requirements — after upload.

<Steps>
  <Step title="Create the property protections configuration" icon="file-code">
    ```ini title="/etc/xavs/glance/property-protections.conf" theme={null}
    [x-image-meta-property-img_signature]
    create = admin
    read = @
    update = admin
    delete = admin

    [x-image-meta-property-xloud_base_image]
    create = admin
    read = @
    update = admin
    delete = admin

    [x-image-meta-property-hw_firmware_type]
    create = @
    read = @
    update = admin
    delete = admin
    ```
  </Step>

  <Step title="Apply the configuration" icon="upload">
    ```bash title="Redeploy glance configuration" theme={null}
    xavs-ansible deploy --tags glance
    ```

    <Check>Property protections are active. Non-admin users cannot modify protected properties.</Check>
  </Step>
</Steps>

***

## Public Image Access Controls

Only users with the `admin` role can mark images as `public`. Enforce this via policy to prevent
accidental or malicious exposure of proprietary images organization-wide.

Verify the policy is active:

```bash title="Check publicize_image policy" theme={null}
openstack registered limit list | grep publicize
```

If the policy needs tightening, add an override:

```yaml title="/etc/xavs/glance/policy.yaml — restrict public image creation" theme={null}
"publicize_image": "role:admin"
"deactivate_image": "role:admin"
"reactivate_image": "role:admin"
```

Apply:

```bash title="Apply policy override" theme={null}
xavs-ansible deploy --tags glance
```

***

## Security Checklist

<AccordionGroup>
  <Accordion title="Signature verification enabled" icon="shield-check">
    Verify that image signature verification is enforced in the compute service
    policy. Test by attempting to launch an unsigned image — it should be rejected.
  </Accordion>

  <Accordion title="Property protections configured" icon="lock">
    Confirm that signature-related properties (`img_signature*`) and platform properties
    (`xloud_base_image`, `hw_firmware_type`) require admin to modify.
  </Accordion>

  <Accordion title="Public image access restricted to admins" icon="eye">
    Verify that non-admin users cannot set images to `public` visibility.
    Test with a project-member account: `openstack image set --public <image-id>`
    should return a policy violation error.
  </Accordion>

  <Accordion title="Audit public and community images" icon="search">
    Regularly audit the public image catalog:

    ```bash title="List all public images" theme={null}
    openstack image list --public --all-projects \
      -c name -c owner -c status -c updated_at
    ```

    Remove or deactivate any images that should not be publicly accessible.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Admin Troubleshooting" href="/services/images/admin-troubleshooting">
    Diagnose signature verification failures and policy enforcement issues.
  </Card>

  <Card title="Quotas" href="/services/images/quotas">
    Combine security controls with quota enforcement for complete image governance.
  </Card>

  <Card title="Identity Admin Guide" href="/services/identity/admin-guide">
    Manage the authentication policies governing image service access.
  </Card>

  <Card title="Metadata" href="/services/images/metadata">
    Define structured property schemas that work with property protection rules.
  </Card>
</CardGroup>


# Image Templates and Catalogs
Source: https://docs.xloud.tech/services/images/image-templates

Use Xloud images as VM templates, manage image catalogs, share community images, export images, and track versions across projects.

## Overview

An image in Xloud serves as more than a boot disk — it is the authoritative template for instance provisioning. A well-managed image catalog ensures consistent deployments, enforces approved operating system versions, and supports multi-project sharing with access control. This guide covers using images as templates, organizing a catalog, publishing community images, exporting images for external use, and managing versions.

***

## Using Images as Templates

Every instance launch derives from an image. Treating images as immutable, versioned templates — rather than ad-hoc uploads — produces consistent, repeatable deployments.

**Template design principles:**

* One image per OS version and configuration profile (e.g., `ubuntu-24.04-base`, `ubuntu-24.04-nginx`)
* Images are never modified after publishing — create a new version instead
* Customization at launch time uses cloud-init user data, not image-level changes
* Golden images include security hardening, required packages, and licensing pre-applied

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    <Steps>
      <Step title="Upload or build the template image" icon="upload">
        Navigate to **Compute > Images** and click **Create Image**.

        Set **Visibility** to `Private` while the image is being validated, then promote to `Shared` or `Community` when ready.
      </Step>

      <Step title="Add template metadata" icon="tag">
        After uploading, click the **More** dropdown on the image row (admin view) and select **Manage Metadata**. Use standard
        image properties to communicate template purpose:

        | Property       | Example Value                         |
        | -------------- | ------------------------------------- |
        | `os_distro`    | `ubuntu`                              |
        | `os_version`   | `24.04`                               |
        | `hw_disk_bus`  | `virtio`                              |
        | `hw_vif_model` | `virtio`                              |
        | `description`  | `Ubuntu 24.04 LTS base template v1.2` |
      </Step>

      <Step title="Launch from the template" icon="play">
        When creating an instance, select the image as the **Start Source** in Step 1 (Base Config). The instance
        inherits all hardware properties defined in the image metadata.

        <Check>Template in use — instances launched from this image receive consistent hardware configuration.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Source your credentials file to authenticate with the Xloud platform:

    ```bash title="Load credentials" theme={null}
    source openrc.sh
    ```

    <Tip>
      Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
    </Tip>

    ```bash title="Upload a template image" theme={null}
    openstack image create \
      --disk-format qcow2 \
      --container-format bare \
      --file ubuntu-24.04-server-cloudimg-amd64.img \
      --property os_distro=ubuntu \
      --property os_version=24.04 \
      --property hw_disk_bus=virtio \
      --property hw_vif_model=virtio \
      --property description="Ubuntu 24.04 LTS base template v1.0" \
      --visibility private \
      ubuntu-24.04-base-v1.0
    ```

    ```bash title="Launch an instance from the template" theme={null}
    openstack server create \
      --image ubuntu-24.04-base-v1.0 \
      --flavor m1.medium \
      --network internal-net \
      my-vm-01
    ```
  </Tab>
</Tabs>

***

## Image Catalog Management

A catalog is the set of approved, maintained images available to projects. Administrators control what appears in the catalog by managing image visibility and metadata.

### Visibility Levels

| Visibility  | Who Can See                            | Use Case                                          |
| ----------- | -------------------------------------- | ------------------------------------------------- |
| `private`   | Owner project only                     | Development, staging images not ready for sharing |
| `shared`    | Owner + explicitly shared projects     | Controlled sharing with specific teams            |
| `community` | All projects (read-only for non-admin) | Approved OS catalog available platform-wide       |
| `public`    | All projects (admin-controlled)        | Platform-wide golden images                       |

<Warning>
  Only users with the `admin` role can set image visibility to `public` or `community`. You can set your own images to `shared` and add project members to the sharing list.
</Warning>

### Promote an Image to the Catalog

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    Navigate to **Compute > Images** (admin view), find the image, click **Edit**, and set **Visibility** to `Community` or `Public`.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Promote image to community visibility" theme={null}
    openstack image set \
      --community \
      ubuntu-24.04-base-v1.0
    ```

    ```bash title="List all community images" theme={null}
    openstack image list \
      --community \
      --long
    ```
  </Tab>
</Tabs>

***

## Community Images

Community images are images made available to all projects without requiring explicit sharing grants. They are managed by administrators and appear in every project's image list.

```bash title="Create a community image" theme={null}
openstack image create \
  --disk-format qcow2 \
  --container-format bare \
  --file rocky-linux-9.qcow2 \
  --property os_distro=rocky \
  --property os_version=9 \
  --community \
  rocky-linux-9-v1.0
```

```bash title="Decommission a community image (hide from catalog)" theme={null}
openstack image set \
  --private \
  rocky-linux-9-v1.0
```

<Note>
  Setting an image back to `private` does not delete running instances that were launched from it. Only future launches are affected.
</Note>

***

## Image Export

Images can be exported for use outside the platform — for migration to another environment, backup archiving, or external distribution.

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    Navigate to **Compute > Images** (admin view), select the image, and use the CLI to download the image (`openstack image save`). The image file is downloaded in its native format (QCOW2 or RAW depending on upload format).
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Download an image to local disk" theme={null}
    openstack image save \
      --file ubuntu-24.04-export.qcow2 \
      ubuntu-24.04-base-v1.0
    ```

    ```bash title="Export and convert format (requires qemu-img)" theme={null}
    openstack image save --file /tmp/image.raw ubuntu-24.04-base-v1.0
    qemu-img convert -f raw -O qcow2 /tmp/image.raw ubuntu-24.04-export.qcow2
    ```

    ```bash title="Verify exported image integrity" theme={null}
    qemu-img check ubuntu-24.04-export.qcow2
    ```
  </Tab>
</Tabs>

For large images, use the Glance image import deactivate/reactivate API to pause an image before export to ensure consistency:

```bash title="Deactivate image before export (prevents modification)" theme={null}
openstack image deactivate <image-id>
openstack image save --file large-image.qcow2 <image-id>
openstack image reactivate <image-id>
```

***

## Image Versioning

Xloud Image Service does not enforce versioning natively — version tracking is implemented through naming conventions and image properties. The recommended approach:

**Naming convention:** `<os>-<version>-<profile>-v<major>.<minor>`

Examples:

* `ubuntu-24.04-base-v1.0`
* `ubuntu-24.04-base-v1.1`
* `ubuntu-24.04-nginx-v2.0`

**Deprecation workflow:**

```bash title="Mark old version as deprecated" theme={null}
openstack image set \
  --deactivated \
  --property deprecated=true \
  --property replacement=ubuntu-24.04-base-v1.1 \
  ubuntu-24.04-base-v1.0
```

```bash title="Hide deprecated image from project catalog" theme={null}
openstack image set \
  --private \
  ubuntu-24.04-base-v1.0
```

<Tip>
  Keep deprecated images in `private` visibility (not deleted) until you verify no instances are running from them. Check with: `openstack server list --all-projects --image <image-id>`
</Tip>

***

## Sharing Images Between Projects

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    Navigate to **Compute > Images**, click the image name, select **Manage Access**, and enter the target project ID.

    The target project administrator must accept the share from the **Shared Images** tab in **Compute > Images**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Share image with another project" theme={null}
    openstack image add project \
      ubuntu-24.04-base-v1.0 \
      <target-project-id>
    ```

    ```bash title="Target project accepts the shared image" theme={null}
    openstack image set \
      --accept \
      ubuntu-24.04-base-v1.0
    ```

    ```bash title="Verify shared image is visible in target project" theme={null}
    openstack image list --shared
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Storage Backends" href="/services/images/storage-backends">
    Configure RBD, Swift, S3, or file store as the image storage backend
  </Card>

  <Card title="Image Properties" href="/services/images/image-properties">
    Set hardware and scheduler properties on images to control instance behavior
  </Card>

  <Card title="Image Security" href="/services/images/image-security">
    Sign images and enforce signature verification at launch time
  </Card>

  <Card title="Upload Image" href="/services/images/upload-image">
    Upload custom images from ISO, QCOW2, RAW, or VMDK formats
  </Card>
</CardGroup>


# Image Service
Source: https://docs.xloud.tech/services/images/index

Upload, manage, and share virtual machine images for Xloud Cloud Platform.

<p>Overview</p>

The Xloud Image Service is the centralized catalog for virtual machine images and instance
snapshots. Every new instance launched on the platform boots from an image registered here.
Images can be public (shared across all projects), private (scoped to your project), or
community-shared with specific projects — giving you full control over your image library.

<Note>
  **Prerequisites**

  * An active Xloud account with project-member privileges or higher
  * Access to the **Xloud Dashboard** or `openstack` CLI
  * For uploading images: image data in a supported format (QCOW2, RAW, VHD, or VMDK)
</Note>

***

<p>What the Image Service Provides</p>

<CardGroup>
  <Card title="Image Catalog" icon="layers" href="/services/images/get-images">
    Centralized storage and discovery of OS images, application images, and snapshots
    available across your Xloud environment.
  </Card>

  <Card title="Multiple Formats" icon="file-code" href="/services/images/image-formats">
    Native support for QCOW2, RAW, VHD, and VMDK formats with automatic metadata
    detection and validation on upload.
  </Card>

  <Card title="Visibility Controls" icon="eye" href="/services/images/share-images">
    Per-image visibility settings — public, private, shared, or community — controlling
    who can discover and use each image.
  </Card>

  <Card title="Instance Snapshots" icon="camera" href="/services/images/create-snapshot">
    Capture a running or stopped instance as an image snapshot. Use snapshots for
    backups, cloning, and golden image workflows.
  </Card>

  <Card title="Image Sharing" icon="share-2" href="/services/images/share-images">
    Share specific images with other projects without making them globally public,
    enabling controlled cross-team image distribution.
  </Card>

  <Card title="Rich Metadata" icon="tag" href="/services/images/image-properties">
    Attach structured properties to images — OS type, version, minimum disk, hardware
    requirements — enabling the scheduler to make informed placement decisions.
  </Card>
</CardGroup>

***

<p>Image Lifecycle</p>

```mermaid theme={null}
graph LR
    A([Upload / Create]) --> B[queued]
    B --> C[saving]
    C --> D[active]
    D --> E([Create Instance])
    D --> F([Create Snapshot])
    F --> D
    D --> G[deactivated]
    G --> D
    D --> H([Delete])
    style D fill:#197560,color:#fff
    style A fill:#3F8F7E,color:#fff
    style E fill:#3F8F7E,color:#fff
```

| State         | Description                                                                       |
| ------------- | --------------------------------------------------------------------------------- |
| `queued`      | Image record created; data upload not yet started                                 |
| `saving`      | Image data is being uploaded and stored                                           |
| `active`      | Image is ready and available for instance launches                                |
| `deactivated` | Image has been disabled by an administrator; cannot be used for new instances     |
| `deleted`     | Image has been removed; the record is retained temporarily before permanent purge |

***

<p>Supported Formats</p>

| Format         | Extension       | Best For                                                                                            |
| -------------- | --------------- | --------------------------------------------------------------------------------------------------- |
| **QCOW2**      | `.qcow2`        | Default format. Supports copy-on-write, compression, and snapshots. Recommended for most workloads. |
| **RAW**        | `.img`          | Maximum performance. No overhead. Preferred when storage backend handles copy-on-write natively.    |
| **VHD / VHDX** | `.vhd`, `.vhdx` | Images exported from Hyper-V environments.                                                          |
| **VMDK**       | `.vmdk`         | Images exported from VMware environments.                                                           |

***

<p>Guides</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/images/user-guide">
    Upload images, manage snapshots, configure visibility, and share images with other
    projects from the Dashboard and CLI.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/images/admin-guide">
    Configure storage backends, metadata schemas, image signing, caching, and quotas
    for production image service deployments.
  </Card>

  <Card title="Compute User Guide" icon="monitor" href="/services/compute/user-guide">
    Learn how images are used when launching compute instances and creating snapshots.
  </Card>

  <Card title="Authentication & CLI" icon="terminal" href="/services/images/cli-reference">
    Configure CLI credentials to manage images from the command line.
  </Card>
</CardGroup>


# Metadata Definitions
Source: https://docs.xloud.tech/services/images/metadata

Define structured metadata namespaces for consistent, validated image property schemas across your Xloud image catalog.

## Overview

Metadata namespaces define structured, validated property schemas for images. When a
namespace is defined, the Dashboard displays a guided property editor instead of a raw
key-value form — ensuring consistent metadata across your image catalog. Namespaces
can be public (visible to all projects) or private (visible only to the creating project).

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Concepts

| Concept           | Description                                                                    |
| ----------------- | ------------------------------------------------------------------------------ |
| **Namespace**     | A named collection of property definitions for a resource type                 |
| **Property**      | A typed key-value field with validation rules (enum, string, boolean, integer) |
| **Object**        | A named group of properties within a namespace                                 |
| **Tag**           | A simple string label associated with images — for filtering and grouping      |
| **Resource type** | The Xloud resource the namespace applies to — `OS::Image::Image` for images    |

***

## Image Tags

Tags are the simplest form of image metadata — plain strings with no key, attached directly to an image. Use them for fast catalog filtering, grouping by OS family, compliance designation, or release milestone.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Images** and click **Edit** (the first row action) on the image.

    In the **Tags** field, type a tag and press **Enter** to add it. Add as many as needed, then click **Confirm**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Add tags to an image" theme={null}
    openstack image set \
      --tag ubuntu \
      --tag lts \
      --tag production-ready \
      ubuntu-24.04-lts
    ```

    ```bash title="Filter images by tag" theme={null}
    openstack image list --tag lts
    ```

    ```bash title="Filter by multiple tags (AND logic)" theme={null}
    openstack image list --tag lts --tag production-ready
    ```

    ```bash title="Remove a specific tag" theme={null}
    openstack image unset --tag production-ready ubuntu-24.04-lts
    ```
  </Tab>
</Tabs>

### Recommended Tag Taxonomy

Use a consistent tagging convention across your image catalog:

| Category           | Example Tags                                          |
| ------------------ | ----------------------------------------------------- |
| **OS family**      | `ubuntu`, `rhel`, `debian`, `windows`, `centos`       |
| **Support tier**   | `lts`, `standard`, `eol`, `preview`                   |
| **Status**         | `production-ready`, `deprecated`, `testing`, `golden` |
| **Compliance**     | `pci-dss`, `hipaa`, `iso27001`, `fips`                |
| **Architecture**   | `x86_64`, `aarch64`                                   |
| **Special config** | `no-cloud-init`, `virtio`, `uefi`, `gpu-compatible`   |

***

## Create a Metadata Namespace

<Steps>
  <Step title="Define the namespace schema" icon="file-code">
    Create a JSON file defining the namespace properties:

    ```json title="namespace-schema.json" theme={null}
    {
      "namespace": "org.xloud.images",
      "display_name": "Xloud Image Properties",
      "description": "Standardized OS and hardware metadata for Xloud images",
      "visibility": "public",
      "protected": true,
      "resource_type_associations": [
        {
          "name": "OS::Image::Image",
          "prefix": ""
        }
      ],
      "properties": {
        "os_lifecycle": {
          "title": "OS Lifecycle",
          "description": "Support lifecycle status of the OS",
          "type": "string",
          "enum": ["lts", "standard", "eol"],
          "operators": ["<or>"]
        },
        "xloud_base_image": {
          "title": "Xloud Base Image",
          "description": "Whether this is an Xloud-maintained base image",
          "type": "boolean"
        },
        "os_security_patch_date": {
          "title": "Last Security Patch Date",
          "description": "Date of the most recent OS security patch applied",
          "type": "string",
          "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"
        }
      }
    }
    ```
  </Step>

  <Step title="Register the namespace" icon="plus">
    ```bash title="Create metadata namespace" theme={null}
    openstack metric namespace create \
      --schema namespace-schema.json \
      --resource-type OS::Image::Image \
      org.xloud.images
    ```
  </Step>

  <Step title="Verify namespace is active" icon="circle-check">
    ```bash title="List registered namespaces" theme={null}
    openstack metric namespace list
    ```

    <Check>The `org.xloud.images` namespace appears in the list.</Check>
  </Step>
</Steps>

***

## Use Namespace Properties

Once a namespace is registered, its properties appear in the Dashboard image editor
as a structured form. Set them via CLI:

```bash title="Set namespace-defined properties on an image" theme={null}
openstack image set \
  --property os_lifecycle=lts \
  --property xloud_base_image=true \
  --property os_security_patch_date=2026-03-15 \
  ubuntu-24.04-lts
```

***

## Manage Namespaces

<CodeGroup>
  ```bash title="List all namespaces" theme={null}
  openstack metric namespace list
  ```

  ```bash title="Show namespace details" theme={null}
  openstack metric namespace show org.xloud.images
  ```

  ```bash title="Delete a namespace" theme={null}
  openstack metric namespace delete org.xloud.images
  ```
</CodeGroup>

<Warning>
  Deleting a namespace removes the property schema definitions but does not remove
  existing property values from images. Images retain their property values even after
  the namespace is deleted.
</Warning>

***

## Property Type Reference

| Type      | Description                                          | Example               |
| --------- | ---------------------------------------------------- | --------------------- |
| `string`  | Free-text string (with optional `enum` or `pattern`) | `"lts"`, `"standard"` |
| `boolean` | True/false value                                     | `true`, `false`       |
| `integer` | Whole number                                         | `4`, `8096`           |
| `number`  | Floating-point number                                | `3.14`                |
| `array`   | List of values                                       | `["tag1", "tag2"]`    |

***

## Next Steps

<CardGroup>
  <Card title="Image Properties" href="/services/images/image-properties">
    Set properties on individual images using the defined namespace schemas.
  </Card>

  <Card title="Security" href="/services/images/image-security">
    Protect critical metadata properties from unauthorized modification.
  </Card>

  <Card title="Storage Backends" href="/services/images/storage-backends">
    Configure the backend that stores the image data alongside its metadata.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/images/admin-troubleshooting">
    Resolve metadata namespace and property validation issues.
  </Card>
</CardGroup>


# Modify Images
Source: https://docs.xloud.tech/services/images/modify-images

Customize existing cloud images offline using virt-customize, guestfish, and guestmount before uploading to Xloud.

## Overview

Modifying a cloud image offline — before uploading — allows you to install packages,
configure software, inject files, and set system parameters without needing to launch
an instance. This approach produces consistent, repeatable golden images for deployment.
All modifications are performed on a copy of the image using libguestfs tools.

<Note>
  **Prerequisites**

  * A Linux workstation or build server with `libguestfs-tools` installed
  * A base cloud image in QCOW2 or RAW format (see [Get Cloud Images](/services/images/get-images))
  * Root or `sudo` access on the build host
  * Sufficient disk space: at least 3x the image size for safe modification
</Note>

<Warning>
  Always work on a copy of the image, not the original. If a modification fails or
  produces an unusable image, the original remains intact.
</Warning>

***

## Tools Reference

| Tool               | Purpose                                        | Best For                                               |
| ------------------ | ---------------------------------------------- | ------------------------------------------------------ |
| **virt-customize** | Apply changes to an image non-interactively    | Installing packages, running scripts, injecting files  |
| **guestfish**      | Interactive filesystem shell for an image      | Manual file edits, inspecting content, complex changes |
| **guestmount**     | Mount an image as a local directory            | Interactive editing with standard filesystem tools     |
| **virt-sysprep**   | Reset and generalize an image for distribution | Removing machine-specific data before capturing        |

***

## Modify with virt-customize

`virt-customize` applies changes to an image from the command line without mounting it.
It is the fastest method for automatable, scripted modifications.

<Steps>
  <Step title="Install libguestfs-tools" icon="download">
    ```bash title="Install on Ubuntu / Debian" theme={null}
    apt-get update && apt-get install -y libguestfs-tools
    ```

    ```bash title="Install on CentOS / AlmaLinux / Rocky Linux" theme={null}
    dnf install -y libguestfs-tools
    ```
  </Step>

  <Step title="Copy the base image" icon="copy">
    ```bash title="Create a working copy of the image" theme={null}
    cp ubuntu-24.04-lts.qcow2 ubuntu-24.04-lts-custom.qcow2
    ```
  </Step>

  <Step title="Install packages" icon="package">
    Install software into the image using the native package manager:

    ```bash title="Install packages with virt-customize" theme={null}
    virt-customize \
      -a ubuntu-24.04-lts-custom.qcow2 \
      --install nginx,curl,htop \
      --update
    ```

    The `--update` flag runs a full package upgrade before installing the specified packages.
  </Step>

  <Step title="Inject an SSH key" icon="key">
    Pre-authorize an SSH public key for the default user:

    ```bash title="Inject SSH public key" theme={null}
    virt-customize \
      -a ubuntu-24.04-lts-custom.qcow2 \
      --ssh-inject ubuntu:file:/home/user/.ssh/id_ed25519.pub
    ```
  </Step>

  <Step title="Run a configuration script" icon="terminal">
    Execute a shell script inside the image to perform complex configuration:

    ```bash title="Run a script inside the image" theme={null}
    virt-customize \
      -a ubuntu-24.04-lts-custom.qcow2 \
      --run-command 'systemctl enable nginx' \
      --run-command 'echo "CUSTOM_IMAGE=true" >> /etc/environment'
    ```
  </Step>

  <Step title="Set the root password" icon="lock">
    <Warning>
      Setting a root password is only appropriate for testing or emergency access images.
      Production images should rely on SSH key injection via cloud-init.
    </Warning>

    ```bash title="Set a root password" theme={null}
    virt-customize \
      -a ubuntu-24.04-lts-custom.qcow2 \
      --root-password password:YourSecurePassword123
    ```
  </Step>
</Steps>

***

## Modify with guestfish

`guestfish` provides an interactive shell for directly editing files inside an image.
Use it for one-off edits, configuration file changes, or inspecting image contents.

<Steps>
  <Step title="Open the image in guestfish" icon="folder-open">
    ```bash title="Open image in interactive mode" theme={null}
    guestfish --rw -a ubuntu-24.04-lts-custom.qcow2
    ```

    At the `><fs>` prompt, run:

    ```text title="Mount the root filesystem" theme={null}
    ><fs> run
    ><fs> list-filesystems
    ><fs> mount /dev/sda1 /
    ```
  </Step>

  <Step title="Edit a configuration file" icon="edit">
    Use the `edit` command to open a file in your default editor:

    ```text title="Edit a file inside the image" theme={null}
    ><fs> edit /etc/cloud/cloud.cfg
    ```

    Or write content directly with `write`:

    ```text title="Write content to a file" theme={null}
    ><fs> write /etc/motd "Xloud Custom Image - $(date +%Y-%m)\n"
    ```
  </Step>

  <Step title="Copy files into the image" icon="file-input">
    Upload a local file into the image:

    ```text title="Copy local file into the image" theme={null}
    ><fs> upload /local/path/custom-config.conf /etc/app/custom-config.conf
    ```
  </Step>

  <Step title="Exit and close" icon="log-out">
    ```text title="Exit guestfish" theme={null}
    ><fs> umount /
    ><fs> exit
    ```

    <Check>Changes are written to the image file when guestfish exits.</Check>
  </Step>
</Steps>

***

## Mount with guestmount

`guestmount` mounts the image filesystem as a local directory, enabling standard
filesystem tools (`cp`, `vim`, `chmod`) to operate on image contents.

<Steps>
  <Step title="Create a mount point" icon="folder-plus">
    ```bash title="Create a temporary mount directory" theme={null}
    mkdir -p /mnt/image
    ```
  </Step>

  <Step title="Mount the image" icon="hard-drive">
    ```bash title="Mount the image read-write" theme={null}
    guestmount -a ubuntu-24.04-lts-custom.qcow2 -i --rw /mnt/image
    ```

    The `-i` flag auto-detects the root filesystem.
  </Step>

  <Step title="Make modifications" icon="edit">
    The image contents are accessible as a regular directory:

    ```bash title="Edit a file in the mounted image" theme={null}
    echo "net.ipv4.tcp_tw_reuse = 1" >> /mnt/image/etc/sysctl.d/99-custom.conf
    cp /local/app/configs/* /mnt/image/etc/app/
    chmod 640 /mnt/image/etc/app/*.conf
    ```
  </Step>

  <Step title="Unmount when complete" icon="eject">
    ```bash title="Unmount the image" theme={null}
    guestunmount /mnt/image
    ```

    <Check>All changes are written to the image file. Verify with `guestfish` if needed.</Check>
  </Step>
</Steps>

***

## Generalize with virt-sysprep

Before distributing or uploading an image as a golden template, use `virt-sysprep` to
remove machine-specific identifiers. This ensures each launched instance is treated as
a fresh system rather than a clone of the build host.

```bash title="Generalize the image for distribution" theme={null}
virt-sysprep \
  -a ubuntu-24.04-lts-custom.qcow2 \
  --operations defaults,-ssh-hostkeys \
  --selinux-relabel
```

`virt-sysprep` removes:

* Machine ID (`/etc/machine-id`)
* Shell history files
* Temporary files and package caches
* Log files
* NetworkManager connection caches

<Tip>
  Use `--operations defaults,-ssh-hostkeys` to keep cloud-init's SSH key regeneration
  intact. The `-ssh-hostkeys` exclusion prevents virt-sysprep from removing keys that
  cloud-init would regenerate anyway.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Image Requirements" href="/services/images/image-requirements">
    Verify your modified image meets all Xloud Compute compatibility requirements.
  </Card>

  <Card title="Convert Formats" href="/services/images/convert-formats">
    Convert VMDK, VHD, or RAW images to QCOW2 after modification.
  </Card>

  <Card title="Upload an Image" href="/services/images/upload-image">
    Upload the customized image to the Xloud Image Service.
  </Card>

  <Card title="Image Properties" href="/services/images/image-properties">
    Tag uploaded images with OS metadata and hardware requirements.
  </Card>
</CardGroup>


# Image Service Quotas
Source: https://docs.xloud.tech/services/images/quotas

Configure per-project image count and storage size limits to prevent catalog storage exhaustion in Xloud.

## Overview

The Xloud Image Service enforces per-project quotas on image count and total storage size.
Quotas prevent a single project from consuming the entire image catalog storage and ensure
equitable resource distribution across teams.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Quota Reference

| Quota Field             | Default   | Description                                                |
| ----------------------- | --------- | ---------------------------------------------------------- |
| `images`                | Unlimited | Maximum number of images (including snapshots) per project |
| `image_size_total`      | Unlimited | Maximum total image storage in bytes per project           |
| `image_staging_total`   | Unlimited | Maximum staging area usage per project                     |
| `image_count_total`     | Unlimited | Combined count of uploaded and staging images              |
| `image_count_uploading` | Unlimited | Maximum simultaneous uploads per project                   |

<Note>
  Default quotas are unlimited unless explicitly configured. Set quotas proactively
  for multi-project deployments to prevent runaway storage consumption.
</Note>

***

## View Image Quotas

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to the admin quota settings and review the Image section quota fields.
    Per-project overrides are visible in **Identity > Projects** (admin view) > Modify Quotas.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show current image quota for a project" theme={null}
    openstack quota show --project backend-prod
    ```

    ```bash title="Show quota usage for a project" theme={null}
    openstack quota show --usage --project backend-prod
    ```

    ```bash title="Count images in a project" theme={null}
    openstack image list \
      --project backend-prod \
      --format json | python3 -c \
      "import json,sys; imgs=json.load(sys.stdin); print(f'{len(imgs)} images')"
    ```
  </Tab>
</Tabs>

***

## Set Project Quotas

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to project quota settings" icon="settings">
        Navigate to **Identity > Projects** (admin view). Find the project and click
        **Modify Quotas**.
      </Step>

      <Step title="Set image quota values" icon="gauge">
        Update the **Images** field to the maximum allowed image count and optionally
        set **Image Storage** in bytes.

        | Field             | Example        | Description                     |
        | ----------------- | -------------- | ------------------------------- |
        | **Images**        | `100`          | Maximum image count per project |
        | **Image Storage** | `107374182400` | Maximum total bytes (100 GB)    |
      </Step>

      <Step title="Save quotas" icon="circle-check">
        Click **Save**. The quota takes effect immediately.
        <Check>Project quota is updated. New image uploads that exceed the limit will be rejected.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Set image count quota for a project" theme={null}
    openstack quota set --images 100 backend-prod
    ```

    ```bash title="View updated quotas" theme={null}
    openstack quota show backend-prod | grep image
    ```

    <Tip>
      Storage-based quotas (byte limits) are set at the Image Service configuration level,
      not via the standard quota API. Contact your platform administrator to set
      `user_storage_quota` in the Image Service configuration.
    </Tip>
  </Tab>
</Tabs>

***

## Monitor Quota Usage

Set up regular quota monitoring to catch projects approaching their limits:

```bash title="List all projects with image counts" theme={null}
openstack image list --all-projects \
  -c project_id \
  -f json | python3 -c "
import json, sys, collections
imgs = json.load(sys.stdin)
counts = collections.Counter(i['project_id'] for i in imgs)
for proj_id, count in sorted(counts.items(), key=lambda x: -x[1])[:10]:
    print(f'{count:4d}  {proj_id}')
"
```

***

## Next Steps

<CardGroup>
  <Card title="Security" href="/services/images/image-security">
    Control access to public image creation alongside quota enforcement.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/images/admin-troubleshooting">
    Resolve quota exceeded errors and investigate storage consumption.
  </Card>

  <Card title="Image Cache" href="/services/images/image-cache">
    Cache configuration also consumes storage — factor cache size into capacity planning.
  </Card>

  <Card title="Storage Backends" href="/services/images/storage-backends">
    Review backend capacity alongside quota limits.
  </Card>
</CardGroup>


# Share Images
Source: https://docs.xloud.tech/services/images/share-images

Share private images with specific projects in Xloud without making them globally public.

## Overview

Private images can be shared with specific projects without making them globally public.
This enables controlled cross-team image distribution — a platform team can maintain
golden images and share them selectively with application teams. The target project must
accept the share before the image becomes visible in their catalog.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Image Visibility Options

| Visibility  | Who Can See                       | Who Can Set | Use Case                                                 |
| ----------- | --------------------------------- | ----------- | -------------------------------------------------------- |
| `private`   | Owning project only               | Image owner | Default. Development and test images.                    |
| `shared`    | Owning project + accepted members | Image owner | Controlled cross-team sharing.                           |
| `community` | All projects (discoverable)       | Image owner | Broadly useful images without admin approval.            |
| `public`    | All projects (visible by default) | Admin only  | Platform-wide OS images maintained by the platform team. |

***

## Share an Image

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Set image visibility to Shared">
        The image must have **Shared** visibility before it can be shared with
        other projects. If the image is currently Private:

        * **During creation**: Select **Shared** in the Visibility radio and choose
          the target projects from the project table
        * **After creation** (admin only): Click **Edit** on the image row and check
          **Public**, or use the CLI to set visibility to `shared`
      </Step>

      <Step title="Manage Access (admin only)">
        In the admin view, navigate to **Compute > Images** (admin view). Click
        the **More** dropdown on the image row and select **Manage Access**.

        <Note>
          The **Manage Access** action only appears when the image has **Shared**
          visibility. It is available to administrators only.
        </Note>

        The dialog shows:

        | Field       | Type               | Description                     |
        | ----------- | ------------------ | ------------------------------- |
        | **Name**    | Read-only          | The image being shared          |
        | **Project** | Multi-select table | Select projects to grant access |

        Select one or more projects and click **Confirm**.
      </Step>

      <Step title="Verify in target project">
        Users in the target project can see the shared image under the
        **Shared Images** tab in **Compute > Images**.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Change visibility to shared" icon="share-2">
        ```bash title="Set image visibility to shared" theme={null}
        openstack image set --shared ubuntu-24.04-lts
        ```
      </Step>

      <Step title="Add the target project as a member" icon="user-plus">
        ```bash title="Share image with target project" theme={null}
        openstack image add project ubuntu-24.04-lts <TARGET_PROJECT_ID>
        ```
      </Step>

      <Step title="Verify member was added" icon="search">
        ```bash title="List current image members" theme={null}
        openstack image member list ubuntu-24.04-lts
        ```

        The target project appears with status `pending`.
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Accept a Shared Image (Target Project)

The target project administrator must accept the share before the image is usable.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Log in as a user in the target project. Navigate to **Compute > Images**.
    Find the shared image in the **Shared Images** tab. Shared images appear
    automatically — no explicit accept action is needed in the Dashboard.

    <Note>
      To explicitly accept a shared image via CLI, run:
      `openstack image set --accept <image-id>`
    </Note>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Source credentials for the target project, then accept the share:

    ```bash title="Accept shared image (run as target project)" theme={null}
    export OS_PROJECT_NAME=<target-project>
    source openrc.sh

    openstack image set --accept <IMAGE_ID>
    ```

    Verify the image is now accessible:

    ```bash title="Confirm image is visible" theme={null}
    openstack image show <IMAGE_ID> -c visibility -c status
    ```

    <Check>Image visibility shows `shared` and status shows `active` — it is ready to use.</Check>
  </Tab>
</Tabs>

***

## Reject or Remove a Share

<Tabs>
  <Tab title="Remove a member (image owner)" icon="gauge">
    ```bash title="Remove project from image members" theme={null}
    openstack image remove project ubuntu-24.04-lts <TARGET_PROJECT_ID>
    ```

    The image immediately disappears from the target project's catalog.
  </Tab>

  <Tab title="Reject share (target project)" icon="terminal">
    ```bash title="Reject a pending share (target project)" theme={null}
    openstack image set --reject <IMAGE_ID>
    ```
  </Tab>
</Tabs>

***

## Community Images

Community images are discoverable by all projects without explicit sharing, but are not
pushed into every project's default catalog. You must search for them explicitly.

```bash title="Set image visibility to community" theme={null}
openstack image set --community my-app-image
```

```bash title="List community images" theme={null}
openstack image list --community
```

<Note>
  Community images are visible to all authenticated users but do not appear in the
  default image list unless searched. They are useful for widely-useful base images
  that any team can discover and use without platform team intervention.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Image Properties" href="/services/images/image-properties">
    Add metadata to shared images to communicate hardware requirements and OS details.
  </Card>

  <Card title="Upload an Image" href="/services/images/upload-image">
    Upload new images to share with your organization.
  </Card>

  <Card title="Image Admin Guide — Quotas" href="/services/images/quotas">
    Configure per-project image count and storage quotas.
  </Card>

  <Card title="Troubleshooting" href="/services/images/troubleshooting">
    Resolve shared image visibility and acceptance issues.
  </Card>
</CardGroup>


# Image Storage Backends
Source: https://docs.xloud.tech/services/images/storage-backends

Configure Xloud Distributed Storage (RBD), file store, Swift, S3, VMware datastore, Cinder, or HTTP as the Xloud Image Service backend.

## Overview

The Xloud Image Service stores image data in a pluggable storage backend. The selection of backend affects availability, performance, and instance launch times. Multiple backends can be registered simultaneously — administrators select which backend stores each image at upload time. This guide covers configuration of all supported backends via XDeploy globals.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Supported Backends

| Backend                          | Use Case                                             | HA  | Notes                                        |
| -------------------------------- | ---------------------------------------------------- | --- | -------------------------------------------- |
| **Distributed Storage (RBD)**    | Production — fast clones, no single point of failure | Yes | Recommended for all multi-node deployments   |
| **File Store**                   | Single-node or development deployments               | No  | Default backend; not HA                      |
| **Xloud Object Storage (Swift)** | Scalable distributed image storage                   | Yes | Suitable when Swift is already deployed      |
| **S3-Compatible**                | AWS S3 or S3-compatible object stores                | Yes | Works with AWS, Ceph RGW, MinIO, Dell ECS    |
| **VMware Datastore**             | VMware vSphere integration environments              | Yes | Stores images directly in vSphere datastores |
| **Cinder (Block Storage)**       | Block storage as image store                         | Yes | Volumes used as image backing devices        |
| **HTTP (read-only)**             | Remote image sources without upload                  | N/A | Read-only; images fetched from external URLs |

***

## Xloud Distributed Storage (RBD) — Recommended

RBD (RADOS Block Device) is the recommended backend for high-availability multi-node deployments. Images are stored as RBD objects in the Xloud Distributed Storage cluster, providing built-in redundancy and copy-on-write clones for fast instance launches.

<Steps>
  <Step title="Configure RBD backend in XDeploy globals" icon="settings">
    ```yaml title="RBD storage backend configuration" theme={null}
    glance_backend_ceph: "yes"
    glance_backend_file: "no"
    ceph_glance_keyring: ceph.client.glance.keyring
    ceph_glance_pool_name: images
    ```
  </Step>

  <Step title="Deploy the configuration" icon="upload">
    ```bash title="Apply glance configuration" theme={null}
    xavs-ansible deploy --tags glance
    ```
  </Step>

  <Step title="Verify the images pool exists" icon="circle-check">
    ```bash title="Verify RBD pool" theme={null}
    ceph osd pool ls | grep images
    ```

    <Check>The `images` pool appears in the Ceph pool list.</Check>
  </Step>
</Steps>

<Tip>
  When using RBD as the image backend and RBD as the volume backend, instance launches
  perform a zero-copy RBD clone from the image pool into the volume pool — resulting in
  near-instantaneous boot times regardless of image size.
</Tip>

***

## File Store

The file store writes image data to a local or NFS-mounted directory. It is the default backend and suitable for single-node deployments.

```yaml title="File store configuration" theme={null}
glance_backend_file: "yes"
glance_file_datadir_volume: glance
```

The image directory is mounted as a Docker named volume (`glance`) accessible at `/var/lib/glance/images/` inside the Image API container.

To store images in a custom directory (e.g., on a dedicated volume):

```ini title="glance-api.conf — custom directory" theme={null}
[glance_store]
default_backend = file
filesystem_store_datadir = /mnt/glance_images/
```

Prepare the host path with the correct ownership:

```bash title="Prepare custom image directory" theme={null}
mkdir -p /mnt/glance_images/
chown 42424:42424 /mnt/glance_images/
```

Apply the configuration:

```bash title="Apply glance configuration" theme={null}
xavs-ansible reconfigure --tags glance
```

<Warning>
  The file store is not highly available. If the node running the Image API fails,
  images stored locally are unavailable until the node recovers. Use RBD for
  production HA deployments.
</Warning>

***

## Xloud Object Storage (Swift)

Images can be stored in Xloud Object Storage (Swift) containers. This backend is suitable when Object Storage is already deployed and disk-to-disk copy overhead is acceptable.

```yaml title="Swift storage backend configuration" theme={null}
glance_backend_swift: "yes"
glance_backend_file: "no"
glance_swift_store_container: glance
glance_swift_store_create_container_on_put: "true"
```

The full `glance-api.conf` parameters for Swift:

```ini title="glance-api.conf — Swift backend" theme={null}
[glance_store]
default_backend = swift
swift_store_auth_address = http://10.0.1.71:5000/v3
swift_store_user = service:glance
swift_store_key = <glance-service-password>
swift_store_container = glance
swift_store_create_container_on_put = true
swift_store_multi_tenant = false
```

***

## S3-Compatible Object Storage

The S3 backend driver supports any S3-compatible endpoint including AWS S3, Ceph RADOS Gateway, MinIO, and Dell ECS. This enables off-site or cloud-based image storage.

```ini title="glance-api.conf — S3 backend" theme={null}
[glance_store]
default_backend = s3
s3_store_host = https://s3.example.com
s3_store_access_key = <access-key>
s3_store_secret_key = <secret-key>
s3_store_bucket = glance-images
s3_store_create_bucket_on_put = true
s3_store_large_object_size = 100
s3_store_large_object_chunk_size = 10
```

Apply via XDeploy globals:

```yaml title="XDeploy globals — S3 backend" theme={null}
glance_backend_s3: "yes"
glance_backend_file: "no"
glance_s3_store_host: "https://s3.example.com"
glance_s3_store_access_key: "<access-key>"
glance_s3_store_secret_key: "<secret-key>"
glance_s3_store_bucket: "glance-images"
```

***

## VMware Datastore

The VMware datastore backend stores images in vSphere datastores. This is required when the compute layer uses the VMware vSphere driver and images must reside on vCenter-managed datastores.

```ini title="glance-api.conf — VMware datastore backend" theme={null}
[glance_store]
default_backend = vsphere
vmware_server_host = 10.0.10.20
vmware_server_username = administrator@vsphere.local
vmware_server_password = <password>
vmware_datastores = DatacenterName:DatastoreName:100
vmware_store_image_dir = /openstack_glance
vmware_api_insecure = false
vmware_task_poll_interval = 5
```

<Note>
  The `vmware_datastores` value follows the format `DatacenterName:DatastoreName:Weight`. Multiple datastores can be listed to distribute images across datastores.
</Note>

***

## Cinder (Block Storage as Image Store)

The Cinder backend stores images as Cinder volumes. Each image is backed by a block volume in the configured Cinder pool. This backend is useful when block storage is the primary shared storage medium.

```ini title="glance-api.conf — Cinder backend" theme={null}
[glance_store]
default_backend = cinder
cinder_catalog_info = volumev3::internalURL
cinder_volume_type = image-store
cinder_store_user_name = glance
cinder_store_auth_address = http://10.0.1.71:5000/v3
cinder_store_project_name = service
```

The Cinder volume type (`image-store`) must exist and map to a backend with sufficient capacity.

***

## HTTP (Read-Only Remote)

The HTTP backend allows the Image Service to serve images stored at external URLs without downloading or storing them locally. This backend is read-only — images cannot be uploaded through the HTTP backend.

```ini title="glance-api.conf — HTTP backend" theme={null}
[glance_store]
default_backend = http
```

To add an image from a remote URL:

```bash title="Register an image from an HTTP URL" theme={null}
openstack image create \
  --disk-format qcow2 \
  --container-format bare \
  --location http://example.com/images/ubuntu-24.04.qcow2 \
  ubuntu-24.04-remote
```

<Warning>
  Images registered with the HTTP backend depend on the availability of the remote URL. If the URL becomes unreachable, image-backed instances cannot be launched. Use this backend only for read-only catalog images or testing.
</Warning>

***

## Verify Backend Configuration

After deployment, confirm the Image API is using the expected backend:

```bash title="Check Image API configuration" theme={null}
docker exec glance_api grep -A 5 "\[glance_store\]" /etc/glance/glance-api.conf
```

```bash title="Upload a test image and verify storage" theme={null}
openstack image create \
  --disk-format qcow2 \
  --container-format bare \
  --file /dev/zero \
  --min-disk 1 \
  test-backend-check
```

```bash title="Check RBD pool contents (if using RBD)" theme={null}
rbd ls images
```

***

## Next Steps

<CardGroup>
  <Card title="Architecture" href="/services/images/architecture">
    Understand how the storage backend fits into the overall Image Service topology.
  </Card>

  <Card title="Image Templates" href="/services/images/image-templates">
    Use images as templates, manage catalogs, and export images.
  </Card>

  <Card title="Image Cache" href="/services/images/image-cache">
    Layer a local cache over your storage backend to reduce launch times.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/images/admin-troubleshooting">
    Diagnose storage connectivity and upload failure issues.
  </Card>
</CardGroup>


# Image Service Troubleshooting
Source: https://docs.xloud.tech/services/images/troubleshooting

Diagnose and resolve image upload failures, stuck images, shared image visibility issues, and launch errors in Xloud Image Service.

## Overview

This guide covers the most common issues encountered when working with Xloud Image Service
— including upload failures, stuck images, launch errors caused by image properties, and
shared image visibility problems.

<Note>
  For platform-level issues such as storage backend connectivity or Image API container
  failures, refer to the [Image Admin Guide — Troubleshooting](/services/images/admin-troubleshooting).
</Note>

***

## Upload Issues

<AccordionGroup>
  <Accordion title="Image stuck in 'saving' status" icon="clock">
    **Cause**: The upload stalled due to a client-side network interruption or the
    image service ran out of available storage.

    **Diagnose**: Check the image status:

    ```bash title="Check image status" theme={null}
    openstack image show <IMAGE_ID> -c status -c size
    ```

    **Resolution**: Delete the stuck image record and re-upload:

    ```bash title="Delete the stuck image" theme={null}
    openstack image delete <IMAGE_ID>
    ```

    Then re-upload with the `--progress` flag to monitor the transfer:

    ```bash title="Re-upload with progress indicator" theme={null}
    openstack image create \
      --disk-format qcow2 \
      --container-format bare \
      --file image.qcow2 \
      --progress \
      my-image
    ```
  </Accordion>

  <Accordion title="Upload fails with 413 Request Entity Too Large" icon="upload">
    **Cause**: HAProxy or the load balancer is enforcing an upload size limit smaller
    than the image file.

    **Resolution**: Contact your platform administrator to increase the upload limit.
    This is a platform-level configuration change. See the
    [Image Admin Guide — Troubleshooting](/services/images/admin-troubleshooting) for
    administrator instructions.
  </Accordion>

  <Accordion title="Upload fails midway through a large file" icon="wifi-off">
    **Cause**: Network interruption during a large file upload. Standard HTTP PUT has
    no resumption capability.

    **Resolution**: Use the chunked import API for images larger than 5 GB:

    ```bash title="Create image record" theme={null}
    openstack image create \
      --disk-format qcow2 \
      --container-format bare \
      --import-method web-download \
      large-image
    ```

    Then import from a URL hosted on a stable server — the Image Service fetches the
    file server-side and handles resumption internally.
  </Accordion>
</AccordionGroup>

***

## Launch Failures

<AccordionGroup>
  <Accordion title="Launch fails: minimum disk requirement not met" icon="hard-drive">
    **Cause**: The selected flavor's root disk is smaller than the image's `min_disk`
    property.

    **Diagnose**: Check the image's minimum disk setting:

    ```bash title="Show image min_disk" theme={null}
    openstack image show <IMAGE_ID> -c min_disk
    ```

    **Resolution**: Either select a flavor with a larger root disk, or reduce `min_disk`
    if it was set too conservatively:

    ```bash title="Update min_disk property" theme={null}
    openstack image set --min-disk 20 my-image
    ```
  </Accordion>

  <Accordion title="Instance fails to boot — OS not found" icon="monitor">
    **Cause**: Incorrect `hw_disk_bus` or `hw_firmware_type` property. A UEFI image
    launched with BIOS firmware (or vice versa) will fail to boot.

    **Diagnose**: Verify properties match the image's actual configuration:

    ```bash title="Show image hardware properties" theme={null}
    openstack image show <IMAGE_ID> -c properties
    ```

    **Resolution**: Correct the firmware and machine type properties:

    ```bash title="Correct UEFI image properties" theme={null}
    openstack image set \
      --property hw_firmware_type=uefi \
      --property hw_machine_type=q35 \
      <IMAGE_ID>
    ```
  </Accordion>
</AccordionGroup>

***

## Sharing Issues

<AccordionGroup>
  <Accordion title="Shared image not visible in target project" icon="eye-off">
    **Cause**: The target project has not accepted the image share.

    **Diagnose**: Check the membership status:

    ```bash title="List image members and their status" theme={null}
    openstack image member list <IMAGE_ID>
    ```

    If status shows `pending`, the target project must accept:

    ```bash title="Accept the share (run as target project)" theme={null}
    openstack image set --accept <IMAGE_ID>
    ```

    <Check>Image is now visible in the target project's image catalog.</Check>
  </Accordion>

  <Accordion title="Cannot share an image — permission denied" icon="lock">
    **Cause**: Only the image owner can modify its members. If the image is `public`,
    membership cannot be modified (it is already visible to all).

    **Diagnose**: Verify you are authenticated as the image owner's project:

    ```bash title="Check image ownership" theme={null}
    openstack image show <IMAGE_ID> -c owner
    ```

    The `owner` must match your current project ID (`openstack token issue -c project_id`).
  </Accordion>
</AccordionGroup>

***

## Image Status Reference

| Status        | Meaning                            | Action                          |
| ------------- | ---------------------------------- | ------------------------------- |
| `queued`      | Record created, upload not started | Upload data or delete and retry |
| `saving`      | Upload in progress                 | Wait, or delete if stalled      |
| `active`      | Ready for use                      | No action needed                |
| `deactivated` | Disabled by admin                  | Contact platform administrator  |
| `killed`      | Upload failed with an error        | Delete and re-upload            |

***

## Next Steps

<CardGroup>
  <Card title="Upload an Image" href="/services/images/upload-image">
    Re-upload after resolving the issue using best practices.
  </Card>

  <Card title="Image Properties" href="/services/images/image-properties">
    Correct hardware properties to resolve launch failures.
  </Card>

  <Card title="Share Images" href="/services/images/share-images">
    Review the correct sharing workflow to resolve visibility issues.
  </Card>

  <Card title="Image Admin Troubleshooting" href="/services/images/admin-troubleshooting">
    Platform-level diagnostics for Image API and storage backend issues.
  </Card>
</CardGroup>


# Upload an Image
Source: https://docs.xloud.tech/services/images/upload-image

Upload virtual machine disk images to the Xloud Image Service from a local file or URL using the Dashboard or CLI.

## Overview

The Xloud Image Service accepts virtual machine disk images in several formats for use
as boot sources when launching new instances. Upload images through the Dashboard's
Create Image form or via the CLI.

<iframe title="How to Upload an Image on Xloud" />

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Create an Image

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Images">
        Navigate to **Compute > Images** in the sidebar. Click **Create Image**.
        The create form opens as a full page.
      </Step>

      <Step title="Configure basic information">
        | Field                | Type        | Required       | Description                                                          |
        | -------------------- | ----------- | -------------- | -------------------------------------------------------------------- |
        | **Name**             | Text        | Yes            | Image display name (e.g., `ubuntu-24.04-lts`)                        |
        | **Upload Type**      | Radio       | No             | **Upload File** (default) or **File URL**                            |
        | **File**             | File upload | If Upload File | Select the local disk image file                                     |
        | **File URL**         | Text        | If File URL    | URL starting with `http://` or `https://`                            |
        | **Disk Format**      | Dropdown    | Yes            | RAW, QCOW2, ISO (admin users also see AKI, ARI, AMI, VDI, VHD, VMDK) |
        | **Container Format** | Dropdown    | Conditional    | Bare or Docker (shown when multiple formats are available)           |

        <Note>
          Administrators see an additional **Owned Project** selector to upload
          images on behalf of other projects.
        </Note>
      </Step>

      <Step title="Configure OS details (bare container format)">
        When the container format is **Bare** (default), these OS fields appear:

        | Field          | Type     | Required | Description                                                            |
        | -------------- | -------- | -------- | ---------------------------------------------------------------------- |
        | **OS**         | Dropdown | Yes      | CentOS, Ubuntu, Fedora, Windows, Debian, CoreOS, Arch, FreeBSD, Others |
        | **OS Version** | Text     | Yes      | Version string (e.g., `24.04`, `9.3`)                                  |
        | **OS Admin**   | Text     | Yes      | Default admin user (`root` for Linux, `Administrator` for Windows)     |

        <Tip>
          Setting the OS Admin correctly enables password injection. The instance
          create wizard uses this field to pre-populate the login username.
        </Tip>
      </Step>

      <Step title="Set resource requirements">
        | Field                     | Type           | Description                                     |
        | ------------------------- | -------------- | ----------------------------------------------- |
        | **Min System Disk (GiB)** | Number (0-500) | Minimum root disk for launching (0 = unlimited) |
        | **Min Memory (GiB)**      | Number (0-500) | Minimum RAM for launching (0 = unlimited)       |
      </Step>

      <Step title="Configure visibility and usage">
        | Field           | Type               | Description                                                                                                                               |
        | --------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
        | **Visibility**  | Radio (admin only) | Public (default for admin), Private (default for users), or Shared                                                                        |
        | **Project**     | Multi-select table | Select projects to share with (shown when Shared visibility)                                                                              |
        | **Protected**   | Checkbox           | Prevent accidental deletion                                                                                                               |
        | **Usage Type**  | Dropdown           | Common Server (default), Bare Metal. Administrators also see: Bare Metal Enroll, Load Balancer, Database, Container, Application Template |
        | **Description** | Text area          | Optional notes (max 255 characters)                                                                                                       |
      </Step>

      <Step title="Advanced options (optional)">
        Click **Advanced Options** to expand:

        | Field                          | Type     | Default | Description                                             |
        | ------------------------------ | -------- | ------- | ------------------------------------------------------- |
        | **qemu\_guest\_agent enabled** | Radio    | Yes     | Enable the QEMU guest agent for password change support |
        | **CPU Policy**                 | Dropdown | Not set | shared, dedicated (for CPU pinning)                     |
        | **CPU Thread Policy**          | Dropdown | Not set | prefer, isolate, require                                |
      </Step>

      <Step title="Submit">
        Click **Confirm**. The image enters `Saving` status while data transfers.

        <Check>
          Image status transitions to **Active** when upload completes. Ready for
          launching instances.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Upload a QCOW2 image" theme={null}
      openstack image create \
        --disk-format qcow2 \
        --container-format bare \
        --file /path/to/ubuntu-24.04.qcow2 \
        --min-disk 10 \
        --min-ram 1024 \
        --property os_type=linux \
        --property os_distro=ubuntu \
        --property os_version=24.04 \
        --property os_admin_user=root \
        ubuntu-24.04-lts
      ```

      ```bash title="Import from URL" theme={null}
      openstack image create \
        --disk-format qcow2 \
        --container-format bare \
        --uri https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img \
        ubuntu-24.04-noble
      ```

      ```bash title="Upload a Windows ISO" theme={null}
      openstack image create \
        --disk-format iso \
        --container-format bare \
        --file /path/to/windows-2022.iso \
        --min-disk 40 \
        --min-ram 2048 \
        --property os_type=windows \
        --property os_distro=windows \
        --property os_admin_user=Administrator \
        windows-2022-iso
      ```
    </CodeGroup>

    ```bash title="Verify image is active" theme={null}
    openstack image show ubuntu-24.04-lts -c status -c size
    ```

    <Check>Status shows `active` — ready for use.</Check>
  </Tab>
</Tabs>

***

## View Images

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Images**. The list shows images in tabs:

    | Tab                        | Shows                            |
    | -------------------------- | -------------------------------- |
    | **Current Project Images** | Images owned by your project     |
    | **Public Images**          | Publicly visible images          |
    | **Shared Images**          | Images shared with your project  |
    | **All Images**             | All images (admin role required) |

    List columns:

    | Column               | Description                                     |
    | -------------------- | ----------------------------------------------- |
    | **ID/Name**          | Image identifier (clickable to view details)    |
    | **Description**      | Optional description                            |
    | **Use Type**         | Common Server, Bare Metal, Load Balancer, etc.  |
    | **Container Format** | Bare or Docker                                  |
    | **Type**             | OS distribution (CentOS, Ubuntu, Windows, etc.) |
    | **Status**           | Active, Saving, Queued, Deactivated, etc.       |
    | **Visibility**       | Public, Private, or Shared                      |
    | **Disk Format**      | RAW, QCOW2, ISO, etc.                           |
    | **Size**             | Image file size                                 |
    | **Created At**       | Upload timestamp                                |

    Filter by **Name**, **Status**, or **Visibility**.

    **User actions**:

    | Action                     | Location              | Description                                      |
    | -------------------------- | --------------------- | ------------------------------------------------ |
    | **Edit**                   | First row action      | Edit name, OS details, protection                |
    | **Create Instance**        | More dropdown         | Launch a VM from this image                      |
    | **Create Ironic Instance** | More dropdown         | Launch a bare metal instance (if Ironic enabled) |
    | **Create Volume**          | More dropdown         | Create a block storage volume from the image     |
    | **Delete**                 | More dropdown / batch | Delete the image                                 |

    **Admin actions** (completely different set):

    | Action              | Location         | Description                                      |
    | ------------------- | ---------------- | ------------------------------------------------ |
    | **Edit**            | First row action | Edit name, OS details, visibility, protection    |
    | **Delete**          | More dropdown    | Delete the image                                 |
    | **Manage Access**   | More dropdown    | Share with projects (requires Shared visibility) |
    | **Manage Metadata** | More dropdown    | Edit image metadata key-value pairs              |
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List all images" theme={null}
      openstack image list
      ```

      ```bash title="List active images" theme={null}
      openstack image list --status active
      ```

      ```bash title="Show image details" theme={null}
      openstack image show <IMAGE_NAME>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Image Detail

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click an image name to open the detail page. The header shows Name, Status,
    Project ID, Description, Created At, and Updated At.

    The **Detail** tab shows:

    | Section               | Fields                                                                            |
    | --------------------- | --------------------------------------------------------------------------------- |
    | **Base Info**         | Size, Min System Disk, Min Memory, Disk Format, OS, OS Version, Container Format  |
    | **Security Info**     | Owner (copyable), Filename (copyable), Visibility, Protected, Checksum (copyable) |
    | **Custom Properties** | All image metadata key-value pairs                                                |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show full image details" theme={null}
    openstack image show <IMAGE_NAME> -f json
    ```
  </Tab>
</Tabs>

***

## Upload Best Practices

<AccordionGroup>
  <Accordion title="Always set min_disk and min_ram">
    Setting minimum disk and RAM prevents instance launch failures on flavors
    too small for the image:

    | OS             | Recommended min\_disk | Recommended min\_ram |
    | -------------- | --------------------- | -------------------- |
    | Ubuntu 22.04+  | 10 GiB                | 1 GiB                |
    | CentOS/RHEL 9  | 15 GiB                | 1 GiB                |
    | Windows Server | 40 GiB                | 2 GiB                |
  </Accordion>

  <Accordion title="Enable qemu_guest_agent">
    The QEMU guest agent enables password changes and other management operations
    from the Dashboard. Set `hw_qemu_guest_agent=yes` in Advanced Options (enabled
    by default in the Dashboard form).
  </Accordion>

  <Accordion title="Name images with version and date">
    Include OS version and build date for easy lifecycle management:
    `ubuntu-24.04-lts-2026-03`, `windows-2022-std-2026-03`.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Image Properties" href="/services/images/image-properties">
    Set hardware requirements and OS metadata on uploaded images
  </Card>

  <Card title="Share Images" href="/services/images/share-images">
    Share images with other projects in your organization
  </Card>

  <Card title="Create a Snapshot" href="/services/images/create-snapshot">
    Capture running instances as golden images
  </Card>

  <Card title="Troubleshooting" href="/services/images/troubleshooting">
    Resolve upload failures and stuck images
  </Card>
</CardGroup>


# Image Service User Guide
Source: https://docs.xloud.tech/services/images/user-guide

Upload, discover, and manage virtual machine images and snapshots in Xloud Cloud Platform.

<p>Overview</p>

The Xloud Image Service stores and delivers virtual machine images to the compute service
at instance launch time. Use the guides below to upload images, capture snapshots, share
images with other projects, set metadata properties, and troubleshoot common issues.

<CardGroup>
  <Card title="Upload an Image" icon="upload" href="/services/images/upload-image">
    Upload OS images and VM disk files from your workstation or via URL import.
  </Card>

  <Card title="Create a Snapshot" icon="camera" href="/services/images/create-snapshot">
    Capture a running or stopped instance as a reusable image snapshot.
  </Card>

  <Card title="Share Images" icon="share-2" href="/services/images/share-images">
    Share private images with specific projects without making them globally public.
  </Card>

  <Card title="Image Properties" icon="tag" href="/services/images/image-properties">
    Set hardware requirements, OS metadata, and scheduler hints on images.
  </Card>

  <Card title="Image Formats" icon="file-code" href="/services/images/image-formats">
    Compare QCOW2, RAW, VHD, and VMDK formats and choose the right one for your workload.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/images/troubleshooting">
    Resolve upload failures, stuck images, and launch errors caused by image issues.
  </Card>

  <Card title="Get Images" icon="download" href="/services/images/get-images">
    Download pre-built cloud images for Ubuntu, AlmaLinux, Rocky Linux, and more.
  </Card>

  <Card title="Image Requirements" icon="circle-check" href="/services/images/image-requirements">
    Understand cloud-init, SSH, and disk requirements for Xloud-compatible images.
  </Card>

  <Card title="Modify Images" icon="edit" href="/services/images/modify-images">
    Customize images offline using virt-customize, guestfish, and guestmount.
  </Card>

  <Card title="Convert Formats" icon="repeat" href="/services/images/convert-formats">
    Convert VMDK, VHD, or RAW images to QCOW2 using qemu-img.
  </Card>
</CardGroup>

***

<p>Key Concepts</p>

| Concept                   | Description                                                                                                                                                    |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Image**                 | A file containing a virtual disk with a pre-installed operating system or application. Used as the boot source for new instances.                              |
| **Snapshot**              | An image captured from a running or stopped instance. Preserves the instance disk state at the moment of capture.                                              |
| **Metadata / Properties** | Key-value pairs attached to an image. Describe OS type, version, minimum hardware requirements, and custom attributes.                                         |
| **Visibility**            | Access scope for the image. Options: `public` (all projects), `private` (owner only), `shared` (specific projects), `community` (discoverable but not pushed). |
| **Disk Format**           | The storage format of the image file (QCOW2, RAW, VHD, VMDK).                                                                                                  |
| **Min Disk / Min RAM**    | Minimum flavor requirements enforced at instance launch.                                                                                                       |

***

<p>Image Lifecycle</p>

```mermaid theme={null}
graph LR
    A([Upload / Create]) --> B[queued]
    B --> C[saving]
    C --> D[active]
    D --> E([Create Instance])
    D --> F([Create Snapshot])
    F --> D
    D --> G[deactivated]
    G --> D
    D --> H([Delete])
    style D fill:#197560,color:#fff
    style A fill:#3F8F7E,color:#fff
    style E fill:#3F8F7E,color:#fff
```

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Image Admin Guide" icon="settings" href="/services/images/admin-guide">
    Configure storage backends, metadata schemas, image caching, and access policies.
  </Card>

  <Card title="Compute User Guide" icon="server" href="/services/compute/user-guide">
    Launch instances from the images you upload and manage.
  </Card>
</CardGroup>


# Instance HA Architecture
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/architecture

Instance HA architecture — component roles, communication flows, deployment topology, and service integration.

## Overview

Xloud Instance HA is a fault-detection and automated recovery service deployed alongside
the compute cluster. Its architecture separates detection (monitors), event routing
(notification engine), decision-making (recovery engine), and execution (Compute API
calls) into independently scalable components. Understanding this separation helps
administrators plan deployments, diagnose failures, and tune recovery behaviour.

<Warning>
  This guide requires administrator privileges. Changes to the Instance HA deployment
  affect all active recovery workflows cluster-wide.
</Warning>

***

## Component Diagram

```mermaid theme={null}
graph TD
    subgraph Detection
        HM[Host Monitor<br/>IPMI / SSH]
        IM[Instance Monitor<br/>Guest Heartbeat]
    end

    subgraph Event Routing
        NE[Notification Engine<br/>NovaNotificationDriver]
    end

    subgraph Decision & Planning
        RE[Recovery Engine]
        PL[Recovery Planner]
        DB[(Instance HA DB<br/>Segments / Hosts)]
    end

    subgraph Execution
        CA[Xloud Compute API]
    end

    HM -->|Host fault| NE
    IM -->|Instance fault| NE
    NE -->|Notification| RE
    RE <--> DB
    RE --> PL
    PL -->|Evacuate| CA
    CA -->|Restart instances| CN[Healthy Compute Host]
```

***

## Components

<AccordionGroup>
  <Accordion title="Host Monitor" icon="radar">
    Polls each registered compute host at a configurable interval using IPMI or SSH.
    Declares a host unreachable after a configurable number of consecutive failures
    and emits a `COMPUTE_HOST` fault notification.

    Deployed as: `masakari-hostmonitor` service on the controller node.
  </Accordion>

  <Accordion title="Instance Monitor" icon="heart-pulse">
    Monitors running instances for guest-level heartbeat failures, independent of the
    host state. Emits `COMPUTE_INSTANCE` fault notifications when a guest stops
    responding.

    Deployed as: `masakari-instancemonitor` service on each compute host.
  </Accordion>

  <Accordion title="Notification Engine" icon="bell">
    Receives raw fault signals from monitors, deduplicates events within a configurable
    window, and routes structured notifications to the Recovery Engine via the message bus.

    The default driver is `NovaNotificationDriver`, which also listens to the Xloud
    Compute message bus for host and instance failure events.
  </Accordion>

  <Accordion title="Recovery Engine" icon="rotate">
    The central decision-making component. On receiving a notification, it:

    1. Queries the Instance HA database to identify the affected segment
    2. Retrieves all protected instances on the failed host
    3. Applies the segment's recovery method to select evacuation targets
    4. Invokes the Compute API to initiate evacuation

    Deployed as: `masakari-engine` service on the controller node.
  </Accordion>

  <Accordion title="Instance HA Database" icon="database">
    Stores all segment definitions, host registrations, reserved host flags, and
    notification history. Backed by the platform database (MySQL/MariaDB).

    Schema includes: `segments`, `hosts`, `notifications`, `vm_moves` tables.
  </Accordion>
</AccordionGroup>

***

## Deployment Topology

<Tree>
  <Tree.Folder name="Controller Nodes">
    <Tree.File name="masakari-api — REST API for segment / host management" />

    <Tree.File name="masakari-engine — Recovery Engine + Notification Engine" />

    <Tree.File name="masakari-hostmonitor — Host Monitor daemon" />
  </Tree.Folder>

  <Tree.Folder name="Compute Nodes">
    <Tree.File name="masakari-instancemonitor — Per-host instance monitor daemon" />
  </Tree.Folder>
</Tree>

<Note>
  In XDeploy-managed deployments, all Instance HA components are deployed as Docker
  containers. Configuration files are managed via the `/etc/xavs/instance-ha/` overlay
  directory.
</Note>

***

## Integration with Xloud Services

| Service                   | Integration                           | Purpose                                                             |
| ------------------------- | ------------------------------------- | ------------------------------------------------------------------- |
| Xloud Compute             | Evacuation API (`/os-evacuate`)       | Executes instance migrations to healthy hosts                       |
| Xloud Identity            | Service account authentication        | Authenticates Instance HA API calls                                 |
| AMQP Message Bus          | `NovaNotificationDriver` subscription | Receives host/instance failure events from the Compute message bus  |
| Xloud Distributed Storage | Shared instance disk backend          | Required for live evacuation — local disk instances cannot be moved |

***

## Data Flow: Host Failure to Recovery

```mermaid theme={null}
sequenceDiagram
    participant HM as Host Monitor
    participant NE as Notification Engine
    participant DB as Instance HA DB
    participant RE as Recovery Engine
    participant NOVA as Xloud Compute API

    HM->>NE: IPMI / SSH timeout — host unreachable
    NE->>DB: Create notification record (status: new)
    NE->>RE: Dispatch notification
    RE->>DB: Query segment — find protected instances
    RE->>DB: Update notification (status: running)
    RE->>NOVA: POST /os-evacuate (per instance)
    NOVA->>NOVA: Restart on target host
    NOVA-->>RE: Evacuation complete
    RE->>DB: Update notification (status: finished)
```

***

## High Availability for Instance HA

To avoid a single point of failure in the recovery infrastructure:

<CardGroup>
  <Card title="Active/Passive API" icon="server">
    Deploy multiple `masakari-api` instances behind the load balancer. The API is
    stateless — all state is in the database.
  </Card>

  <Card title="Engine Leader Election" icon="crown">
    Run `masakari-engine` on two controller nodes. The engine uses Tooz-based
    distributed locking to elect a leader — only one engine processes notifications
    at a time.
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="Failover Segments" href="/services/instance-ha/admin-guide/failover-segments">
    Create and manage failover segments and register compute hosts.
  </Card>

  <Card title="Host Monitors" href="/services/instance-ha/admin-guide/host-monitors">
    Configure IPMI and SSH host monitors for your compute nodes.
  </Card>

  <Card title="Engine Configuration" href="/services/instance-ha/admin-guide/engine-config">
    Tune recovery engine timing, retry intervals, and instance failure behaviour.
  </Card>

  <Card title="Security" href="/services/instance-ha/admin-guide/security">
    Configure RBAC policies and credential management for the Instance HA service.
  </Card>
</CardGroup>


# Engine Configuration
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/engine-config

Tune the Xloud Instance HA recovery engine — configure detection timeouts, retry intervals, instance failure behaviour, and service endpoints.

## Overview

The Instance HA engine processes fault notifications and orchestrates the recovery
workflow. Its timing and behaviour parameters determine how quickly recovery begins,
how many retries are attempted, and how edge-case scenarios (instances in ERROR state,
short-lived faults) are handled. This page documents all key configuration parameters
and their production recommendations.

<Warning>
  Engine configuration changes require a service restart. The engine will not process
  new notifications during the restart window. Schedule configuration changes during
  low-risk periods.
</Warning>

***

## Configuration File Location

<Tree>
  <Tree.Folder name="etc">
    <Tree.Folder name="xavs">
      <Tree.Folder name="instance-ha">
        <Tree.File name="instance-ha.conf — Primary configuration file" />
      </Tree.Folder>
    </Tree.Folder>
  </Tree.Folder>
</Tree>

Apply changes by restarting the engine container:

```bash title="Restart Instance HA engine" theme={null}
docker restart masakari_engine
```

***

## Core Parameters

### DEFAULT Section

| Parameter                          | Default      | Description                                                                                                          |
| ---------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------- |
| `host`                             | `<hostname>` | Service identifier used for distributed locking                                                                      |
| `long_rpc_timeout`                 | `300`        | Max seconds to wait for a Compute RPC call                                                                           |
| `wait_period_after_service_update` | `180`        | Seconds to wait after a host service update before triggering recovery — prevents false alarms from planned restarts |
| `notification_service_endpoint`    | —            | External webhook endpoint for incoming notifications                                                                 |

### \[host\_failure] Section

| Parameter                        | Default | Description                                                                |
| -------------------------------- | ------- | -------------------------------------------------------------------------- |
| `host_failure_recovery_interval` | `17`    | Seconds between recovery retry attempts                                    |
| `ignore_lease_seconds`           | `0`     | Seconds after host boot to suppress failure notifications                  |
| `evacuate_all_instances`         | `True`  | Evacuate all instances from failed host, not just those with HA protection |

### \[instance\_failure] Section

| Parameter                          | Default | Description                                             |
| ---------------------------------- | ------- | ------------------------------------------------------- |
| `recover_ignoring_error_instances` | `False` | Attempt recovery for instances already in `ERROR` state |
| `recover_instance_failure_method`  | `auto`  | Recovery method for instance-level faults               |

***

## Example Production Configuration

<Tabs>
  <Tab title="XDeploy" icon="gauge">
    <Steps>
      <Step title="Enable Host HA" icon="toggle-right">
        In XDeploy, navigate to **Configuration → Advance Features** and toggle
        **Enable Host HA** to **Yes**. Click **Save Configuration**.

        <Info>
          Database connection strings (`[database]`) and Xloud Identity credentials
          (`[keystone_authtoken]`) are **auto-managed** by XDeploy. Do not edit these
          sections manually -- they are generated during deployment and kept in sync
          with the cluster identity service automatically.
        </Info>
      </Step>

      <Step title="Customize engine parameters (optional)" icon="file-code">
        To tune timing or behaviour parameters beyond the defaults, open
        **Advanced Configuration** in XDeploy. In the **Service Tree**, select
        **masakari** and open (or create) `instance-ha.conf`.

        Edit the parameters in the Code Editor:

        ```ini title="Engine parameters in XDeploy Advanced Configuration" theme={null}
        [DEFAULT]
        host = controller-01
        long_rpc_timeout = 300
        wait_period_after_service_update = 180

        [host_failure]
        host_failure_recovery_interval = 17
        ignore_lease_seconds = 60
        evacuate_all_instances = True

        [instance_failure]
        recover_ignoring_error_instances = False
        recover_instance_failure_method = auto
        ```

        Click **Save Current File**.
      </Step>

      <Step title="Apply changes" icon="play">
        Navigate to **Operations** and run a **reconfigure** action to apply the
        updated engine configuration.

        <Check>Engine restarts with the new parameters. Verify via container logs.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly and restart the engine container:

    ```ini title="/etc/xavs/instance-ha/instance-ha.conf" theme={null}
    [DEFAULT]
    host = controller-01
    long_rpc_timeout = 300
    wait_period_after_service_update = 180

    [host_failure]
    host_failure_recovery_interval = 17
    ignore_lease_seconds = 60
    evacuate_all_instances = True

    [instance_failure]
    recover_ignoring_error_instances = False
    recover_instance_failure_method = auto

    [database]
    connection = mysql+pymysql://masakari:password@10.0.1.70/masakari

    [keystone_authtoken]
    auth_url = http://10.0.1.70:5000/v3
    project_name = service
    username = masakari
    password = <service-account-password>
    ```

    ```bash title="Restart the engine" theme={null}
    docker restart masakari_engine
    ```

    <Warning>
      The `[database]` and `[keystone_authtoken]` sections are generated by
      xavs-ansible during deployment. Edit them only if you are managing the
      configuration entirely through CLI without XDeploy.
    </Warning>
  </Tab>
</Tabs>

***

## Timing Tuning Guidance

<AccordionGroup>
  <Accordion title="Reduce false positives from planned restarts" icon="clock">
    Increase `wait_period_after_service_update` and `ignore_lease_seconds` to prevent
    recovery from triggering during planned host reboots:

    ```ini title="Recommended for environments with frequent planned maintenance" theme={null}
    [DEFAULT]
    wait_period_after_service_update = 300

    [host_failure]
    ignore_lease_seconds = 120
    ```

    This adds up to 2 minutes of tolerance for hosts coming back online after a reboot
    before Instance HA declares them permanently failed.
  </Accordion>

  <Accordion title="Faster recovery for latency-sensitive workloads" icon="zap">
    Reduce retry intervals for faster recovery at the cost of increased sensitivity
    to transient network partitions:

    ```ini title="Faster recovery (higher false-positive risk)" theme={null}
    [host_failure]
    host_failure_recovery_interval = 10
    ignore_lease_seconds = 30
    ```

    <Warning>
      Reducing intervals increases the risk of unnecessary evacuations during brief
      network interruptions. Use conservative values in shared or multi-tenant clusters.
    </Warning>
  </Accordion>

  <Accordion title="Enable recovery for instances in ERROR state" icon="circle-x">
    For environments where instances frequently enter `ERROR` state due to transient
    issues, enable recovery for error-state instances:

    ```ini title="Recover ERROR-state instances" theme={null}
    [instance_failure]
    recover_ignoring_error_instances = True
    ```

    This setting is disabled by default because attempting to evacuate an instance
    that is in `ERROR` due to a configuration issue (rather than a host failure)
    may repeatedly fail and generate noise in the notification log.
  </Accordion>
</AccordionGroup>

***

## Verify Engine Configuration

```bash title="View active configuration" theme={null}
docker exec masakari_engine \
  python3 -c "from masakari.conf import CONF; CONF(['--config-file', '/etc/masakari/masakari.conf']); print(CONF.long_rpc_timeout)"
```

```bash title="Check engine logs for configuration errors" theme={null}
docker logs masakari_engine | grep -E "ERROR|WARNING|ConfigFileNotFound"
```

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance-HA > Notifications (admin view)** after a test event.
    Verify that recovery workflow timing aligns with the configured parameters.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Check engine service status" theme={null}
    docker ps --filter name=masakari_engine
    ```

    ```bash title="Confirm engine is processing notifications" theme={null}
    docker logs masakari_engine | tail -50
    ```

    <Check>Engine shows active log output and no configuration-related error messages.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Recovery Methods" href="/services/instance-ha/admin-guide/recovery-methods">
    Configure how instances are evacuated after fault detection.
  </Card>

  <Card title="Security" href="/services/instance-ha/admin-guide/security">
    Manage service credentials and RBAC policies for Instance HA.
  </Card>

  <Card title="Troubleshooting" href="/services/instance-ha/admin-guide/troubleshooting">
    Diagnose engine startup failures and notification processing issues.
  </Card>

  <Card title="Notification Drivers" href="/services/instance-ha/admin-guide/notification-drivers">
    Configure the notification driver that feeds fault events to the engine.
  </Card>
</CardGroup>


# Failover Segments
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/failover-segments

Create and manage Xloud Instance HA failover segments — configure recovery methods, register compute hosts, and designate reserved standby nodes.

## Overview

Failover segments group compute hosts into logical fault domains. Each segment has its
own recovery method — determining how instances are relocated when a host fails. Creating
well-designed segments is the most important administrative task for Instance HA. Incorrect
segment design (too few hosts, wrong recovery method) is the primary cause of failed
automatic recovery.

<Warning>
  Segment configuration changes take effect immediately and affect all active recovery
  workflows. Plan segment structure carefully before production deployment.
</Warning>

<Note>
  **Prerequisites**

  * Administrator role in Xloud Identity
  * Compute hosts registered and reachable
  * Instance HA service deployed via XDeploy
</Note>

***

## Segment Design Principles

<CardGroup>
  <Card title="Match Physical Topology" icon="network">
    Group hosts that share a failure domain — the same power circuit, network switch,
    or rack. Hosts in the same fault domain should not be in the same segment.
  </Card>

  <Card title="Reserve Enough Headroom" icon="gauge">
    For `auto` recovery, maintain 20–30% unused capacity across all hosts in the segment.
    For `reserved_host`, the reserved node must absorb all instances from the largest host.
  </Card>

  <Card title="Separate Critical Workloads" icon="shield-check">
    Place SLA-critical instances in segments with `reserved_host` recovery. Use `auto`
    segments for standard workloads where recovery capacity is shared.
  </Card>

  <Card title="Single-Segment Host Membership" icon="server">
    A compute host can belong to only one segment. Plan segment boundaries before
    registering hosts to avoid re-registration overhead.
  </Card>
</CardGroup>

***

## Create a Failover Segment

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Instance HA administration" icon="compass">
        Navigate to
        **Instance-HA > Segments (admin view)**.
      </Step>

      <Step title="Create the segment" icon="plus">
        Click **Create Segment** and complete the form:

        | Field               | Description                 | Example                 |
        | ------------------- | --------------------------- | ----------------------- |
        | **Name**            | Unique identifier           | `prod-zone-a`           |
        | **Recovery Method** | Evacuation algorithm        | `auto`                  |
        | **Enabled**         | Activate immediately        | Checked                 |
        | **Description**     | Optional documentation note | `Production AZ-A hosts` |
      </Step>

      <Step title="Save" icon="circle-check">
        Click **Create Segment**. The segment appears in the list with status `ENABLED`.

        <Check>Segment created and ready for host registration.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate as admin" icon="key">
        ```bash title="Load admin credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Create the segment" icon="plus">
        ```bash title="Create failover segment" theme={null}
        openstack segment create \
          --recovery_method auto \
          --enabled True \
          --description "Production Zone A" \
          prod-zone-a
        ```

        Recovery method options:

        | Method          | Behaviour                                      |
        | --------------- | ---------------------------------------------- |
        | `auto`          | Evacuate to any healthy host in the segment    |
        | `reserved_host` | Evacuate only to pre-designated reserved hosts |
        | `rh_priority`   | Prefer reserved; fall back to `auto`           |
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="Show segment" theme={null}
        openstack segment show prod-zone-a
        ```

        <Check>Segment shows `enabled: True` and the correct `recovery_method`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Register Hosts in a Segment

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the segment" icon="eye">
        Navigate to **Instance-HA > Segments (admin view)** and click the segment name.
      </Step>

      <Step title="Add host" icon="plus">
        Click **Add Host** and fill in:

        | Field                  | Description                                                                  |
        | ---------------------- | ---------------------------------------------------------------------------- |
        | **Name**               | Compute hostname — must match the hostname registered in the Compute service |
        | **Type**               | `COMPUTE` for compute nodes                                                  |
        | **Control Attributes** | JSON object with IPMI or SSH connection parameters                           |
        | **On Maintenance**     | Temporarily exclude host from recovery targets                               |
        | **Reserved**           | Designate as a standby node for `reserved_host` / `rh_priority` methods      |
      </Step>

      <Step title="Confirm registration" icon="circle-check">
        The host appears in the segment host list with `ON_MAINTENANCE: False`.

        <Check>Host registered and available as a recovery target.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Register a compute host" theme={null}
    openstack segment host create \
      --type COMPUTE \
      --control_attributes '{"host": "compute-01.xloud.local"}' \
      --on_maintenance False \
      --reserved False \
      <segment-uuid>
    ```

    ```bash title="Designate a reserved standby host" theme={null}
    openstack segment host update \
      --reserved True \
      <segment-uuid> <host-uuid>
    ```

    ```bash title="List hosts in segment" theme={null}
    openstack segment host list <segment-uuid>
    ```

    <Tip>
      Reserved hosts must be in the same segment as the instances they recover.
      A host cannot belong to more than one segment.
    </Tip>
  </Tab>
</Tabs>

***

## Manage Segment Lifecycle

<Tabs>
  <Tab title="Disable a Segment" icon="pause-circle">
    Temporarily disable a segment to suppress recovery during maintenance windows.

    ```bash title="Disable segment" theme={null}
    openstack segment update --enabled False <segment-uuid>
    ```

    Re-enable after maintenance is complete:

    ```bash title="Re-enable segment" theme={null}
    openstack segment update --enabled True <segment-uuid>
    ```

    <Warning>
      Disabling a segment suppresses all automatic recovery for hosts in that segment.
      Any host failure during the window requires manual evacuation.
    </Warning>
  </Tab>

  <Tab title="Place Host on Maintenance" icon="wrench">
    Flag a specific host as on maintenance to exclude it from recovery targets without
    affecting the rest of the segment.

    ```bash title="Set host maintenance flag" theme={null}
    openstack segment host update \
      --on_maintenance True \
      <segment-uuid> <host-uuid>
    ```

    Clear the flag when maintenance is complete:

    ```bash title="Clear maintenance flag" theme={null}
    openstack segment host update \
      --on_maintenance False \
      <segment-uuid> <host-uuid>
    ```
  </Tab>

  <Tab title="Delete a Segment" icon="trash">
    <Danger>
      Deleting a segment removes all host registrations and historical notification
      data associated with it. Instances on registered hosts will no longer be
      automatically recovered.
    </Danger>

    ```bash title="Delete segment" theme={null}
    openstack segment delete <segment-uuid>
    ```

    Deregister all hosts from the segment before deletion:

    ```bash title="Delete a host from segment" theme={null}
    openstack segment host delete <segment-uuid> <host-uuid>
    ```
  </Tab>
</Tabs>

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance-HA > Segments (admin view)**. Verify:

    * All production segments have `Status: ENABLED`
    * Each segment lists the expected compute hosts
    * Reserved hosts are correctly flagged for `reserved_host` segments

    <Check>Segments are enabled and all compute hosts are registered.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all segments" theme={null}
    openstack segment list
    ```

    ```bash title="Verify host registration for each segment" theme={null}
    for seg in $(openstack segment list -f value -c uuid); do
      echo "=== Segment: $seg ==="
      openstack segment host list $seg
    done
    ```

    <Check>All segments show `enabled: True` and expected hosts are listed.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Host Monitors" href="/services/instance-ha/admin-guide/host-monitors">
    Configure IPMI and SSH monitors for hosts registered in your segments.
  </Card>

  <Card title="Recovery Methods" href="/services/instance-ha/admin-guide/recovery-methods">
    Deep-dive into recovery method selection and reserved host configuration.
  </Card>

  <Card title="Engine Configuration" href="/services/instance-ha/admin-guide/engine-config">
    Tune detection timeouts, retry intervals, and engine behaviour.
  </Card>

  <Card title="Architecture" href="/services/instance-ha/admin-guide/architecture">
    Review the full Instance HA component architecture and deployment topology.
  </Card>
</CardGroup>


# Host Monitors
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/host-monitors

Configure Xloud Instance HA host monitors — IPMI out-of-band and SSH in-band monitoring for compute hosts, with timing parameters and credential management.

## Overview

Host monitors are the detection layer of Instance HA. They continuously poll registered
compute hosts and emit fault notifications when a host becomes unreachable. Xloud Instance
HA supports two monitor types: IPMI (out-of-band, recommended for production) and SSH
(in-band, for environments without IPMI access). This page covers configuration for both
types and the timing parameters that control detection sensitivity.

<Warning>
  Misconfigured monitor credentials or unreachable IPMI endpoints will cause false-negative
  detections — the monitor reports success even when the host has failed. Validate all
  monitor connections before enabling production workloads.
</Warning>

***

## Monitor Types

<CardGroup>
  <Card title="IPMI (Recommended)" icon="microchip">
    Uses out-of-band management hardware. Detects failures even when the host OS,
    kernel, or all network interfaces are completely unresponsive.
  </Card>

  <Card title="SSH" icon="terminal">
    Attempts an SSH TCP connection to the host. Simpler to set up but dependent on
    host network stack — may miss hardware failures that leave the SSH port unreachable.
  </Card>
</CardGroup>

***

## IPMI Host Monitor

The IPMI monitor uses the host's `control_attributes` JSON field, set when registering
the host in a segment.

### Configure IPMI Credentials

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    When adding a host to a segment, set the **Control Attributes** field to a JSON
    object with the IPMI endpoint:

    ```json title="IPMI control attributes" theme={null}
    {
      "host": "192.168.10.11",
      "username": "admin",
      "password": "ipmi-password"
    }
    ```

    | Key        | Description                                    |
    | ---------- | ---------------------------------------------- |
    | `host`     | IPMI management IP address                     |
    | `username` | IPMI user with chassis status read permissions |
    | `password` | IPMI user password                             |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Register host with IPMI credentials" theme={null}
    openstack segment host create \
      --type COMPUTE \
      --control_attributes '{"host": "192.168.10.11", "username": "admin", "password": "ipmi-password"}' \
      --on_maintenance False \
      <segment-uuid>
    ```

    <Warning>
      IPMI credentials are stored in the Instance HA database. Restrict database access
      to the Instance HA service account only. Consider using Xloud Key Management to
      manage IPMI secrets and inject them via a custom notification driver.
    </Warning>
  </Tab>
</Tabs>

### Validate IPMI Connectivity

Before registering hosts, validate IPMI access from the controller node:

```bash title="Test IPMI connectivity" theme={null}
ipmitool -I lanplus \
  -H <ipmi-ip> \
  -U <username> \
  -P <password> \
  chassis status
```

Expected output: `System Power State: on` confirms IPMI access is working.

<Tip>
  Ensure UDP port 623 is permitted between the Instance HA controller and all IPMI
  management interfaces. IPMI uses RMCP+ protocol over UDP 623 by default.
</Tip>

***

## SSH Host Monitor

The SSH monitor attempts a TCP connection to port 22. It uses the SSH key configured
for the Instance HA service account — no password authentication is used.

### Deploy SSH Keys

<Steps>
  <Step title="Locate the service key" icon="key">
    The Instance HA host monitor generates an SSH key pair at service startup. Locate
    the public key on the controller node:

    ```bash title="Find service public key" theme={null}
    cat /etc/xavs/instance-ha/id_rsa.pub
    ```
  </Step>

  <Step title="Deploy to compute hosts" icon="upload">
    Append the public key to the `authorized_keys` of the user the monitor will connect
    as (typically `root` or a dedicated service account):

    ```bash title="Deploy key to compute host" theme={null}
    ssh-copy-id -i /etc/xavs/instance-ha/id_rsa.pub root@<compute-host>
    ```
  </Step>

  <Step title="Register with SSH attributes" icon="terminal">
    When registering the host in the segment, use the IP address only — no credentials:

    ```bash title="Register host for SSH monitoring" theme={null}
    openstack segment host create \
      --type COMPUTE \
      --control_attributes '{"host": "10.0.1.72"}' \
      <segment-uuid>
    ```
  </Step>

  <Step title="Validate connectivity" icon="circle-check">
    ```bash title="Test SSH from controller" theme={null}
    ssh -i /etc/xavs/instance-ha/id_rsa root@<compute-host> hostname
    ```

    <Check>SSH connection succeeds without password prompt.</Check>
  </Step>
</Steps>

***

## Timing Parameters

Adjust monitoring sensitivity through the parameters below:

| Section          | Parameter                          | Default | Description                                                                                                                 |
| ---------------- | ---------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------- |
| `[DEFAULT]`      | `wait_period_after_service_update` | `180`   | Seconds to wait after a host enters maintenance before triggering recovery -- prevents false alarms during planned restarts |
| `[DEFAULT]`      | `long_rpc_timeout`                 | `300`   | Maximum seconds to wait for a Compute RPC call to complete before declaring it failed                                       |
| `[host_failure]` | `host_failure_recovery_interval`   | `17`    | Seconds between recovery retry attempts when the first evacuation attempt fails                                             |
| `[host_failure]` | `ignore_lease_seconds`             | `0`     | Seconds after host boot to suppress fault notifications -- set to 60-120 to avoid startup noise                             |

<Tabs>
  <Tab title="XDeploy" icon="gauge">
    <Steps>
      <Step title="Open Advanced Configuration" icon="settings">
        In XDeploy, navigate to **Advanced Configuration**. In the **Service Tree**,
        select **masakari**.
      </Step>

      <Step title="Edit timing parameters" icon="file-code">
        Select or create `instance-ha.conf` in the Code Editor. Add or modify the
        timing parameters:

        ```ini title="Host monitor timing in XDeploy Advanced Configuration" theme={null}
        [DEFAULT]
        wait_period_after_service_update = 180
        long_rpc_timeout = 300

        [host_failure]
        host_failure_recovery_interval = 17
        ignore_lease_seconds = 60
        ```

        Click **Save Current File**.
      </Step>

      <Step title="Apply changes" icon="play">
        Navigate to **Operations** and run a **reconfigure** action. The host monitor
        service restarts automatically with the updated parameters.

        <Check>Host monitor is running with the new timing configuration.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly and restart the host monitor container:

    ```bash title="Open Instance HA configuration" theme={null}
    vi /etc/xavs/instance-ha/instance-ha.conf
    ```

    ```ini title="Example timing configuration" theme={null}
    [DEFAULT]
    wait_period_after_service_update = 180
    long_rpc_timeout = 300

    [host_failure]
    host_failure_recovery_interval = 17
    ignore_lease_seconds = 60
    ```

    ```bash title="Restart host monitor" theme={null}
    docker restart masakari_hostmonitor
    ```
  </Tab>
</Tabs>

***

## Monitor Health Check

Verify the host monitor is running and detecting hosts correctly:

```bash title="Check monitor service status" theme={null}
docker ps --filter name=masakari_hostmonitor
```

```bash title="View monitor logs" theme={null}
docker logs -f masakari_hostmonitor
```

Look for log entries confirming successful polls:

```
INFO masakari.hostmonitor: Host compute-01 is ALIVE
INFO masakari.hostmonitor: Host compute-02 is ALIVE
```

A repeated `UNREACHABLE` log for a running host indicates a credential or network
configuration issue — not a genuine host failure.

***

## Next Steps

<CardGroup>
  <Card title="Instance Monitors" href="/services/instance-ha/admin-guide/instance-monitors">
    Configure guest-level instance heartbeat monitoring for per-instance fault detection.
  </Card>

  <Card title="Failover Segments" href="/services/instance-ha/admin-guide/failover-segments">
    Register and manage compute hosts within protection segments.
  </Card>

  <Card title="Engine Configuration" href="/services/instance-ha/admin-guide/engine-config">
    Tune recovery engine timing and retry parameters.
  </Card>

  <Card title="Security" href="/services/instance-ha/admin-guide/security">
    Secure IPMI credentials and restrict access to Instance HA APIs.
  </Card>
</CardGroup>


# Instance Monitors
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/instance-monitors

Configure Xloud Instance HA instance-level monitors — guest heartbeat detection, notification types, and per-instance fault handling independent of host state.

## Overview

Instance monitors detect failures at the guest level — independent of whether the
underlying compute host is healthy. When an instance stops responding to heartbeat
checks, an instance-level fault notification is generated and the recovery engine
attempts to restart the affected instance. This complements host monitoring by
handling scenarios such as OS crashes, guest kernel panics, and runaway processes
that consume all available resources without taking down the host.

<Note>
  **Prerequisites**

  * Administrator privileges
  * Instance HA service deployed and running
  * XAVS Guest Agent or heartbeat capability enabled in the instance image
</Note>

***

## Instance Monitor Architecture

```mermaid theme={null}
graph TD
    subgraph Compute Host
        IM[Instance Monitor Daemon<br/>masakari-instancemonitor]
        IM -->|Poll each instance| G1[Guest 1 — heartbeat OK]
        IM -->|Poll each instance| G2[Guest 2 — heartbeat FAIL]
    end

    G2 -->|Instance fault| NE[Notification Engine]
    NE -->|COMPUTE_INSTANCE notification| RE[Recovery Engine]
    RE -->|Restart instance| NOVA[Xloud Compute API]
```

The instance monitor runs on each compute host and monitors all running instances.
It operates independently of the host monitor — both can run simultaneously.

***

## Notification Types

Instance HA distinguishes between host-level and instance-level faults using the
notification `type` field.

| Type               | Source           | Trigger                                       |
| ------------------ | ---------------- | --------------------------------------------- |
| `COMPUTE_HOST`     | Host Monitor     | Host becomes unreachable (IPMI / SSH timeout) |
| `COMPUTE_INSTANCE` | Instance Monitor | Guest heartbeat stops responding              |
| `COMPUTE_PROCESS`  | Process Monitor  | Critical compute process (nova-compute) dies  |

***

## View Instance-Level Notifications

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance-HA > Notifications (admin view)**.

    Filter by **Type: COMPUTE\_INSTANCE** to display only instance-level fault events.
    Each row shows the affected instance UUID, the source host, and the current recovery
    status.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List instance-level notifications" theme={null}
    openstack notification list --type COMPUTE_INSTANCE
    ```

    ```bash title="Show instance notification details" theme={null}
    openstack notification show <notification-uuid>
    ```

    The detail view includes the `source_host_uuid`, `payload` (with instance UUID),
    `generated_time`, and `status`.
  </Tab>
</Tabs>

***

## Configure the Instance Monitor

The instance monitor daemon runs on each compute host. Configure it via the Instance
HA configuration overlay.

<Tree>
  <Tree.Folder name="etc">
    <Tree.Folder name="xavs">
      <Tree.Folder name="instance-ha">
        <Tree.File name="instance-ha.conf" />
      </Tree.Folder>
    </Tree.Folder>
  </Tree.Folder>
</Tree>

Key instance monitor parameters:

| Section              | Parameter                          | Default | Description                                             |
| -------------------- | ---------------------------------- | ------- | ------------------------------------------------------- |
| `[instance_failure]` | `recover_ignoring_error_instances` | `False` | Attempt recovery for instances already in `ERROR` state |
| `[instance_failure]` | `recover_instance_failure_method`  | `auto`  | Recovery method for instance-level faults               |
| `[DEFAULT]`          | `instance_check_interval`          | `30`    | Seconds between instance heartbeat polls                |

<Tabs>
  <Tab title="XDeploy" icon="gauge">
    <Steps>
      <Step title="Open Advanced Configuration" icon="settings">
        In XDeploy, navigate to **Advanced Configuration**. In the **Service Tree**,
        select **masakari**.
      </Step>

      <Step title="Edit instance monitor parameters" icon="file-code">
        Select or create `instance-ha.conf` in the Code Editor. Add or modify the
        instance monitor parameters:

        ```ini title="Instance monitor settings in XDeploy Advanced Configuration" theme={null}
        [instance_failure]
        recover_ignoring_error_instances = False
        recover_instance_failure_method = auto

        [DEFAULT]
        instance_check_interval = 30
        ```

        Click **Save Current File**.
      </Step>

      <Step title="Apply changes" icon="play">
        Navigate to **Operations** and run a **reconfigure** action. The instance
        monitor restarts automatically on each compute host with the updated parameters.

        <Check>Instance monitor is running on all compute hosts with the new configuration.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly and restart the instance monitor on each
    compute host:

    ```ini title="/etc/xavs/instance-ha/instance-ha.conf" theme={null}
    [instance_failure]
    recover_ignoring_error_instances = False
    recover_instance_failure_method = auto

    [DEFAULT]
    instance_check_interval = 30
    ```

    ```bash title="Restart instance monitor on each compute host" theme={null}
    docker restart masakari_instancemonitor
    ```
  </Tab>
</Tabs>

***

## Enable Guest Heartbeat in Instances

Instance-level detection requires the instance image to have the `masakari-instancemonitor`
XAVS Guest Agent or a compatible heartbeat mechanism installed. The XAVS Guest Agent includes a VSS provider for Windows application-consistent snapshots. Check whether the agent is running inside an instance:

```bash title="Verify XAVS Guest Agent inside instance (SSH)" theme={null}
systemctl status masakari-processmonitor
```

<Tip>
  For instances using the standard Xloud images, the guest heartbeat is enabled by default.
  For custom images, install the `python3-masakari` package and enable the
  `masakari-processmonitor` service at boot.
</Tip>

***

## Difference Between Host and Instance Recovery

| Scenario                                    | Monitor Used     | Recovery Scope                                   |
| ------------------------------------------- | ---------------- | ------------------------------------------------ |
| Physical host failure, OS crash, power loss | Host Monitor     | All instances on the failed host are evacuated   |
| Single guest OS crash, kernel panic         | Instance Monitor | Only the crashed instance is restarted           |
| nova-compute process dies on a healthy host | Process Monitor  | nova-compute restarted; instances remain on host |

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance-HA > Notifications (admin view)** and confirm that
    instance-level notifications appear and transition to `finished` status when
    instance faults are detected and resolved.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Check instance monitor service" theme={null}
    docker ps --filter name=masakari_instancemonitor
    ```

    ```bash title="View instance monitor logs" theme={null}
    docker logs -f masakari_instancemonitor
    ```

    <Check>Instance monitor is running on all compute hosts and logs confirm active polling.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Notification Drivers" href="/services/instance-ha/admin-guide/notification-drivers">
    Configure the notification driver that routes fault events to the recovery engine.
  </Card>

  <Card title="Host Monitors" href="/services/instance-ha/admin-guide/host-monitors">
    Configure IPMI and SSH host-level monitors for your compute nodes.
  </Card>

  <Card title="Recovery Methods" href="/services/instance-ha/admin-guide/recovery-methods">
    Select and configure the recovery method for each failover segment.
  </Card>

  <Card title="Troubleshooting" href="/services/instance-ha/admin-guide/troubleshooting">
    Diagnose monitor failures, notification delivery issues, and recovery errors.
  </Card>
</CardGroup>


# Notification Drivers
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/notification-drivers

Configure Instance HA notification drivers — Nova driver, TaskFlow driver, and custom webhook integration.

## Overview

Notification drivers are the bridge between external monitoring systems and the Instance HA
recovery engine. They receive fault signals in various formats, translate them into
structured Instance HA notifications, and route them to the Recovery Engine. Xloud Instance
HA ships with the `NovaNotificationDriver` as the default. Custom drivers allow integration
with existing monitoring infrastructure such as Prometheus Alertmanager or Nagios.

***

## Built-in Drivers

| Driver                   | Source                                 | Protocol     | When to Use                                          |
| ------------------------ | -------------------------------------- | ------------ | ---------------------------------------------------- |
| `NovaNotificationDriver` | Xloud Compute message bus              | AMQP         | Default — all standard deployments                   |
| `TaskFlowDriver`         | TaskFlow workflow engine               | Internal RPC | Advanced workflow orchestration                      |
| Custom webhook driver    | Third-party tools (Prometheus, Nagios) | HTTP POST    | Environments with existing monitoring infrastructure |

***

## NovaNotificationDriver (Default)

The `NovaNotificationDriver` is enabled by default in all Xloud Instance HA deployments.
It subscribes to the Xloud Compute AMQP message bus and listens for `compute.host.error`
and `compute.instance.error` notification events.

<AccordionGroup>
  <Accordion title="How it works" icon="bell">
    When a compute host enters a failure state, the Compute service publishes an error
    notification on the AMQP message bus. The `NovaNotificationDriver` receives this
    message, extracts the affected host information, and creates an Instance HA
    notification record to trigger the recovery workflow.

    This driver requires no additional configuration beyond what is provided by the
    standard XDeploy deployment.
  </Accordion>

  <Accordion title="Verify the driver is active" icon="circle-check">
    ```bash title="Check driver configuration" theme={null}
    grep -i notification_driver \
      /etc/xavs/instance-ha/instance-ha.conf
    ```

    Expected output:

    ```
    notification_drivers = nova_notification
    ```

    ```bash title="Confirm AMQP connectivity" theme={null}
    docker logs masakari_engine | grep -i "notification"
    ```
  </Accordion>
</AccordionGroup>

***

## Webhook Notification Driver

For environments that use Prometheus, Nagios, or other external monitoring tools as the
primary fault detection system, Instance HA exposes an HTTP notification endpoint that
accepts structured fault payloads.

### Endpoint

```
POST /v1/notifications
Authorization: Bearer <token>
Content-Type: application/json
```

### Payload Format

```json title="Host fault notification payload" theme={null}
{
  "hostname": "compute-01.xloud.local",
  "type": "COMPUTE_HOST",
  "payload": {
    "event": "STOPPED",
    "cluster_status": "OFFLINE",
    "host_status": "NORMAL"
  }
}
```

| Field                    | Type   | Description                                              |
| ------------------------ | ------ | -------------------------------------------------------- |
| `hostname`               | string | The compute hostname as registered in the segment        |
| `type`                   | string | `COMPUTE_HOST`, `COMPUTE_INSTANCE`, or `COMPUTE_PROCESS` |
| `payload.event`          | string | `STOPPED` or `STARTED`                                   |
| `payload.cluster_status` | string | `ONLINE` or `OFFLINE`                                    |
| `payload.host_status`    | string | `NORMAL` or `UNKNOWN`                                    |

### Example: Prometheus Alertmanager Webhook

Configure an Alertmanager receiver that calls the Instance HA notification endpoint:

```yaml title="alertmanager.yml — webhook receiver" theme={null}
receivers:
  - name: "instance-ha-webhook"
    webhook_configs:
      - url: "http://<instance-ha-api>:15868/v1/notifications"
        http_config:
          bearer_token: "<service-token>"
```

<Warning>
  The Instance HA API uses Xloud Identity token authentication. Generate a service
  token for the alertmanager integration using a dedicated service account with
  the `admin` role. Do not use personal user tokens in production.
</Warning>

***

## TaskFlowDriver

The TaskFlow driver enables advanced workflow orchestration for recovery actions.
It is used internally when the default recovery workflow requires multi-step
sequencing with retry and rollback support.

This driver operates transparently alongside the `NovaNotificationDriver` and
does not require separate configuration in standard deployments. To customize
the TaskFlow task pipeline, implement the `BaseTask` interface and register the
plugin in the configuration.

<Tabs>
  <Tab title="XDeploy" icon="gauge">
    <Steps>
      <Step title="Open Advanced Configuration" icon="settings">
        In XDeploy, navigate to **Advanced Configuration**. In the **Service Tree**,
        select **masakari**.
      </Step>

      <Step title="Edit workflow targets" icon="file-code">
        Select or create `instance-ha.conf` in the Code Editor. Add the custom
        workflow targets:

        ```ini title="TaskFlow workflow in XDeploy Advanced Configuration" theme={null}
        [recovery_workflow_on_stop]
        targets = disableComputeNodeTask, PrepareHAEnabled, EvacuateHost
        ```

        Click **Save Current File**.
      </Step>

      <Step title="Apply changes" icon="play">
        Navigate to **Operations** and run a **reconfigure** action. The recovery
        engine restarts with the updated workflow pipeline.

        <Check>Engine logs confirm the custom TaskFlow targets are loaded.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly and restart the engine container:

    ```ini title="/etc/xavs/instance-ha/instance-ha.conf" theme={null}
    [recovery_workflow_on_stop]
    targets = disableComputeNodeTask, PrepareHAEnabled, EvacuateHost
    ```

    ```bash title="Restart recovery engine" theme={null}
    docker restart masakari_engine
    ```
  </Tab>
</Tabs>

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance-HA > Notifications (admin view)**.

    Simulate a notification by creating one manually (test environments only):

    * Click **Create Notification** (admin view)
    * Set type to `COMPUTE_HOST`, hostname to a registered host, event to `STOPPED`
    * Confirm the notification appears and transitions to `running`

    <Check>Notification is received, logged, and triggers the recovery workflow.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a test notification (test environments only)" theme={null}
    openstack notification create \
      --hostname compute-01.xloud.local \
      --type COMPUTE_HOST \
      --payload '{"event": "STOPPED", "cluster_status": "OFFLINE", "host_status": "NORMAL"}'
    ```

    ```bash title="Monitor notification status" theme={null}
    openstack notification list --status new
    ```

    <Warning>
      Creating test notifications in production triggers real recovery workflows.
      Only use this in isolated test environments.
    </Warning>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Recovery Methods" href="/services/instance-ha/admin-guide/recovery-methods">
    Configure how instances are evacuated after a notification triggers recovery.
  </Card>

  <Card title="Instance Monitors" href="/services/instance-ha/admin-guide/instance-monitors">
    Configure guest-level monitoring independent of the notification driver.
  </Card>

  <Card title="Engine Configuration" href="/services/instance-ha/admin-guide/engine-config">
    Tune recovery engine timing, retries, and workflow task ordering.
  </Card>

  <Card title="Security" href="/services/instance-ha/admin-guide/security">
    Secure the notification API endpoint and service account credentials.
  </Card>
</CardGroup>


# Recovery Methods
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/recovery-methods

Configure Xloud Instance HA recovery methods per segment — auto, reserved_host, and rh_priority evacuation strategies with capacity planning guidance.

## Overview

The recovery method defines how the Instance HA engine selects evacuation targets when
a host fails. The method is configured at the segment level and applies to all hosts
and instances within that segment. Choosing the right method for each workload tier is
critical to meeting recovery time and availability objectives.

<Note>
  **Prerequisites**

  * Administrator privileges
  * At least one failover segment created
  * Compute hosts registered in the segment
</Note>

***

## Method Comparison

| Method          | Target Selection                  | Capacity Guarantee              | Cost                 | Best For                          |
| --------------- | --------------------------------- | ------------------------------- | -------------------- | --------------------------------- |
| `auto`          | Any healthy host in segment       | None — first-come, first-served | Lowest               | General-purpose workloads         |
| `reserved_host` | Pre-designated standby hosts only | Guaranteed                      | Highest (idle nodes) | SLA-critical, regulated workloads |
| `rh_priority`   | Reserved hosts first, then `auto` | Best-effort                     | Moderate             | Mixed critical and standard       |

***

## auto — Evacuate to Any Host

The `auto` method instructs the recovery engine to select any available host in the
segment with sufficient vCPU and memory to accept the evacuating instances.

<AccordionGroup>
  <Accordion title="How target selection works" icon="shuffle">
    The engine queries all registered, non-maintenance hosts in the segment and selects
    those with the most available capacity. Instances are distributed across multiple
    target hosts if no single host can accept all evacuees.

    Selection order: hosts with the most free vCPU are preferred, then memory, then
    any remaining host with capacity above the minimum threshold.
  </Accordion>

  <Accordion title="Capacity planning for auto segments" icon="gauge">
    Maintain a minimum 20–30% unused vCPU and memory headroom across all hosts in
    the segment. Calculate the headroom needed to absorb the largest host's workload:

    ```
    Headroom needed = max(host vCPU) / total segment vCPU
    ```

    Example: segment with 4 hosts × 40 vCPU = 160 vCPU total.
    Largest host uses 32 vCPU → required headroom = 32/160 = 20%.
  </Accordion>
</AccordionGroup>

### Create an auto Segment

```bash title="Create auto-recovery segment" theme={null}
openstack segment create \
  --recovery_method auto \
  --enabled True \
  prod-general
```

***

## reserved\_host — Dedicated Standby

The `reserved_host` method restricts evacuation to hosts explicitly designated as
reserved standby nodes. Reserved hosts do not accept regular instance scheduling —
they remain idle until a failover event.

<AccordionGroup>
  <Accordion title="Reserved host sizing" icon="server">
    A reserved host must have sufficient vCPU and memory to absorb all instances from
    the largest non-reserved host in the segment. Size the reserved host generously
    to handle burst workloads:

    ```
    Reserved vCPU >= max(non-reserved host vCPU used)
    Reserved RAM >= max(non-reserved host RAM used)
    ```

    For a host running 20 × `m1.large` (4 vCPU, 8 GB each): the reserved host needs
    80 vCPU and 160 GB RAM minimum.
  </Accordion>

  <Accordion title="Designate a reserved host" icon="shield-check">
    ```bash title="Create segment with reserved_host method" theme={null}
    openstack segment create \
      --recovery_method reserved_host \
      --enabled True \
      prod-critical
    ```

    ```bash title="Register compute hosts in segment" theme={null}
    openstack segment host create \
      --type COMPUTE \
      --control_attributes '{"host": "compute-01"}' \
      --reserved False \
      <segment-uuid>

    openstack segment host create \
      --type COMPUTE \
      --control_attributes '{"host": "compute-standby"}' \
      --reserved True \
      <segment-uuid>
    ```

    <Warning>
      The reserved host must not be a target for regular workload scheduling.
      Apply a compute service aggregate or availability zone restriction to prevent
      the scheduler from placing non-HA instances on it.
    </Warning>
  </Accordion>
</AccordionGroup>

***

## rh\_priority — Reserved First, Fall Back

The `rh_priority` method attempts reserved hosts first. If all reserved hosts are at
capacity, it falls back to the `auto` behaviour and selects any available host.

```bash title="Create rh_priority segment" theme={null}
openstack segment create \
  --recovery_method rh_priority \
  --enabled True \
  prod-mixed
```

This method is suitable for segments with heterogeneous workloads where some instances
need guaranteed failover capacity and others can tolerate best-effort recovery.

<Tip>
  Use `rh_priority` as the default method when you have at least one reserved host
  but want recovery to succeed even if the reserved host is exhausted.
</Tip>

***

## Change Recovery Method on an Existing Segment

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance-HA > Segments (admin view)**, click the segment,
    and select **Edit Segment**. Change the **Recovery Method** field and save.

    <Warning>
      Changing the recovery method on an active segment takes effect immediately.
      Ongoing recovery workflows complete with the previous method. New fault events
      use the updated method.
    </Warning>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Update segment recovery method" theme={null}
    openstack segment update \
      --recovery_method rh_priority \
      <segment-uuid>
    ```

    ```bash title="Verify the change" theme={null}
    openstack segment show <segment-uuid> \
      -f value -c recovery_method
    ```
  </Tab>
</Tabs>

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance-HA > Segments (admin view)** and verify:

    * Each segment shows the intended `Recovery Method`
    * Reserved hosts are flagged with `RESERVED: True` in the host list

    <Check>Segments are configured with correct methods and reserved hosts are designated.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Verify all segments and their methods" theme={null}
    openstack segment list \
      -f table -c name -c recovery_method -c enabled
    ```

    ```bash title="Verify reserved hosts in a segment" theme={null}
    openstack segment host list <segment-uuid> \
      -f table -c name -c reserved -c on_maintenance
    ```

    <Check>All segments show expected recovery methods and reserved hosts are flagged.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Failover Segments" href="/services/instance-ha/admin-guide/failover-segments">
    Create segments and register compute hosts within them.
  </Card>

  <Card title="Engine Configuration" href="/services/instance-ha/admin-guide/engine-config">
    Tune recovery timing, retry intervals, and instance failure behaviour.
  </Card>

  <Card title="Host Monitors" href="/services/instance-ha/admin-guide/host-monitors">
    Configure the IPMI and SSH monitors that trigger recovery workflows.
  </Card>

  <Card title="Security" href="/services/instance-ha/admin-guide/security">
    Secure segment access and enforce role-based recovery policies.
  </Card>
</CardGroup>


# Instance HA Security
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/security

Secure Xloud Instance HA deployments — RBAC policy enforcement, service account credential management, IPMI credential handling, and audit logging.

## Overview

Instance HA operates with elevated compute privileges — it can initiate instance
evacuations, modify host states, and access IPMI credentials stored in the database.
Proper security configuration limits the blast radius of a compromised service account,
prevents unauthorized recovery triggers, and protects sensitive infrastructure credentials.

<Warning>
  Incorrectly configured RBAC policies may allow project users to trigger instance
  evacuations across other projects. Review the default policy rules before production deployment.
</Warning>

***

## RBAC Policy Enforcement

Instance HA enforces role-based access control via the Oslo policy engine. Default roles:

| Role     | Permissions                                                               |
| -------- | ------------------------------------------------------------------------- |
| `admin`  | Full access — create/modify/delete segments, approve and trigger recovery |
| `member` | Read access to notifications and segment listings                         |
| `reader` | Read-only access to all Instance HA resources                             |

### Review Default Policies

```bash title="List effective policies" theme={null}
docker exec masakari_api \
  oslopolicy-list-redundant --config-file /etc/masakari/masakari.conf
```

### Restrict Segment Management

To restrict segment creation and deletion to cloud administrators only, verify the
default policy rules are not overridden in your deployment:

```bash title="Check policy overrides" theme={null}
cat /etc/xavs/instance-ha/policy.yaml
```

If the file does not exist or is empty, the built-in defaults apply. The built-in
defaults correctly restrict destructive operations to the `admin` role.

***

## Service Account Credentials

Instance HA authenticates to Xloud Identity and the Compute API using a dedicated
`masakari` service account. Manage these credentials securely.

<AccordionGroup>
  <Accordion title="Rotate service account password" icon="rotate">
    <Steps>
      <Step title="Generate a new password" icon="key">
        ```bash title="Generate secure password" theme={null}
        openssl rand -base64 32
        ```
      </Step>

      <Step title="Update the Identity service" icon="fingerprint">
        ```bash title="Update service account password" theme={null}
        openstack user set --password <new-password> masakari
        ```
      </Step>

      <Step title="Update the configuration file" icon="file">
        ```bash title="Edit configuration" theme={null}
        vi /etc/xavs/instance-ha/instance-ha.conf
        ```

        Update the `[keystone_authtoken]` section:

        ```ini theme={null}
        [keystone_authtoken]
        password = <new-password>
        ```
      </Step>

      <Step title="Restart the service" icon="rotate">
        ```bash title="Restart Instance HA containers" theme={null}
        docker restart masakari_api masakari_engine masakari_hostmonitor
        ```

        <Check>Service restarts without authentication errors in logs.</Check>
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Minimum required roles for service account" icon="shield-check">
    The `masakari` service account requires the following minimum roles:

    | Service        | Role      | Purpose                                |
    | -------------- | --------- | -------------------------------------- |
    | Xloud Compute  | `admin`   | Initiate evacuations, query host state |
    | Xloud Identity | `service` | Authenticate API tokens                |

    Do not grant broader roles than required. The `admin` role on Compute is necessary
    for evacuation operations and cannot be reduced.
  </Accordion>
</AccordionGroup>

***

## IPMI Credential Security

IPMI credentials are stored in the Instance HA database within the `hosts.control_attributes`
column. These credentials grant physical access to compute hardware.

<AccordionGroup>
  <Accordion title="Database access restriction" icon="database">
    Limit database access to the Instance HA service account only:

    ```sql title="Grant minimum database privileges" theme={null}
    GRANT SELECT, INSERT, UPDATE, DELETE ON masakari.* TO 'masakari'@'%';
    REVOKE ALL ON masakari.* FROM 'root'@'%';
    ```

    Verify no other service accounts have access to the `masakari` database.
  </Accordion>

  <Accordion title="Disable debug logging in production" icon="eye-off">
    Debug logging may write `control_attributes` contents (including IPMI passwords)
    to log files. Ensure debug logging is disabled:

    ```ini title="/etc/xavs/instance-ha/instance-ha.conf" theme={null}
    [DEFAULT]
    debug = False
    ```

    ```bash title="Verify debug is disabled" theme={null}
    grep -i debug /etc/xavs/instance-ha/instance-ha.conf
    ```

    <Warning>
      Enabling `debug = True` in production exposes IPMI credentials in log output.
      Never enable debug logging on production controllers without log file access controls.
    </Warning>
  </Accordion>

  <Accordion title="Key Management integration (advanced)" icon="key">
    For the highest security posture, manage IPMI credentials using Xloud Key Management
    (Barbican) and inject them into Instance HA via a custom notification driver that
    fetches credentials at runtime rather than storing them in the database.

    This requires a custom `control_attributes` resolver plugin — consult the
    [Instance HA architecture](/services/instance-ha/admin-guide/architecture) page
    for the plugin interface documentation.
  </Accordion>
</AccordionGroup>

***

## Network Access Controls

<CardGroup>
  <Card title="IPMI Network Isolation" icon="network">
    Place IPMI management interfaces on a dedicated management VLAN. Only the Instance
    HA controller should have network access to IPMI interfaces (UDP port 623).
  </Card>

  <Card title="API Endpoint Protection" icon="shield-check">
    The Instance HA API (port 15868) should only be accessible from trusted management
    networks. Do not expose it to project tenant networks.
  </Card>
</CardGroup>

***

## Audit Logging

All Instance HA operations are logged with the requesting user's token identity.
Retain API access logs for at least 90 days to support incident investigations.

```bash title="Check API access logs" theme={null}
docker logs masakari_api | grep -E "POST|DELETE|PUT"
```

```bash title="Review recent notification events" theme={null}
openstack notification list --limit 100 -f json
```

***

## Next Steps

<CardGroup>
  <Card title="Engine Configuration" href="/services/instance-ha/admin-guide/engine-config">
    Tune recovery parameters including debug logging settings.
  </Card>

  <Card title="Troubleshooting" href="/services/instance-ha/admin-guide/troubleshooting">
    Diagnose authentication failures and policy enforcement issues.
  </Card>

  <Card title="Architecture" href="/services/instance-ha/admin-guide/architecture">
    Review the full Instance HA deployment topology and trust boundaries.
  </Card>

  <Card title="Notification Drivers" href="/services/instance-ha/admin-guide/notification-drivers">
    Secure the webhook notification endpoint with appropriate authentication.
  </Card>
</CardGroup>


# Instance HA Admin Troubleshooting
Source: https://docs.xloud.tech/services/instance-ha/admin-guide/troubleshooting

Diagnose Instance HA platform issues — engine failures, monitor connectivity, and notification delivery problems.

## Overview

This guide covers administrator-level troubleshooting for Xloud Instance HA — from
service startup failures to notification processing issues and capacity-related recovery
failures. For user-facing issues such as individual instance recovery failures, see the
[Instance HA User Troubleshooting](/services/instance-ha/user-guide/troubleshooting) guide.

<Warning>
  Several diagnostic commands in this guide inspect live recovery state. Run them on
  the controller node and avoid interfering with in-progress recovery workflows.
</Warning>

***

## Common Issues

<AccordionGroup>
  <Accordion title="Notification received but no recovery triggered" icon="bell-off">
    **Cause**: The segment may be disabled, or the failed host is not registered in
    any segment. Also occurs if the engine is not running.

    **Resolution**:

    ```bash title="Check engine status" theme={null}
    docker ps --filter name=masakari_engine
    docker logs masakari_engine | tail -30
    ```

    ```bash title="Check segment and host registration" theme={null}
    openstack segment list
    openstack segment host list <segment-uuid>
    ```

    If the segment is disabled, re-enable it:

    ```bash title="Re-enable segment" theme={null}
    openstack segment update --enabled True <segment-uuid>
    ```

    If the host is missing from the segment, register it:

    ```bash title="Register host" theme={null}
    openstack segment host create \
      --type COMPUTE \
      --control_attributes '{"host": "compute-01"}' \
      <segment-uuid>
    ```
  </Accordion>

  <Accordion title="Recovery fails with capacity error" icon="server">
    **Cause**: No healthy host in the segment has sufficient vCPU or memory to
    accept the evacuated instances.

    **Resolution**:

    ```bash title="Check host capacity" theme={null}
    openstack host list --service compute
    openstack host show <hostname>
    ```

    ```bash title="Check per-host utilization" theme={null}
    openstack hypervisor list --long
    ```

    Add compute capacity or add additional hosts to the segment. For `reserved_host`
    segments, verify the reserved host has sufficient headroom:

    ```bash title="Check reserved host utilization" theme={null}
    openstack host show <reserved-hostname>
    ```
  </Accordion>

  <Accordion title="Instances stuck in UNKNOWN state after evacuation" icon="circle-help">
    **Cause**: The compute database still associates the instance with the failed host.
    The evacuation may have been partially completed.

    **Resolution**:

    ```bash title="Force instance state to active" theme={null}
    openstack server set --state active <instance-uuid>
    ```

    If the instance remains stuck after state reset, manually evacuate:

    ```bash title="Manual evacuation" theme={null}
    openstack server evacuate <instance-uuid> --host <healthy-host>
    ```
  </Accordion>

  <Accordion title="Host monitor not detecting failures" icon="radar">
    **Cause**: IPMI or SSH credentials are incorrect, the monitor cannot reach the
    management network, or a firewall is blocking the monitoring port.

    **Resolution**:

    ```bash title="Check host monitor logs" theme={null}
    docker logs -f masakari_hostmonitor
    ```

    Test IPMI connectivity manually:

    ```bash title="Test IPMI connection" theme={null}
    ipmitool -I lanplus \
      -H <ipmi-ip> \
      -U <username> \
      -P <password> \
      chassis status
    ```

    Test SSH connectivity:

    ```bash title="Test SSH connection" theme={null}
    ssh -i /etc/xavs/instance-ha/id_rsa root@<compute-host> hostname
    ```

    Confirm firewall rules permit UDP 623 (IPMI) and TCP 22 (SSH) from the
    Instance HA controller to all monitored hosts.
  </Accordion>

  <Accordion title="Engine fails to start" icon="circle-alert">
    **Cause**: Database connectivity failure, Identity authentication error, or a
    configuration file syntax error.

    **Resolution**:

    ```bash title="Check engine startup logs" theme={null}
    docker logs masakari_engine | grep -E "ERROR|CRITICAL"
    ```

    Common log patterns and their resolution:

    | Log Message                                                       | Cause                        | Fix                                                    |
    | ----------------------------------------------------------------- | ---------------------------- | ------------------------------------------------------ |
    | `OperationalError: (pymysql)`                                     | Database unreachable         | Check DB connection string and service status          |
    | `Unauthorized: The request you have made requires authentication` | Invalid Identity credentials | Rotate service account password                        |
    | `ConfigFileNotFound`                                              | Missing config file          | Verify `/etc/xavs/instance-ha/instance-ha.conf` exists |
    | `ImportError: No module named`                                    | Missing Python dependency    | Reinstall the Instance HA container image              |
  </Accordion>

  <Accordion title="Notifications permanently stuck in running state" icon="clock">
    **Cause**: The recovery workflow has stalled — the engine is waiting for a Compute
    RPC call that never completes, or the Taskflow state machine is stuck.

    **Resolution**:

    ```bash title="Check engine logs for stalled workflows" theme={null}
    docker logs masakari_engine | grep -E "stuck|timeout|waiting"
    ```

    If a notification has been `running` for more than 15 minutes, manually reset it:

    ```bash title="Reset stalled notification" theme={null}
    openstack notification update \
      --status error \
      <notification-uuid>
    ```

    Then run a manual evacuation for any instances that were not recovered:

    ```bash title="Manual evacuation" theme={null}
    openstack server evacuate <instance-uuid>
    ```

    Restart the engine after resolving the root cause:

    ```bash title="Restart engine" theme={null}
    docker restart masakari_engine
    ```
  </Accordion>
</AccordionGroup>

***

## Diagnostic Commands Reference

```bash title="Check all Instance HA service container statuses" theme={null}
docker ps --filter name=masakari
```

```bash title="View engine logs (last 100 lines)" theme={null}
docker logs --tail 100 masakari_engine
```

```bash title="List all notifications with status" theme={null}
openstack notification list -f table -c uuid -c hostname -c type -c status
```

```bash title="Show full notification payload" theme={null}
openstack notification show <notification-uuid> -f json
```

```bash title="Check Compute service status for all hosts" theme={null}
openstack compute service list --service nova-compute
```

***

## Next Steps

<CardGroup>
  <Card title="Engine Configuration" href="/services/instance-ha/admin-guide/engine-config">
    Tune engine timing parameters to reduce false positives and improve recovery speed.
  </Card>

  <Card title="Host Monitors" href="/services/instance-ha/admin-guide/host-monitors">
    Validate and reconfigure IPMI and SSH monitor connectivity.
  </Card>

  <Card title="Failover Segments" href="/services/instance-ha/admin-guide/failover-segments">
    Review segment configuration and host registration.
  </Card>

  <Card title="User Troubleshooting" href="/services/instance-ha/user-guide/troubleshooting">
    Guide for project users experiencing individual instance recovery failures.
  </Card>
</CardGroup>


# Instance HA CLI Reference
Source: https://docs.xloud.tech/services/instance-ha/cli-reference

Complete openstack ha CLI commands for managing failover segments, hosts, and notifications in Xloud Instance HA.

## Overview

The `openstack ha` command group manages Instance HA — failover segments, segment hosts, and notification drivers.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python masakariclient installed: `pip install python-masakariclient`
  * Admin role required for all HA operations
</Note>

***

## Failover Segments

<CodeGroup>
  ```bash title="List segments" theme={null}
  openstack ha segment list
  ```

  ```bash title="Create segment" theme={null}
  openstack ha segment create \
    --recovery-method auto \
    --service-type COMPUTE \
    primary-zone
  ```

  ```bash title="Show segment" theme={null}
  openstack ha segment show primary-zone
  ```

  ```bash title="Update segment" theme={null}
  openstack ha segment update \
    --recovery-method reserved_host \
    primary-zone
  ```

  ```bash title="Delete segment" theme={null}
  openstack ha segment delete primary-zone
  ```
</CodeGroup>

***

## Segment Hosts

<CodeGroup>
  ```bash title="List hosts in a segment" theme={null}
  openstack ha host list primary-zone
  ```

  ```bash title="Add host to segment" theme={null}
  openstack ha host create \
    --segment primary-zone \
    --name compute-01 \
    --type COMPUTE \
    --control-attributes "SSH"
  ```

  ```bash title="Show host" theme={null}
  openstack ha host show primary-zone compute-01
  ```

  ```bash title="Update host" theme={null}
  openstack ha host update primary-zone compute-01 \
    --on-maintenance True
  ```

  ```bash title="Remove host from segment" theme={null}
  openstack ha host delete primary-zone compute-01
  ```
</CodeGroup>

***

## Notifications

<CodeGroup>
  ```bash title="List notifications" theme={null}
  openstack ha notification list
  ```

  ```bash title="Show notification" theme={null}
  openstack ha notification show <notification-uuid>
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="How Instance HA Works" href="/services/instance-ha/user-guide/how-it-works">
    Understand the automatic recovery workflow
  </Card>

  <Card title="Failover Segments Guide" href="/services/instance-ha/admin-guide/failover-segments">
    Configure segments and assign hosts
  </Card>
</CardGroup>


# Instance High Availability
Source: https://docs.xloud.tech/services/instance-ha/index

Automatically detect and recover failed instances and compute hosts with Xloud Instance HA — zero-touch failover for mission-critical workloads.

Protect your workloads from compute host failures with automatic detection and recovery.
Xloud Instance HA continuously monitors compute nodes and instances, triggering evacuation
and restart workflows the moment a fault is detected — without manual intervention.

<Card title="XAVS — Advanced Virtualization Solutions" icon="external-link" href="https://xloud.tech/xavs">
  Product details on xloud.tech
</Card>

***

<p>Instance High Availability</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/instance-ha/user-guide">
    Understand protection segments, instance protection policies, and how to monitor
    recovery workflows for your running workloads.
  </Card>

  <Card title="Admin Guide" icon="shield-check" href="/services/instance-ha/admin-guide">
    Configure failover segments, host and instance monitors, notification drivers, and
    integrate Instance HA with your compute cluster.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/instance-ha/cli-reference">
    Complete command reference for managing failover segments, hosts, and recovery
    notifications using the openstack CLI.
  </Card>

  <Card title="Compute Service" icon="server" href="/services/compute">
    Xloud Compute provides the hypervisor layer that Instance HA monitors and manages
    during host failover events.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Host Failure Detection" icon="radar" href="/services/instance-ha/admin-guide/host-monitors">
    IPMI and SSH-based monitors detect unreachable hosts in seconds and immediately
    trigger evacuation of all protected instances.
  </Card>

  <Card title="Automatic Instance Recovery" icon="rotate" href="/services/instance-ha/user-guide/recovery-workflows">
    Failed instances are automatically restarted on healthy hosts within the same
    protection segment, respecting affinity rules.
  </Card>

  <Card title="Reserved Host Failover" icon="server" href="/services/instance-ha/admin-guide/failover-segments">
    Designate standby compute hosts that remain idle until a failover event occurs —
    guaranteeing resource availability for recovery.
  </Card>

  <Card title="Protection Segments" icon="shield-check" href="/services/instance-ha/user-guide/protection-segments">
    Group hosts and instances into logical fault domains. Each segment has its own
    recovery policy, monitors, and notification targets.
  </Card>

  <Card title="Notification Drivers" icon="bell" href="/services/instance-ha/admin-guide/notification-drivers">
    Integrate with IPMI, SSH, and custom notification sources to receive precise
    fault signals from infrastructure monitoring tools.
  </Card>

  <Card title="Audit Trail" icon="clipboard-list" href="/services/instance-ha/user-guide/monitoring-status">
    Every recovery event is logged with timestamps, affected instances, and resolution
    outcomes — fully queryable via the Dashboard and CLI.
  </Card>
</CardGroup>

***

<p>How It Works</p>

```mermaid theme={null}
graph TD
    A[Compute Host] -->|Heartbeat monitored| B[Host Monitor]
    B -->|Fault detected| C[Notification Engine]
    C -->|Triggers recovery| D[Instance HA Engine]
    D -->|Evacuates instances| E[Healthy Compute Hosts]
    D -->|Logs event| F[Recovery Audit Log]
    E -->|Instances restarted| G[Protected Instances Active]
```

***

<p>Platform Resilience</p>

<Info>**Xloud-Developed** — These resilience capabilities are developed by Xloud and ship with XAVS.</Info>

<CardGroup>
  <Card title="VM High Availability" icon="heart-pulse">
    Automatic instance restart on host failure. Configurable per-instance priority. Failover segments for per-group recovery policies. Requires XAVS.
  </Card>

  <Card title="Power Recovery Automation" icon="bolt">
    9-phase automated recovery playbook. Target recovery time: 7-13 minutes. Sequential service startup with health gates between each phase. Requires XAVS.
  </Card>

  <Card title="Container Self-Healing" icon="rotate">
    Three-tier autoheal daemon with dependency-aware restart ordering. Circuit breaker pattern prevents restart loops. Exponential backoff. Requires XAVS.
  </Card>

  <Card title="Proactive Monitoring" icon="bell">
    Pre-configured alert rules across 13 groups covering storage, database, message queue, compute, networking, containers, APIs, system resources, disk, memory, security, and capacity. Predictive alerts for capacity forecasting. Requires XAVS.
  </Card>

  <Card title="Network Resilience" icon="network">
    L3 high availability and DHCP high availability with sub-3-second failover. Automatic ARP gratuitous announcements for fast VIP convergence. Requires XAVS.
  </Card>

  <Card title="Rolling Upgrades with Rollback" icon="arrow-up-circle">
    Per-service container upgrades with 2-10 second swap time. Canary deployment (first node only). Image tag rollback mechanism. Previous images cached locally. Requires XAVS.
  </Card>
</CardGroup>

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    The hypervisor layer monitored and managed by Instance HA
  </Card>

  <Card title="Resource Optimizer" icon="trending-up" href="/services/optimization/index">
    Rebalances workloads after recovery to restore cluster efficiency
  </Card>

  <Card title="Xloud Block Storage" icon="hard-drive" href="/services/storage/index">
    Persistent volumes that survive host failover when using shared storage
  </Card>
</CardGroup>


# How Instance HA Works
Source: https://docs.xloud.tech/services/instance-ha/user-guide/how-it-works

Understand Instance HA detection and recovery — host monitors, notifications, and the recovery engine.

## Overview

Xloud Instance HA delivers zero-touch recovery for protected compute workloads. When a
compute host becomes unreachable, the service detects the fault, identifies all protected
instances on that host, and automatically evacuates them to healthy nodes — without any
manual intervention. This page explains the end-to-end detection and recovery flow.

<Note>
  **Prerequisites**

  * An active Xloud account
  * Instance HA enabled on your platform by an administrator
  * At least one failover segment configured with registered compute hosts
</Note>

***

## Core Components

<CardGroup>
  <Card title="Host Monitor">
    Continuously polls compute hosts using IPMI out-of-band management or SSH.
    Declares a host unreachable when it fails to respond within the configured timeout.
  </Card>

  <Card title="Notification Engine">
    Receives fault signals from monitors, deduplicates events, and routes them to
    the Recovery Engine as structured notifications.
  </Card>

  <Card title="Recovery Engine">
    The decision-making core. Identifies protected instances on the failed host and
    determines the evacuation target based on the segment's recovery method.
  </Card>

  <Card title="Compute API">
    Executes the evacuation. Instances are restarted on the selected healthy host
    using the same image, volume, and network configuration.
  </Card>
</CardGroup>

***

## Recovery Flow

```mermaid theme={null}
sequenceDiagram
    participant HM as Host Monitor
    participant NE as Notification Engine
    participant RE as Recovery Engine
    participant CN as Healthy Compute Host
    participant Dashboard

    HM->>NE: Host unreachable (IPMI / SSH timeout)
    NE->>RE: Failure notification (type: COMPUTE_HOST)
    RE->>RE: Query segment — identify protected instances
    RE->>CN: Evacuate instances via Compute API
    CN->>CN: Restart instances on healthy node
    RE->>Dashboard: Log recovery event with status
    Dashboard->>Dashboard: Update instance status to ACTIVE
```

The recovery process starts within seconds of fault detection. No human action is
required for instances enrolled in an active protection segment.

***

## Detection Methods

<AccordionGroup>
  <Accordion title="IPMI (Out-of-Band)">
    The preferred detection method. The host monitor connects to the server's IPMI
    interface — which operates independently of the host OS — to verify whether the
    physical node is powered and responsive.

    IPMI detection is more reliable than SSH because it does not depend on the host
    network stack or OS. A host that has kernel-panicked or lost all network interfaces
    is still detectable via IPMI.

    | Advantage                          | Disadvantage                              |
    | ---------------------------------- | ----------------------------------------- |
    | Works even when OS is unresponsive | Requires IPMI hardware and network access |
    | Detects power failures             | Requires IPMI credentials per host        |
  </Accordion>

  <Accordion title="SSH (In-Band)">
    The SSH monitor attempts a TCP connection to the host on port 22. Use this method
    when IPMI hardware is unavailable.

    SSH monitoring is susceptible to false positives caused by SSH service restarts,
    temporary network partitions, or high host load. The monitor implements a configurable
    retry interval to reduce spurious alerts.

    | Advantage                    | Disadvantage                        |
    | ---------------------------- | ----------------------------------- |
    | No special hardware required | Dependent on host network and OS    |
    | Easy to deploy               | May miss physical hardware failures |
  </Accordion>

  <Accordion title="Pacemaker">
    Uses Pacemaker cluster monitoring to detect node failures. Pacemaker tracks cluster
    membership and triggers a notification when a node is fenced or goes offline.

    This method integrates with existing Pacemaker/Corosync clusters and leverages
    STONITH fencing for reliable failure detection.

    | Advantage                                       | Disadvantage                      |
    | ----------------------------------------------- | --------------------------------- |
    | Integrates with existing cluster infrastructure | Requires Pacemaker/Corosync setup |
    | Reliable fencing-based detection                | More complex configuration        |
  </Accordion>
</AccordionGroup>

***

## Recovery Methods

Each failover segment uses one of four recovery methods. The method is selected when
creating the segment.

| Method          | Behaviour                                     | Best Suited For                  |
| --------------- | --------------------------------------------- | -------------------------------- |
| `auto`          | Evacuate to any healthy host in the segment   | General workloads                |
| `auto_priority` | Evacuate using priority-based host selection  | Workloads with preferred targets |
| `reserved_host` | Evacuate only to pre-designated standby hosts | SLA-critical workloads           |
| `rh_priority`   | Prefer reserved hosts, fall back to any host  | Mixed environments               |

<Info>
  The recovery method is configured per segment by your administrator. Contact your
  administrator to understand which method applies to your protection segment. Your
  administrator can configure this through [XDeploy](/deployment).
</Info>

***

## Notification Types

Instance HA generates different notification types depending on the source of the failure:

| Type              | Color  | Description                                                |
| ----------------- | ------ | ---------------------------------------------------------- |
| **COMPUTE\_HOST** | Red    | Physical compute host failure detected by the host monitor |
| **VM**            | Orange | Individual VM failure detected by the instance monitor     |
| **PROCESS**       | Blue   | Service process failure (e.g., nova-compute crash)         |
| **pacemaker**     | Purple | Failure detected by Pacemaker cluster monitoring           |

***

## Instance Lifecycle During Recovery

```mermaid theme={null}
graph TD
    A[Instance ACTIVE on Host A] -->|Host A fails| B[Instance status: UNKNOWN]
    B -->|Recovery Engine evacuates| C[Instance evacuating to Host B]
    C -->|Evacuation complete| D[Instance restarting on Host B]
    D -->|Restart complete| E[Instance ACTIVE on Host B]
    E -->|Recovery logged| F[Notification: FINISHED]
```

The instance `ID`, `name`, attached volumes, and network configuration are preserved
across the recovery. Only the physical host changes.

***

## Monitoring in the Dashboard

The Xloud Dashboard provides dedicated pages for monitoring Instance HA:

| Page              | Path                        | Purpose                                          |
| ----------------- | --------------------------- | ------------------------------------------------ |
| **Segments**      | Instance HA > Segments      | View and manage failover segments and hosts      |
| **Hosts**         | Instance HA > Hosts         | View all registered hosts across all segments    |
| **Notifications** | Instance HA > Notifications | Track recovery events with real-time progress    |
| **VM Moves**      | Instance HA > VM Moves      | View all VM evacuations across all notifications |

<Tip>
  The **Notifications** detail page includes a **Recovery Progress** tab that shows
  real-time VM evacuation status with auto-refresh every 5 seconds during active
  recovery. See [Monitoring Status](/services/instance-ha/user-guide/monitoring-status)
  for details.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Protection Segments" href="/services/instance-ha/user-guide/protection-segments">
    Create segments, add hosts, and configure recovery methods
  </Card>

  <Card title="Recovery Workflows" href="/services/instance-ha/user-guide/recovery-workflows">
    Understand recovery methods and how the engine selects evacuation targets
  </Card>

  <Card title="Monitoring Status" href="/services/instance-ha/user-guide/monitoring-status">
    Track active and historical recovery events in the Dashboard
  </Card>

  <Card title="Instance HA Overview" href="/services/instance-ha/index">
    Return to the Instance HA service overview page
  </Card>
</CardGroup>


# Monitoring Recovery Status
Source: https://docs.xloud.tech/services/instance-ha/user-guide/monitoring-status

Track Instance HA recovery events, review notification history, manage hosts, and monitor VM evacuations from the Xloud Dashboard and CLI.

## Overview

Xloud Instance HA logs every fault detection and recovery event as a notification.
The Dashboard provides four dedicated pages for monitoring the entire HA lifecycle:
**Segments**, **Hosts**, **Notifications**, and **VM Moves**. This page explains how
to use each monitoring view.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * Instance HA service enabled on the platform
</Note>

***

## Notifications

The Notifications page is the primary monitoring interface for recovery events.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Notifications">
        Navigate to **Instance HA > Notifications** in the sidebar.
      </Step>

      <Step title="Review the notification list">
        Each row represents one fault event.

        | Column             | Description                                                         |
        | ------------------ | ------------------------------------------------------------------- |
        | **UUID**           | Notification identifier (clickable to view details, copyable)       |
        | **Source Host**    | UUID of the compute host that triggered the notification (copyable) |
        | **Type**           | Failure type as a colored tag                                       |
        | **Status**         | Recovery status as a colored tag                                    |
        | **Generated Time** | When the fault was first detected                                   |
        | **Updated At**     | Last status update timestamp                                        |

        **Notification types** (color-coded):

        | Type              | Color  | Description                    |
        | ----------------- | ------ | ------------------------------ |
        | **COMPUTE\_HOST** | Red    | Physical compute host failure  |
        | **VM**            | Orange | Individual VM failure          |
        | **PROCESS**       | Blue   | Service process failure        |
        | **pacemaker**     | Purple | Pacemaker cluster node failure |

        **Notification statuses** (color-coded):

        | Status       | Color  | Meaning                                  |
        | ------------ | ------ | ---------------------------------------- |
        | **New**      | Blue   | Notification received; recovery queued   |
        | **Running**  | Orange | Recovery workflow in progress            |
        | **Finished** | Green  | All instances recovered successfully     |
        | **Error**    | Red    | Recovery encountered errors              |
        | **Failed**   | Red    | Recovery failed completely               |
        | **Ignored**  | Grey   | Segment disabled or duplicate suppressed |
      </Step>

      <Step title="Filter notifications">
        Use the search/filter bar to narrow results:

        | Filter     | Type     | Options                               |
        | ---------- | -------- | ------------------------------------- |
        | **Host**   | Text     | Source host UUID                      |
        | **UUID**   | Text     | Notification UUID                     |
        | **Status** | Dropdown | new, running, finished, error, failed |
        | **Type**   | Dropdown | COMPUTE\_HOST, VM, PROCESS            |
      </Step>
    </Steps>

    <Note>
      The Notifications page is read-only — there are no create, update, or delete
      actions. Notifications are generated automatically by the HA engine.
    </Note>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="List all notifications" theme={null}
      openstack notification list
      ```

      ```bash title="Filter by status" theme={null}
      openstack notification list --status error
      ```

      ```bash title="Show notification detail" theme={null}
      openstack notification show <notification-uuid>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Notification Detail

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a notification UUID in the list to open the detail page. The detail page
    header shows the notification **Type** and **Status** as colored tags.

    Two tabs are available:

    **Detail tab** — Notification metadata:

    | Section                             | Fields                                                                                                     |
    | ----------------------------------- | ---------------------------------------------------------------------------------------------------------- |
    | **Notification Detail** (left card) | Notification UUID (copyable), Source Host (copyable), Type, Status, Generated Time, Created At, Updated At |
    | **Payload** (right card)            | Dynamic fields from the notification payload — shows all key-value pairs from the fault event data         |

    **Recovery Progress tab** — Real-time VM evacuation tracking:

    | Field                    | Description                                                                                              |
    | ------------------------ | -------------------------------------------------------------------------------------------------------- |
    | **Summary**              | Notification status, total VMs, succeeded count (green), failed count (red), circular progress indicator |
    | **VM Evacuations table** | Per-VM status with source host, destination host, type, status icon, start/end time, error message       |

    **VM evacuation statuses**:

    | Status    | Color | Icon    | Meaning                             |
    | --------- | ----- | ------- | ----------------------------------- |
    | Pending   | Grey  | Clock   | Queued, not started                 |
    | Running   | Blue  | Spinner | In progress                         |
    | Succeeded | Green | Check   | Successfully recovered              |
    | Failed    | Red   | Close   | Failed — manual intervention needed |

    <Tip>
      When the notification is in **Running** status, the Recovery Progress tab
      **auto-refreshes every 5 seconds**. A tag reading "Auto-refreshing every 5s"
      is displayed. Watch VM evacuations complete in real time without manual page refresh.
    </Tip>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show full notification detail" theme={null}
    openstack notification show <notification-uuid>
    ```

    The output includes `source_host_uuid`, `type`, `generated_time`, `status`,
    and the full payload with affected instance UUIDs.
  </Tab>
</Tabs>

***

## Hosts Monitoring

The Hosts page provides a cross-segment view of all registered compute hosts.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Hosts">
        Navigate to **Instance HA > Hosts** in the sidebar.
      </Step>

      <Step title="Review the host list">
        All hosts across all segments are displayed in a single list.

        | Column                | Description                                      |
        | --------------------- | ------------------------------------------------ |
        | **Name**              | Host name (clickable to view details)            |
        | **UUID**              | Unique host identifier                           |
        | **Reserved**          | Whether the host is a standby node (Yes/No)      |
        | **Type**              | Host type identifier                             |
        | **Control Attribute** | Monitoring attributes (SSH, IPMI, etc.)          |
        | **On Maintenance**    | Whether the host is in maintenance mode (Yes/No) |
        | **Failover Segment**  | Link to the parent segment                       |
      </Step>

      <Step title="Filter hosts">
        | Filter             | Type                                |
        | ------------------ | ----------------------------------- |
        | **Segment ID**     | Text — filter by parent segment     |
        | **Type**           | Text — filter by host type          |
        | **On Maintenance** | Text — filter by maintenance status |
        | **Reserved**       | Text — filter by reserved status    |
      </Step>
    </Steps>

    **Host actions**:

    | Action     | Location                  | Description                                                               |
    | ---------- | ------------------------- | ------------------------------------------------------------------------- |
    | **Update** | Row action (first button) | Edit host properties (reserved, type, control attributes, on maintenance) |
    | **Delete** | More dropdown / batch     | Remove host from its segment                                              |
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List all hosts in a segment" theme={null}
      openstack segment host list <segment-uuid>
      ```

      ```bash title="Show host detail" theme={null}
      openstack segment host show <segment-uuid> <host-uuid>
      ```

      ```bash title="Update host maintenance mode" theme={null}
      openstack segment host update <segment-uuid> <host-uuid> \
        --on-maintenance true
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Update a Host

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Update dialog">
        Navigate to **Instance HA > Hosts**. Click the **Update** action on a host row.
      </Step>

      <Step title="Edit host properties">
        | Field                 | Type            | Editable | Description           |
        | --------------------- | --------------- | -------- | --------------------- |
        | **Host Name**         | Text (disabled) | No       | Cannot be changed     |
        | **Reserved**          | Toggle switch   | Yes      | Standby designation   |
        | **Type**              | Text input      | Yes      | Host type identifier  |
        | **Control Attribute** | Text input      | Yes      | Monitoring attributes |
        | **On Maintenance**    | Toggle switch   | Yes      | Maintenance mode flag |

        <Warning>
          Setting **On Maintenance** to `Yes` prevents the host from being used as
          an evacuation target. The host will also not trigger recovery notifications
          while in maintenance mode.
        </Warning>
      </Step>

      <Step title="Save">
        Click **Confirm** to save the changes.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Set host to reserved" theme={null}
      openstack segment host update <segment-uuid> <host-uuid> \
        --reserved true
      ```

      ```bash title="Enable maintenance mode" theme={null}
      openstack segment host update <segment-uuid> <host-uuid> \
        --on-maintenance true
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Host Detail Page

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a host name in the Hosts list to open the detail page.

    | Field                 | Description                |
    | --------------------- | -------------------------- |
    | **UUID**              | Unique host identifier     |
    | **Failover Segment**  | Link to the parent segment |
    | **Reserved**          | Yes/No                     |
    | **On Maintenance**    | Yes/No                     |
    | **Type**              | Host type identifier       |
    | **Control Attribute** | Monitoring attributes      |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show host detail" theme={null}
    openstack segment host show <segment-uuid> <host-uuid>
    ```
  </Tab>
</Tabs>

***

## VM Moves

The VM Moves page provides a consolidated view of all VM evacuations across all
recent notifications.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance HA > VM Moves** in the sidebar.

    | Column               | Description                                              |
    | -------------------- | -------------------------------------------------------- |
    | **VM Name**          | Instance name (falls back to UUID)                       |
    | **Instance ID**      | VM UUID (copyable, truncated)                            |
    | **Notification**     | Parent notification UUID (copyable, truncated)           |
    | **Source Host**      | Failed compute host                                      |
    | **Destination Host** | Target recovery host, or `-` if pending                  |
    | **Type**             | Evacuation type (typically `evacuation`)                 |
    | **Status**           | Colored tag with icon (Succeeded/Failed/Running/Pending) |
    | **Start Time**       | When the evacuation started (default sort, descending)   |
    | **End Time**         | When the evacuation completed, or `-`                    |
    | **Message**          | Error message if failed (red text), or `-`               |

    Click **Refresh** to reload the latest data.

    <Note>
      VM Moves is a read-only page that aggregates evacuations from up to 50 recent
      notifications. No actions are available on individual VM moves.
    </Note>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List VM moves for a specific notification" theme={null}
    curl -s -H "X-Auth-Token: $TOKEN" \
      "$MASAKARI_ENDPOINT/v1/notifications/<notification-uuid>/vmoves" \
      | python3 -m json.tool
    ```
  </Tab>
</Tabs>

***

## Verify Instance Recovery

After a recovery event, verify that each affected instance has returned to `ACTIVE` status.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Instances**. Check the instance status column.

    Instances in a completed recovery show:

    * **Status**: `ACTIVE`
    * **Host**: Updated to the new compute host
    * **Power State**: `Running`

    <Check>All protected instances are `ACTIVE` on their new hosts after recovery completes.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List instances and their current hosts" theme={null}
    openstack server list --all-projects \
      -f table -c ID -c Name -c Status -c "OS-EXT-SRV-ATTR:host"
    ```

    ```bash title="Show a specific instance" theme={null}
    openstack server show <instance-id> \
      -f value -c status -c "OS-EXT-SRV-ATTR:host"
    ```

    <Check>Instance is `ACTIVE` and the host field reflects the target recovery node.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Troubleshooting" href="/services/instance-ha/user-guide/troubleshooting">
    Resolve failed notifications and manually recover instances
  </Card>

  <Card title="Recovery Workflows" href="/services/instance-ha/user-guide/recovery-workflows">
    Understand the recovery stages, methods, and expected timelines
  </Card>

  <Card title="Protection Segments" href="/services/instance-ha/user-guide/protection-segments">
    Create segments and manage host registrations
  </Card>

  <Card title="Instance HA Admin Guide" href="/services/instance-ha/admin-guide">
    Configure recovery policies, monitors, and engine settings
  </Card>
</CardGroup>


# Protection Segments
Source: https://docs.xloud.tech/services/instance-ha/user-guide/protection-segments

View and manage failover segments in Instance HA. Create segments, add hosts, and configure recovery methods.

## Overview

Protection segments are logical groupings of compute hosts that share a recovery policy.
When a host in a segment fails, Instance HA evacuates all protected instances to other
healthy hosts within the same segment. Segments define the fault domain boundary —
instances can only be recovered to hosts within the same segment.

<Note>
  **Prerequisites**

  * An active Xloud account with administrator access
  * Instance HA service enabled on the platform
  * Compute hosts available to register in segments
</Note>

***

## View Segments

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Segments">
        Navigate to
        **Instance HA > Segments** in the sidebar.
      </Step>

      <Step title="Review the segment list">
        The segment list displays all protection groups on the platform.

        | Column              | Description                                      |
        | ------------------- | ------------------------------------------------ |
        | **Name**            | Segment identifier (clickable to view details)   |
        | **UUID**            | Unique segment identifier                        |
        | **Recovery Method** | How instances are relocated after a host failure |
        | **Service Type**    | The type of service protected (e.g., `compute`)  |
        | **Description**     | Optional note about the segment's purpose        |

        Use the search/filter bar to narrow the list:

        | Filter              | Type        |
        | ------------------- | ----------- |
        | **Recovery Method** | Text search |
        | **Service Type**    | Text search |
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="List all segments" theme={null}
      openstack segment list
      ```

      ```bash title="Show segment details" theme={null}
      openstack segment show <segment-uuid>
      ```

      ```bash title="List hosts in a segment" theme={null}
      openstack segment host list <segment-uuid>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Create a Segment

The Dashboard provides a two-step wizard for creating a segment and adding hosts
in a single workflow.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Create Segment wizard">
        Navigate to **Instance HA > Segments** and click **Create Segment**.
        The wizard opens with two steps.
      </Step>

      <Step title="Step 1 — Configure the segment">
        Fill in the segment details:

        | Field               | Type       | Required | Default   | Description                                |
        | ------------------- | ---------- | -------- | --------- | ------------------------------------------ |
        | **Segment Name**    | Text input | Yes      | —         | Human-readable name for the segment        |
        | **Recovery Method** | Dropdown   | Yes      | `auto`    | Algorithm for selecting evacuation targets |
        | **Service Type**    | Text input | Yes      | `compute` | Service type (fixed, not editable)         |
        | **Description**     | Text area  | No       | —         | Optional notes about the segment           |

        **Recovery Method options**:

        | Value           | Description                                   |
        | --------------- | --------------------------------------------- |
        | `auto`          | Evacuate to any healthy host in the segment   |
        | `auto_priority` | Evacuate using priority-based host selection  |
        | `reserved_host` | Evacuate only to pre-designated standby hosts |
        | `rh_priority`   | Prefer reserved hosts, fall back to any host  |

        Click **Next** to proceed. The segment is created at this step.
      </Step>

      <Step title="Step 2 — Add hosts to the segment">
        A table of available compute hosts is displayed. Select one or more hosts
        to add to the segment.

        The host selection table shows:

        | Column                 | Description                             | Editable            |
        | ---------------------- | --------------------------------------- | ------------------- |
        | **Name**               | Compute host name                       | No                  |
        | **Zone**               | Availability zone of the host           | No                  |
        | **Updated**            | Last update timestamp                   | No                  |
        | **Reserved**           | Designate as standby host for failover  | Yes (toggle switch) |
        | **Type**               | Host type identifier                    | Yes (text input)    |
        | **Control Attributes** | Monitoring attributes (e.g., SSH, IPMI) | Yes (text input)    |
        | **On Maintenance**     | Whether the host is in maintenance mode | Yes (toggle switch) |

        <Note>
          Only compute hosts that are not already assigned to another segment
          appear in the selection table. Each host can belong to only one segment.
        </Note>

        <Tip>
          Set **Reserved** to `Yes` for hosts that should remain idle as standby
          capacity. Reserved hosts are used by the `reserved_host` and `rh_priority`
          recovery methods.
        </Tip>

        Select your hosts and click **Confirm** to complete the wizard.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create the segment">
        ```bash title="Create a failover segment" theme={null}
        openstack segment create \
          --recovery-method auto \
          --service-type compute \
          --description "Production compute segment" \
          prod-segment
        ```
      </Step>

      <Step title="Add hosts to the segment">
        ```bash title="Add a compute host" theme={null}
        openstack segment host create <segment-uuid> \
          --name <compute-hostname> \
          --type compute \
          --control-attributes "SSH" \
          --reserved false \
          --on-maintenance false
        ```

        Repeat for each host in the segment.
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## View Segment Details

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a segment name in the list to open the detail page. Two tabs are available:

    **Detail tab** — Segment configuration:

    | Field               | Description                            |
    | ------------------- | -------------------------------------- |
    | **Recovery Method** | Current recovery algorithm             |
    | **Service Type**    | Protected service type                 |
    | **Enabled**         | Whether the segment is active (Yes/No) |
    | **Created At**      | Creation timestamp                     |
    | **Updated At**      | Last modification timestamp            |

    **Hosts tab** — All hosts registered in this segment:

    | Column                | Description                                      |
    | --------------------- | ------------------------------------------------ |
    | **Name**              | Host name (clickable to view host details)       |
    | **UUID**              | Unique host identifier                           |
    | **Reserved**          | Whether the host is a standby node (Yes/No)      |
    | **Type**              | Host type identifier                             |
    | **Control Attribute** | Monitoring attributes                            |
    | **On Maintenance**    | Whether the host is in maintenance mode (Yes/No) |
    | **Failover Segment**  | Link back to the parent segment                  |
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Show segment details" theme={null}
      openstack segment show <segment-uuid>
      ```

      ```bash title="List hosts in the segment" theme={null}
      openstack segment host list <segment-uuid>
      ```

      ```bash title="Show a specific host" theme={null}
      openstack segment host show <segment-uuid> <host-uuid>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Edit a Segment

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the edit dialog">
        Navigate to **Instance HA > Segments**. Click the **Update** action
        (the first action button) on the segment row.
      </Step>

      <Step title="Update segment fields">
        You can update:

        | Field               | Editable                       |
        | ------------------- | ------------------------------ |
        | **Segment Name**    | Yes                            |
        | **Recovery Method** | Yes (same 4 options as create) |
        | **Description**     | Yes                            |
      </Step>

      <Step title="Save changes">
        Click **Confirm** to save the updated segment.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Update segment recovery method" theme={null}
    openstack segment update <segment-uuid> \
      --recovery-method reserved_host
    ```
  </Tab>
</Tabs>

***

## Add a Host to an Existing Segment

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Add Host dialog">
        Navigate to **Instance HA > Segments**. Click the **More** dropdown on a
        segment row and select **Add Host**.
      </Step>

      <Step title="Configure the host">
        | Field                  | Type            | Required | Default         | Description                         |
        | ---------------------- | --------------- | -------- | --------------- | ----------------------------------- |
        | **Segment Name**       | Text (disabled) | —        | Current segment | Read-only reference                 |
        | **Host Name**          | Dropdown        | Yes      | —               | Select from available compute hosts |
        | **Reserved**           | Toggle switch   | No       | Off             | Standby host designation            |
        | **Type**               | Text input      | Yes      | —               | Host type identifier                |
        | **Control Attributes** | Text input      | Yes      | —               | Monitoring attributes               |
        | **On Maintenance**     | Toggle switch   | No       | Off             | Maintenance mode flag               |

        <Note>
          Only compute hosts not already assigned to any segment appear in the
          Host Name dropdown.
        </Note>
      </Step>

      <Step title="Add the host">
        Click **Confirm**. The host appears in the segment's Hosts tab.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Add host to segment" theme={null}
    openstack segment host create <segment-uuid> \
      --name <compute-hostname> \
      --type compute \
      --control-attributes "SSH" \
      --reserved false \
      --on-maintenance false
    ```
  </Tab>
</Tabs>

***

## Delete a Segment

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Instance HA > Segments**. Click the **More** dropdown on a
    segment row and select **Delete**, or select multiple segments using
    checkboxes and click **Delete** in the batch actions bar.

    Confirm the deletion in the dialog.

    <Danger>
      Deleting a segment removes all host registrations within it. Instances
      on those hosts will no longer be automatically recovered on host failure.
    </Danger>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a segment" theme={null}
    openstack segment delete <segment-uuid>
    ```
  </Tab>
</Tabs>

***

## Recovery Method Reference

| Method          | Evacuation Target                      | Capacity Guarantee                   | Use Case                             |
| --------------- | -------------------------------------- | ------------------------------------ | ------------------------------------ |
| `auto`          | Any available host in the segment      | None — depends on current load       | General-purpose workloads            |
| `auto_priority` | Priority-based host selection          | None — best-effort prioritization    | Workloads with preferred targets     |
| `reserved_host` | Reserved standby hosts only            | Guaranteed — reserved hosts are idle | SLA-critical workloads               |
| `rh_priority`   | Reserved hosts first; any host if full | Best-effort with preference          | Mixed critical and general workloads |

<Tip>
  For most deployments, `auto` is the recommended starting point. Use
  `reserved_host` or `rh_priority` only when you have dedicated standby
  capacity and strict recovery time requirements.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Recovery Workflows" href="/services/instance-ha/user-guide/recovery-workflows">
    Understand how the recovery engine executes failover and tracks progress
  </Card>

  <Card title="Monitoring Status" href="/services/instance-ha/user-guide/monitoring-status">
    Track live and historical recovery events in the Dashboard
  </Card>

  <Card title="How It Works" href="/services/instance-ha/user-guide/how-it-works">
    Understand the end-to-end detection and recovery architecture
  </Card>

  <Card title="Instance HA Admin Guide" href="/services/instance-ha/admin-guide">
    Configure recovery policies, monitors, and engine settings
  </Card>
</CardGroup>


# Recovery Workflows
Source: https://docs.xloud.tech/services/instance-ha/user-guide/recovery-workflows

Understand Instance HA recovery workflows — real-time VM evacuations, VM Moves, and recovery progress.

## Overview

A recovery workflow is the ordered sequence of actions the Instance HA engine takes
after a host failure notification is received. The workflow covers instance evacuation,
restart on a healthy host, and post-recovery status reporting. The Dashboard provides
real-time tracking of each VM evacuation through the Recovery Progress tab and a
consolidated VM Moves page.

<Note>
  **Prerequisites**

  * Instance HA protection enabled on your instances
  * At least one failover segment configured with registered hosts
</Note>

***

## Recovery Workflow Stages

```mermaid theme={null}
graph TD
    A[Host Monitor: fault detected] --> B[Notification Engine: event received]
    B --> C{Segment enabled?}
    C -->|No| D[Notification ignored]
    C -->|Yes| E[Recovery Engine: identify protected instances]
    E --> F{Recovery method}
    F -->|auto| G[Select any healthy host in segment]
    F -->|auto_priority| GA[Select host by priority]
    F -->|reserved_host| H[Select reserved standby host]
    F -->|rh_priority| I[Prefer reserved; fall back to any]
    G --> J[Evacuate instances via Compute API]
    GA --> J
    H --> J
    I --> J
    J --> K[Instances restarted on target host]
    K --> L[Notification status: finished]
```

***

## Recovery Progress — Real-Time Tracking

When a recovery is in progress, the Dashboard provides real-time tracking of each
individual VM evacuation through the **Recovery Progress** tab on the notification
detail page.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the notification detail">
        Navigate to **Instance HA > Notifications**. Click a notification UUID
        to open the detail page.
      </Step>

      <Step title="View the Recovery Progress tab">
        Click the **Recovery Progress** tab. This tab shows:

        **Summary card** at the top:

        | Field                   | Description                                    |
        | ----------------------- | ---------------------------------------------- |
        | **Notification Status** | Current status as a colored tag                |
        | **Total VMs**           | Total number of VMs being evacuated            |
        | **Succeeded**           | Count of successfully recovered VMs (green)    |
        | **Failed**              | Count of failed evacuations (red, if any)      |
        | **Progress**            | Circular progress indicator showing completion |

        **VM Evacuations table** below the summary:

        | Column               | Description                                          |
        | -------------------- | ---------------------------------------------------- |
        | **VM Name**          | Instance name (falls back to UUID if no name)        |
        | **Source Host**      | The failed compute host                              |
        | **Destination Host** | Target recovery host ("Pending" if not yet assigned) |
        | **Type**             | Evacuation type (typically `evacuation`)             |
        | **Status**           | Evacuation status with icon                          |
        | **Start Time**       | When the evacuation started                          |
        | **End Time**         | When the evacuation completed                        |
        | **Message**          | Error message if the evacuation failed (red text)    |

        **VM evacuation status values**:

        | Status        | Color | Icon            | Meaning                                        |
        | ------------- | ----- | --------------- | ---------------------------------------------- |
        | **Pending**   | Grey  | Clock           | Evacuation queued, not yet started             |
        | **Running**   | Blue  | Loading spinner | Evacuation in progress                         |
        | **Succeeded** | Green | Check circle    | VM successfully recovered                      |
        | **Failed**    | Red   | Close circle    | Evacuation failed — manual intervention needed |

        <Tip>
          When a notification is in **Running** status, the Recovery Progress tab
          **auto-refreshes every 5 seconds**, showing a "Auto-refreshing every 5s"
          indicator. You can watch evacuations complete in real time.
        </Tip>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Show notification detail" theme={null}
      openstack notification show <notification-uuid>
      ```

      ```bash title="List VM moves for a notification" theme={null}
      # VM moves are available via the Masakari API
      curl -s -H "X-Auth-Token: $TOKEN" \
        $MASAKARI_ENDPOINT/v1/notifications/<notification-uuid>/vmoves | python3 -m json.tool
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## VM Moves — Consolidated View

The VM Moves page provides a single view of all VM evacuations across all
notifications, making it easy to review recovery history.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to VM Moves">
        Navigate to **Instance HA > VM Moves** in the sidebar.
      </Step>

      <Step title="Review the VM moves list">
        The page displays all VM evacuations from recent notifications (up to the
        last 50 notifications), sorted by start time.

        | Column               | Description                                              |
        | -------------------- | -------------------------------------------------------- |
        | **VM Name**          | Instance name (falls back to UUID)                       |
        | **Instance ID**      | VM UUID (copyable, truncated display)                    |
        | **Notification**     | Parent notification UUID (copyable, truncated display)   |
        | **Source Host**      | The failed compute host                                  |
        | **Destination Host** | Target recovery host, or `-` if pending                  |
        | **Type**             | Evacuation type (typically `evacuation`)                 |
        | **Status**           | Colored tag with icon (Succeeded/Failed/Running/Pending) |
        | **Start Time**       | When the evacuation started (default sort, descending)   |
        | **End Time**         | When the evacuation completed, or `-`                    |
        | **Message**          | Error message if failed (red text)                       |

        Use the **Refresh** button to reload the latest data.

        <Note>
          The VM Moves page is read-only — it provides a consolidated view for
          monitoring and auditing. No actions are available on individual VM moves.
        </Note>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List VM moves across all recent notifications" theme={null}
    for notif in $(curl -s -H "X-Auth-Token: $TOKEN" \
      "$MASAKARI_ENDPOINT/v1/notifications?sort_key=updated_at&sort_dir=desc&limit=10" \
      | python3 -c "import sys,json; [print(n['notification_uuid']) for n in json.load(sys.stdin)['notifications']]"); do
      echo "=== Notification: $notif ==="
      curl -s -H "X-Auth-Token: $TOKEN" \
        "$MASAKARI_ENDPOINT/v1/notifications/$notif/vmoves" | python3 -m json.tool
    done
    ```
  </Tab>
</Tabs>

***

## Recovery Methods in Detail

<AccordionGroup>
  <Accordion title="auto — Evacuate to Any Host">
    The `auto` method selects the healthiest available host in the segment based on
    current vCPU and memory availability. Instances are distributed across multiple
    target hosts if no single host has sufficient capacity for all evacuees.

    **Characteristics**:

    * No pre-reserved capacity required
    * Recovery succeeds as long as aggregate free capacity in the segment is sufficient
    * Most flexible option for mixed workloads

    **Risk**: Recovery may fail if all remaining hosts are near capacity when the fault
    occurs. Maintain a minimum headroom of 20-30% unused capacity across the segment.
  </Accordion>

  <Accordion title="auto_priority — Priority-Based Selection">
    Similar to `auto`, but uses priority-based host selection. The engine evaluates
    hosts based on configured priority attributes and selects the highest-priority
    available host for each evacuation.

    **Characteristics**:

    * Allows administrators to influence target host selection
    * Still best-effort — no guaranteed standby capacity
    * Useful when certain hosts are preferred targets
  </Accordion>

  <Accordion title="reserved_host — Dedicated Standby">
    One or more hosts in the segment are designated as reserved standby nodes. These
    hosts remain idle until a failover event occurs, ensuring guaranteed capacity for
    recovery.

    **Characteristics**:

    * Guaranteed recovery capacity regardless of current cluster load
    * Reserved hosts do not accept regular instance scheduling
    * Higher infrastructure cost (idle nodes consume resources)

    **Best for**: Mission-critical applications, financial systems, and workloads with
    strict RTO requirements.
  </Accordion>

  <Accordion title="rh_priority — Prefer Reserved, Fall Back">
    The engine attempts recovery to reserved hosts first. If reserved hosts are full,
    it falls back to the `auto` behaviour and selects any available host in the segment.

    **Characteristics**:

    * Balances guaranteed capacity for high-priority workloads with flexibility
    * Works well in mixed segments that contain both critical and standard workloads
    * Requires at least one reserved host in the segment

    **Best for**: Environments with heterogeneous workloads where some instances need
    guaranteed failover and others can tolerate best-effort recovery.
  </Accordion>
</AccordionGroup>

***

## Instance State During Recovery

| Phase                  | Instance Status | Description                                                     |
| ---------------------- | --------------- | --------------------------------------------------------------- |
| Normal operation       | `ACTIVE`        | Instance running on original host                               |
| Fault detected         | `UNKNOWN`       | Host unreachable; compute service cannot confirm instance state |
| Evacuation in progress | `MIGRATING`     | Instance being moved to target host                             |
| Restarting             | `BUILD`         | Instance starting up on target host                             |
| Recovery complete      | `ACTIVE`        | Instance fully operational on new host                          |
| Recovery failed        | `ERROR`         | Manual intervention required                                    |

<Warning>
  Instances in `ERROR` or `SHUTOFF` state at the time of the host failure may not be
  automatically recovered, depending on your administrator's configuration.
</Warning>

***

## Notification Status Reference

Every recovery event creates a notification record. The notification `status` field
tracks progress through the workflow.

| Status     | Color  | Meaning                                                    |
| ---------- | ------ | ---------------------------------------------------------- |
| `new`      | Blue   | Fault notification received; recovery not yet started      |
| `running`  | Orange | Recovery workflow in progress                              |
| `finished` | Green  | All instances recovered successfully                       |
| `error`    | Red    | Recovery encountered errors                                |
| `failed`   | Red    | Recovery failed completely                                 |
| `ignored`  | Grey   | Notification was de-duplicated or the segment was disabled |

***

## Recovery Time Expectations

Recovery time depends on several factors:

| Factor                              | Typical Impact                                                  |
| ----------------------------------- | --------------------------------------------------------------- |
| Host monitor detection timeout      | 30-120 seconds to declare host unreachable                      |
| Instance count on failed host       | Each instance adds 30-120 seconds to total recovery time        |
| Instance disk size (shared storage) | Minimal — shared storage volumes are reattached, not copied     |
| Target host boot overhead           | Constant per instance — determined by instance flavor and image |

<Tip>
  Use shared storage (Xloud Distributed Storage) for all protected instances. Instances
  backed by local ephemeral disk cannot be evacuated and will be lost on host failure.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Monitoring Status" href="/services/instance-ha/user-guide/monitoring-status">
    View notifications, hosts, and VM moves in the Dashboard
  </Card>

  <Card title="Troubleshooting" href="/services/instance-ha/user-guide/troubleshooting">
    Resolve stuck or failed recovery workflows
  </Card>

  <Card title="Protection Segments" href="/services/instance-ha/user-guide/protection-segments">
    Create segments and manage host registrations
  </Card>

  <Card title="Instance HA Admin Guide" href="/services/instance-ha/admin-guide">
    Configure recovery policies, monitors, and engine settings
  </Card>
</CardGroup>


# Instance HA Troubleshooting — User Guide
Source: https://docs.xloud.tech/services/instance-ha/user-guide/troubleshooting

Resolve Instance HA issues — stuck notifications, failed recoveries, and manual evacuation procedures.

## Overview

This page covers common Instance HA issues encountered by project users — instances that
did not recover, notifications stuck in error or running states, and protection settings
that are not visible in the Dashboard. For platform-level issues such as monitor failures
or engine misconfiguration, refer to the
[Instance HA Admin Troubleshooting](/services/instance-ha/admin-guide/troubleshooting) guide.

<Note>
  **Prerequisites**

  * Project access to the Xloud Dashboard or CLI
  * Knowledge of the affected instance IDs and the compute host involved
</Note>

***

## Common Issues

<AccordionGroup>
  <Accordion title="Instance did not recover after host failure" icon="circle-alert">
    **Cause**: Instance HA protection may not be enabled for the instance, the compute
    host may not be registered in any segment, or the segment may be disabled.

    **Resolution**:

    Confirm the segment is enabled and the host is registered:

    ```bash title="Check segment status" theme={null}
    openstack segment list
    ```

    ```bash title="List hosts in segment" theme={null}
    openstack segment host list <segment-uuid>
    ```

    If the failed host is missing from the segment, contact your administrator to
    register it. Your administrator can configure this through [XDeploy](/deployment). If the segment is disabled (`enabled: False`), your administrator
    must re-enable it.

    <Tip>
      After the root cause is resolved, manually evacuate the instance to restore service:

      ```bash title="Manual evacuation" theme={null}
      openstack server evacuate <instance-id> --host <healthy-host>
      ```
    </Tip>
  </Accordion>

  <Accordion title="Notification status is error" icon="circle-x">
    **Cause**: Automatic recovery failed. Common causes: insufficient capacity on remaining
    hosts, an instance stuck in `ERROR` state that the engine cannot recover, or a network
    issue during evacuation.

    **Resolution**:

    ```bash title="Show notification detail" theme={null}
    openstack notification show <notification-uuid>
    ```

    Review the payload for specific failure information. Then check instance state:

    ```bash title="Show instance status" theme={null}
    openstack server show <instance-id> -f value -c status
    ```

    If the instance is in `ERROR` state, attempt a manual reset and evacuation:

    ```bash title="Reset instance state" theme={null}
    openstack server set --state active <instance-id>
    ```

    ```bash title="Manually evacuate" theme={null}
    openstack server evacuate <instance-id> --host <healthy-host>
    ```

    Contact your administrator if the instance cannot be recovered through the above steps. Your administrator can configure this through [XDeploy](/deployment).
  </Accordion>

  <Accordion title="Notification stuck in running state" icon="clock">
    **Cause**: The recovery engine is waiting for the target host to accept the instance,
    a workflow step has timed out, or the target host is under load.

    **Resolution**:

    Check how long the notification has been in `running` state:

    ```bash title="Show notification timestamps" theme={null}
    openstack notification show <notification-uuid> \
      -f value -c generated_time -c status
    ```

    If the notification has been `running` for more than 10 minutes, contact your
    administrator. They can inspect the Instance HA engine logs and reset the workflow
    if it has genuinely stalled. Your administrator can configure this through [XDeploy](/deployment).
  </Accordion>

  <Accordion title="Protection segment not visible in Dashboard" icon="eye-off">
    **Cause**: No failover segment has been created for your environment, or the segments
    that exist have not been made accessible to your project.

    **Resolution**: Contact your Xloud administrator and request that a failover segment
    be created and that the compute hosts used by your project be registered. Your administrator can configure this through [XDeploy](/deployment). See the
    [Instance HA Admin Guide — Failover Segments](/services/instance-ha/admin-guide/failover-segments)
    for the configuration steps.
  </Accordion>

  <Accordion title="Instance shows UNKNOWN status after host failure" icon="circle-help">
    **Cause**: The compute service cannot confirm the instance state because the host
    is unreachable. This is expected immediately after a host fault is detected.

    **Resolution**: Wait for the recovery workflow to complete. The instance transitions
    from `UNKNOWN` → `MIGRATING` → `BUILD` → `ACTIVE` as recovery proceeds.

    If the instance remains `UNKNOWN` for more than 5 minutes without a recovery
    notification being created, verify that the failed host is registered in an enabled
    segment and that the host monitor can reach the Instance HA notification endpoint.
  </Accordion>

  <Accordion title="Instance restarted on wrong host" icon="server">
    **Cause**: The `auto` recovery method placed the instance on any available host
    rather than a specific preferred host. This is expected behaviour for the `auto` method.

    **Resolution**: If your workloads require guaranteed placement on specific hosts, ask
    your administrator to configure a segment with `reserved_host` or `rh_priority`
    recovery method and designate the preferred host as a reserved standby.
  </Accordion>
</AccordionGroup>

***

## Manual Recovery Procedure

If automatic recovery fails, use the following procedure to restore service manually.

<Steps>
  <Step title="Identify the affected instances" icon="search">
    ```bash title="Find instances on the failed host" theme={null}
    openstack server list \
      --host <failed-hostname> \
      -f table -c ID -c Name -c Status
    ```
  </Step>

  <Step title="Reset instance state if stuck in ERROR" icon="rotate">
    ```bash title="Reset instance state" theme={null}
    openstack server set --state active <instance-id>
    ```
  </Step>

  <Step title="Evacuate to a healthy host" icon="arrow-right">
    ```bash title="Evacuate to a specific host" theme={null}
    openstack server evacuate <instance-id> --host <healthy-host>
    ```

    Omit `--host` to let the scheduler choose any available host in the segment:

    ```bash title="Evacuate to any available host" theme={null}
    openstack server evacuate <instance-id>
    ```
  </Step>

  <Step title="Verify recovery" icon="circle-check">
    ```bash title="Confirm instance is ACTIVE" theme={null}
    openstack server show <instance-id> \
      -f value -c status -c "OS-EXT-SRV-ATTR:host"
    ```

    <Check>Instance shows `ACTIVE` and the host field reflects the new compute node.</Check>
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Monitoring Status" href="/services/instance-ha/user-guide/monitoring-status">
    Track live and historical recovery notifications.
  </Card>

  <Card title="Recovery Workflows" href="/services/instance-ha/user-guide/recovery-workflows">
    Understand recovery methods and workflow stages.
  </Card>

  <Card title="Instance HA Admin Troubleshooting" href="/services/instance-ha/admin-guide/troubleshooting">
    Administrator-level troubleshooting for monitors, engine failures, and capacity issues.
  </Card>

  <Card title="Compute User Guide" href="/services/compute/user-guide">
    Manage and recover instances manually using core compute operations.
  </Card>
</CardGroup>


# VM Moves
Source: https://docs.xloud.tech/services/instance-ha/user-guide/vm-moves

Track VM evacuations across recovery events. View source/destination hosts, status, timing, and error details.

## Overview

The VM Moves page provides a consolidated view of every VM evacuation performed by
the Instance HA engine across all recent recovery notifications. Unlike the
per-notification Recovery Progress tab, VM Moves aggregates evacuations from up to
50 recent notifications into a single sortable, searchable list — making it the
primary tool for recovery auditing and troubleshooting.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * Instance HA service enabled with at least one completed or in-progress recovery event
</Note>

***

## View VM Moves

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to VM Moves">
        Navigate to **Instance HA > VM Moves** in the sidebar.

        The page loads all VM evacuations from recent notifications automatically.
      </Step>

      <Step title="Review the evacuation list">
        The table shows every individual VM evacuation, sorted by start time
        (most recent first).

        | Column               | Description                                                              |
        | -------------------- | ------------------------------------------------------------------------ |
        | **VM Name**          | Instance name. Falls back to instance UUID if no name is set             |
        | **Instance ID**      | VM UUID (copyable, truncated display showing first 8 characters)         |
        | **Notification**     | Parent notification UUID (copyable, truncated display)                   |
        | **Source Host**      | The failed compute host the VM was evacuated from                        |
        | **Destination Host** | The healthy host the VM was moved to. Shows `-` if not yet assigned      |
        | **Type**             | Evacuation type — typically `evacuation`                                 |
        | **Status**           | Colored tag with icon indicating the evacuation result                   |
        | **Start Time**       | When the evacuation started (default sort column, descending)            |
        | **End Time**         | When the evacuation completed, or `-` if still in progress               |
        | **Message**          | Error details if the evacuation failed (displayed in red), otherwise `-` |
      </Step>

      <Step title="Understand evacuation statuses">
        Each VM evacuation has one of four statuses:

        | Status        | Color | Icon            | Meaning                                                  |
        | ------------- | ----- | --------------- | -------------------------------------------------------- |
        | **Pending**   | Grey  | Clock           | Evacuation is queued but has not started                 |
        | **Running**   | Blue  | Loading spinner | Evacuation is actively in progress                       |
        | **Succeeded** | Green | Check circle    | VM was successfully recovered on the destination host    |
        | **Failed**    | Red   | Close circle    | Evacuation failed — check the Message column for details |

        <Warning>
          VMs with **Failed** status require manual intervention. Check the error
          message, verify the destination host has sufficient capacity, and attempt
          a manual evacuation if needed. See
          [Troubleshooting](/services/instance-ha/user-guide/troubleshooting) for
          resolution steps.
        </Warning>
      </Step>

      <Step title="Refresh the data">
        Click the **Refresh** button in the page header to reload the latest
        VM move data. This fetches evacuations from the most recent 50 notifications.

        <Tip>
          During an active recovery, refresh periodically to see new VM evacuations
          appear as the engine processes each instance. For real-time auto-refreshing,
          use the **Recovery Progress** tab on the individual notification detail page
          instead.
        </Tip>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    The CLI does not have a dedicated command for listing VM moves across all
    notifications. Use the Masakari API directly:

    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="List VM moves for a specific notification" theme={null}
    curl -s -H "X-Auth-Token: $TOKEN" \
      "$MASAKARI_ENDPOINT/v1/notifications/<notification-uuid>/vmoves" \
      | python3 -m json.tool
    ```

    ```bash title="List VM moves across recent notifications" theme={null}
    for notif in $(curl -s -H "X-Auth-Token: $TOKEN" \
      "$MASAKARI_ENDPOINT/v1/notifications?sort_key=updated_at&sort_dir=desc&limit=10" \
      | python3 -c "import sys,json; [print(n['notification_uuid']) for n in json.load(sys.stdin)['notifications']]"); do
      echo "=== Notification: $notif ==="
      curl -s -H "X-Auth-Token: $TOKEN" \
        "$MASAKARI_ENDPOINT/v1/notifications/$notif/vmoves" | python3 -m json.tool
    done
    ```
  </Tab>
</Tabs>

***

## VM Moves vs Recovery Progress

The Dashboard provides two ways to view VM evacuations. Choose based on your use case:

| Feature           | VM Moves Page                                          | Recovery Progress Tab                                   |
| ----------------- | ------------------------------------------------------ | ------------------------------------------------------- |
| **Location**      | Instance HA > VM Moves                                 | Notification detail > Recovery Progress                 |
| **Scope**         | All evacuations across up to 50 notifications          | Single notification only                                |
| **Auto-refresh**  | Manual refresh button                                  | Auto-refreshes every 5 seconds when running             |
| **Summary stats** | Not shown                                              | Total, succeeded, failed counts with progress indicator |
| **Best for**      | Auditing, historical review, cross-notification search | Real-time monitoring of an active recovery              |

<Tip>
  Use **Recovery Progress** when you want to watch a single recovery event complete
  in real time. Use **VM Moves** when you need to review evacuation history across
  multiple events, or search for a specific VM's recovery outcome.
</Tip>

***

## Common Scenarios

<AccordionGroup>
  <Accordion title="Find all failed evacuations">
    Open **Instance HA > VM Moves** and look for rows with a red **Failed** status
    tag. The **Message** column shows the error reason. Common causes:

    * **Insufficient capacity**: No destination host has enough vCPU/memory
    * **Shared storage not available**: Instance uses local ephemeral disk
    * **Compute service down**: Target host's nova-compute is not running

    For each failed VM, attempt a manual evacuation:

    ```bash title="Manually evacuate a failed instance" theme={null}
    openstack server evacuate <instance-id> --host <target-host>
    ```
  </Accordion>

  <Accordion title="Verify a specific VM was recovered">
    Open **Instance HA > VM Moves** and locate the VM by name or instance ID.
    Check that:

    * **Status** is `Succeeded` (green)
    * **Destination Host** shows a valid compute host
    * **End Time** is populated

    Then verify the instance is running:

    ```bash title="Confirm instance is active" theme={null}
    openstack server show <instance-id> -c status -c "OS-EXT-SRV-ATTR:host"
    ```

    <Check>Status is `ACTIVE` and the host matches the Destination Host from VM Moves.</Check>
  </Accordion>

  <Accordion title="Review recovery history for a time period">
    Open **Instance HA > VM Moves** and sort by **Start Time**. The default
    sort is descending (most recent first). Scroll through to find evacuations
    in your time window.

    For CLI-based historical analysis:

    ```bash title="Export VM moves to JSON for analysis" theme={null}
    curl -s -H "X-Auth-Token: $TOKEN" \
      "$MASAKARI_ENDPOINT/v1/notifications?sort_key=updated_at&sort_dir=desc&limit=50" \
      | python3 -c "
    import sys, json, requests, os
    token = os.environ['TOKEN']
    endpoint = os.environ['MASAKARI_ENDPOINT']
    notifs = json.load(sys.stdin)['notifications']
    all_moves = []
    for n in notifs:
        r = requests.get(f'{endpoint}/v1/notifications/{n[\"notification_uuid\"]}/vmoves',
                        headers={'X-Auth-Token': token})
        all_moves.extend(r.json().get('vmoves', []))
    json.dump(all_moves, sys.stdout, indent=2)
    " > vm-moves-history.json
    ```
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Recovery Workflows" href="/services/instance-ha/user-guide/recovery-workflows">
    Understand recovery methods and the Recovery Progress real-time view
  </Card>

  <Card title="Monitoring Status" href="/services/instance-ha/user-guide/monitoring-status">
    View notifications, hosts, and notification details
  </Card>

  <Card title="Troubleshooting" href="/services/instance-ha/user-guide/troubleshooting">
    Resolve failed evacuations and stuck recovery workflows
  </Card>

  <Card title="Protection Segments" href="/services/instance-ha/user-guide/protection-segments">
    Manage segments and host registrations
  </Card>
</CardGroup>


# Key Manager
Source: https://docs.xloud.tech/services/key-manager

Securely store and manage secrets, certificates, and encryption keys in your Xloud private cloud with Xloud Key Manager — enterprise key management as a.

Centralized, secure secret and certificate management for your entire Xloud cloud infrastructure.

<Card title="Xloud Key Manager — Enterprise KMS" icon="lock" href="https://xloud.tech/xavs">
  Product details and datasheet on xloud.tech
</Card>

***

<p>Xloud Key Manager</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/key-manager/user-guide">
    Store secrets and credentials, manage certificate containers, issue certificate orders, and configure access control lists for your Xloud Key Manager resources.
  </Card>

  <Card title="Admin Guide" icon="shield-check" href="/services/key-manager/admin-guide">
    Configure secret store backends, manage transport keys, enforce quotas, and apply security hardening policies for the Key Manager service.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/key-manager/cli-reference">
    `openstack secret` commands for managing secrets, containers, orders, and ACLs from the command line.
  </Card>

  <Card title="TLS Integration" icon="shield">
    Store TLS certificates in Key Manager and reference them directly from Load Balancer HTTPS listeners for centralized certificate lifecycle management.
  </Card>
</CardGroup>

***

<p>Key Features</p>

<CardGroup>
  <Card title="Secret Storage" icon="vault" href="/services/key-manager/store-secrets">
    Securely store passwords, API keys, encryption keys, and arbitrary binary secrets. All secrets are encrypted at rest using the configured backend store.
  </Card>

  <Card title="Certificate Management" icon="certificate" href="/services/key-manager/certificates">
    Store and manage TLS/SSL certificates with their associated private keys and certificate chains. Reference directly from Load Balancer and other services.
  </Card>

  <Card title="Access Control Lists" icon="list-checks" href="/services/key-manager/acl">
    Fine-grained ACLs control which users and projects can read or manage each secret. Delegate access without exposing credentials.
  </Card>

  <Card title="Certificate Orders" icon="clipboard-list" href="/services/key-manager/certificates">
    Automate certificate issuance through configured Certificate Authority plugins. Track order status and retrieve issued certificates programmatically.
  </Card>

  <Card title="Transport Key Encryption" icon="key" href="/services/key-manager/transport-keys">
    Client-side secret encryption using transport keys prevents secrets from ever appearing in plaintext on the network — even during upload.
  </Card>

  <Card title="Multi-Backend Support" icon="database" href="/services/key-manager/backend-config">
    Plug in industry-standard backends including local encryption, hardware security modules (HSMs), and KMIP-compliant key management appliances.
  </Card>
</CardGroup>

***

<p>Key Manager Components</p>

| Component     | Description                                                                                      |
| ------------- | ------------------------------------------------------------------------------------------------ |
| Secret        | An encrypted payload — passwords, API keys, certificates, private keys, or arbitrary binary data |
| Container     | A named grouping of related secrets (e.g., a certificate + private key + CA chain)               |
| Order         | An asynchronous request to generate or issue a key or certificate via a CA plugin                |
| Transport Key | An asymmetric key pair used to encrypt secrets client-side before transmission                   |
| ACL           | Access Control List defining per-user and per-project read/write permissions on a secret         |
| Secret Store  | The backend encryption provider (simple crypto, PKCS#11 HSM, KMIP)                               |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Load Balancer" icon="combine" href="/services/load-balancer">
    Reference TLS certificate containers in HTTPS listener configuration
  </Card>

  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Encrypt instance storage volumes with keys managed in Key Manager
  </Card>

  <Card title="Xloud DNS" icon="globe" href="/services/dns">
    Store DNSSEC signing keys as secrets for automated zone signing
  </Card>

  <Card title="Xloud Object Storage" icon="box" href="/services/object-storage">
    Server-side encryption of object containers with customer-managed keys
  </Card>

  <Card title="Xloud Identity" icon="fingerprint" href="/cli-setup">
    RBAC policies and trust delegation for Key Manager resource access
  </Card>

  <Card title="Xloud Block Storage" icon="hard-drive" href="/services/compute">
    Volume encryption using keys managed and rotated through Key Manager
  </Card>
</CardGroup>

***

<p>Getting Started</p>

<CardGroup>
  <Card title="Authentication" icon="key" href="/cli-setup">
    Configure Dashboard access and CLI credentials before working with Key Manager
  </Card>

  <Card title="User Guide" icon="book-open" href="/services/key-manager/user-guide">
    Step-by-step instructions for storing your first secret
  </Card>
</CardGroup>


# Secret Access Control (ACL)
Source: https://docs.xloud.tech/services/key-manager/acl

Configure ACLs on secrets and containers in Xloud Key Manager to grant per-user and per-project access without sharing credentials.

## Overview

By default, secrets and containers are private to the project that created them. ACLs
grant specific users read access to secrets and containers across project boundaries,
enabling secure credential sharing without exposing the payload itself.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## ACL Concepts

| Concept            | Description                                                                                         |
| ------------------ | --------------------------------------------------------------------------------------------------- |
| **read**           | Allows the grantee to retrieve the secret payload or container contents                             |
| **per-user ACL**   | Grants access to specific user IDs — most restrictive and recommended for sensitive secrets         |
| **project-access** | Grants all users in the secret's own project read access — use only for non-sensitive shared config |

***

## View Current ACL

```bash title="Show ACL on a secret" theme={null}
openstack acl get <secret-href>
```

```bash title="Show ACL on a container" theme={null}
openstack acl get <container-href>
```

***

## Grant Access

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Key Manager > Secrets**, select a secret, and click the
    **Access Control** tab. Click **Add ACL** to grant access to a specific user.

    | Field              | Description                                                   |
    | ------------------ | ------------------------------------------------------------- |
    | **Operation**      | `read` — allows the user to retrieve the secret payload       |
    | **Users**          | Xloud user IDs to grant the permission                        |
    | **Project Access** | Toggle to grant all users in the secret's project read access |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Grant read access to a specific user" theme={null}
    openstack acl submit \
      --user <user-id> \
      --operation read \
      <secret-href>
    ```

    ```bash title="Grant project-wide read access" theme={null}
    openstack acl submit \
      --project-access \
      <secret-href>
    ```

    ```bash title="Grant access to multiple users" theme={null}
    openstack acl submit \
      --user <user-id-1> \
      --user <user-id-2> \
      --operation read \
      <secret-href>
    ```

    <Warning>
      Granting `--project-access` makes the secret readable by all users in the project.
      Reserve this setting for non-sensitive shared configuration. Use per-user ACLs for
      credentials, private keys, and certificates.
    </Warning>
  </Tab>
</Tabs>

***

## Revoke Access

```bash title="Revoke all ACL entries on a secret" theme={null}
openstack acl delete <secret-href>
```

```bash title="Update ACL to remove a specific user" theme={null}
openstack acl submit \
  --user <remaining-user-id> \
  --operation read \
  <secret-href>
```

<Note>
  `openstack acl submit` replaces the entire ACL. To remove one user, resubmit the ACL
  with only the users that should retain access. There is no append/remove operation.
</Note>

***

## Find Your User ID

```bash title="Get the current user's ID" theme={null}
openstack token issue -c user_id -f value
```

```bash title="Look up another user's ID (admin)" theme={null}
openstack user show <username> -c id -f value
```

***

## Next Steps

<CardGroup>
  <Card title="Store Secrets" href="/services/key-manager/store-secrets">
    Create secrets before configuring ACL access
  </Card>

  <Card title="Containers" href="/services/key-manager/containers">
    Apply ACLs at the container level for grouped secret access
  </Card>

  <Card title="Troubleshooting" href="/services/key-manager/troubleshooting">
    Resolve 403 errors and ACL propagation issues
  </Card>

  <Card title="Key Manager Admin Guide" href="/services/key-manager/admin-guide">
    Configure platform-wide access policies and quotas
  </Card>
</CardGroup>


# Key Manager Admin Guide
Source: https://docs.xloud.tech/services/key-manager/admin-guide

Administer Xloud Key Manager — configure secret store backends, manage transport keys, enforce quotas, apply security hardening, and troubleshoot platform.

<p>Overview</p>

This guide covers platform-level administration of the Xloud Key Manager service.
Administrators configure the backend encryption store, manage transport keys for
client-side encryption, define per-project quotas, and enforce security hardening
policies. The Key Manager service is a critical security component — changes to its
configuration affect secret accessibility across all services that reference it.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Tabs>
  <Tab title="XDeploy" icon="server">
    The Key Management service is enabled through the XDeploy Configuration panel:

    <Steps>
      <Step title="Open Configuration" icon="settings">
        Navigate to **XDeploy → Configuration** and select the **Advance Features** tab.
      </Step>

      <Step title="Enable KMS" icon="toggle-left">
        Set **Enable KMS** to **Yes**. This deploys the Key Management service and
        configures integration with all dependent services (Block Storage encryption,
        K8SaaS certificate storage, Load Balancer TLS).
      </Step>

      <Step title="Save and deploy" icon="rocket">
        Click **Save Configuration**, then navigate to **XDeploy → Operations** and
        run a **Deploy** for the Key Management service.

        <Check>Key Management service is deployed and accessible to all platform services.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Configure the Key Management service by editing `barbican.conf` directly at
    `/etc/xavs/config/barbican/barbican.conf`. See the individual topic guides below
    for backend configuration, secret stores, and security hardening parameters.
  </Tab>
</Tabs>

***

<p>Topics in This Guide</p>

<CardGroup>
  <Card title="Architecture" icon="network" href="/services/key-manager/architecture">
    Key Manager service topology — API, worker, metadata DB, and secret store backends
  </Card>

  <Card title="Backend Configuration" icon="database" href="/services/key-manager/backend-config">
    Configure simple crypto, PKCS#11 HSM, and KMIP secret store backends
  </Card>

  <Card title="Secret Stores" icon="vault" href="/services/key-manager/secret-stores">
    Manage multiple secret store backends and assign stores to projects
  </Card>

  <Card title="Transport Keys" icon="key" href="/services/key-manager/transport-keys">
    View and rotate the RSA transport key for client-side encryption
  </Card>

  <Card title="Quotas" icon="gauge" href="/services/key-manager/quotas">
    Set per-project limits for secrets, containers, orders, and CAs
  </Card>

  <Card title="Security" icon="shield" href="/services/key-manager/security">
    Protect master keys, audit secret access, and enforce network controls
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/key-manager/admin-troubleshooting">
    Diagnose backend failures, pending certificate orders, and ACL issues
  </Card>
</CardGroup>

***

<p>Prerequisites</p>

<Note>
  **Required before proceeding**

  * Administrator credentials sourced via `openrc.sh`
  * Access to XDeploy for service configuration changes
  * Understanding of key management concepts (HSM, PKCS#11, KMIP, symmetric encryption)
</Note>

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Key Manager User Guide" icon="book-open" href="/services/key-manager/user-guide">
    Step-by-step instructions for managing secrets, containers, and ACLs
  </Card>

  <Card title="Load Balancer Admin Guide" icon="shield-check" href="/services/load-balancer/admin-guide">
    Configure TLS termination using Key Manager certificates
  </Card>

  <Card title="Identity" icon="fingerprint" href="/cli-setup">
    Configure service accounts and RBAC policies for Key Manager access
  </Card>

  <Card title="Object Storage Admin Guide" icon="box" href="/services/object-storage/admin-guide">
    Configure server-side encryption using Key Manager-managed keys
  </Card>
</CardGroup>


# Key Manager Admin Troubleshooting
Source: https://docs.xloud.tech/services/key-manager/admin-troubleshooting

Diagnose and resolve platform-level Xloud Key Manager issues — backend connectivity failures, pending certificate orders, ACL propagation delays, and.

## Overview

This guide covers platform-level Key Manager issues that require administrator access.
For user-facing issues such as 403 errors or expired secrets, see the
[Key Manager Troubleshooting](/services/key-manager/troubleshooting) guide.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Diagnostic Checklist

```bash title="Check Key Manager container status" theme={null}
docker ps --filter name=barbican
```

```bash title="Verify Key Manager API is responding" theme={null}
openstack secret list --limit 1
```

```bash title="Check Key Manager API logs" theme={null}
docker logs barbican-api --tail 100
```

***

## Platform Issues

<AccordionGroup>
  <Accordion title="Secret retrieval returns 500 Internal Server Error" icon="server">
    **Cause**: The secret store backend is unavailable — HSM connectivity lost,
    KMIP server unreachable, or master key file inaccessible.

    **Diagnosis**:

    ```bash title="Check Key Manager worker logs for backend errors" theme={null}
    docker logs barbican-worker --tail 100 | grep -i "error\|backend\|connect"
    ```

    **Resolution by backend type**:

    | Backend         | Check                               | Resolution                              |
    | --------------- | ----------------------------------- | --------------------------------------- |
    | `simple_crypto` | Master key file readable            | `ls -la /etc/xavs/key-manager/kek.conf` |
    | `pkcs11`        | HSM online and partition accessible | Verify HSM dashboard status             |
    | `kmip`          | Network to KMIP server on port 5696 | `nc -zv <kmip-host> 5696`               |
  </Accordion>

  <Accordion title="Certificate orders remain in PENDING indefinitely" icon="clock">
    **Cause**: The CA plugin is unreachable or misconfigured.

    **Diagnosis**:

    ```bash title="Check order status and error detail" theme={null}
    openstack secret order show <order-href>
    ```

    Review the `error_status_code` and `error_reason` fields. Common causes:

    * CA plugin service is not running — check container status via XDeploy
    * Certificate subject DN contains invalid characters or fields rejected by the CA
    * CA connectivity timeout — verify network access from Key Manager to the CA endpoint

    ```bash title="Check CA plugin container" theme={null}
    docker ps --filter name=barbican
    docker logs barbican-worker --tail 50 | grep -i "ca\|order\|cert"
    ```
  </Accordion>

  <Accordion title="ACL not taking effect" icon="list-x">
    **Cause**: ACL changes require a short propagation delay, or the caller is
    authenticated under a different user identity than expected.

    **Diagnosis**:

    ```bash title="Verify ACL on the secret" theme={null}
    openstack acl get <secret-href>
    ```

    Confirm the user ID in the ACL matches the authenticated user's actual ID:

    ```bash title="Get current user ID" theme={null}
    openstack token issue -c user_id -f value
    ```

    **Resolution**: If the user ID does not match, ensure the correct user identity
    is being used in the API call. ACL entries reference user IDs, not usernames —
    a renamed user retains the same ID.
  </Accordion>

  <Accordion title="Key Manager service fails to start" icon="server">
    **Cause**: Database connectivity failure, missing master key file, or configuration
    error preventing service initialization.

    **Diagnosis**:

    ```bash title="Check startup logs" theme={null}
    docker logs barbican-api --tail 200 | grep -i "error\|critical\|warn"
    ```

    Common startup failures:

    | Error                      | Cause                   | Resolution                         |
    | -------------------------- | ----------------------- | ---------------------------------- |
    | `database not reachable`   | DB host unreachable     | Check DB container status          |
    | `No such file: kek.conf`   | Master key file missing | Restore from backup or re-generate |
    | `PKCS11 library not found` | HSM library missing     | Verify library path in config      |
    | `KMIP connection refused`  | KMIP server down        | Check KMIP server connectivity     |
  </Accordion>

  <Accordion title="High secret creation latency" icon="gauge">
    **Cause**: The secret store backend is under load or the encryption operation is
    slow (common with PKCS#11 HSM under high request rates).

    **Resolution**:

    * Check HSM health and current load from the HSM management interface
    * Consider scaling Key Manager worker replicas via XDeploy to parallelize requests
    * For KMIP backends, verify network latency to the KMIP server
    * Review Key Manager worker logs for timeout or retry events
  </Accordion>
</AccordionGroup>

***

## Log Locations

| Component                     | Log Command                              |
| ----------------------------- | ---------------------------------------- |
| Key Manager API               | `docker logs barbican-api`               |
| Key Manager Worker            | `docker logs barbican-worker`            |
| Key Manager Keystone Listener | `docker logs barbican-keystone-listener` |

***

## Next Steps

<CardGroup>
  <Card title="Key Manager Troubleshooting (User)" href="/services/key-manager/troubleshooting">
    User-facing Key Manager issues — 403 errors, expired secrets, ACL problems
  </Card>

  <Card title="Backend Configuration" href="/services/key-manager/backend-config">
    Verify and update secret store backend configuration
  </Card>

  <Card title="Architecture" href="/services/key-manager/architecture">
    Understand component roles to narrow down failure scope
  </Card>

  <Card title="Security" href="/services/key-manager/security">
    Security hardening to prevent recurrence
  </Card>
</CardGroup>


# Key Manager Architecture
Source: https://docs.xloud.tech/services/key-manager/architecture

Understand the Xloud Key Manager service topology — API layer, worker service, metadata database, secret store backends, and CA plugin integration.

## Overview

The Key Manager service separates the API layer from the secret store backend, allowing
the encryption backend to be swapped or scaled independently. Secret payloads are never
stored in the metadata database — only encrypted references. The actual ciphertext
resides exclusively in the configured secret store backend.

***

## Service Topology

```mermaid theme={null}
graph TD
    Client["API Client / Service"] --> API["Key Manager API\n:9311"]
    API --> SVC["Key Manager Worker"]
    SVC --> DB[("Metadata DB\n(secret references, ACLs)")]
    SVC --> Store["Secret Store Backend"]

    subgraph "Secret Store Options"
        Store --> Simple["Simple Crypto\n(software AES)"]
        Store --> PKCS11["PKCS#11 HSM\n(hardware)"]
        Store --> KMIP["KMIP Server\n(enterprise KMS)"]
    end

    SVC --> Plugin["CA Plugin\n(cert issuance)"]
    Plugin --> LocalCA["Local CA"]
    Plugin --> ExtCA["External CA\n(ACME / EJBCA)"]
```

***

## Component Descriptions

| Component                | Role                                                                       | Port              |
| ------------------------ | -------------------------------------------------------------------------- | ----------------- |
| **Key Manager API**      | REST API for secrets, containers, orders, ACLs                             | 9311              |
| **Key Manager Worker**   | Orchestrates secret lifecycle and CA plugin communication                  | Internal          |
| **Metadata DB**          | Stores secret references, container metadata, ACLs — no secret payloads    | Internal          |
| **Secret Store Backend** | Stores encrypted secret ciphertext                                         | Varies by backend |
| **CA Plugin**            | Integrates with Certificate Authorities for automated certificate issuance | Internal          |

***

## Secret Storage Flow

```mermaid theme={null}
sequenceDiagram
    participant User
    participant API as Key Manager API
    participant Worker as Key Manager Worker
    participant DB as Metadata DB
    participant Store as Secret Store Backend

    User->>API: POST /v1/secrets {payload: "secret", type: "passphrase"}
    API->>Worker: Encrypt and store secret
    Worker->>Store: Store encrypted ciphertext
    Store-->>Worker: Storage reference
    Worker->>DB: Store secret metadata + reference
    DB-->>Worker: Secret UUID
    Worker-->>API: Secret href
    API-->>User: 201 Created {secret_ref: "https://.../secrets/<uuid>"}
```

***

## Security Separation

<Info>
  The metadata database contains only encrypted references and ACL metadata — never
  plaintext secret payloads. Even if the metadata database is compromised, secret
  payloads cannot be extracted without also compromising the secret store backend.
</Info>

| Data            | Location             | Contains                                         |
| --------------- | -------------------- | ------------------------------------------------ |
| Secret metadata | Metadata DB          | Name, type, content type, expiration, ACLs       |
| Secret payload  | Secret store backend | Encrypted ciphertext only                        |
| Encryption keys | Secret store backend | Master wrapping keys (HSM) or key files (simple) |

***

## Next Steps

<CardGroup>
  <Card title="Backend Configuration" href="/services/key-manager/backend-config">
    Configure simple crypto, PKCS#11, and KMIP backends
  </Card>

  <Card title="Secret Stores" href="/services/key-manager/secret-stores">
    Manage multiple backends and per-project store assignments
  </Card>

  <Card title="Security" href="/services/key-manager/security">
    Harden the Key Manager service and protect master keys
  </Card>

  <Card title="Troubleshooting" href="/services/key-manager/admin-troubleshooting">
    Diagnose and resolve Key Manager platform issues
  </Card>
</CardGroup>


# Key Manager Backend Configuration
Source: https://docs.xloud.tech/services/key-manager/backend-config

Configure Barbican secret store backends for Xloud Key Manager — Simple Crypto, PKCS#11 HSM, KMIP, HashiCorp Vault, Dogtag KRA, and multi-backend deployments.

## Overview

The Key Manager (Barbican) secret store backend determines how secret payloads are encrypted and where ciphertext is stored. Xloud Key Manager supports five backend types — from a software-only AES plugin for development, to hardware HSMs, external KMIP servers, HashiCorp Vault, and Dogtag KRA for enterprise deployments. Configuration is managed through XDeploy; changing the backend after secrets exist requires a migration operation.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Supported Backends

| Backend             | Plugin Name     | External System                                           | Best For                         |
| ------------------- | --------------- | --------------------------------------------------------- | -------------------------------- |
| **Simple Crypto**   | `simple_crypto` | None — DB-stored encrypted blobs                          | Development, non-HSM deployments |
| **PKCS#11 HSM**     | `p11_crypto`    | Hardware Security Module (Thales, nCipher, Utimaco, ATOS) | FIPS 140-2 / production HSM      |
| **KMIP**            | `kmip_plugin`   | Any OASIS KMIP server (Thales, SafeNet, Vormetric, IBM)   | Enterprise KMS integration       |
| **HashiCorp Vault** | `vault_plugin`  | HashiCorp Vault (self-hosted or HCP)                      | Cloud-native secrets management  |
| **Dogtag KRA**      | `dogtag_plugin` | Red Hat Certificate System / FreeIPA                      | Software-HSM without hardware    |

***

## View Current Backend

The active backend is visible in the XDeploy configuration panel under **Key Manager → Secret Store**. Verify the API is operational:

```bash title="Verify Key Manager API" theme={null}
openstack secret list --limit 1
```

***

## Backend Configuration Reference

<Tabs>
  <Tab title="Simple Crypto" icon="cpu">
    Software-based AES-256 encryption. The Key Encryption Key (KEK) is base64-encoded and stored in `barbican.conf`. Secrets are encrypted at rest in the Barbican database.

    ```ini title="barbican.conf" theme={null}
    [secretstore]
    enabled_secretstore_plugins = store_crypto
    enabled_crypto_plugins = simple_crypto

    [simple_crypto_plugin]
    # 32-byte base64-encoded key encryption key
    kek = dGhpcnR5X3R3b19ieXRlX2tleWtleWtleWtleWs=
    ```

    | Parameter | Description                                                 |
    | --------- | ----------------------------------------------------------- |
    | `kek`     | 32-byte AES key, base64-encoded — protects all project KEKs |

    <Warning>
      The `kek` value is stored in plaintext in `barbican.conf`. Any host compromise exposes all secrets. This backend is suitable only for development and testing environments. Use PKCS#11, KMIP, or Vault for production.
    </Warning>
  </Tab>

  <Tab title="PKCS#11 HSM" icon="microchip">
    Stores the Master KEK and HMAC signing key inside a hardware HSM. Per-project KEKs are wrapped by the MKEK and signed with HMAC before being stored in the database. Supports Thales Luna, nCipher nShield, ATOS Bull, and Utimaco HSMs.

    ```ini title="barbican.conf" theme={null}
    [secretstore]
    enabled_secretstore_plugins = store_crypto
    enabled_crypto_plugins = p11_crypto

    [p11_crypto_plugin]
    library_path = /usr/lib/libCryptoki2_64.so
    token_labels = barbican_token
    login = <hsm-partition-pin>
    mkek_label = barbican_mkek
    mkek_length = 32
    hmac_label = barbican_hmac
    encryption_mechanism = CKM_AES_CBC
    hmac_mechanism = CKM_SHA256_HMAC
    key_wrap_mechanism = CKM_AES_KEY_WRAP_KWP
    pkek_length = 32
    pkek_cache_ttl = 900
    pkek_cache_limit = 100
    ```

    | Parameter              | Description                                                                  |
    | ---------------------- | ---------------------------------------------------------------------------- |
    | `library_path`         | Path to HSM vendor's PKCS#11 `.so` shared library                            |
    | `token_labels`         | Label identifying the HSM token                                              |
    | `login`                | HSM partition PIN                                                            |
    | `mkek_label`           | Label of the Master KEK inside the HSM                                       |
    | `hmac_label`           | Label of the HMAC signing key inside the HSM (must differ from `mkek_label`) |
    | `encryption_mechanism` | Encryption cipher, e.g. `CKM_AES_CBC`                                        |
    | `hmac_mechanism`       | HMAC algorithm, e.g. `CKM_SHA256_HMAC`                                       |
    | `key_wrap_mechanism`   | Project KEK wrapping algorithm                                               |
    | `pkek_cache_ttl`       | Unwrapped project KEK cache lifetime (seconds)                               |

    Generate master keys on first deployment:

    ```bash title="Generate MKEK and HMAC keys on HSM" theme={null}
    barbican-manage hsm gen_mkek --label barbican_mkek
    barbican-manage hsm gen_hmac --label barbican_hmac
    ```

    <Warning>
      `mkek_label` and `hmac_label` must be different values. Identical labels cause authentication failures.
    </Warning>
  </Tab>

  <Tab title="KMIP" icon="server">
    Secrets are stored directly on the external KMIP device — Barbican's database holds only a reference/locator. Supports mutual TLS authentication. Compatible with any OASIS KMIP-compliant server.

    ```ini title="barbican.conf" theme={null}
    [secretstore]
    enabled_secretstore_plugins = kmip_plugin

    [kmip_plugin]
    host = kmip.internal.<your-domain>
    port = 5696
    keyfile = /etc/barbican/certs/client.key
    certfile = /etc/barbican/certs/client.crt
    ca_certs = /etc/barbican/certs/ca.crt
    # Optional: username/password if required by the KMIP device
    username = barbican-svc
    password = <kmip-password>
    ```

    | Parameter  | Description                             |
    | ---------- | --------------------------------------- |
    | `host`     | KMIP server hostname or IP              |
    | `port`     | KMIP port (default `5696`)              |
    | `keyfile`  | Client TLS private key path             |
    | `certfile` | Client TLS certificate path             |
    | `ca_certs` | CA certificate for server validation    |
    | `username` | KMIP authentication username (optional) |
    | `password` | KMIP authentication password (optional) |

    <Tip>
      Client certificates and private keys must be readable only by the `barbican` service user (`chmod 0400`). Store them on an encrypted volume.
    </Tip>
  </Tab>

  <Tab title="HashiCorp Vault" icon="vault">
    Secrets are stored in Vault's KV secrets engine. Barbican authenticates using either a root token (development) or AppRole (production). The Vault server can be self-hosted or HCP Vault.

    ```ini title="barbican.conf" theme={null}
    [secretstore]
    enabled_secretstore_plugins = vault_plugin

    [vault_plugin]
    vault_url = https://vault.internal.<your-domain>:8200
    use_ssl = true
    ssl_ca_crt_file = /etc/barbican/certs/vault-ca.crt
    # Production: use AppRole (recommended)
    approle_role_id = <vault-role-id>
    approle_secret_id = <vault-secret-id>
    kv_mountpoint = secret
    # Development only (not recommended for production):
    # root_token_id = <vault-root-token>
    ```

    | Parameter           | Description                                       |
    | ------------------- | ------------------------------------------------- |
    | `vault_url`         | Vault server address                              |
    | `use_ssl`           | Enable TLS for Vault communication                |
    | `ssl_ca_crt_file`   | CA certificate for Vault TLS verification         |
    | `approle_role_id`   | AppRole role ID (recommended for production)      |
    | `approle_secret_id` | AppRole secret ID paired with role ID             |
    | `kv_mountpoint`     | KV secrets engine mount point (default: `secret`) |
    | `root_token_id`     | Root token auth — development only                |

    <Tip>
      Use AppRole authentication in production. Root token auth works but is not rotatable and grants full Vault access. AppRole provides scoped, rotatable credentials.
    </Tip>
  </Tab>

  <Tab title="Dogtag KRA" icon="certificate">
    Integrates with Red Hat Certificate System's Key Recovery Authority. Secrets are stored in the KRA with master keys held in an NSS certificate database (software) or HSM. Supports optional FreeIPA integration.

    ```ini title="barbican.conf" theme={null}
    [secretstore]
    enabled_secretstore_plugins = dogtag_plugin

    [dogtag_plugin]
    dogtag_host = kra.internal.<your-domain>
    dogtag_port = 8443
    pem_path = /etc/barbican/kra_admin_cert.pem
    nss_db_path = /etc/barbican/alias
    nss_password = <nss-db-password>
    ```

    | Parameter      | Description                         |
    | -------------- | ----------------------------------- |
    | `dogtag_host`  | KRA server hostname                 |
    | `dogtag_port`  | KRA server port (default `8443`)    |
    | `pem_path`     | KRA admin certificate PEM file path |
    | `nss_db_path`  | NSS certificate database directory  |
    | `nss_password` | NSS database password               |

    <Note>
      Dogtag is the only backend offering software-HSM security (NSS database) without requiring physical HSM hardware. It is the recommended middle ground between Simple Crypto (no hardware) and PKCS#11 (requires HSM).
    </Note>
  </Tab>
</Tabs>

***

## Multi-Backend Deployments

Deploy multiple backends simultaneously — e.g., KMIP as the global default with Vault available for specific projects.

```ini title="barbican.conf — multi-backend example" theme={null}
[secretstore]
enable_multiple_secret_stores = True
stores_lookup_suffix = software, kmip, vault

[secretstore:software]
secret_store_plugin = store_crypto
crypto_plugin = simple_crypto

[secretstore:kmip]
secret_store_plugin = kmip_plugin
global_default = True

[secretstore:vault]
secret_store_plugin = vault_plugin
```

| Rule                                   | Description                                                                         |
| -------------------------------------- | ----------------------------------------------------------------------------------- |
| `enable_multiple_secret_stores = True` | Activates multi-backend mode                                                        |
| `stores_lookup_suffix`                 | Comma-separated list — each maps to a `[secretstore:SUFFIX]` section                |
| `global_default = True`                | **Exactly one** backend must have this — handles secrets with no project preference |
| `enabled_secretstore_plugins`          | **Ignored** when multi-backend mode is active                                       |

<Warning>
  When switching from single-backend to multi-backend, keep the existing backend configuration as the `global_default` to maintain access to existing secrets.
</Warning>

***

## Apply Backend Configuration

```bash title="Deploy Key Manager configuration via XDeploy" theme={null}
xavs-ansible deploy -t barbican
```

```bash title="Verify the service is healthy after deployment" theme={null}
openstack secret list --limit 1
```

***

## Next Steps

<CardGroup>
  <Card title="Secret Stores" href="/services/key-manager/secret-stores">
    Configure multiple backends and assign preferred stores to projects
  </Card>

  <Card title="Security" href="/services/key-manager/security">
    Protect master keys, audit secret access, and certificate management
  </Card>

  <Card title="Architecture" href="/services/key-manager/architecture">
    Understand Key Manager service topology and secret lifecycle
  </Card>

  <Card title="Admin Troubleshooting" href="/services/key-manager/admin-troubleshooting">
    Diagnose backend connectivity and startup failures
  </Card>
</CardGroup>


# Certificate Management
Source: https://docs.xloud.tech/services/key-manager/certificates

Store externally issued TLS certificates and order new certificates through CA plugins in Xloud Key Manager. Manage the full certificate lifecycle from.

## Overview

Xloud Key Manager supports two certificate workflows: storing externally issued
certificates from your existing CA, and ordering certificates through a configured
CA plugin for automated issuance. Both workflows produce a certificate container
that can be consumed by the Load Balancer service for HTTPS termination.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Store an Existing Certificate

Use this workflow when you have an externally issued certificate (Let's Encrypt, DigiCert,
your enterprise CA, etc.) and want to store it in Key Manager.

<Steps>
  <Step title="Store the certificate secret">
    ```bash title="Store the X.509 certificate" theme={null}
    openstack secret store \
      --name app-tls-cert \
      --secret-type certificate \
      --payload-content-type "application/pkix-cert" \
      --payload-content-encoding base64 \
      --payload "$(base64 -w 0 certificate.pem)"
    ```
  </Step>

  <Step title="Store the private key secret">
    ```bash title="Store the private key" theme={null}
    openstack secret store \
      --name app-tls-key \
      --secret-type private \
      --payload-content-type "application/pkcs8" \
      --payload-content-encoding base64 \
      --payload "$(base64 -w 0 private_key.pem)"
    ```
  </Step>

  <Step title="Store the CA chain (recommended)">
    ```bash title="Store CA chain" theme={null}
    openstack secret store \
      --name app-ca-chain \
      --secret-type certificate \
      --payload-content-type "application/pkix-cert" \
      --payload-content-encoding base64 \
      --payload "$(base64 -w 0 ca-chain.pem)"
    ```
  </Step>

  <Step title="Create a certificate container">
    ```bash title="Bundle into certificate container" theme={null}
    openstack secret container create \
      --name app-tls-bundle \
      --type certificate \
      --secret "certificate=<cert-href>" \
      --secret "private_key=<key-href>" \
      --secret "intermediates=<ca-chain-href>"
    ```

    <Check>Container is ready to reference in Load Balancer HTTPS listener configuration.</Check>
  </Step>
</Steps>

***

## Order a Certificate

Certificate orders automate issuance through a Certificate Authority plugin configured
by your administrator.

<Tabs>
  <Tab title="Create an order" icon="clipboard-list">
    ```bash title="Create a certificate order" theme={null}
    openstack secret order create certificate \
      --name app-cert-order \
      --algorithm rsa \
      --bit-length 2048 \
      --subject-dn "CN=app.example.com,O=Example Corp,C=US"
    ```

    ```bash title="Check order status" theme={null}
    openstack secret order show <order-href>
    ```

    When the order status reaches `ACTIVE`, retrieve the issued certificate container:

    ```bash title="Get the issued certificate container" theme={null}
    openstack secret order show <order-href> -c container_ref
    ```

    <Info>
      Certificate order availability depends on your platform's CA plugin configuration.
      Contact your administrator to verify which CA backends are enabled.
    </Info>
  </Tab>

  <Tab title="List and manage orders" icon="list">
    <CodeGroup>
      ```bash title="List all certificate orders" theme={null}
      openstack secret order list
      ```

      ```bash title="Show order detail" theme={null}
      openstack secret order show <order-href>
      ```

      ```bash title="Delete an order" theme={null}
      openstack secret order delete <order-href>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Certificate Lifecycle Management

| Stage          | Action                                       | Notes                               |
| -------------- | -------------------------------------------- | ----------------------------------- |
| **Issuance**   | Store or order via CA plugin                 | Creates certificate + key secrets   |
| **Deployment** | Create container, reference in Load Balancer | Bundles cert + key + chain          |
| **Monitoring** | Track expiration date externally             | Key Manager sends no alerts         |
| **Renewal**    | Store new certificate, update container      | Update Load Balancer reference      |
| **Revocation** | Delete old secrets after transition          | Update all service references first |

<Tip>
  Set calendar reminders at 60 days, 30 days, and 7 days before certificate expiration.
  Renew the certificate and update the Load Balancer listener reference at least 14 days
  before expiry to allow for propagation and testing.
</Tip>

***

## Verify a Certificate

```bash title="Retrieve certificate and check expiry" theme={null}
openstack secret get <cert-href> --payload | \
  openssl x509 -noout -dates -subject
```

```bash title="Verify certificate matches private key" theme={null}
openstack secret get <cert-href> --payload > /tmp/cert.pem
openstack secret get <key-href> --payload > /tmp/key.pem
diff <(openssl x509 -noout -modulus -in /tmp/cert.pem | md5sum) \
     <(openssl rsa -noout -modulus -in /tmp/key.pem | md5sum)
```

<Check>If both `md5sum` values match, the certificate and private key are a valid pair.</Check>

***

## Next Steps

<CardGroup>
  <Card title="Containers" href="/services/key-manager/containers">
    Bundle certificates into containers for Load Balancer use
  </Card>

  <Card title="ACL" href="/services/key-manager/acl">
    Control which users and services can access certificate secrets
  </Card>

  <Card title="Store Secrets" href="/services/key-manager/store-secrets">
    Store other secret types alongside certificates
  </Card>

  <Card title="Troubleshooting" href="/services/key-manager/troubleshooting">
    Resolve certificate container and order issues
  </Card>
</CardGroup>


# Key Manager CLI Reference
Source: https://docs.xloud.tech/services/key-manager/cli-reference

Complete openstack secret CLI commands for managing secrets, containers, certificates, and ACLs in Xloud Key Manager.

## Overview

The `openstack secret` command group manages secrets, containers, and access control policies in the Xloud Key Manager service.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python barbicanclient installed: `pip install python-barbicanclient`
</Note>

***

## Secrets

<CodeGroup>
  ```bash title="List secrets" theme={null}
  openstack secret list
  ```

  ```bash title="Store a passphrase" theme={null}
  openstack secret store \
    --name db-password \
    --secret-type passphrase \
    --payload "my-secure-password"
  ```

  ```bash title="Store a symmetric key" theme={null}
  openstack secret order create \
    --name aes-key \
    --algorithm aes \
    --bit-length 256 \
    --mode cbc \
    key
  ```

  ```bash title="Store from file" theme={null}
  openstack secret store \
    --name tls-cert \
    --secret-type certificate \
    --file /path/to/cert.pem \
    --payload-content-type "application/octet-stream"
  ```

  ```bash title="Show secret metadata" theme={null}
  openstack secret get <secret-href>
  ```

  ```bash title="Retrieve secret payload" theme={null}
  openstack secret get --payload <secret-href>
  ```

  ```bash title="Delete secret" theme={null}
  openstack secret delete <secret-href>
  ```
</CodeGroup>

***

## Containers

<CodeGroup>
  ```bash title="List containers" theme={null}
  openstack secret container list
  ```

  ```bash title="Create certificate container" theme={null}
  openstack secret container create \
    --name my-tls \
    --type certificate \
    --secret "certificate=<cert-href>" \
    --secret "private_key=<key-href>"
  ```

  ```bash title="Show container" theme={null}
  openstack secret container get <container-href>
  ```

  ```bash title="Delete container" theme={null}
  openstack secret container delete <container-href>
  ```
</CodeGroup>

***

## Orders (Key Generation)

<CodeGroup>
  ```bash title="List orders" theme={null}
  openstack secret order list
  ```

  ```bash title="Generate AES key" theme={null}
  openstack secret order create \
    --name my-key \
    --algorithm aes \
    --bit-length 256 \
    key
  ```

  ```bash title="Show order" theme={null}
  openstack secret order get <order-href>
  ```

  ```bash title="Delete order" theme={null}
  openstack secret order delete <order-href>
  ```
</CodeGroup>

***

## ACLs

<CodeGroup>
  ```bash title="Show ACL for secret" theme={null}
  openstack acl get <secret-href>
  ```

  ```bash title="Grant read access to a user" theme={null}
  openstack acl submit \
    --user <user-id> \
    --operation-type read \
    <secret-href>
  ```

  ```bash title="Delete ACL" theme={null}
  openstack acl delete <secret-href>
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Store Secrets Guide" href="/services/key-manager/store-secrets">
    Guide to storing and retrieving secrets securely
  </Card>

  <Card title="Certificates Guide" href="/services/key-manager/certificates">
    Manage TLS certificates and certificate orders
  </Card>
</CardGroup>


# Secret Containers
Source: https://docs.xloud.tech/services/key-manager/containers

Create and manage secret containers in Xloud Key Manager. Bundle certificates, keys, and secrets into named groups for use with Load Balancer TLS and.

## Overview

Containers group related secrets into a named bundle. The most common use case is
bundling a TLS certificate with its private key for use with the Load Balancer
service. Containers reference secrets by UUID — they do not copy secret payloads.

Like secrets, containers are **project-scoped** — they are visible only to users
within the project that created them unless shared via [ACL](/services/key-manager/acl).

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  **Project Scope** — Containers belong to the project that created them.
  The Load Balancer service accesses containers through the project's service
  credentials. If you create a container in Project A and configure a listener
  in Project B, the listener cannot access the container unless an ACL is set.
</Note>

***

## Container Types

| Container Type  | Contents                                               | Primary Use Case                                  |
| --------------- | ------------------------------------------------------ | ------------------------------------------------- |
| **certificate** | Certificate + private key + intermediates + passphrase | TLS termination for Load Balancer HTTPS listeners |
| **rsa**         | Public key + private key + passphrase                  | RSA key pair management                           |
| **generic**     | Any combination of secrets                             | API credential bundles, configuration groups      |

***

## Create a Certificate Container

The Dashboard provides a dedicated **Create Certificate** workflow that creates
both the secrets and the container in a single step. This is the recommended
approach for TLS certificate management.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Certificate Manager">
        Navigate to **Network > Certificates** in the sidebar.

        Click **Create Certificate** in the upper-right corner.
      </Step>

      <Step title="Enter the certificate name">
        Enter a **Certificate Name** for the container. This name identifies the
        certificate bundle in the Load Balancer listener configuration.

        <Note>
          The name must contain only letters, numbers, and hyphens. Special characters
          and spaces are not permitted.
        </Note>
      </Step>

      <Step title="Select the certificate type">
        Choose the **Certificate Type**:

        | Type       | Description                                                           | Required Fields                  |
        | ---------- | --------------------------------------------------------------------- | -------------------------------- |
        | **Server** | A server certificate with its private key for TLS termination         | Certificate Content, Private Key |
        | **CA**     | A Certificate Authority certificate for client certificate validation | Certificate Content only         |

        <Tip>
          Use **Server** for HTTPS listener termination — this is the most common
          use case. Use **CA** when you need to validate client certificates in
          mutual TLS (mTLS) configurations.
        </Tip>
      </Step>

      <Step title="Provide the certificate content">
        Paste the certificate content into the **Certificate Content** text area,
        or click the upload button to load it from a `.crt` or `.pem` file.

        The certificate must be in PEM format:

        ```text title="Expected format" theme={null}
        -----BEGIN CERTIFICATE-----
        MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhki...
        -----END CERTIFICATE-----
        ```

        <Warning>
          The form validates that the content starts with `-----BEGIN CERTIFICATE-----`
          and ends with `-----END CERTIFICATE-----`. Certificates in other formats
          (DER, PKCS#7) must be converted to PEM first.
        </Warning>
      </Step>

      <Step title="Provide the private key (Server type only)">
        For **Server** type certificates, paste the private key into the **Private Key**
        text area, or upload a `.key` or `.pem` file.

        The key must be in PEM format:

        ```text title="Expected format" theme={null}
        -----BEGIN RSA PRIVATE KEY-----
        MIIEpAIBAAKCAQEA2a2rwplBQLYV0...
        -----END RSA PRIVATE KEY-----
        ```

        <Note>
          This field is hidden for CA type certificates. The private key must be
          in RSA format — ECDSA and Ed25519 keys must be converted first.
        </Note>
      </Step>

      <Step title="Add domain names (optional, Server type only)">
        For SNI (Server Name Indication) certificates, enter the **Domain Name(s)**
        that this certificate covers.

        | Rule                 | Limit                           |
        | -------------------- | ------------------------------- |
        | Multiple domains     | Separated by commas             |
        | Maximum domains      | 30 per certificate              |
        | Single domain length | 100 characters max              |
        | Total length         | 1024 characters max             |
        | Valid characters     | Letters, numbers, hyphens, dots |

        <Tip>
          Domain names are used for SNI-based certificate selection in the Load
          Balancer. If your listener serves multiple domains, specify all of them
          here so the correct certificate is presented for each domain.
        </Tip>
      </Step>

      <Step title="Set an expiration date (optional)">
        Use the **Expires At** date picker to set an optional expiration date.
        Only future dates are selectable.

        <Warning>
          This is the container-level expiration, independent of the certificate's
          own validity period. Set it to match your certificate's actual expiry date
          for consistency.
        </Warning>
      </Step>

      <Step title="Create the certificate">
        Click **Confirm** to create the certificate container. The system stores the
        certificate and private key as separate secrets and bundles them into a
        container automatically.

        <Check>
          The certificate container appears in the Certificates list and is ready
          to reference in Load Balancer HTTPS listener configuration.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    With the CLI, you create the individual secrets first, then bundle them into
    a container. Source your project credentials first:

    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <Steps>
      <Step title="Store the certificate secret">
        ```bash title="Store TLS certificate" theme={null}
        CERT_HREF=$(openstack secret store \
          --name app-tls-cert \
          --secret-type certificate \
          --payload-content-type "application/pkix-cert" \
          --payload-content-encoding base64 \
          --payload "$(base64 -w 0 certificate.pem)" \
          -f value -c Secret\ href)
        ```
      </Step>

      <Step title="Store the private key secret">
        ```bash title="Store private key" theme={null}
        KEY_HREF=$(openstack secret store \
          --name app-tls-key \
          --secret-type private \
          --payload-content-type "application/pkcs8" \
          --payload-content-encoding base64 \
          --payload "$(base64 -w 0 private_key.pem)" \
          -f value -c Secret\ href)
        ```
      </Step>

      <Step title="Store the CA chain (recommended)">
        ```bash title="Store CA chain" theme={null}
        CA_HREF=$(openstack secret store \
          --name app-ca-chain \
          --secret-type certificate \
          --payload-content-type "application/pkix-cert" \
          --payload-content-encoding base64 \
          --payload "$(base64 -w 0 ca-chain.pem)" \
          -f value -c Secret\ href)
        ```
      </Step>

      <Step title="Create the certificate container">
        ```bash title="Bundle into certificate container" theme={null}
        openstack secret container create \
          --name app-tls-bundle \
          --type certificate \
          --secret "certificate=$CERT_HREF" \
          --secret "private_key=$KEY_HREF" \
          --secret "intermediates=$CA_HREF"
        ```

        <Check>Container is ready to reference in Load Balancer HTTPS listener configuration.</Check>
      </Step>
    </Steps>

    <AccordionGroup>
      <Accordion title="Create an RSA container">
        ```bash title="RSA key pair container" theme={null}
        openstack secret container create \
          --name app-signing-keypair \
          --type rsa \
          --secret "public_key=$PUB_HREF" \
          --secret "private_key=$KEY_HREF" \
          --secret "private_key_passphrase=$PASS_HREF"
        ```
      </Accordion>

      <Accordion title="Create a generic container">
        ```bash title="Generic container for grouped credentials" theme={null}
        openstack secret container create \
          --name app-credentials \
          --type generic \
          --secret "db_password=$DB_HREF" \
          --secret "api_key=$API_HREF" \
          --secret "encryption_key=$ENC_HREF"
        ```
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>

***

## View Container Details

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to containers">
        Go to **Project > Key Manager > Containers**. The list shows all
        containers in your current project.

        | Column      | Description                                      |
        | ----------- | ------------------------------------------------ |
        | **Name**    | Container identifier (clickable to view details) |
        | **Type**    | Container type: certificate, rsa, or generic     |
        | **Status**  | Active or Error                                  |
        | **Secrets** | Number of secret references in the container     |
        | **Created** | Creation timestamp                               |
      </Step>

      <Step title="View detail page">
        Click a container name to view its detail page. The detail page shows:

        | Field                 | Description                             |
        | --------------------- | --------------------------------------- |
        | **Type**              | Container type                          |
        | **Status**            | Current status                          |
        | **Secret References** | List of name and secret reference pairs |
        | **Created**           | Creation timestamp                      |
        | **Updated**           | Last modification timestamp             |
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List all containers" theme={null}
      openstack secret container list
      ```

      ```bash title="Show container detail" theme={null}
      openstack secret container show <container-href>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Delete a Container

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Select containers to delete">
        Navigate to **Project > Key Manager > Containers**. Select one or more
        containers using the checkboxes, then click **Delete** in the batch actions bar.

        Alternatively, click the **More** menu on a single container row and
        select **Delete Container**.
      </Step>

      <Step title="Confirm deletion">
        Confirm the deletion in the dialog.

        <Danger>
          Deleting a container through the Dashboard also deletes all associated
          secrets within it. Verify that no services reference these secrets
          before proceeding.
        </Danger>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a container" theme={null}
    openstack secret container delete <container-href>
    ```

    <Note>
      Deleting a container via the CLI does **not** delete the secrets it
      references. The secrets remain in Key Manager and must be deleted
      separately if no longer needed.
    </Note>
  </Tab>
</Tabs>

***

## Project Scope and Access

Containers follow the same project-scoping rules as secrets:

| Behavior                  | Description                                                                     |
| ------------------------- | ------------------------------------------------------------------------------- |
| **Ownership**             | Containers belong to the project that created them                              |
| **Visibility**            | Only visible to users within the same project                                   |
| **Service access**        | The Load Balancer accesses containers through the project's service credentials |
| **Cross-project sharing** | Requires an [ACL](/services/key-manager/acl) on the container                   |
| **Contained secrets**     | Secrets within the container must also be accessible to the consuming service   |

<Warning>
  When sharing a container via ACL, you must also set ACLs on each secret
  referenced by the container. Granting access to the container alone does
  not grant access to the secret payloads within it.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Certificates" icon="shield" href="/services/key-manager/certificates">
    Manage the full certificate lifecycle from storage to renewal
  </Card>

  <Card title="Access Control (ACL)" icon="lock" href="/services/key-manager/acl">
    Share containers and secrets across projects
  </Card>

  <Card title="Store Secrets" icon="lock" href="/services/key-manager/store-secrets">
    Create individual secrets to populate containers
  </Card>

  <Card title="Load Balancer" icon="combine" href="/services/load-balancer/user-guide">
    Use certificate containers for HTTPS listener configuration
  </Card>
</CardGroup>


# Key Manager Quotas
Source: https://docs.xloud.tech/services/key-manager/quotas

Manage per-project Key Manager resource limits in Xloud — view and set quotas for secrets, containers, certificate orders, consumers, and CA configurations.

## Overview

Key Manager quotas prevent individual projects from creating excessive secrets,
containers, or orders. Default values are set platform-wide; administrators override
them per project.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Default Quota Reference

| Resource     | Default Limit | Description                                      |
| ------------ | ------------- | ------------------------------------------------ |
| `secrets`    | 20            | Secrets per project                              |
| `orders`     | 20            | Certificate orders per project                   |
| `containers` | 20            | Containers per project                           |
| `consumers`  | 20            | Service consumers per container                  |
| `cas`        | 10            | Certificate Authority configurations per project |

***

## View Quotas

<Tabs>
  <Tab title="Platform defaults" icon="eye">
    ```bash title="Show platform-wide quota defaults" theme={null}
    openstack secret quota show
    ```
  </Tab>

  <Tab title="Project quotas (admin)" icon="shield">
    ```bash title="Show quotas for a specific project" theme={null}
    openstack secret quota show <project-id>
    ```
  </Tab>
</Tabs>

***

## Set Quotas

```bash title="Set project-specific Key Manager quotas" theme={null}
openstack secret quota set \
  --secrets 100 \
  --orders 50 \
  --containers 50 \
  <project-id>
```

```bash title="Reset to platform defaults" theme={null}
openstack secret quota delete <project-id>
```

<Tip>
  Production projects running automated certificate rotation or large service meshes
  may require `secrets` quotas in the hundreds. Monitor usage quarterly to right-size
  quota allocations.
</Tip>

***

## Monitor Quota Usage

```bash title="Count secrets in a project" theme={null}
openstack secret list --project <project-id> | wc -l
```

```bash title="Count containers in a project" theme={null}
openstack secret container list --project <project-id> | wc -l
```

***

## Next Steps

<CardGroup>
  <Card title="Security" href="/services/key-manager/security">
    Apply Key Manager security hardening policies
  </Card>

  <Card title="Backend Configuration" href="/services/key-manager/backend-config">
    Configure the underlying secret store backend
  </Card>

  <Card title="Admin Troubleshooting" href="/services/key-manager/admin-troubleshooting">
    Diagnose quota-related errors and service issues
  </Card>

  <Card title="Key Manager User Guide" href="/services/key-manager/user-guide">
    User-facing secret and container management
  </Card>
</CardGroup>


# Secret Store Management
Source: https://docs.xloud.tech/services/key-manager/secret-stores

Manage multiple secret store backends in Xloud Key Manager. List available stores, assign preferred stores to projects, and manage tiered security levels.

## Overview

Key Manager supports multiple simultaneous secret store backends. Different stores can
be assigned to different projects, providing tiered security levels — e.g., routing
regulated projects to HSM-backed storage while using software crypto for development projects.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## List Secret Stores

```bash title="List all configured secret stores" theme={null}
openstack secret store list
```

```bash title="Show the platform-default secret store" theme={null}
openstack secret store get preferred
```

***

## Assign a Store to a Project

Assign a specific secret store to a project to override the platform default:

```bash title="Set preferred store for a project" theme={null}
openstack secret store set preferred \
  --secret-store-id <store-id>
```

```bash title="Show the preferred store for a project" theme={null}
openstack secret store get preferred --project <project-id>
```

```bash title="Reset to platform default" theme={null}
openstack secret store unset preferred
```

<Warning>
  Changing a project's preferred store does not migrate existing secrets. Secrets
  created before the change remain in the original store. Only new secrets use the
  newly assigned store.
</Warning>

***

## Multi-Store Design Patterns

| Pattern                    | Configuration                                         | Use Case                            |
| -------------------------- | ----------------------------------------------------- | ----------------------------------- |
| **Single store (default)** | One `simple_crypto` or PKCS#11 backend                | Homogeneous environment             |
| **Tiered security**        | `simple_crypto` for dev, PKCS#11 for production       | PCI-DSS, HIPAA, regulated workloads |
| **Geographic isolation**   | Per-region KMIP servers as separate stores            | Data residency requirements         |
| **Workload separation**    | HSM store for TLS keys, software store for API tokens | Cost optimization                   |

***

## Migration Between Stores

Migrating existing secrets from one store to another is a manual process:

<Steps>
  <Step title="Retrieve existing secrets">
    Export all secret payloads from the current store. This requires read access to
    every secret in the project.
  </Step>

  <Step title="Assign the new store to the project">
    ```bash title="Set new preferred store" theme={null}
    openstack secret store set preferred \
      --secret-store-id <new-store-id>
    ```
  </Step>

  <Step title="Re-create secrets in the new store">
    Store each payload as a new secret. The new secrets will be encrypted by the new store.
  </Step>

  <Step title="Update all references">
    Update any services, Load Balancer listeners, or containers that reference the old
    secret HREFs to point to the new secret HREFs.
  </Step>

  <Step title="Delete old secrets">
    Once all references are updated and verified, delete the original secrets from the
    old store.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Backend Configuration" href="/services/key-manager/backend-config">
    Configure the underlying backend for each secret store
  </Card>

  <Card title="Transport Keys" href="/services/key-manager/transport-keys">
    Manage RSA transport keys for client-side encryption
  </Card>

  <Card title="Security" href="/services/key-manager/security">
    Apply hardening policies for each store type
  </Card>

  <Card title="Quotas" href="/services/key-manager/quotas">
    Set per-project limits for secret creation
  </Card>
</CardGroup>


# Key Manager Security
Source: https://docs.xloud.tech/services/key-manager/security

Harden Xloud Key Manager — protect master encryption keys, configure HSM access controls, audit secret access logs, enforce expiration policies, and.

## Overview

The Key Manager service is a critical security component — it stores the credentials
that protect all other services. A compromise of the Key Manager backend could expose
secrets used across your entire platform. This guide covers the key hardening areas.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Hardening Guidelines

<AccordionGroup>
  <Accordion title="Protect the master encryption key" icon="lock">
    The `simple_crypto` backend encrypts secrets using a master key. This key must be
    treated as the most sensitive credential in your platform:

    * Store master key files with `0400` permissions, owned by the Key Manager service user
    * Back up the master key to offline, encrypted media stored in a physically secure location
    * Never include the master key in container images, version control, or backup snapshots
    * Rotate the master key annually — plan a maintenance window for the re-encryption operation

    ```bash title="Verify master key file permissions" theme={null}
    ls -la /etc/xavs/key-manager/kek.conf
    # Expected: -r-------- key-manager:key-manager
    ```
  </Accordion>

  <Accordion title="HSM access control" icon="microchip">
    For PKCS#11 deployments:

    * Assign a dedicated HSM partition exclusively to Key Manager — never share partitions
      with other applications
    * Use separate HSM credentials for Key Manager vs. HSM administration tasks
    * Enable HSM audit logging and forward logs to your SIEM
    * Test HSM failover procedures quarterly — Key Manager unavailability during HSM outage
      blocks secret retrieval across all dependent services
    * Restrict physical HSM access to authorized datacenter staff only
  </Accordion>

  <Accordion title="Audit secret access" icon="file-text">
    All Key Manager API requests are logged. Forward logs to your centralized logging
    platform for:

    * Secret creation and deletion events
    * ACL modifications — track when access is granted or revoked
    * Secret retrieval (`GET /v1/secrets/<id>/payload`) — every access by every caller
    * Failed access attempts — potential indicators of unauthorized access attempts

    <Tip>
      Correlate Key Manager retrieval logs with the services that should legitimately
      access each secret. Unexplained retrieval events from unfamiliar callers warrant
      immediate investigation.
    </Tip>
  </Accordion>

  <Accordion title="Secret expiration policy" icon="clock">
    Enforce expiration on time-sensitive secrets:

    * Set expiration dates on all API tokens and temporary credentials
    * TLS certificates: track expiration in an external calendar — Key Manager does not
      send expiration alerts
    * Implement rotation workflows: create a new secret, update all references, then
      delete the old secret
    * Review and purge secrets without expiration dates quarterly — accumulation of
      orphaned secrets increases the blast radius of a breach
  </Accordion>

  <Accordion title="Restrict API network access" icon="network">
    The Key Manager API should not be exposed on public networks:

    * Bind the API to the internal management network only
    * Configure firewall rules limiting port 9311 access to authorized service hosts
    * Use HAProxy frontend ACLs to restrict source IPs if Key Manager is behind a load balancer
    * Enable HTTPS on the Key Manager API endpoint — never transmit secrets over plain HTTP
    * Disable direct external access — all API calls should originate from within the platform
  </Accordion>
</AccordionGroup>

***

## Security Checklist

| Control                     | Frequency     | Notes                               |
| --------------------------- | ------------- | ----------------------------------- |
| Master key file permissions | At deployment | `0400`, owned by service user       |
| Master key offline backup   | At rotation   | Encrypted media, physically secured |
| Master key rotation         | Annually      | Plan maintenance window             |
| HSM partition isolation     | At setup      | Dedicated partition per service     |
| Audit log forwarding        | Continuous    | SIEM integration                    |
| Orphaned secret review      | Quarterly     | Purge secrets without expiration    |
| Network access controls     | At deployment | Firewall rules on port 9311         |
| HTTPS on API endpoint       | Always        | TLS termination at HAProxy          |

***

## Next Steps

<CardGroup>
  <Card title="Backend Configuration" href="/services/key-manager/backend-config">
    Configure HSM and KMIP backends for production deployments
  </Card>

  <Card title="Quotas" href="/services/key-manager/quotas">
    Set per-project limits to control resource consumption
  </Card>

  <Card title="Admin Troubleshooting" href="/services/key-manager/admin-troubleshooting">
    Diagnose security-related Key Manager service issues
  </Card>

  <Card title="DNS Security" href="/services/dns/security">
    Secure DNSSEC keys stored in Key Manager
  </Card>
</CardGroup>


# Store Secrets
Source: https://docs.xloud.tech/services/key-manager/store-secrets

Create, retrieve, and manage secrets in Xloud Key Manager using the Dashboard or CLI.

## Overview

Secrets are the fundamental resource in Xloud Key Manager. Each secret stores an
encrypted payload with type and algorithm metadata. Secrets are **project-scoped** —
they are visible only to users within the project that created them unless shared
via [ACL](/services/key-manager/acl). Secrets are referenced by UUID and payloads
are never returned in API responses outside of an explicit retrieve operation.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  **Project Scope** — All secrets created in Key Manager belong to the project
  selected at the time of creation. Users in other projects cannot see or access
  your secrets unless you explicitly grant access through an ACL. If you switch
  projects in the Dashboard, you will see a different set of secrets.
</Note>

***

## Secret Types

Xloud Key Manager supports six secret types. The type you select determines which
algorithm, key length, and encryption mode options are available.

| Type            | Use Case                                                    | Example Payload             |
| --------------- | ----------------------------------------------------------- | --------------------------- |
| **Opaque**      | Arbitrary data — API keys, passwords, config values, tokens | Any text or binary data     |
| **Symmetric**   | Encryption keys for AES, DES, 3DES                          | Base64-encoded key material |
| **Public**      | RSA, DSA, or EC public keys                                 | PEM-encoded public key      |
| **Private**     | RSA, DSA, or EC private keys                                | PEM-encoded private key     |
| **Certificate** | X.509 TLS/SSL certificates                                  | PEM-encoded certificate     |
| **Passphrase**  | Passwords and passphrase strings                            | Plain text passphrase       |

***

## Create a Secret

<iframe title="How to Create and Manage Secrets on Xloud" />

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Create secrets through the Xloud Dashboard with a form that adapts
    based on the selected secret type.

    <Steps>
      <Step title="Navigate to Key Manager">
        Navigate to **Key Manager > Secrets** in the sidebar.

        Click **Create Secret** in the upper-right corner.
      </Step>

      <Step title="Enter the secret name">
        Enter a descriptive **Name** for your secret. This is a required field and serves
        as a human-readable identifier (e.g., `db-root-password`, `app-tls-private-key`).
      </Step>

      <Step title="Select the secret type">
        Choose the **Secret Type** from the dropdown. This selection controls which
        additional fields appear in the form:

        | Secret Type     | Additional Fields Shown        |
        | --------------- | ------------------------------ |
        | **Opaque**      | No additional algorithm fields |
        | **Symmetric**   | Algorithm, Bit Length, Mode    |
        | **Public**      | Algorithm, Bit Length          |
        | **Private**     | Algorithm, Bit Length          |
        | **Certificate** | Algorithm, Bit Length          |
        | **Passphrase**  | No additional algorithm fields |
      </Step>

      <Step title="Configure algorithm options (if applicable)">
        For secret types that support cryptographic metadata, configure the following
        fields. These fields appear dynamically based on your secret type selection:

        **Algorithm** — Select the cryptographic algorithm:

        | Secret Type | Available Algorithms |
        | ----------- | -------------------- |
        | Symmetric   | AES, DES, 3DES       |
        | Public      | RSA, DSA, EC         |
        | Private     | RSA, DSA, EC         |
        | Certificate | RSA, EC              |

        **Bit Length** — Select the key size (appears after algorithm selection):

        | Algorithm | Available Bit Lengths |
        | --------- | --------------------- |
        | AES       | 128, 192, 256         |
        | DES       | 56                    |
        | 3DES      | 168                   |
        | RSA       | 2048, 3072, 4096      |
        | DSA       | 2048, 3072            |
        | EC        | 256, 384, 521         |

        **Mode** — Select the block cipher mode (symmetric keys only):

        | Mode | Description                                                       |
        | ---- | ----------------------------------------------------------------- |
        | CBC  | Cipher Block Chaining — standard mode for block encryption        |
        | CTR  | Counter mode — enables parallel encryption                        |
        | GCM  | Galois/Counter Mode — provides both encryption and authentication |

        <Tip>
          For symmetric encryption keys, AES-256 with GCM mode is recommended for
          most use cases. It provides both strong encryption and built-in integrity
          verification.
        </Tip>
      </Step>

      <Step title="Select a secret store backend (optional)">
        If your platform has multiple secret store backends configured, a **Secret Store
        Backend** dropdown appears. Select which backend should store this secret.

        The default backend is pre-selected and marked with **(Default)**.

        <Note>
          This field only appears when the administrator has configured multiple
          secret store backends. Most deployments use a single backend.
        </Note>
      </Step>

      <Step title="Set an expiration date (optional)">
        Use the **Expiration** date-time picker to set an optional expiration date for
        the secret. After this date, the secret is no longer usable.

        <Warning>
          Key Manager does not send expiration alerts. Set external calendar reminders
          to renew secrets before they expire, especially for certificates and
          encryption keys used by running services.
        </Warning>
      </Step>

      <Step title="Enter the payload">
        Enter the secret value in the **Payload** text area. The form displays a
        format hint based on the selected secret type:

        | Secret Type | Payload Format Hint                                      |
        | ----------- | -------------------------------------------------------- |
        | Opaque      | Any text or data: API key, password, config value, token |
        | Symmetric   | Base64-encoded key. Generate: `openssl rand -base64 32`  |
        | Public      | PEM format: `-----BEGIN PUBLIC KEY-----`                 |
        | Private     | PEM format: `-----BEGIN RSA PRIVATE KEY-----`            |
        | Certificate | PEM format: `-----BEGIN CERTIFICATE-----`                |
        | Passphrase  | A passphrase or password string                          |

        <Note>
          The payload field is optional at creation time. You can create a secret
          without a payload and add it later via the API. However, most use cases
          require providing the payload during creation.
        </Note>
      </Step>

      <Step title="Select payload content type">
        If you entered a payload, select the **Payload Content Type**:

        | Content Type               | When to Use                            |
        | -------------------------- | -------------------------------------- |
        | `text/plain`               | Passphrases, API keys, plain text data |
        | `application/octet-stream` | Binary data, symmetric keys            |
        | `application/pkix-cert`    | X.509 certificates, public keys        |

        <Tip>
          If you do not select a content type, the system auto-selects based on
          your secret type: `application/octet-stream` for symmetric keys,
          `application/pkix-cert` for certificates, and `text/plain` for all others.
        </Tip>
      </Step>

      <Step title="Create the secret">
        Click **Confirm** to create the secret. It appears in the Secrets list with
        status **Active**.

        <Check>Secret appears in the Secrets list with its UUID and Active status.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Store secrets using the `openstack secret store` command. Source your
    project credentials first:

    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Store a passphrase" theme={null}
      openstack secret store \
        --name db-root-password \
        --secret-type passphrase \
        --payload "S3cur3P@ssw0rd!" \
        --payload-content-type "text/plain"
      ```

      ```bash title="Store an AES-256 symmetric key" theme={null}
      openstack secret store \
        --name data-encryption-key \
        --secret-type symmetric \
        --algorithm aes \
        --bit-length 256 \
        --mode gcm \
        --payload-content-type "application/octet-stream" \
        --payload-content-encoding base64 \
        --payload "$(openssl rand -base64 32)"
      ```

      ```bash title="Store a private key from file" theme={null}
      openstack secret store \
        --name app-tls-private-key \
        --secret-type private \
        --algorithm rsa \
        --bit-length 4096 \
        --payload-content-type "application/pkcs8" \
        --payload-content-encoding base64 \
        --payload "$(base64 -w 0 private_key.pem)"
      ```

      ```bash title="Store a certificate" theme={null}
      openstack secret store \
        --name app-tls-cert \
        --secret-type certificate \
        --algorithm rsa \
        --bit-length 2048 \
        --payload-content-type "application/pkix-cert" \
        --payload-content-encoding base64 \
        --payload "$(base64 -w 0 certificate.pem)"
      ```

      ```bash title="Store an opaque secret with expiration" theme={null}
      openstack secret store \
        --name temp-api-key \
        --secret-type opaque \
        --payload "temp-token-value-abc123" \
        --payload-content-type "text/plain" \
        --expiration "2026-12-31T23:59:59"
      ```
    </CodeGroup>

    <Note>
      For binary payloads (private keys, certificates, symmetric keys), use
      `--payload-content-encoding base64` and base64-encode the payload.
      For text payloads (passphrases, API keys), use `text/plain` without encoding.
    </Note>
  </Tab>
</Tabs>

***

## Retrieve a Secret

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="View secret metadata">
        Navigate to **Project > Key Manager > Secrets**. Click the secret name
        to open the detail page.

        The detail page shows:

        | Field           | Description                             |
        | --------------- | --------------------------------------- |
        | **Secret Type** | The type selected at creation           |
        | **Status**      | Active or Error                         |
        | **Mode**        | Block cipher mode (symmetric keys only) |
        | **Bit Length**  | Key size in bits                        |
        | **Created**     | Creation timestamp                      |
        | **Updated**     | Last modification timestamp             |
        | **Expiration**  | Expiration date, or `-` if none set     |
      </Step>

      <Step title="Retrieve the payload">
        <Warning>
          The Dashboard does not display secret payloads after creation. To retrieve
          a secret payload, use the CLI or API.
        </Warning>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all secrets in your project" theme={null}
    openstack secret list
    ```

    ```bash title="Show secret metadata (no payload)" theme={null}
    openstack secret show <secret-href>
    ```

    ```bash title="Retrieve the secret payload" theme={null}
    openstack secret get <secret-href> --payload
    ```

    <Warning>
      Treat retrieved payloads with the same care as any plaintext credential. Do not
      log, store in environment variables without restriction, or pipe to commands that
      might expose the value in process listings.
    </Warning>
  </Tab>
</Tabs>

***

## Delete a Secret

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Select secrets to delete">
        Navigate to **Project > Key Manager > Secrets**. Select one or more
        secrets using the checkboxes, then click **Delete** in the actions menu.

        Alternatively, click the **More** menu on a single secret row and
        select **Delete Secret**.
      </Step>

      <Step title="Confirm deletion">
        Confirm the deletion in the dialog. This action is permanent.

        <Danger>
          Deleting a secret is irreversible. If the secret is referenced by
          containers, Load Balancer listeners, or volume encryption, those
          references will break immediately. Update all references before deleting.
        </Danger>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Delete a secret" theme={null}
      openstack secret delete <secret-href>
      ```

      ```bash title="Delete by name (if unique)" theme={null}
      openstack secret delete $(openstack secret list --name db-root-password -f value -c "Secret href")
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Project Scope and Access

Secrets in Key Manager are **project-scoped** by default:

| Behavior                 | Description                                                                            |
| ------------------------ | -------------------------------------------------------------------------------------- |
| **Visibility**           | Secrets are visible only to users within the project that created them                 |
| **Cross-project access** | Not permitted unless explicitly granted via [ACL](/services/key-manager/acl)           |
| **Project switching**    | Switching projects in the Dashboard shows a different set of secrets                   |
| **Service access**       | Services like the Load Balancer access secrets using the project's service credentials |
| **Admin access**         | Platform administrators can view all secrets across projects from the admin panel      |

<Tip>
  To share a certificate or encryption key with another project, use an ACL
  rather than storing duplicate copies. This ensures a single source of truth
  and simplifies key rotation.
</Tip>

***

## Secret Type Reference

<AccordionGroup>
  <Accordion title="Opaque" icon="file">
    **Use case**: Store any arbitrary data — API keys, database passwords, configuration
    values, OAuth tokens, or binary blobs.

    No algorithm or key length metadata is required. This is the most flexible type
    and the default selection.

    ```bash title="CLI example" theme={null}
    openstack secret store \
      --name github-api-token \
      --secret-type opaque \
      --payload "ghp_xxxxxxxxxxxxxxxxxxxx" \
      --payload-content-type "text/plain"
    ```
  </Accordion>

  <Accordion title="Symmetric" icon="shield-check">
    **Use case**: Store symmetric encryption keys for AES, DES, or 3DES encryption.
    Used for volume encryption, object storage encryption, and application-level encryption.

    | Field      | Options                              |
    | ---------- | ------------------------------------ |
    | Algorithm  | AES, DES, 3DES                       |
    | Bit Length | AES: 128/192/256, DES: 56, 3DES: 168 |
    | Mode       | CBC, CTR, GCM                        |

    ```bash title="CLI example — AES-256 GCM" theme={null}
    openstack secret store \
      --name volume-encryption-key \
      --secret-type symmetric \
      --algorithm aes --bit-length 256 --mode gcm \
      --payload-content-type "application/octet-stream" \
      --payload-content-encoding base64 \
      --payload "$(openssl rand -base64 32)"
    ```
  </Accordion>

  <Accordion title="Public Key" icon="key">
    **Use case**: Store RSA, DSA, or EC public keys for key pair management,
    signature verification, or encryption.

    | Field      | Options                                              |
    | ---------- | ---------------------------------------------------- |
    | Algorithm  | RSA, DSA, EC                                         |
    | Bit Length | RSA: 2048/3072/4096, DSA: 2048/3072, EC: 256/384/521 |

    ```bash title="CLI example — RSA 4096 public key" theme={null}
    openstack secret store \
      --name ssh-public-key \
      --secret-type public \
      --algorithm rsa --bit-length 4096 \
      --payload-content-type "application/pkix-cert" \
      --payload-content-encoding base64 \
      --payload "$(base64 -w 0 id_rsa.pub)"
    ```
  </Accordion>

  <Accordion title="Private Key" icon="lock">
    **Use case**: Store RSA, DSA, or EC private keys for TLS, SSH, or code signing.
    Always pair with a corresponding public key or certificate.

    | Field      | Options                                              |
    | ---------- | ---------------------------------------------------- |
    | Algorithm  | RSA, DSA, EC                                         |
    | Bit Length | RSA: 2048/3072/4096, DSA: 2048/3072, EC: 256/384/521 |

    ```bash title="CLI example — RSA 4096 private key" theme={null}
    openstack secret store \
      --name app-tls-key \
      --secret-type private \
      --algorithm rsa --bit-length 4096 \
      --payload-content-type "application/pkcs8" \
      --payload-content-encoding base64 \
      --payload "$(base64 -w 0 private_key.pem)"
    ```
  </Accordion>

  <Accordion title="Certificate" icon="shield">
    **Use case**: Store X.509 TLS/SSL certificates for HTTPS termination,
    mTLS authentication, or certificate chain management.

    | Field      | Options                              |
    | ---------- | ------------------------------------ |
    | Algorithm  | RSA, EC                              |
    | Bit Length | RSA: 2048/3072/4096, EC: 256/384/521 |

    ```bash title="CLI example — TLS certificate" theme={null}
    openstack secret store \
      --name app-tls-cert \
      --secret-type certificate \
      --algorithm rsa --bit-length 2048 \
      --payload-content-type "application/pkix-cert" \
      --payload-content-encoding base64 \
      --payload "$(base64 -w 0 certificate.pem)"
    ```
  </Accordion>

  <Accordion title="Passphrase" icon="keyboard">
    **Use case**: Store passwords, passphrases, PINs, or other human-readable
    credential strings. No algorithm metadata is required.

    ```bash title="CLI example" theme={null}
    openstack secret store \
      --name db-admin-password \
      --secret-type passphrase \
      --payload "MyS3cur3P@ssw0rd!" \
      --payload-content-type "text/plain"
    ```
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Containers" icon="layers" href="/services/key-manager/containers">
    Bundle secrets into named containers for TLS and key pair management
  </Card>

  <Card title="Certificates" icon="shield" href="/services/key-manager/certificates">
    Store and manage TLS certificates using Key Manager
  </Card>

  <Card title="Access Control (ACL)" icon="lock" href="/services/key-manager/acl">
    Share secrets across projects with fine-grained access control
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/key-manager/troubleshooting">
    Resolve 403 errors, payload retrieval failures, and expired secret issues
  </Card>
</CardGroup>


# Transport Keys
Source: https://docs.xloud.tech/services/key-manager/transport-keys

Manage the RSA transport key in Xloud Key Manager for client-side encryption. View the current transport key and understand the rotation process.

## Overview

Transport keys enable clients to encrypt secret payloads before transmission,
ensuring secrets never appear in plaintext on the network or in intermediary processes.
The transport key is an RSA public key published by the Key Manager API. Clients
encrypt their secret payload with this key before POSTing to the API — the Key Manager
service uses its corresponding private key to unwrap the payload on the server side.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## How Transport Keys Work

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant API as Key Manager API
    participant Store as Secret Store

    Client->>API: GET /v1/transport_key (fetch RSA public key)
    API-->>Client: RSA public key
    Client->>Client: Encrypt payload with RSA public key
    Client->>API: POST /v1/secrets {payload: <encrypted>, transport_key_ref: <href>}
    API->>API: Decrypt payload with RSA private key
    API->>Store: Encrypt and store with master key
    Store-->>API: Storage reference
    API-->>Client: 201 Created {secret_ref: <href>}
```

<Info>
  Transport keys only protect secrets in transit. Stored secrets are encrypted using
  the secret store backend's master key, not the transport key.
</Info>

***

## View the Transport Key

```bash title="Get the current transport key" theme={null}
openstack secret transport key get
```

The output contains an RSA public key in PEM format. API clients use this to wrap
(encrypt) their secret payload before submission.

***

## Use a Transport Key in Secret Creation

```bash title="Create secret with transport key wrapping" theme={null}
# Fetch transport key
TRANSPORT_KEY_REF=$(openstack secret transport key get -c "Transport Key href" -f value)

# Encrypt payload with the transport key
WRAPPED_PAYLOAD=$(echo -n "S3cur3P@ss" | \
  openssl rsautl -encrypt -pubin -inkey <(openstack secret transport key get --payload) | \
  base64 -w 0)

# Submit encrypted payload
openstack secret store \
  --transport-key-ref "$TRANSPORT_KEY_REF" \
  --payload-content-encoding base64 \
  --payload "$WRAPPED_PAYLOAD" \
  --name secure-credential
```

***

## Transport Key Rotation

Transport key rotation is managed through XDeploy service configuration. After
generating a new RSA key pair:

<Steps>
  <Step title="Generate a new RSA key pair">
    Generate a new RSA key pair for the transport key in XDeploy Key Manager configuration.
  </Step>

  <Step title="Update Key Manager configuration">
    Update the Key Manager configuration to reference the new key pair.
  </Step>

  <Step title="Deploy the updated configuration">
    ```bash title="Deploy Key Manager configuration" theme={null}
    xavs-ansible deploy -t barbican
    ```
  </Step>

  <Step title="Notify API clients">
    Notify API clients that use transport key wrapping to retrieve the new transport key
    from the API:

    ```bash title="Fetch updated transport key" theme={null}
    openstack secret transport key get
    ```
  </Step>
</Steps>

<Warning>
  Existing secrets encrypted with the old transport key remain accessible — they are
  stored using the secret store backend encryption, not the transport key. Transport keys
  only protect secrets in transit during the creation request.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Backend Configuration" href="/services/key-manager/backend-config">
    Configure the backend that stores encrypted secret payloads
  </Card>

  <Card title="Security" href="/services/key-manager/security">
    Full Key Manager security hardening guidelines
  </Card>

  <Card title="Secret Stores" href="/services/key-manager/secret-stores">
    Manage multiple secret store backends
  </Card>

  <Card title="Admin Troubleshooting" href="/services/key-manager/admin-troubleshooting">
    Diagnose transport key and backend connectivity issues
  </Card>
</CardGroup>


# Key Manager Troubleshooting
Source: https://docs.xloud.tech/services/key-manager/troubleshooting

Resolve common Xloud Key Manager issues — 403 Forbidden errors, secret payload retrieval failures, Load Balancer TLS container rejections, and expired secrets.

## Overview

This guide covers user-facing Key Manager issues. For platform-level issues such as
backend connectivity failures or CA plugin errors, see the
[Admin Troubleshooting](/services/key-manager/admin-troubleshooting) guide.

***

## Common Issues

<AccordionGroup>
  <Accordion title="Secret creation returns 403 Forbidden" icon="lock">
    **Cause**: The current user's role does not include the Key Manager creator policy.

    **Diagnosis**:

    ```bash title="Check role assignments" theme={null}
    openstack role assignment list --user $OS_USERNAME --project $OS_PROJECT_NAME
    ```

    **Resolution**: Verify your project role assignment includes `member` or a custom
    role with Key Manager create permissions. Contact your administrator to assign the
    appropriate role.
  </Accordion>

  <Accordion title="Cannot retrieve secret payload" icon="eye-off">
    **Cause**: The secret has an ACL that does not include your user, or the secret
    belongs to a different project.

    **Diagnosis**:

    ```bash title="Show secret ACL" theme={null}
    openstack acl get <secret-href>
    ```

    **Resolution**: If your user is not listed and project access is disabled, request
    ACL modification from the secret owner or an administrator.

    ```bash title="Get your user ID for ACL comparison" theme={null}
    openstack token issue -c user_id -f value
    ```
  </Accordion>

  <Accordion title="Load Balancer rejects the TLS container" icon="shield-alert">
    **Cause**: The certificate container is missing the private key reference, the
    certificate is expired, or the certificate does not match the private key.

    **Diagnosis**:

    ```bash title="Verify container contents" theme={null}
    openstack secret container show <container-href>
    ```

    Confirm both `certificate` and `private_key` references are present.

    ```bash title="Check certificate expiry" theme={null}
    openstack secret get <cert-href> --payload | \
      openssl x509 -noout -dates
    ```

    ```bash title="Verify cert/key pair match" theme={null}
    diff <(openstack secret get <cert-href> --payload | openssl x509 -noout -modulus | md5sum) \
         <(openstack secret get <key-href> --payload | openssl rsa -noout -modulus | md5sum)
    ```

    **Resolution**:

    * If the container is missing the private key, delete and recreate it with both secrets
    * If the certificate is expired, store a renewed certificate and create a new container
    * If the cert/key pair do not match, verify you are using the correct private key file
  </Accordion>

  <Accordion title="Secret expired — access returns 404" icon="clock">
    **Cause**: The secret was created with an expiration date that has passed. Expired
    secrets are deleted automatically.

    **Resolution**: Create a new secret with the updated payload. If the secret is
    referenced by containers or services (e.g., Load Balancer), update each reference
    to point to the new secret or container.

    <Tip>
      Set calendar reminders for certificate and key expiration dates. Xloud Key Manager
      does not send expiration notifications — lifecycle management is the owner's responsibility.
    </Tip>
  </Accordion>

  <Accordion title="Order returns error — certificate issuance failed" icon="badge-x">
    **Cause**: The CA plugin is unreachable, the subject DN contains invalid fields, or
    the requested algorithm is not supported by the configured CA.

    **Diagnosis**:

    ```bash title="Show order error detail" theme={null}
    openstack secret order show <order-href>
    ```

    Review the `error_status_code` and `error_reason` fields.

    **Resolution**: Contact your platform administrator to verify CA plugin configuration
    and network connectivity. See the [Admin Troubleshooting](/services/key-manager/admin-troubleshooting)
    guide for CA plugin diagnostics.
  </Accordion>
</AccordionGroup>

***

## Diagnostic Commands

<CodeGroup>
  ```bash title="List all secrets in the project" theme={null}
  openstack secret list
  ```

  ```bash title="Show secret metadata" theme={null}
  openstack secret show <secret-href>
  ```

  ```bash title="Check ACL on a secret" theme={null}
  openstack acl get <secret-href>
  ```

  ```bash title="List certificate orders" theme={null}
  openstack secret order list
  ```

  ```bash title="List containers" theme={null}
  openstack secret container list
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Admin Troubleshooting" href="/services/key-manager/admin-troubleshooting">
    Platform-level issues — backend failures, CA plugin errors, ACL propagation
  </Card>

  <Card title="ACL" href="/services/key-manager/acl">
    Review and update access control lists on secrets
  </Card>

  <Card title="Certificates" href="/services/key-manager/certificates">
    Renew and replace expired certificates
  </Card>

  <Card title="Store Secrets" href="/services/key-manager/store-secrets">
    Create replacement secrets after expiration
  </Card>
</CardGroup>


# Key Manager User Guide
Source: https://docs.xloud.tech/services/key-manager/user-guide

Manage secrets, containers, certificates, and ACLs in Xloud Key Manager. Securely store credentials, private keys, and TLS certificates for your cloud.

<p>Overview</p>

Xloud Key Manager provides a centralized, encrypted store for secrets, certificates,
and cryptographic keys used across your private cloud. Secrets stored in Key Manager
are encrypted at rest and access-controlled independently of the resources that consume
them.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  Secrets stored in Key Manager are encrypted at rest. The secret payload is never
  logged, echoed in API responses after creation, or exposed in plain text outside
  of an explicit retrieve operation by an authorized caller.
</Note>

***

<p>Topics in This Guide</p>

<CardGroup>
  <Card title="Store Secrets" icon="vault" href="/services/key-manager/store-secrets">
    Store passwords, API tokens, private keys, and binary payloads with type metadata
  </Card>

  <Card title="Containers" icon="package" href="/services/key-manager/containers">
    Group related secrets into named bundles — certificate, RSA, and generic types
  </Card>

  <Card title="Certificates" icon="badge-check" href="/services/key-manager/certificates">
    Store externally issued TLS certificates or order new ones through a CA plugin
  </Card>

  <Card title="Access Control (ACL)" icon="list-checks" href="/services/key-manager/acl">
    Grant per-user or per-project read access to secrets and containers
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/key-manager/troubleshooting">
    Resolve 403 errors, expired secrets, and Load Balancer TLS container issues
  </Card>
</CardGroup>

***

<p>Key Concepts</p>

| Concept           | Description                                                                             |
| ----------------- | --------------------------------------------------------------------------------------- |
| **Secret**        | An encrypted payload — passwords, API keys, private keys, certificates, or binary blobs |
| **Container**     | A named group of related secrets — commonly certificate + private key + CA chain        |
| **Order**         | An async request to generate a key or issue a certificate through a CA plugin           |
| **ACL**           | Per-secret or per-container permission rules for cross-user or cross-project access     |
| **Transport Key** | An RSA public key used to encrypt secrets before upload for zero-plaintext transmission |

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Key Manager Admin Guide" icon="shield-check" href="/services/key-manager/admin-guide">
    Configure secret store backends, transport keys, and quotas
  </Card>

  <Card title="Load Balancer" icon="combine" href="/services/load-balancer/user-guide">
    Use TLS certificate containers in HTTPS listener configuration
  </Card>

  <Card title="DNS User Guide" icon="globe" href="/services/dns/user-guide">
    Configure DNSSEC with signing keys stored in Key Manager
  </Card>

  <Card title="Object Storage" icon="box" href="/services/object-storage/user-guide">
    Encrypt object containers with customer-managed keys from Key Manager
  </Card>
</CardGroup>


# Kubernetes (K8SaaS) Architecture
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/architecture

Understand the Xloud K8SaaS architecture — component roles, provisioning flow, infrastructure dependencies, and Conductor lifecycle management.

## Overview

Xloud Kubernetes as a Service (K8SaaS) automates the full lifecycle of Kubernetes clusters
on top of Xloud infrastructure services. The platform is built around a Conductor that
orchestrates cluster create, update, delete, and upgrade operations by composing calls to
Compute, Load Balancer, DNS, and Networking services through Xloud Orchestration templates.

<Warning>
  This guide requires administrator privileges. Misconfiguring the cluster driver or
  Orchestration integration affects all clusters across the platform.
</Warning>

***

## Component Architecture

```mermaid theme={null}
graph TD
    subgraph K8SaaS
        API[K8SaaS API<br/>REST endpoints]
        COND[Conductor<br/>Lifecycle orchestration]
        DB[(K8SaaS DB<br/>Clusters / Templates / CAs)]
    end

    subgraph Provisioning
        ORCH[Xloud Orchestration<br/>Heat Templates]
    end

    subgraph Infrastructure
        COMPUTE[Xloud Compute<br/>VM instances]
        LB[Xloud Load Balancer<br/>Master API VIPs]
        DNS[Xloud DNS<br/>Cluster endpoints]
        NET[Xloud Networking<br/>Tenant subnets]
    end

    subgraph Cluster
        MASTER[Master Nodes<br/>kube-apiserver, etcd, scheduler]
        WORKER[Worker Nodes<br/>kubelet, kube-proxy]
    end

    API <--> DB
    API -->|Dispatch lifecycle task| COND
    COND -->|Create stack| ORCH
    ORCH -->|Launch VMs| COMPUTE
    ORCH -->|Allocate VIPs| LB
    ORCH -->|Register records| DNS
    ORCH -->|Allocate subnets| NET
    COMPUTE --> MASTER
    COMPUTE --> WORKER
    MASTER -->|Kubernetes API| KUBECTL[kubectl / Users]
```

***

## Components

<AccordionGroup>
  <Accordion title="K8SaaS API" icon="code">
    RESTful API that accepts cluster lifecycle requests (create, update, delete, upgrade,
    config). Validates requests, stores state in the K8SaaS database, and dispatches
    async tasks to the Conductor.

    Deployed as: `magnum_api` container on controller nodes, behind the load balancer.
  </Accordion>

  <Accordion title="Conductor" icon="cpu">
    Long-running worker that executes cluster lifecycle operations. For each operation,
    it creates or updates an Orchestration stack, monitors stack progress, and updates
    cluster status in the database.

    Deployed as: `magnum_conductor` container on controller nodes. Multiple conductors
    can run for horizontal scaling — each claims tasks from the queue.
  </Accordion>

  <Accordion title="Cluster Database" icon="database">
    Stores cluster definitions, template configurations, node group state, and cluster
    CA private keys. Backed by the platform MariaDB instance.

    Schema includes: `cluster`, `cluster_template`, `nodegroup`, `x509keypair` tables.
  </Accordion>

  <Accordion title="Orchestration Templates" icon="layers">
    Each cluster driver (kubernetes) ships Heat templates that describe the full cluster
    resource stack: VM instances, network ports, floating IPs, security groups, LB members,
    and node bootstrap scripts.

    Template location: `/usr/lib/python3/dist-packages/magnum/drivers/k8s_fedora_coreos_v1/templates/`
  </Accordion>
</AccordionGroup>

***

## Cluster Provisioning Flow

```mermaid theme={null}
sequenceDiagram
    participant User
    participant API as K8SaaS API
    participant COND as Conductor
    participant ORCH as Xloud Orchestration
    participant COMPUTE as Xloud Compute

    User->>API: POST /v1/clusters
    API->>DB: Store cluster record (status: CREATE_IN_PROGRESS)
    API->>COND: Dispatch create_cluster task
    COND->>ORCH: Create Heat stack
    ORCH->>COMPUTE: Launch master + worker VMs
    COMPUTE-->>ORCH: VMs created
    ORCH-->>COND: Stack CREATE_COMPLETE
    COND->>DB: Update cluster (status: CREATE_COMPLETE)
    COND-->>User: Cluster ready
```

***

## Infrastructure Dependencies

| Service              | Role                                            | Minimum Version |
| -------------------- | ----------------------------------------------- | --------------- |
| Xloud Compute        | VM instances for master and worker nodes        | 2025.1          |
| Xloud Orchestration  | Stack management for cluster resources          | 2025.1          |
| Xloud Load Balancer  | API server VIP and Kubernetes service LBs       | 2025.1          |
| Xloud Networking     | Tenant subnet allocation for cluster nodes      | 2025.1          |
| Xloud DNS            | Endpoint records for ingress and services       | Optional        |
| Xloud Block Storage  | Persistent volume claims for stateful workloads | 2025.1          |
| Xloud Key Management | Cluster CA private key storage (optional)       | Optional        |

***

## Deployment Topology

<Tree>
  <Tree.Folder name="Controller Nodes">
    <Tree.File name="magnum_api — REST API" />

    <Tree.File name="magnum_conductor — Lifecycle orchestration" />
  </Tree.Folder>

  <Tree.Folder name="Cluster Nodes (per cluster)">
    <Tree.Folder name="Master Nodes">
      <Tree.File name="kube-apiserver" />

      <Tree.File name="etcd" />

      <Tree.File name="kube-scheduler" />

      <Tree.File name="kube-controller-manager" />
    </Tree.Folder>

    <Tree.Folder name="Worker Nodes">
      <Tree.File name="kubelet" />

      <Tree.File name="kube-proxy" />

      <Tree.File name="CNI plugin (calico or flannel)" />
    </Tree.Folder>
  </Tree.Folder>
</Tree>

***

## Next Steps

<CardGroup>
  <Card title="Cluster Drivers" href="/services/kubernetes/admin-guide/cluster-drivers">
    Configure and verify the Kubernetes cluster driver.
  </Card>

  <Card title="Template Management" href="/services/kubernetes/admin-guide/template-management">
    Create and manage public cluster templates for project teams.
  </Card>

  <Card title="Quotas" href="/services/kubernetes/admin-guide/quotas">
    Set per-project cluster and node count limits.
  </Card>

  <Card title="Security" href="/services/kubernetes/admin-guide/security">
    Configure TLS, RBAC, and node security groups.
  </Card>
</CardGroup>


# Certificate Management
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/certificates

Manage Xloud K8SaaS cluster certificate authorities — view cluster CAs, rotate certificates, and handle post-rotation kubeconfig refresh for project users.

## Overview

Every K8SaaS cluster has a dedicated certificate authority (CA) generated at provisioning
time. The CA signs all cluster component certificates (API server, etcd, kubelet) and
kubeconfig client certificates. Administrators rotate the CA when certificates approach
expiry, when credentials are suspected compromised, or after a security incident.

<Warning>
  Rotating a cluster CA invalidates **all existing kubeconfigs** for that cluster.
  All project users with access to the cluster must re-download their credentials
  after rotation. Notify all users before performing a CA rotation.
</Warning>

***

## View Cluster CA

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Container (admin view) > Clusters**, click a cluster, and select
    **Actions → Show CA** to view the cluster's current CA certificate (public key only).
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show cluster CA certificate" theme={null}
    openstack coe ca show prod-cluster-01
    ```

    The output displays the PEM-encoded CA certificate. Check the expiry date:

    ```bash title="Check CA certificate expiry" theme={null}
    openstack coe ca show prod-cluster-01 \
      | grep -A 20 "BEGIN CERTIFICATE" \
      | openssl x509 -noout -dates
    ```
  </Tab>
</Tabs>

***

## Rotate the Cluster CA

<Warning>
  CA rotation is irreversible. Existing kubeconfigs stop working immediately after rotation.
  Plan rotation during a maintenance window and prepare user communication in advance.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Notify users" icon="bell">
        Send advance notice to all users with cluster access that their kubeconfigs
        will be invalidated at the scheduled rotation time.
      </Step>

      <Step title="Initiate rotation" icon="rotate">
        Navigate to **Container (admin view) > Clusters**, click the cluster, and select
        **Actions → Rotate CA**. Confirm the rotation.

        The cluster enters `UPDATE_IN_PROGRESS` during CA regeneration.
      </Step>

      <Step title="Wait for completion" icon="clock">
        Wait for the cluster to return to `UPDATE_COMPLETE` status.
      </Step>

      <Step title="Direct users to refresh" icon="arrow-down">
        Notify all users to re-download their kubeconfig:

        ```bash title="Refresh kubeconfig (user command)" theme={null}
        openstack coe cluster config prod-cluster-01 \
          --dir ~/.kube \
          --force
        ```

        <Check>All users can connect to the cluster with the new kubeconfig.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Rotate cluster CA" theme={null}
    openstack coe ca rotate prod-cluster-01
    ```

    ```bash title="Monitor rotation progress" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status
    ```

    Wait for status to return to `UPDATE_COMPLETE`.

    ```bash title="Verify new CA certificate" theme={null}
    openstack coe ca show prod-cluster-01
    ```

    After rotation, all users must refresh their kubeconfig:

    ```bash title="Refresh kubeconfig" theme={null}
    openstack coe cluster config prod-cluster-01 \
      --dir ~/.kube \
      --force
    ```

    <Check>New CA is in place and kubectl connects with the refreshed kubeconfig.</Check>
  </Tab>
</Tabs>

***

## Certificate Expiry Planning

Plan CA rotations proactively to avoid service disruption from expired certificates.

| Certificate               | Default Validity      | Action Required                                |
| ------------------------- | --------------------- | ---------------------------------------------- |
| Cluster CA                | 10 years              | Rotate 90 days before expiry                   |
| API server TLS            | 1 year (auto-renewed) | Monitor via `openssl x509 -noout -dates`       |
| Node kubelet certificates | 1 year (auto-renewed) | Ensure `auto_healing_enabled=true` in template |

```bash title="Audit all cluster CA expiry dates" theme={null}
for cluster in $(openstack coe cluster list -f value -c name); do
  echo -n "$cluster: "
  openstack coe ca show $cluster \
    | grep -A 20 "BEGIN CERTIFICATE" \
    | openssl x509 -noout -enddate 2>/dev/null
done
```

***

## Xloud Key Management Integration

For the highest security posture, store cluster CA private keys in Xloud Key Management
rather than in the K8SaaS database.

<Note>
  Xloud Key Management must be deployed and the `magnum` service account must have the
  `creator` role on the Key Management service before enabling this integration.
</Note>

<Tabs>
  <Tab title="XDeploy" icon="server">
    <Steps>
      <Step title="Open XDeploy Configuration" icon="settings">
        Navigate to **XDeploy → Configuration** and select the **Advance Features** tab.
      </Step>

      <Step title="Enable KMS" icon="toggle-left">
        Set **Enable KMS** to **Yes**. This deploys the Xloud Key Management service
        and configures service account integration.
      </Step>

      <Step title="Set certificate manager type" icon="key">
        Open **XDeploy → Advanced Configuration**, select **magnum** in the Service Tree,
        then open or create `kubernetes.conf`. Add the following in the Code Editor:

        ```ini title="kubernetes.conf" theme={null}
        [certificate]
        cert_manager_type = barbican
        ```

        Click **Save Current File**.
      </Step>

      <Step title="Deploy the changes" icon="rocket">
        Navigate to **XDeploy → Operations** and run a **Reconfigure** for the
        Kubernetes and Key Management services.

        <Check>K8SaaS is configured to store CA private keys in Xloud Key Management.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```ini title="/etc/xavs/kubernetes/kubernetes.conf" theme={null}
    [certificate]
    cert_manager_type = barbican
    ```

    ```bash title="Restart API and Conductor after config change" theme={null}
    docker restart magnum_api magnum_conductor
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Security" href="/services/kubernetes/admin-guide/security">
    Configure RBAC and node security groups alongside certificate management.
  </Card>

  <Card title="Monitoring" href="/services/kubernetes/admin-guide/monitoring">
    Monitor certificate expiry and cluster health across all projects.
  </Card>

  <Card title="Cluster Upgrades (User)" href="/services/kubernetes/user-guide/cluster-upgrades">
    Advise users on refreshing kubeconfigs after upgrades and CA rotations.
  </Card>

  <Card title="Key Manager" href="/services/key-manager">
    Configure Xloud Key Management for secure CA private key storage.
  </Card>
</CardGroup>


# Cluster Drivers
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/cluster-drivers

Configure and manage Xloud K8SaaS cluster drivers — the provisioning engine that templates and deploys Kubernetes clusters on the platform infrastructure.

## Overview

Cluster drivers are the provisioning engine behind K8SaaS. Each driver defines the
orchestration templates and bootstrap scripts used to deploy and manage a specific
type of cluster. Xloud K8SaaS ships with the `kubernetes` driver as the supported
default. You can enable, disable, or configure driver behaviour through
the K8SaaS service configuration file.

<Warning>
  Changing driver configuration affects all future cluster deployments. Existing
  clusters continue to use the driver version they were provisioned with.
</Warning>

***

## Available Drivers

| Driver       | Status                 | Description                                                               |
| ------------ | ---------------------- | ------------------------------------------------------------------------- |
| `kubernetes` | <Badge>Default</Badge> | Provisions Kubernetes clusters using Xloud Orchestration (Heat) templates |

The `kubernetes` driver is the only supported driver in standard Xloud K8SaaS deployments.
Third-party drivers (e.g., `swarm`, `mesos`) are not enabled by default and are not
supported in production configurations.

***

## Verify the Driver is Active

```bash title="List registered driver entry points" theme={null}
openstack coe cluster template list --public
```

The output shows public cluster templates. If templates exist and are usable, the driver
is active.

```bash title="Check driver configuration" theme={null}
grep -E "disabled_drivers|driver" \
  /etc/xavs/kubernetes/kubernetes.conf
```

If `disabled_drivers` is empty or not present, all built-in drivers are enabled.

***

## Driver Configuration

Driver behaviour is configurable in the K8SaaS service configuration file.

<Tree>
  <Tree.Folder name="etc">
    <Tree.Folder name="xavs">
      <Tree.Folder name="kubernetes">
        <Tree.File name="kubernetes.conf — K8SaaS service configuration" />
      </Tree.Folder>
    </Tree.Folder>
  </Tree.Folder>
</Tree>

<Tabs>
  <Tab title="XDeploy" icon="server">
    <Steps>
      <Step title="Open Advanced Configuration" icon="settings">
        Navigate to **XDeploy → Advanced Configuration** and select **magnum** in the
        Service Tree.
      </Step>

      <Step title="Edit kubernetes.conf" icon="file-code">
        Select or create `kubernetes.conf` in the file list. Use the Code Editor to
        add or modify driver and conductor settings:

        ```ini title="kubernetes.conf — Disable experimental drivers" theme={null}
        [drivers]
        disabled_drivers = swarm,mesos
        ```

        ```ini title="kubernetes.conf — Increase conductor workers" theme={null}
        [DEFAULT]
        workers = 4
        ```

        Click **Save Current File** after each change.
      </Step>

      <Step title="Deploy the changes" icon="rocket">
        Navigate to **XDeploy → Operations** and run a **Reconfigure** for the
        Kubernetes service.

        <Check>Driver restrictions and conductor worker count are applied.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ### Disable Specific Drivers

    To restrict the platform to only the `kubernetes` driver and prevent experimental
    drivers from being registered:

    ```ini title="/etc/xavs/kubernetes/kubernetes.conf" theme={null}
    [drivers]
    disabled_drivers = swarm,mesos
    ```

    ### Conductor Worker Count

    Increase the number of Conductor workers to handle more concurrent cluster operations:

    ```ini title="/etc/xavs/kubernetes/kubernetes.conf" theme={null}
    [DEFAULT]
    workers = 4
    ```

    Restart the Conductor after changes:

    ```bash title="Restart Conductor" theme={null}
    docker restart magnum_conductor
    ```
  </Tab>
</Tabs>

***

## Driver Template Location

The `kubernetes` driver ships Orchestration templates that define the full cluster
resource stack. These templates are embedded in the K8SaaS container image.

```bash title="List driver templates inside the container" theme={null}
docker exec magnum_conductor find \
  /usr/lib/python3/dist-packages/magnum/drivers \
  -name "*.yaml" -maxdepth 4
```

To inspect a specific template:

```bash title="View base cluster template" theme={null}
docker exec magnum_conductor cat \
  /usr/lib/python3/dist-packages/magnum/drivers/k8s_fedora_coreos_v1/templates/cluster.yaml
```

***

## Validate Driver Functionality

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Container (admin view) > Cluster Templates** and create a test public
    template using the `kubernetes` driver. If the template is created successfully
    without errors, the driver is functioning correctly.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Verify Conductor is processing tasks" theme={null}
    docker logs magnum_conductor | tail -20
    ```

    ```bash title="Create a test template" theme={null}
    openstack coe cluster template create driver-test \
      --coe kubernetes \
      --image fedora-coreos-39 \
      --keypair admin-keypair \
      --flavor m1.medium \
      --external-network public
    ```

    ```bash title="Delete test template" theme={null}
    openstack coe cluster template delete driver-test
    ```

    <Check>Template creates and deletes without errors — driver is operational.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Template Management" href="/services/kubernetes/admin-guide/template-management">
    Create and publish public cluster templates for project teams.
  </Card>

  <Card title="Container Runtime" href="/services/kubernetes/admin-guide/container-runtime">
    Configure the container runtime for cluster templates.
  </Card>

  <Card title="Network Drivers" href="/services/kubernetes/admin-guide/network-drivers">
    Select and configure CNI plugins for production clusters.
  </Card>

  <Card title="Architecture" href="/services/kubernetes/admin-guide/architecture">
    Review the full K8SaaS component architecture.
  </Card>
</CardGroup>


# Container Runtime
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/container-runtime

Configure the container runtime for Xloud K8SaaS cluster templates — containerd recommendations, deprecated Docker support, and runtime configuration labels.

## Overview

The container runtime is specified in the cluster template and determines how container
images are pulled, started, and managed on cluster nodes. Xloud K8SaaS supports
`containerd` as the recommended runtime for all Kubernetes versions 1.24 and above.
Docker runtime support was removed from Kubernetes upstream in version 1.24.

***

## Supported Runtimes

| Runtime      | Status                     | Kubernetes Support | Recommended For                             |
| ------------ | -------------------------- | ------------------ | ------------------------------------------- |
| `containerd` | <Badge>Recommended</Badge> | 1.20+              | All production clusters on Kubernetes 1.24+ |
| `docker`     | <Badge>Deprecated</Badge>  | Removed in 1.24    | Legacy clusters only                        |

<Warning>
  Do not create new cluster templates with the `docker` runtime. Docker as the Kubernetes
  container runtime was removed in Kubernetes 1.24. All new templates must use `containerd`.
</Warning>

***

## Configure Runtime in a Template

Set the container runtime via the `container_runtime` label in the cluster template:

```bash title="Create template with containerd runtime" theme={null}
openstack coe cluster template create k8s-1.29-standard \
  --coe kubernetes \
  --image fedora-coreos-39 \
  --labels container_runtime=containerd \
  ...
```

```bash title="Verify runtime label on existing template" theme={null}
openstack coe cluster template show k8s-1.29-standard \
  -f value -c labels
```

Expected output includes `container_runtime=containerd`.

***

## containerd Configuration

The `containerd` runtime is pre-configured in the cluster node bootstrap script.
Default containerd settings suitable for most deployments:

| Setting           | Default                           | Description                                 |
| ----------------- | --------------------------------- | ------------------------------------------- |
| CRI socket        | `/run/containerd/containerd.sock` | Standard CRI socket path                    |
| Pause image       | Configured by K8SaaS bootstrap    | Kubernetes pause container image            |
| Sandbox image     | `registry.k8s.io/pause:3.9`       | Infrastructure sandbox container            |
| Image pull policy | `IfNotPresent`                    | Default pull policy for workload containers |

***

## Private Registry Configuration

If your organization uses an internal container registry, configure it in the cluster
template using the `insecure_registry` label:

```bash title="Template with internal registry" theme={null}
openstack coe cluster template create k8s-internal-registry \
  --coe kubernetes \
  --labels container_runtime=containerd \
  --labels insecure_registry=registry.xloud.local:5000 \
  ...
```

For HTTPS-enabled internal registries, configure the CA certificate via a custom
bootstrap script or a ConfigMap deployed to the cluster after provisioning.

***

## Verify Runtime on Running Nodes

After cluster deployment, confirm `containerd` is active on all nodes:

```bash title="Check runtime on all nodes" theme={null}
kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,RUNTIME:.status.nodeInfo.containerRuntimeVersion'
```

Expected output for each node: `containerd://1.7.x`

***

## Next Steps

<CardGroup>
  <Card title="Network Drivers" href="/services/kubernetes/admin-guide/network-drivers">
    Configure the CNI plugin for cluster network policy enforcement.
  </Card>

  <Card title="Template Management" href="/services/kubernetes/admin-guide/template-management">
    Create and publish public templates with the correct runtime configuration.
  </Card>

  <Card title="Security" href="/services/kubernetes/admin-guide/security">
    Harden container runtime configuration for production clusters.
  </Card>

  <Card title="Cluster Drivers" href="/services/kubernetes/admin-guide/cluster-drivers">
    Review the provisioning driver that uses the template runtime configuration.
  </Card>
</CardGroup>


# Monitoring Clusters
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/monitoring

Monitor Xloud K8SaaS cluster health, resource usage, and lifecycle status across all projects — admin-level cluster observability and health auditing.

## Overview

Administrators monitor the health and status of all Kubernetes clusters across all projects
from a single view. This includes tracking cluster lifecycle states, node health, control
plane availability, and identifying clusters that require attention — stuck in a non-terminal
state, unhealthy, or consuming unexpected resources.

***

## Admin Cluster Overview

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Container (admin view) > Clusters** to view all clusters across all projects.

    | Column            | Description                                                                     |
    | ----------------- | ------------------------------------------------------------------------------- |
    | **Name**          | Cluster identifier                                                              |
    | **Status**        | Lifecycle state: `CREATE_COMPLETE`, `UPDATE_IN_PROGRESS`, `CREATE_FAILED`, etc. |
    | **Health Status** | Kubernetes-level health: `HEALTHY`, `UNHEALTHY`, `UNKNOWN`                      |
    | **Master Count**  | Number of control plane nodes                                                   |
    | **Node Count**    | Number of worker nodes                                                          |
    | **Project**       | Owning project                                                                  |
    | **Created**       | Provisioning timestamp                                                          |

    <Tip>
      Filter by Status to quickly identify clusters in non-terminal states that require
      operator attention (e.g., `CREATE_IN_PROGRESS` for more than 30 minutes).
    </Tip>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all clusters across all projects" theme={null}
    openstack coe cluster list --all
    ```

    ```bash title="Filter for non-healthy clusters" theme={null}
    openstack coe cluster list --all \
      -f json | jq '.[] | select(.health_status != "HEALTHY")'
    ```

    ```bash title="Show detailed status for a specific cluster" theme={null}
    openstack coe cluster show <cluster-name> -f json
    ```

    ```bash title="List clusters stuck in a transitional state" theme={null}
    openstack coe cluster list --all \
      | grep -v -E "CREATE_COMPLETE|UPDATE_COMPLETE|DELETE_COMPLETE"
    ```
  </Tab>
</Tabs>

***

## Cluster Health States

| Status               | Meaning                            | Operator Action                          |
| -------------------- | ---------------------------------- | ---------------------------------------- |
| `CREATE_COMPLETE`    | Cluster deployed and healthy       | None required                            |
| `UPDATE_COMPLETE`    | Last update succeeded              | None required                            |
| `CREATE_IN_PROGRESS` | Provisioning in progress           | Monitor; investigate if >30 min          |
| `UPDATE_IN_PROGRESS` | Update (scale/upgrade) in progress | Monitor                                  |
| `CREATE_FAILED`      | Provisioning failed                | Investigate `status_reason`, assist user |
| `UPDATE_FAILED`      | Scale or upgrade failed            | Investigate and assist user              |
| `DELETE_IN_PROGRESS` | Cluster being deleted              | Monitor                                  |
| `DELETE_FAILED`      | Deletion failed                    | Manual stack cleanup required            |

***

## Check Control Plane Availability

For high-availability clusters (3 master nodes), verify the control plane load balancer
and all master nodes are healthy:

```bash title="Show cluster API address" theme={null}
openstack coe cluster show <cluster-name> \
  -f value -c api_address
```

```bash title="Test API server availability" theme={null}
curl -sk https://<api-address>:6443/healthz
```

Expected: `ok`

***

## Identify Unhealthy Clusters

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Container (admin view) > Clusters** and sort by **Health Status**.
    Clusters with `UNHEALTHY` or `UNKNOWN` health status should be investigated
    and the project owner notified.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Find unhealthy clusters" theme={null}
    openstack coe cluster list --all \
      -f json \
      | jq -r '.[] | select(.health_status != "HEALTHY") | [.name, .status, .health_status] | @tsv'
    ```

    For each unhealthy cluster, check the associated compute instances:

    ```bash title="List instances for a cluster" theme={null}
    openstack server list \
      --name <cluster-name> \
      -f table -c ID -c Name -c Status
    ```
  </Tab>
</Tabs>

***

## Audit Inactive Clusters

Identify clusters that may have been abandoned by project teams to reclaim compute
resources:

```bash title="List all clusters with creation date" theme={null}
openstack coe cluster list --all \
  -f table -c name -c project_id -c created_at -c status
```

Contact the project owner for clusters that have been in `CREATE_COMPLETE` status for
an extended period without recent activity, and confirm whether they are still needed.

***

## Next Steps

<CardGroup>
  <Card title="Quotas" href="/services/kubernetes/admin-guide/quotas">
    Manage per-project cluster limits to prevent resource exhaustion.
  </Card>

  <Card title="Troubleshooting" href="/services/kubernetes/admin-guide/troubleshooting">
    Diagnose failed clusters and stuck lifecycle states.
  </Card>

  <Card title="Security" href="/services/kubernetes/admin-guide/security">
    Audit cluster security groups and RBAC configuration.
  </Card>

  <Card title="Certificates" href="/services/kubernetes/admin-guide/certificates">
    Monitor and rotate cluster certificate authorities.
  </Card>
</CardGroup>


# Network Drivers
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/network-drivers

Configure Xloud K8SaaS network drivers (CNI plugins) — choose between Calico and Flannel, understand NetworkPolicy support, and plan for production.

## Overview

The network driver (CNI plugin) determines how Pod-to-Pod and Pod-to-Service networking
works within Kubernetes clusters. The driver is selected in the cluster template and
cannot be changed after cluster deployment. Xloud K8SaaS supports two drivers: Calico
for production workloads requiring NetworkPolicy enforcement, and Flannel for simplified
development environments.

***

## Driver Comparison

| Driver    | NetworkPolicy                 | Performance            | Encryption         | Recommended For                                   |
| --------- | ----------------------------- | ---------------------- | ------------------ | ------------------------------------------------- |
| `calico`  | Full Kubernetes NetworkPolicy | BGP (native routing)   | Optional WireGuard | Production clusters requiring pod-level isolation |
| `flannel` | None                          | VXLAN overlay (simple) | None               | Development / test environments                   |

<Tip>
  Use `calico` for all production templates. Flannel is appropriate only for isolated
  development environments where NetworkPolicy is not required.
</Tip>

***

## Calico Configuration

Calico is the recommended CNI for production Xloud K8SaaS clusters. It supports
Kubernetes NetworkPolicy resources and provides BGP-based native routing for optimal
performance in datacenter environments.

### Create Template with Calico

```bash title="Create production template with Calico" theme={null}
openstack coe cluster template create k8s-1.29-prod \
  --coe kubernetes \
  --network-driver calico \
  ...
```

### Verify Calico is Running

After cluster deployment, confirm Calico components are healthy:

```bash title="Check Calico pods" theme={null}
kubectl get pods -n kube-system \
  | grep -E "calico|bird"
```

Expected: `calico-node` pods on every node, all `Running`.

```bash title="Check Calico node status" theme={null}
kubectl exec -n kube-system \
  $(kubectl get pod -n kube-system -l k8s-app=calico-node -o name | head -1) \
  -- calicoctl node status
```

***

## Flannel Configuration

Flannel provides a simple VXLAN overlay network. No NetworkPolicy support — all pods can
communicate with all other pods across the cluster.

```bash title="Create development template with Flannel" theme={null}
openstack coe cluster template create k8s-dev \
  --coe kubernetes \
  --network-driver flannel \
  ...
```

### Verify Flannel is Running

```bash title="Check Flannel pods" theme={null}
kubectl get pods -n kube-system \
  | grep flannel
```

Expected: `kube-flannel` DaemonSet pods on every node, all `Running`.

***

## Applying NetworkPolicy (Calico clusters only)

After deploying a Calico cluster, you can apply Kubernetes NetworkPolicy resources to
restrict Pod communication. Example policy to allow only intra-namespace traffic:

```yaml title="Default deny-all ingress policy" theme={null}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
```

```bash title="Apply the policy" theme={null}
kubectl apply -f default-deny-ingress.yaml
```

***

## Network Driver Immutability

The CNI driver is set at cluster template creation and **cannot be changed** after a
cluster is deployed. To switch CNI plugins:

1. Deploy a new cluster from a template with the desired driver
2. Migrate workloads to the new cluster
3. Delete the old cluster

<Warning>
  There is no in-place CNI migration path. Plan your driver selection carefully
  before deploying production clusters.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Container Runtime" href="/services/kubernetes/admin-guide/container-runtime">
    Configure the container runtime alongside the network driver in templates.
  </Card>

  <Card title="Security" href="/services/kubernetes/admin-guide/security">
    Apply node security groups and restrict Kubernetes API server access.
  </Card>

  <Card title="Template Management" href="/services/kubernetes/admin-guide/template-management">
    Publish templates with the correct network driver for project teams.
  </Card>

  <Card title="Troubleshooting" href="/services/kubernetes/admin-guide/troubleshooting">
    Diagnose CNI-related node NotReady issues and network failures.
  </Card>
</CardGroup>


# Kubernetes Quotas
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/quotas

Manage Xloud K8SaaS per-project cluster quotas — set cluster count limits, monitor usage, and coordinate with compute quotas for node capacity planning.

## Overview

K8SaaS enforces per-project cluster quotas independent of compute quotas. A cluster quota
limits the total number of clusters a project can create. Cluster nodes also consume
compute quota (vCPU, RAM, and storage) — both quota systems must have sufficient headroom
for cluster creation to succeed. This page covers setting K8SaaS quotas, monitoring usage,
and coordinating limits across both services.

<Note>
  **Prerequisites**

  * Cloud administrator role
  * Project IDs for the projects you wish to quota
</Note>

***

## Default Quota

By default, K8SaaS applies a platform-wide quota of **20 clusters** per project. Projects
without an explicit quota record use this default.

***

## Set a Project Quota

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Quotas" icon="compass">
        Navigate to
        **Container (admin view) > Quotas**.
      </Step>

      <Step title="Create or edit quota" icon="sliders">
        Click **Create Quota** and fill in:

        | Field                     | Description              | Example           |
        | ------------------------- | ------------------------ | ----------------- |
        | **Project**               | Target project           | `production-team` |
        | **Hard Limit (Clusters)** | Maximum clusters allowed | `10`              |
      </Step>

      <Step title="Save" icon="circle-check">
        Click **Create**. The quota takes effect immediately for the selected project.

        <Check>Project quota is set and visible in the quota list.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a quota for a project" theme={null}
    openstack coe quota create \
      --project <project-id> \
      --hard-limit 10 \
      kubernetes
    ```

    ```bash title="Show quota for a project" theme={null}
    openstack coe quota show \
      --project <project-id> \
      kubernetes
    ```

    ```bash title="List all project quotas" theme={null}
    openstack coe quota list
    ```

    <Tip>
      Set conservative limits for development projects (3–5 clusters) and higher
      limits for production projects (10–20). Coordinate with compute quota to
      ensure sufficient resources for the maximum expected cluster footprint.
    </Tip>
  </Tab>
</Tabs>

***

## Compute Quota Coordination

K8SaaS cluster nodes consume compute resources. Use the following formula to estimate
the compute quota needed for a project's K8SaaS allocation:

```
Total vCPU needed = (master_flavor_vcpu × master_count + worker_flavor_vcpu × worker_count) × max_clusters
Total RAM needed  = (master_flavor_ram_gb × master_count + worker_flavor_ram_gb × worker_count) × max_clusters
```

Example: 5 clusters × (3 masters × 8 vCPU + 6 workers × 4 vCPU) = 5 × (24 + 24) = 240 vCPU

```bash title="Check current project compute quota" theme={null}
openstack quota show --detail <project-id>
```

```bash title="Increase compute quota for a project" theme={null}
openstack quota set \
  --cores 240 \
  --ram 491520 \
  <project-id>
```

***

## Monitor Quota Usage

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Container (admin view) > Quotas** to view current usage versus limit
    for all projects. Click a project to see individual cluster details.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show quota usage for a specific project" theme={null}
    openstack coe quota show \
      --project <project-id> \
      kubernetes
    ```

    The output shows `hard_limit` and `in_use` values:

    ```
    hard_limit = 10
    in_use = 3
    ```

    ```bash title="Count active clusters in a project" theme={null}
    openstack coe cluster list --all \
      | grep <project-name> | wc -l
    ```
  </Tab>
</Tabs>

***

## Update an Existing Quota

```bash title="Update an existing cluster quota" theme={null}
openstack coe quota update \
  --project <project-id> \
  --hard-limit 20 \
  kubernetes
```

***

## Next Steps

<CardGroup>
  <Card title="Monitoring" href="/services/kubernetes/admin-guide/monitoring">
    Monitor cluster health and resource consumption across all projects.
  </Card>

  <Card title="Template Management" href="/services/kubernetes/admin-guide/template-management">
    Create public templates that standardize cluster resource usage.
  </Card>

  <Card title="Security" href="/services/kubernetes/admin-guide/security">
    Apply RBAC policies to restrict cluster operations per role.
  </Card>

  <Card title="Troubleshooting" href="/services/kubernetes/admin-guide/troubleshooting">
    Resolve quota-related cluster creation failures.
  </Card>
</CardGroup>


# Kubernetes Security
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/security

Secure Xloud K8SaaS deployments — TLS configuration, Kubernetes RBAC enforcement, node security groups, and hardened node image practices.

## Overview

K8SaaS security encompasses multiple layers: TLS encryption for all cluster API
communication, Kubernetes RBAC within clusters, network-level restrictions via security
groups, and hardened node images. This page covers the key security controls and how to
verify they are correctly applied across your K8SaaS deployment.

<Warning>
  Security misconfigurations in K8SaaS can expose Kubernetes API servers to unauthorized
  access or allow cross-tenant privilege escalation. Review each control before deploying
  production clusters.
</Warning>

***

## TLS Configuration

All K8SaaS cluster API communication is TLS-encrypted. Each cluster has a dedicated
CA generated at provisioning time.

<AccordionGroup>
  <Accordion title="Verify TLS on the API server" icon="lock">
    ```bash title="Check API server TLS certificate" theme={null}
    openssl s_client -connect <api-address>:6443 \
      -showcerts 2>/dev/null \
      | openssl x509 -noout -issuer -subject -dates
    ```

    Verify the certificate is issued by the cluster's CA (not a self-signed root)
    and has not expired.
  </Accordion>

  <Accordion title="Key Management integration" icon="key">
    Enable Xloud Key Management to store CA private keys outside the K8SaaS database.
    This prevents CA private keys from being accessible to anyone with database
    read access.

    <Tabs>
      <Tab title="XDeploy" icon="server">
        <Steps>
          <Step title="Enable KMS" icon="toggle-left">
            Navigate to **XDeploy → Configuration → Advance Features** and set
            **Enable KMS** to **Yes**.
          </Step>

          <Step title="Configure certificate manager" icon="key">
            Navigate to **XDeploy → Advanced Configuration**, select **magnum** in the
            Service Tree, then open or create `kubernetes.conf`. Add the following:

            ```ini title="kubernetes.conf" theme={null}
            [certificate]
            cert_manager_type = barbican
            ```

            Click **Save Current File**.
          </Step>

          <Step title="Deploy" icon="rocket">
            Run **XDeploy → Operations → Reconfigure** for both the Key Management
            and Kubernetes services.
          </Step>
        </Steps>
      </Tab>

      <Tab title="CLI" icon="terminal">
        ```ini title="/etc/xavs/kubernetes/kubernetes.conf" theme={null}
        [certificate]
        cert_manager_type = barbican
        ```

        ```bash title="Restart services" theme={null}
        docker restart magnum_api magnum_conductor
        ```
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

***

## Kubernetes RBAC

Kubernetes RBAC is enabled by default on all K8SaaS clusters. The `--authorization-mode`
flag is set to `Node,RBAC` in the Kubernetes API server configuration.

<AccordionGroup>
  <Accordion title="Enforce least-privilege service accounts" icon="shield-check">
    Do not grant the `cluster-admin` role to application service accounts. Use
    namespace-scoped roles and bindings:

    ```yaml title="Namespace-scoped role example" theme={null}
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      namespace: production
      name: app-reader
    rules:
    - apiGroups: [""]
      resources: ["pods", "services"]
      verbs: ["get", "list", "watch"]
    ```

    ```bash title="Check for overly broad cluster-admin bindings" theme={null}
    kubectl get clusterrolebindings \
      -o json | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name'
    ```
  </Accordion>

  <Accordion title="Audit RBAC bindings" icon="clipboard-list">
    Periodically audit cluster role bindings to identify unauthorized privilege escalation:

    ```bash title="List all cluster-level role bindings" theme={null}
    kubectl get clusterrolebindings -o wide
    ```

    ```bash title="Find service accounts with admin privileges" theme={null}
    kubectl get clusterrolebindings \
      -o json | jq -r '.items[] | select(.roleRef.name | startswith("cluster-admin")) | "\(.metadata.name): \(.subjects[].name)"'
    ```
  </Accordion>
</AccordionGroup>

***

## Node Security Groups

Each K8SaaS cluster creates dedicated security groups for master and worker nodes.
Review and restrict these groups to permit only required traffic.

```bash title="List cluster security groups" theme={null}
openstack security group list \
  | grep <cluster-name>
```

### Required Ports

| Port        | Protocol | Direction          | Purpose                                       |
| ----------- | -------- | ------------------ | --------------------------------------------- |
| 6443        | TCP      | Inbound to masters | Kubernetes API server (kubectl, kubelets)     |
| 2379-2380   | TCP      | Master to master   | etcd peer and client communication            |
| 10250       | TCP      | Master to workers  | kubelet API                                   |
| 10255       | TCP      | Any (optional)     | kubelet read-only API (disable if not needed) |
| VXLAN / BGP | UDP/TCP  | Node to node       | CNI overlay (Flannel) or BGP peering (Calico) |

```bash title="Restrict master API port to management CIDR only" theme={null}
openstack security group rule create \
  --protocol tcp \
  --dst-port 6443 \
  --remote-ip <management-cidr> \
  <master-security-group-id>
```

```bash title="Remove any overly permissive rule" theme={null}
openstack security group rule delete <rule-id>
```

***

## Node Image Hardening

Use CIS-benchmarked or hardened node images for production cluster templates.

<CardGroup>
  <Card title="Use Hardened Images" icon="shield-check">
    Use images with CIS Level 1 benchmark applied. Fedora CoreOS provides a minimal,
    immutable OS — a good baseline for Kubernetes node hardening.
  </Card>

  <Card title="Rotate Images on Security Updates" icon="rotate">
    When node OS security updates are released, create a new cluster template with
    the updated image and upgrade existing clusters to replace all nodes.
  </Card>
</CardGroup>

***

## Validation Checklist

<Steps>
  <Step title="TLS verified" icon="lock">
    API server TLS certificate is valid, issued by the cluster CA, and not expired.
  </Step>

  <Step title="RBAC audited" icon="shield-check">
    No unnecessary `cluster-admin` bindings exist for application service accounts.
  </Step>

  <Step title="Security groups restricted" icon="network">
    Master API port 6443 is restricted to management CIDR only.
  </Step>

  <Step title="Key Management configured" icon="key">
    CA private keys stored in Xloud Key Management, not the K8SaaS database.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Certificates" href="/services/kubernetes/admin-guide/certificates">
    Manage and rotate cluster CA certificates.
  </Card>

  <Card title="Monitoring" href="/services/kubernetes/admin-guide/monitoring">
    Monitor cluster health and audit security events.
  </Card>

  <Card title="Network Drivers" href="/services/kubernetes/admin-guide/network-drivers">
    Configure Calico for NetworkPolicy enforcement in production clusters.
  </Card>

  <Card title="Key Manager" href="/services/key-manager">
    Configure Xloud Key Management for cluster CA storage.
  </Card>
</CardGroup>


# Template Management
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/template-management

Create and manage Xloud K8SaaS public cluster templates — publish standardized Kubernetes configurations for project teams, and enforce platform-wide defaults.

## Overview

Administrators create and publish **public** cluster templates that are shared across
all projects. Public templates provide standardized, pre-approved Kubernetes configurations
— ensuring project teams use consistent Kubernetes versions, flavors, and network drivers
without needing to configure templates themselves. This page covers creating, updating,
and retiring public templates.

<Warning>
  Changes to public templates do not affect clusters already deployed from those templates.
  However, all new clusters created from an updated template will use the new configuration.
</Warning>

***

## Public vs Private Templates

| Type    | Visibility           | Created By    | Used By          |
| ------- | -------------------- | ------------- | ---------------- |
| Public  | All projects         | Administrator | Any project team |
| Private | Current project only | Any user      | Own project only |

***

## Create a Public Template

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Cluster Templates (admin)" icon="compass">
        Navigate to
        **Container (admin view) > Cluster Templates**.
      </Step>

      <Step title="Create template" icon="plus">
        Click **Create Cluster Template** and fill in the fields. Set **Public** to `True`.

        | Field                   | Recommended Value | Notes                                    |
        | ----------------------- | ----------------- | ---------------------------------------- |
        | **Master Flavor**       | `m1.xlarge`       | Control plane: 8 vCPU / 16 GB minimum    |
        | **Master LB Enabled**   | `True`            | Required for HA control plane            |
        | **Network Driver**      | `calico`          | NetworkPolicy enforcement for production |
        | **Volume Driver**       | `cinder`          | Enables PersistentVolumeClaims           |
        | **Docker Volume Size**  | `50` GB           | Container image storage per node         |
        | **Floating IP Enabled** | `True`            | kubectl access via floating IP           |
        | **Public**              | `True`            | Share across all projects                |
      </Step>

      <Step title="Add auto-healing labels" icon="settings">
        In the **Labels** field, add:

        ```
        auto_healing_enabled=true
        ```

        This enables automatic node replacement when a node becomes unhealthy.
      </Step>

      <Step title="Save" icon="circle-check">
        Click **Create Cluster Template**. The template appears in all project template
        lists immediately.

        <Check>Template is visible in all projects and usable for cluster creation.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create public cluster template" theme={null}
    openstack coe cluster template create k8s-1.29-standard \
      --coe kubernetes \
      --image fedora-coreos-39 \
      --keypair admin-keypair \
      --flavor m1.large \
      --master-flavor m1.xlarge \
      --network-driver calico \
      --volume-driver cinder \
      --external-network public \
      --dns-nameserver 8.8.8.8 \
      --docker-volume-size 50 \
      --master-lb-enabled \
      --floating-ip-enabled \
      --public \
      --labels auto_healing_enabled=true,auto_scaling_enabled=true
    ```

    ```bash title="Verify template is public" theme={null}
    openstack coe cluster template show k8s-1.29-standard \
      -f value -c public
    ```

    Expected: `True`
  </Tab>
</Tabs>

***

## Template Version Lifecycle

Maintain a clear versioning strategy for public templates as Kubernetes versions are
released and deprecated.

| Stage                       | Action                                                    |
| --------------------------- | --------------------------------------------------------- |
| New version available       | Create new public template (e.g., `k8s-1.30-standard`)    |
| Previous version stable     | Keep both templates public — users can choose             |
| Previous version deprecated | Mark old template as non-public; notify project teams     |
| Previous version retired    | Delete old template after all clusters have been upgraded |

```bash title="Make an existing template private (deprecation)" theme={null}
openstack coe cluster template update k8s-1.28-standard \
  replace public=False
```

```bash title="Delete a retired template" theme={null}
openstack coe cluster template delete k8s-1.28-standard
```

<Warning>
  Templates referenced by active clusters cannot be deleted. All clusters using a
  template must be upgraded or deleted before the template can be removed.
</Warning>

***

## Recommended Label Configuration

Labels control advanced Kubernetes and platform features at the template level.

| Label                    | Value        | Effect                                                |
| ------------------------ | ------------ | ----------------------------------------------------- |
| `auto_healing_enabled`   | `true`       | Automatically replace unhealthy nodes                 |
| `auto_scaling_enabled`   | `true`       | Enable Cluster Autoscaler                             |
| `container_runtime`      | `containerd` | Set container runtime (required for Kubernetes 1.24+) |
| `cloud_provider_enabled` | `true`       | Enable Xloud cloud provider for LoadBalancer services |
| `cinder_csi_enabled`     | `true`       | Enable Cinder CSI driver for PersistentVolumeClaims   |

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to a non-admin project and check **Container > Cluster Templates**.
    Verify the public template is visible and selectable for cluster creation.

    <Check>Public template is visible in all projects and usable for cluster creation.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all public templates" theme={null}
    openstack coe cluster template list --public
    ```

    ```bash title="Verify from a non-admin project context" theme={null}
    OS_PROJECT_NAME=<user-project> \
      openstack coe cluster template list
    ```

    <Check>Template appears in all project contexts.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Quotas" href="/services/kubernetes/admin-guide/quotas">
    Set per-project limits on cluster and node counts.
  </Card>

  <Card title="Container Runtime" href="/services/kubernetes/admin-guide/container-runtime">
    Configure the recommended container runtime for new templates.
  </Card>

  <Card title="Network Drivers" href="/services/kubernetes/admin-guide/network-drivers">
    Choose CNI plugins and configure NetworkPolicy defaults.
  </Card>

  <Card title="Certificates" href="/services/kubernetes/admin-guide/certificates">
    Manage cluster certificate authorities and perform CA rotations.
  </Card>
</CardGroup>


# Kubernetes Admin Troubleshooting
Source: https://docs.xloud.tech/services/kubernetes/admin-guide/troubleshooting

Diagnose and resolve Xloud K8SaaS platform issues — Conductor failures, Heat stack errors, certificate problems, and cross-project cluster failures.

## Overview

This guide covers administrator-level troubleshooting for the K8SaaS platform — from
Conductor startup failures and Heat stack errors to certificate issues and quota
enforcement problems. For user-facing issues such as individual cluster access failures,
see the [Kubernetes User Troubleshooting](/services/kubernetes/user-guide/troubleshooting) guide.

***

## Common Issues

<AccordionGroup>
  <Accordion title="Clusters fail to create across projects" icon="circle-x">
    **Cause**: The K8SaaS Conductor cannot reach the Orchestration service, or the
    cluster template references an image or flavor that does not exist.

    **Resolution**:

    ```bash title="Check Conductor logs" theme={null}
    docker logs -f magnum_conductor
    ```

    Look for `ConnectionError`, `NotFound`, or `AuthenticationRequired` messages.

    ```bash title="Verify Orchestration service is healthy" theme={null}
    openstack stack list
    ```

    ```bash title="Verify node image exists" theme={null}
    openstack image show fedora-coreos-39
    ```

    If the image is missing, upload it and ask project teams to retry cluster creation.
  </Accordion>

  <Accordion title="Heat stack in FAILED state" icon="layers">
    **Cause**: The Orchestration template failed during resource creation — quota
    exhaustion, a dependency failure (LB or DNS), or a template rendering error.

    **Resolution**:

    ```bash title="Show failed stack events" theme={null}
    openstack stack event list <cluster-stack-name> \
      --nested-depth 3 \
      | grep -i fail
    ```

    ```bash title="Find the stack name for a cluster" theme={null}
    openstack coe cluster show <cluster-name> \
      -f value -c stack_id
    ```

    Address the root cause (quota, network, service availability) and then delete
    the failed cluster before retrying:

    ```bash title="Delete failed cluster" theme={null}
    openstack coe cluster delete <cluster-name>
    ```
  </Accordion>

  <Accordion title="Certificate errors after node replacement" icon="lock">
    **Cause**: Replaced nodes received new TLS certificates that do not match the
    cluster CA recorded in the K8SaaS database.

    **Resolution**: Rotate the cluster CA to regenerate consistent certificates:

    ```bash title="Rotate cluster CA" theme={null}
    openstack coe ca rotate <cluster-name>
    ```

    Notify all project users to refresh their kubeconfig after the rotation completes.
  </Accordion>

  <Accordion title="DELETE_FAILED — cluster cannot be removed" icon="trash-2">
    **Cause**: The underlying Heat stack has resources in a failed state that prevent
    cleanup, or a resource dependency is blocking deletion.

    **Resolution**:

    ```bash title="Show stack deletion error" theme={null}
    openstack stack show <stack-id> -f value -c stack_status_reason
    ```

    Manually delete the blocking resource (e.g., a floating IP still attached to a
    deleted VM):

    ```bash title="List stack resources" theme={null}
    openstack stack resource list <stack-id>
    ```

    ```bash title="Force-delete the Heat stack" theme={null}
    openstack stack delete --yes <stack-id>
    ```

    After manual cleanup, delete the cluster record from K8SaaS:

    ```bash title="Force delete cluster record" theme={null}
    openstack coe cluster delete --force <cluster-name>
    ```
  </Accordion>

  <Accordion title="Conductor not processing cluster tasks" icon="cpu">
    **Cause**: The Conductor is overloaded, has lost database connectivity, or crashed
    due to an unhandled exception.

    **Resolution**:

    ```bash title="Check Conductor status and logs" theme={null}
    docker ps --filter name=magnum_conductor
    docker logs --tail 50 magnum_conductor
    ```

    Restart the Conductor if it shows as unhealthy or has no recent log output:

    ```bash title="Restart Conductor" theme={null}
    docker restart magnum_conductor
    ```

    Increase the worker count if the Conductor is consistently behind:

    ```ini title="/etc/xavs/kubernetes/kubernetes.conf" theme={null}
    [DEFAULT]
    workers = 4
    ```
  </Accordion>
</AccordionGroup>

***

## Diagnostic Commands Reference

```bash title="Check all K8SaaS container statuses" theme={null}
docker ps --filter name=magnum
```

```bash title="List all clusters across all projects" theme={null}
openstack coe cluster list --all
```

```bash title="Show cluster with full detail" theme={null}
openstack coe cluster show <cluster-name> -f json
```

```bash title="Show Orchestration stack events" theme={null}
openstack stack event list <stack-id> --nested-depth 2
```

```bash title="Check K8SaaS API logs" theme={null}
docker logs --tail 100 magnum_api
```

***

## Next Steps

<CardGroup>
  <Card title="Monitoring" href="/services/kubernetes/admin-guide/monitoring">
    Monitor all clusters for failed and stuck lifecycle states.
  </Card>

  <Card title="Certificates" href="/services/kubernetes/admin-guide/certificates">
    Resolve certificate errors with CA rotation.
  </Card>

  <Card title="Quotas" href="/services/kubernetes/admin-guide/quotas">
    Resolve quota-related cluster creation failures.
  </Card>

  <Card title="User Troubleshooting" href="/services/kubernetes/user-guide/troubleshooting">
    User-facing guide for individual cluster access and health issues.
  </Card>
</CardGroup>


# Kubernetes (K8SaaS) CLI Reference
Source: https://docs.xloud.tech/services/kubernetes/cli-reference

Complete openstack coe cluster CLI commands for managing Xloud Kubernetes clusters — create, scale, upgrade, and manage cluster templates.

## Overview

The `openstack coe cluster` and `openstack coe cluster template` command groups manage the full lifecycle of Kubernetes clusters.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python magnumclient installed: `pip install python-magnumclient`
</Note>

***

## Cluster Templates

<CodeGroup>
  ```bash title="List cluster templates" theme={null}
  openstack coe cluster template list
  ```

  ```bash title="Create cluster template" theme={null}
  openstack coe cluster template create \
    --name k8s-ubuntu \
    --image Ubuntu-22.04 \
    --keypair my-keypair \
    --external-network external \
    --dns-nameserver 8.8.8.8 \
    --flavor m1.medium \
    --master-flavor m1.large \
    --docker-volume-size 50 \
    --network-driver flannel \
    --coe kubernetes
  ```

  ```bash title="Show template" theme={null}
  openstack coe cluster template show k8s-ubuntu
  ```

  ```bash title="Delete template" theme={null}
  openstack coe cluster template delete k8s-ubuntu
  ```
</CodeGroup>

***

## Clusters

<CodeGroup>
  ```bash title="List clusters" theme={null}
  openstack coe cluster list
  ```

  ```bash title="Create cluster" theme={null}
  openstack coe cluster create \
    --cluster-template k8s-ubuntu \
    --keypair my-keypair \
    --master-count 1 \
    --node-count 3 \
    my-cluster
  ```

  ```bash title="Show cluster" theme={null}
  openstack coe cluster show my-cluster
  ```

  ```bash title="Get kubeconfig" theme={null}
  openstack coe cluster config my-cluster
  ```

  ```bash title="Scale worker nodes" theme={null}
  openstack coe cluster resize my-cluster --node-count 5
  ```

  ```bash title="Upgrade cluster" theme={null}
  openstack coe cluster upgrade my-cluster <new-template>
  ```

  ```bash title="Delete cluster" theme={null}
  openstack coe cluster delete my-cluster
  ```
</CodeGroup>

***

## Node Groups

<CodeGroup>
  ```bash title="List node groups" theme={null}
  openstack coe nodegroup list my-cluster
  ```

  ```bash title="Create node group" theme={null}
  openstack coe nodegroup create my-cluster \
    --name gpu-workers \
    --flavor m1.gpu \
    --node-count 2 \
    --min-nodes 1 \
    --max-nodes 5
  ```

  ```bash title="Scale node group" theme={null}
  openstack coe nodegroup update my-cluster gpu-workers \
    --node-count 3
  ```

  ```bash title="Delete node group" theme={null}
  openstack coe nodegroup delete my-cluster gpu-workers
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Deploy Cluster Guide" href="/services/kubernetes/user-guide/deploy-cluster">
    Step-by-step guide to deploying your first cluster
  </Card>

  <Card title="Scale Cluster Guide" href="/services/kubernetes/user-guide/scale-cluster">
    Add and remove worker nodes
  </Card>
</CardGroup>


# Kubernetes as a Service
Source: https://docs.xloud.tech/services/kubernetes/index

Deploy and manage production-grade Kubernetes clusters on Xloud infrastructure — automated provisioning, scaling, and lifecycle management for.

Deploy and manage production-grade Kubernetes clusters directly on your Xloud private
cloud. Xloud Kubernetes as a Service (K8SaaS) automates cluster provisioning, node
scaling, certificate management, and upgrades — giving teams a fully managed Kubernetes
experience on dedicated infrastructure.

<Card title="XAVS — Advanced Virtualization Solutions" icon="external-link" href="https://xloud.tech/xavs">
  Product details on xloud.tech
</Card>

***

<p>Kubernetes as a Service</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/kubernetes/user-guide">
    Create cluster templates, deploy Kubernetes clusters, scale node groups, access
    clusters via kubectl, and manage the full cluster lifecycle.
  </Card>

  <Card title="Admin Guide" icon="shield-check" href="/services/kubernetes/admin-guide">
    Configure cluster drivers, manage quotas, administer certificate authorities,
    choose container runtimes, and monitor cluster health across projects.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/kubernetes/cli-reference">
    Create cluster templates, provision clusters, manage node groups, and retrieve
    kubeconfig files using the openstack CLI.
  </Card>

  <Card title="Compute Service" icon="server" href="/services/compute">
    Xloud Compute provides the virtual machine instances that form Kubernetes master
    and worker nodes.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="One-Click Cluster Provisioning" icon="rocket" href="/services/kubernetes/user-guide/deploy-cluster">
    Deploy a fully configured Kubernetes cluster from a template in minutes — nodes,
    networking, certificates, and API access configured automatically.
  </Card>

  <Card title="Elastic Node Scaling" icon="up-right-and-down-left-from-center" href="/services/kubernetes/user-guide/scale-cluster">
    Scale worker node groups up or down on demand. Autoscaling policies adapt cluster
    capacity to workload pressure automatically.
  </Card>

  <Card title="Multi-Node-Group Clusters" icon="layers" href="/services/kubernetes/user-guide/node-groups">
    Create heterogeneous clusters with multiple node groups — differentiated by flavor,
    availability zone, or hardware profile — for GPU, memory-optimized, and general
    workloads.
  </Card>

  <Card title="Automated Certificate Management" icon="lock" href="/services/kubernetes/admin-guide/certificates">
    Cluster certificates are generated, rotated, and managed automatically. No manual
    PKI configuration required.
  </Card>

  <Card title="Network Plugin Choice" icon="network" href="/services/kubernetes/admin-guide/network-drivers">
    Choose between Flannel (simple overlay) and Calico (network policy enforcement)
    at cluster template creation time. Switch plugins between clusters, not within.
  </Card>

  <Card title="Cluster Upgrades" icon="arrow-up" href="/services/kubernetes/user-guide/cluster-upgrades">
    Perform rolling Kubernetes version upgrades with zero downtime — master nodes
    upgraded first, worker nodes drained and replaced sequentially.
  </Card>
</CardGroup>

***

<p>How It Works</p>

```mermaid theme={null}
graph TD
    U[User / Dashboard] -->|Create cluster| API[K8SaaS API]
    API -->|Provision stack| ORCH[Xloud Orchestration]
    ORCH -->|Launch VMs| COMPUTE[Xloud Compute]
    ORCH -->|Allocate VIPs| LB[Xloud Load Balancer]
    ORCH -->|Assign DNS| DNS[Xloud DNS]
    COMPUTE -->|Master nodes| MASTER[Control Plane VMs]
    COMPUTE -->|Worker nodes| WORKER[Worker Node VMs]
    MASTER -->|Join cluster| K8S[Kubernetes API Server]
    WORKER -->|Join cluster| K8S
    K8S -->|Accessible via| KUBECTL[kubectl / Dashboard]
```

***

<p>Supported Kubernetes Versions</p>

<Note>
  Supported Kubernetes versions are determined by the cluster templates deployed on
  your platform. Contact your administrator for the available version matrix.
</Note>

| Channel | Version | Status                 |
| ------- | ------- | ---------------------- |
| Stable  | 1.29.x  | <Badge>GA</Badge>      |
| LTS     | 1.28.x  | <Badge>GA</Badge>      |
| Preview | 1.30.x  | <Badge>Preview</Badge> |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Virtual machine instances that run Kubernetes master and worker nodes
  </Card>

  <Card title="Xloud Load Balancer" icon="combine" href="/services/load-balancer">
    API server and service load balancing for Kubernetes clusters
  </Card>

  <Card title="Xloud Networking" icon="network" href="/services/networking/index">
    Tenant networks, floating IPs, and security groups for cluster nodes
  </Card>

  <Card title="Xloud Block Storage" icon="hard-drive" href="/services/storage/index">
    Persistent volume claims backed by Xloud block storage
  </Card>

  <Card title="Xloud DNS" icon="globe" href="/services/dns">
    DNS records for Kubernetes ingress and service endpoints
  </Card>

  <Card title="Xloud Key Management" icon="lock" href="/services/key-manager">
    Secrets management for cluster certificates and service credentials
  </Card>
</CardGroup>


# Access Your Kubernetes Cluster
Source: https://docs.xloud.tech/services/kubernetes/user-guide/access-cluster

Download kubeconfig, configure kubectl, and verify access to your Xloud K8SaaS cluster — connect to the API server and validate node readiness.

## Overview

After a cluster reaches `CREATE_COMPLETE` status, you connect to it using `kubectl`
and the cluster's kubeconfig file. Xloud K8SaaS generates a unique certificate authority
per cluster and embeds the cluster API server endpoint, CA certificate, and user
credentials into the kubeconfig. This page covers downloading credentials, configuring
`kubectl`, and verifying connectivity.

<Note>
  **Prerequisites**

  * A cluster in `CREATE_COMPLETE` status
  * `kubectl` installed locally ([install guide](https://kubernetes.io/docs/tasks/tools/))
  * Network access from your machine to the cluster API server (port 6443)
</Note>

***

## Download Cluster Credentials

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to the cluster" icon="compass">
        Navigate to
        **Container > Clusters**. Click your cluster name.
      </Step>

      <Step title="Download kubeconfig" icon="download">
        On the cluster detail page, click **Download kubeconfig**. Save the file
        (e.g., `prod-cluster-01-kubeconfig.yaml`) to your machine.
      </Step>

      <Step title="Configure kubectl" icon="terminal">
        Set the `KUBECONFIG` environment variable to point to the downloaded file:

        ```bash title="Set kubeconfig" theme={null}
        export KUBECONFIG=~/Downloads/prod-cluster-01-kubeconfig.yaml
        ```
      </Step>

      <Step title="Verify access" icon="circle-check">
        ```bash title="List cluster nodes" theme={null}
        kubectl get nodes
        ```

        All nodes should show `STATUS: Ready`.

        <Check>kubectl is connected and all cluster nodes are Ready.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Download kubeconfig" icon="download">
        ```bash title="Save kubeconfig to ~/.kube/" theme={null}
        mkdir -p ~/.kube
        openstack coe cluster config prod-cluster-01 \
          --dir ~/.kube \
          --force
        ```

        This writes the kubeconfig to `~/.kube/config` (or a named file in that directory).
      </Step>

      <Step title="Set kubeconfig path" icon="settings">
        ```bash title="Export kubeconfig path" theme={null}
        export KUBECONFIG=~/.kube/config
        ```
      </Step>

      <Step title="Verify connectivity" icon="circle-check">
        ```bash title="Check cluster info" theme={null}
        kubectl cluster-info
        ```

        ```bash title="List all nodes" theme={null}
        kubectl get nodes -o wide
        ```

        Expected: all master and worker nodes show `STATUS: Ready`.

        <Check>kubectl connects to the API server and all nodes are Ready.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Manage Multiple Cluster Contexts

If you access multiple clusters, use `kubectl` contexts to switch between them.

```bash title="View all configured contexts" theme={null}
kubectl config get-contexts
```

```bash title="Switch to a specific cluster context" theme={null}
kubectl config use-context <context-name>
```

```bash title="Merge multiple kubeconfigs" theme={null}
export KUBECONFIG=~/.kube/cluster-01-config:~/.kube/cluster-02-config
kubectl config view --merge --flatten > ~/.kube/config
```

***

## API Server Endpoint

The cluster API server endpoint is accessible via the master load balancer floating IP
or directly via the master node floating IP (for single-master clusters).

```bash title="Show API server endpoint" theme={null}
openstack coe cluster show prod-cluster-01 \
  -f value -c api_address
```

<Info>
  The API server listens on port 6443 (HTTPS). Ensure your workstation's network allows
  outbound TCP to port 6443 on the cluster API server IP.
</Info>

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Container > Clusters** and click your cluster. Verify:

    * **Health Status**: `HEALTHY`
    * All listed nodes show `STATUS: Ready`

    <Check>Cluster is healthy and all nodes are ready to accept workloads.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Full node readiness check" theme={null}
    kubectl get nodes -o wide
    ```

    ```bash title="Check cluster component health" theme={null}
    kubectl get componentstatuses
    ```

    ```bash title="Deploy a test pod to verify scheduling" theme={null}
    kubectl run test-pod \
      --image=nginx \
      --restart=Never \
      --rm \
      -it \
      -- echo "Cluster access confirmed"
    ```

    <Check>Test pod schedules, runs, and exits successfully.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Node Groups" href="/services/kubernetes/user-guide/node-groups">
    Create specialized node pools and schedule workloads to specific groups.
  </Card>

  <Card title="Scale Cluster" href="/services/kubernetes/user-guide/scale-cluster">
    Add or remove worker nodes from your cluster.
  </Card>

  <Card title="Cluster Upgrades" href="/services/kubernetes/user-guide/cluster-upgrades">
    Upgrade your cluster to a newer Kubernetes version.
  </Card>

  <Card title="Troubleshooting" href="/services/kubernetes/user-guide/troubleshooting">
    Resolve kubectl connectivity and node health issues.
  </Card>
</CardGroup>


# Cluster Templates
Source: https://docs.xloud.tech/services/kubernetes/user-guide/cluster-templates

Create and manage Xloud K8SaaS cluster templates — define Kubernetes version, node flavor, network driver, and container runtime for reusable cluster.

## Overview

Cluster templates are reusable blueprints that define the configuration for Kubernetes
clusters. A template captures the Kubernetes version, node image, instance flavor, network
driver, and storage backend. Once created, a template can be used to deploy multiple
clusters with consistent configurations. Your administrator can publish public templates shared
across all projects; you can also create private templates for your own use.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * A keypair created in your project for node SSH access
  * Compute quota sufficient for cluster nodes
  * A project network with external access
</Note>

***

## Template Fields Reference

| Field                      | Required | Description                     | Recommended Value            |
| -------------------------- | -------- | ------------------------------- | ---------------------------- |
| **Name**                   | Yes      | Unique template identifier      | `k8s-1.29-prod`              |
| **Container Infra Driver** | Yes      | Cluster provisioning engine     | `kubernetes`                 |
| **Image**                  | Yes      | Boot image for cluster nodes    | `fedora-coreos-39`           |
| **Keypair**                | Yes      | SSH key for node access         | Your project keypair         |
| **Flavor**                 | Yes      | Instance size for worker nodes  | `m1.large` (4 vCPU / 8 GB)   |
| **Master Flavor**          | Yes      | Instance size for master nodes  | `m1.xlarge` (8 vCPU / 16 GB) |
| **Network Driver**         | Yes      | Container network interface     | `calico` (production)        |
| **Volume Driver**          | Yes      | Persistent volume backend       | `cinder`                     |
| **External Network**       | Yes      | Floating IP source              | `public`                     |
| **DNS Nameserver**         | Yes      | Resolver for cluster nodes      | `8.8.8.8`                    |
| **Master LB Enabled**      | No       | Load balance master nodes       | `True` (HA)                  |
| **Docker Volume Size**     | No       | Container storage per node (GB) | `50`                         |
| **Floating IP Enabled**    | No       | Assign floating IPs to nodes    | `True`                       |

***

## Create a Cluster Template

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Cluster Templates" icon="compass">
        Navigate to
        **Container > Cluster Templates**.
      </Step>

      <Step title="Create the template" icon="plus">
        Click **Create Cluster Template** and fill in the required fields using the
        reference table above.

        <Tip>
          For production clusters, enable **Master LB** to place a load balancer in front
          of master nodes and set **Master Count** to 3 when deploying the cluster.
        </Tip>
      </Step>

      <Step title="Configure advanced options" icon="settings">
        Expand **Advanced** to set optional parameters:

        * **Insecure Registry**: Internal container registry URL (if used)
        * **Fixed Network / Subnet**: Pin cluster nodes to a specific project network
        * **Labels**: Key-value labels for platform-specific options (e.g., `auto_healing_enabled=true`)
      </Step>

      <Step title="Save the template" icon="circle-check">
        Click **Create Cluster Template**. The template appears in the list with status `ACTIVE`.

        <Check>Template created with status `ACTIVE` and visible in the template list.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Create the template" icon="plus">
        ```bash title="Create cluster template" theme={null}
        openstack coe cluster template create k8s-1.29-prod \
          --coe kubernetes \
          --image fedora-coreos-39 \
          --keypair my-keypair \
          --flavor m1.large \
          --master-flavor m1.xlarge \
          --network-driver calico \
          --volume-driver cinder \
          --external-network public \
          --dns-nameserver 8.8.8.8 \
          --docker-volume-size 50 \
          --master-lb-enabled \
          --labels auto_healing_enabled=true
        ```
      </Step>

      <Step title="Verify the template" icon="circle-check">
        ```bash title="List templates" theme={null}
        openstack coe cluster template list
        ```

        ```bash title="Show template details" theme={null}
        openstack coe cluster template show k8s-1.29-prod
        ```

        <Check>Template shows `coe_version` populated and no error messages.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Network Driver Comparison

The network driver (CNI plugin) is configured at template creation and cannot be changed
after cluster deployment.

| Driver    | Network Policy                 | Performance              | Recommended For                              |
| --------- | ------------------------------ | ------------------------ | -------------------------------------------- |
| `calico`  | Full NetworkPolicy enforcement | Moderate                 | Production workloads requiring pod isolation |
| `flannel` | None                           | Higher (simpler overlay) | Development and test clusters                |

<Tip>
  Use `calico` for all production templates. Flannel is appropriate only for isolated
  test environments where Kubernetes NetworkPolicy is not required.
</Tip>

***

## Manage Existing Templates

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Container > Cluster Templates**. From this view you can:

    * Click a template name to view its full configuration
    * Click **Actions → Delete** to remove unused templates (templates in use by clusters cannot be deleted)
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all templates" theme={null}
    openstack coe cluster template list
    ```

    ```bash title="Show a template" theme={null}
    openstack coe cluster template show <template-name>
    ```

    ```bash title="Update a template field" theme={null}
    openstack coe cluster template update <template-name> \
      replace dns_nameserver=1.1.1.1
    ```

    ```bash title="Delete a template" theme={null}
    openstack coe cluster template delete <template-name>
    ```

    <Warning>
      Templates referenced by active clusters cannot be deleted. Delete all clusters
      using the template before removing it.
    </Warning>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Deploy a Cluster" href="/services/kubernetes/user-guide/deploy-cluster">
    Provision a Kubernetes cluster from your template in minutes.
  </Card>

  <Card title="Node Groups" href="/services/kubernetes/user-guide/node-groups">
    Add specialized node pools with different flavors to your cluster.
  </Card>

  <Card title="Access Cluster" href="/services/kubernetes/user-guide/access-cluster">
    Download kubeconfig and connect kubectl to your cluster.
  </Card>

  <Card title="Kubernetes Admin Guide" href="/services/kubernetes/admin-guide">
    Administrator reference for managing public templates and quotas.
  </Card>
</CardGroup>


# Cluster Upgrades
Source: https://docs.xloud.tech/services/kubernetes/user-guide/cluster-upgrades

Upgrade Xloud K8SaaS clusters to newer Kubernetes versions — rolling upgrades, sequential version progression, and post-upgrade validation.

## Overview

Xloud K8SaaS supports rolling Kubernetes version upgrades. Master nodes are upgraded
first, followed by sequential worker node replacement — draining each node before
removing it and adding a replacement with the new version. This approach maintains
cluster availability throughout the upgrade process.

<Warning>
  Kubernetes does not support skipping minor versions. Upgrade sequentially through
  each minor version (e.g., 1.27 → 1.28 → 1.29). Attempting to skip a version
  (e.g., 1.27 → 1.29 directly) is not supported and will fail.
</Warning>

<Note>
  **Prerequisites**

  * A cluster in `CREATE_COMPLETE` or `UPDATE_COMPLETE` status
  * A cluster template with the target Kubernetes version available in your project
  * Sufficient compute quota to temporarily run extra nodes during the rolling replacement
</Note>

***

## Pre-Upgrade Checklist

<Steps>
  <Step title="Verify the target template exists" icon="file-text">
    Confirm a cluster template with the target Kubernetes version is available:

    ```bash title="List available templates" theme={null}
    openstack coe cluster template list
    ```

    If the target version template does not exist, ask your administrator to create
    a public template or create a private one in your project.
  </Step>

  <Step title="Check cluster health" icon="heart-pulse">
    ```bash title="Verify cluster is healthy" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status -c health_status
    ```

    Proceed only when `status = CREATE_COMPLETE` or `UPDATE_COMPLETE` and
    `health_status = HEALTHY`.
  </Step>

  <Step title="Verify node readiness" icon="server">
    ```bash title="Check all nodes are Ready" theme={null}
    kubectl get nodes
    ```

    All nodes must show `STATUS: Ready` before starting the upgrade.
  </Step>

  <Step title="Notify application teams" icon="bell">
    Inform application owners that the cluster will undergo a rolling upgrade.
    While the upgrade is non-destructive, pods will be evicted during node replacement.
    Ensure workloads use multiple replicas and have Pod Disruption Budgets configured.
  </Step>
</Steps>

***

## Perform the Upgrade

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to the cluster" icon="compass">
        Navigate to
        **Container > Clusters**. Click your cluster name.
      </Step>

      <Step title="Start upgrade" icon="arrow-up">
        Click **Actions → Upgrade Cluster**. Select the target cluster template
        with the desired Kubernetes version and confirm.
      </Step>

      <Step title="Monitor progress" icon="clock">
        The cluster enters `UPDATE_IN_PROGRESS` status. The upgrade progresses through:

        1. Master nodes upgraded first (one at a time for HA clusters)
        2. Worker nodes drained and replaced sequentially

        <Tip>
          Use `kubectl get nodes -w` to watch node status changes in real time
          during the rolling replacement.
        </Tip>
      </Step>

      <Step title="Verify completion" icon="circle-check">
        The cluster returns to `UPDATE_COMPLETE` when all nodes are on the new version.

        <Check>All nodes show the new Kubernetes version in `kubectl get nodes`.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Initiate upgrade" icon="arrow-up">
        ```bash title="Upgrade cluster to new template" theme={null}
        openstack coe cluster upgrade prod-cluster-01 \
          k8s-1.30-prod
        ```
      </Step>

      <Step title="Monitor progress" icon="clock">
        ```bash title="Check upgrade status" theme={null}
        watch -n 15 "openstack coe cluster show prod-cluster-01 \
          -f value -c status -c status_reason"
        ```

        ```bash title="Watch node replacement in kubectl" theme={null}
        kubectl get nodes -w
        ```
      </Step>

      <Step title="Verify all nodes on new version" icon="circle-check">
        ```bash title="Check Kubernetes version on all nodes" theme={null}
        kubectl get nodes \
          -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion'
        ```

        All nodes should show the same target version.

        <Check>All nodes show the target Kubernetes version and `STATUS: Ready`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Post-Upgrade Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    After upgrade completes, verify:

    * Cluster shows `Health Status: HEALTHY`
    * All nodes listed in the cluster detail show `STATUS: Ready`

    <Check>Cluster is healthy and all nodes are at the target version.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Verify cluster health" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status -c health_status
    ```

    ```bash title="Verify node versions" theme={null}
    kubectl get nodes -o wide
    ```

    ```bash title="Check system pod health" theme={null}
    kubectl get pods -n kube-system
    ```

    All system pods should show `STATUS: Running` or `Completed`.

    <Check>Cluster is `UPDATE_COMPLETE`, all nodes are `Ready`, and system pods are running.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Troubleshooting" href="/services/kubernetes/user-guide/troubleshooting">
    Resolve upgrade failures and post-upgrade issues.
  </Card>

  <Card title="Node Groups" href="/services/kubernetes/user-guide/node-groups">
    Manage specialized node pools after an upgrade.
  </Card>

  <Card title="Kubernetes Admin Guide — Certificates" href="/services/kubernetes/admin-guide/certificates">
    Rotate cluster CA certificates after major version upgrades.
  </Card>

  <Card title="Access Cluster" href="/services/kubernetes/user-guide/access-cluster">
    Refresh kubeconfig after a cluster upgrade if the CA was rotated.
  </Card>
</CardGroup>


# Deploy a Kubernetes Cluster
Source: https://docs.xloud.tech/services/kubernetes/user-guide/deploy-cluster

Provision a production-grade Kubernetes cluster on Xloud infrastructure — select a template, configure master and worker node counts, and monitor.

## Overview

Deploying a Kubernetes cluster on Xloud K8SaaS provisions the full control plane and
worker node infrastructure from a cluster template. The process creates virtual machine
instances for master and worker nodes, configures the Kubernetes API server, installs
the selected CNI plugin, and allocates a load balancer VIP for master access. A cluster
deployment typically completes in 5–10 minutes depending on node count.

<Note>
  **Prerequisites**

  * A cluster template created in your project (see [Cluster Templates](/services/kubernetes/user-guide/cluster-templates))
  * Compute quota to provision master + worker nodes
  * A project network with external connectivity
  * A keypair available in your project
</Note>

***

## Deploy a Cluster

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Clusters" icon="compass">
        Navigate to
        **Container > Clusters**.
      </Step>

      <Step title="Create a cluster" icon="plus">
        Click **Create Cluster** and complete the form:

        | Field                 | Description                         | Recommended           |
        | --------------------- | ----------------------------------- | --------------------- |
        | **Cluster Name**      | Unique name for this cluster        | `prod-cluster-01`     |
        | **Cluster Template**  | Select a template from your project | `k8s-1.29-prod`       |
        | **Master Count**      | Number of control plane nodes       | `1` (dev) or `3` (HA) |
        | **Node Count**        | Initial number of worker nodes      | `3`                   |
        | **Keypair**           | SSH key for node access             | Your project keypair  |
        | **Availability Zone** | Fault domain for node placement     | `nova`                |
      </Step>

      <Step title="Launch the cluster" icon="rocket">
        Click **Create Cluster**. The status transitions through:

        `CREATE_IN_PROGRESS` → `CREATE_COMPLETE`

        Provisioning typically takes 5–10 minutes. Refresh the cluster list to track progress.

        <Tip>
          For production clusters, use 3 master nodes and ensure the cluster template
          has **Master LB Enabled** set to `True` for control plane high availability.
        </Tip>
      </Step>

      <Step title="Verify completion" icon="circle-check">
        The cluster shows `Status: CREATE_COMPLETE` and `Health Status: HEALTHY` when ready.

        <Check>Cluster is deployed and all nodes are healthy.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Create the cluster" icon="plus">
        ```bash title="Deploy Kubernetes cluster" theme={null}
        openstack coe cluster create prod-cluster-01 \
          --cluster-template k8s-1.29-prod \
          --master-count 3 \
          --node-count 3 \
          --keypair my-keypair
        ```

        The command returns the cluster UUID immediately. Provisioning continues
        asynchronously.
      </Step>

      <Step title="Monitor progress" icon="clock">
        ```bash title="Check cluster status" theme={null}
        openstack coe cluster show prod-cluster-01 \
          -f value -c status -c status_reason
        ```

        Poll until `status` is `CREATE_COMPLETE`. If status shows `CREATE_FAILED`,
        check `status_reason` for the error message.
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="List all clusters" theme={null}
        openstack coe cluster list
        ```

        ```bash title="Show cluster health" theme={null}
        openstack coe cluster show prod-cluster-01 \
          -f value -c health_status -c master_count -c node_count
        ```

        <Check>Status is `CREATE_COMPLETE` and `health_status` is `HEALTHY`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Cluster Sizing Reference

| Cluster Type  | Master Count | Worker Count | Use Case                            |
| ------------- | ------------ | ------------ | ----------------------------------- |
| Development   | `1`          | `2–3`        | Single developer or CI/CD pipelines |
| Staging       | `1`          | `3–5`        | Pre-production testing              |
| Production HA | `3`          | `5+`         | Customer-facing workloads           |

<Warning>
  A single master node is a single point of failure. For production workloads, always
  deploy 3 master nodes with the master load balancer enabled in the cluster template.
</Warning>

***

## Monitor Deployment Progress

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Container > Clusters**. The `Status` column updates in
    real time. Click the cluster name to view node-level detail.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Watch cluster status in a loop" theme={null}
    watch -n 10 "openstack coe cluster show prod-cluster-01 \
      -f value -c status -c status_reason"
    ```

    If a failure occurs, the `status_reason` field contains the error from the
    underlying Orchestration stack:

    ```bash title="Show failure details" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status_reason
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Access Cluster" href="/services/kubernetes/user-guide/access-cluster">
    Download kubeconfig and verify kubectl connectivity to your new cluster.
  </Card>

  <Card title="Scale Cluster" href="/services/kubernetes/user-guide/scale-cluster">
    Add or remove worker nodes from the cluster default node group.
  </Card>

  <Card title="Node Groups" href="/services/kubernetes/user-guide/node-groups">
    Create specialized node pools with different flavors and configurations.
  </Card>

  <Card title="Troubleshooting" href="/services/kubernetes/user-guide/troubleshooting">
    Resolve deployment failures and cluster health issues.
  </Card>
</CardGroup>


# Node Groups
Source: https://docs.xloud.tech/services/kubernetes/user-guide/node-groups

Create and manage Xloud K8SaaS node groups — add heterogeneous worker pools to a cluster, configure autoscaling, and scale individual groups independently.

## Overview

Node groups are named pools of worker nodes within a single Kubernetes cluster. Each
group can have a different instance flavor, enabling heterogeneous clusters — e.g.,
a GPU node pool for machine learning workloads alongside a general-purpose pool for web
services. Node groups are scaled independently, giving fine-grained control over capacity
without affecting other workloads on the cluster.

<Note>
  **Prerequisites**

  * A cluster in `CREATE_COMPLETE` status
  * Sufficient compute quota for the new node group
</Note>

***

## Default Node Group

Every cluster has a default node group created at provisioning time. Its flavor and node
count are set by the cluster template and the initial node count parameter. The default
node group is named `default-worker`.

```bash title="List node groups for a cluster" theme={null}
openstack coe nodegroup list prod-cluster-01
```

***

## Create a Node Group

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to the cluster" icon="compass">
        Navigate to
        **Container > Clusters**. Click your cluster name.
      </Step>

      <Step title="Open Node Groups" icon="layers">
        Click the **Node Groups** tab on the cluster detail page.
      </Step>

      <Step title="Create node group" icon="plus">
        Click **Create Node Group** and fill in the fields:

        | Field          | Description                          | Example       |
        | -------------- | ------------------------------------ | ------------- |
        | **Name**       | Unique name within the cluster       | `gpu-workers` |
        | **Node Count** | Initial number of nodes in the group | `2`           |
        | **Flavor**     | Instance size for this group         | `g1.xlarge`   |
        | **Min Nodes**  | Minimum nodes for autoscaling        | `1`           |
        | **Max Nodes**  | Maximum nodes for autoscaling        | `5`           |
        | **Role**       | `worker` or `infra`                  | `worker`      |
      </Step>

      <Step title="Create" icon="circle-check">
        Click **Create Node Group**. Nodes are provisioned and join the cluster.

        <Check>Node group appears in the list and nodes show `STATUS: Ready` in kubectl.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a GPU node group" theme={null}
    openstack coe nodegroup create prod-cluster-01 \
      --name gpu-workers \
      --node-count 2 \
      --flavor g1.xlarge \
      --min-nodes 1 \
      --max-nodes 5 \
      --role worker
    ```

    ```bash title="List all node groups" theme={null}
    openstack coe nodegroup list prod-cluster-01
    ```

    ```bash title="Show node group details" theme={null}
    openstack coe nodegroup show prod-cluster-01 gpu-workers
    ```

    ```bash title="Verify nodes are Ready in kubectl" theme={null}
    kubectl get nodes -l ng=gpu-workers
    ```

    <Check>New nodes appear in kubectl with `STATUS: Ready`.</Check>
  </Tab>
</Tabs>

***

## Scale a Node Group

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    On the cluster detail page, click the **Node Groups** tab. Find the node group
    and click **Actions → Resize**. Enter the new node count and confirm.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Scale a node group" theme={null}
    openstack coe nodegroup update prod-cluster-01 gpu-workers \
      replace node_count=4
    ```

    ```bash title="Monitor scaling progress" theme={null}
    openstack coe nodegroup show prod-cluster-01 gpu-workers \
      -f value -c status -c node_count
    ```

    Wait for `status` to return to `UPDATE_COMPLETE`.

    <Tip>
      Use node groups for fine-grained scaling: scale GPU nodes up for batch jobs
      and back down when idle, without touching the general-purpose worker pool.
    </Tip>
  </Tab>
</Tabs>

***

## Schedule Workloads to a Specific Node Group

Use Kubernetes node selectors or taints and tolerations to target workloads to a
specific node group.

<CodeGroup>
  ```yaml title="Node selector in Pod spec" theme={null}
  spec:
    nodeSelector:
      node.kubernetes.io/instance-type: g1.xlarge
  ```

  ```yaml title="Toleration for a tainted node group" theme={null}
  spec:
    tolerations:
      - key: "node-type"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
  ```
</CodeGroup>

Apply a taint to all nodes in a node group to ensure only workloads with the matching
toleration are scheduled there:

```bash title="Taint all GPU nodes" theme={null}
kubectl taint nodes -l ng=gpu-workers \
  node-type=gpu:NoSchedule
```

***

## Delete a Node Group

<Danger>
  Deleting a node group removes all nodes in the group and evicts all pods running on
  them. Ensure workloads have been migrated to other node groups before deletion.
</Danger>

```bash title="Delete a node group" theme={null}
openstack coe nodegroup delete prod-cluster-01 gpu-workers
```

***

## Next Steps

<CardGroup>
  <Card title="Scale Cluster" href="/services/kubernetes/user-guide/scale-cluster">
    Resize the default node group for overall cluster capacity changes.
  </Card>

  <Card title="Cluster Upgrades" href="/services/kubernetes/user-guide/cluster-upgrades">
    Upgrade Kubernetes version across all node groups.
  </Card>

  <Card title="Access Cluster" href="/services/kubernetes/user-guide/access-cluster">
    Configure kubectl to connect to your cluster and verify node readiness.
  </Card>

  <Card title="Troubleshooting" href="/services/kubernetes/user-guide/troubleshooting">
    Resolve node group creation and scaling failures.
  </Card>
</CardGroup>


# Scale a Kubernetes Cluster
Source: https://docs.xloud.tech/services/kubernetes/user-guide/scale-cluster

Scale Xloud K8SaaS cluster node counts up or down — resize the default node group, monitor scale progress, and apply best practices for safe scale-down.

## Overview

Scaling a Kubernetes cluster adjusts the worker node count in the default node group.
Scale up to handle increased workload demand; scale down to reduce infrastructure cost
during low-utilization periods. Node scaling is non-disruptive for scale-up operations.
Scale-down operations drain and remove nodes — ensure workloads are properly distributed
before reducing node count.

<Note>
  **Prerequisites**

  * A running cluster in `CREATE_COMPLETE` or `UPDATE_COMPLETE` status
  * Sufficient compute quota for the new node count (scale-up only)
</Note>

***

## Scale Up (Add Nodes)

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to the cluster" icon="compass">
        Navigate to
        **Container > Clusters**.
      </Step>

      <Step title="Resize the cluster" icon="sliders">
        Click **Actions → Resize Cluster** next to your cluster.
        Enter the new total node count in the **Node Count** field.
      </Step>

      <Step title="Confirm" icon="circle-check">
        Click **Resize**. The cluster enters `UPDATE_IN_PROGRESS` status while new
        nodes are provisioned and join the cluster.

        <Check>Cluster returns to `UPDATE_COMPLETE` once nodes are ready.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Scale cluster up" theme={null}
    openstack coe cluster resize prod-cluster-01 \
      --node-count 6
    ```

    ```bash title="Monitor progress" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status -c node_count
    ```

    Wait for `status` to return to `UPDATE_COMPLETE`.

    ```bash title="Verify new nodes" theme={null}
    kubectl get nodes
    ```

    <Check>All new nodes show `STATUS: Ready` in kubectl output.</Check>
  </Tab>
</Tabs>

***

## Scale Down (Remove Nodes)

<Warning>
  Scaling down removes nodes from the cluster. Pods running on removed nodes are
  evicted before the node is deleted. Ensure Pod Disruption Budgets (PDBs) and
  workload replicas are configured to tolerate node removal without service interruption.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Prepare workloads" icon="shield-check">
        Before scaling down, verify that no stateful or single-replica workloads are
        running on the nodes that will be removed. Use the Kubernetes Dashboard or
        `kubectl` to inspect current pod placement.
      </Step>

      <Step title="Resize the cluster" icon="sliders">
        Navigate to **Container > Clusters** → **Actions → Resize Cluster**.
        Enter the reduced node count and click **Resize**.
      </Step>

      <Step title="Verify" icon="circle-check">
        Wait for the cluster to reach `UPDATE_COMPLETE`. Confirm remaining nodes
        are healthy and workloads have been rescheduled.

        <Check>Cluster is `UPDATE_COMPLETE` and all pods are running on remaining nodes.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Identify nodes to remove" icon="search">
        ```bash title="List nodes and their workloads" theme={null}
        kubectl get pods -A -o wide | grep <node-name>
        ```
      </Step>

      <Step title="Drain the target node (optional)" icon="arrow-right">
        If you want to control which nodes are removed, cordon and drain them first:

        ```bash title="Cordon node to prevent new scheduling" theme={null}
        kubectl cordon <node-name>
        ```

        ```bash title="Drain node" theme={null}
        kubectl drain <node-name> \
          --ignore-daemonsets \
          --delete-emptydir-data
        ```
      </Step>

      <Step title="Scale down" icon="minus">
        ```bash title="Scale cluster down" theme={null}
        openstack coe cluster resize prod-cluster-01 \
          --node-count 3
        ```

        ```bash title="Monitor scale-down" theme={null}
        openstack coe cluster show prod-cluster-01 \
          -f value -c status -c node_count
        ```
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="Check remaining nodes" theme={null}
        kubectl get nodes
        ```

        <Check>Node count matches the new target and all remaining nodes are `Ready`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Best Practices

<CardGroup>
  <Card title="Maintain Minimum Replicas" icon="layers">
    Configure `PodDisruptionBudget` resources for stateful workloads to ensure a
    minimum number of replicas remain available during node removal.
  </Card>

  <Card title="Use Node Groups for Fine-Grained Scaling" icon="sliders">
    Use separate node groups for different workload types to scale them independently
    without affecting other workloads on the cluster.
  </Card>

  <Card title="Scale During Low-Traffic Windows" icon="clock">
    Perform scale-down operations during low-traffic periods when evicted pods have
    minimal user impact.
  </Card>

  <Card title="Check Compute Quota Before Scale-Up" icon="gauge">
    Verify sufficient compute quota before adding nodes to avoid partial scale-up failures.
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="Node Groups" href="/services/kubernetes/user-guide/node-groups">
    Create and manage separate node pools for specialized workloads.
  </Card>

  <Card title="Access Cluster" href="/services/kubernetes/user-guide/access-cluster">
    Verify cluster access after scaling via kubectl.
  </Card>

  <Card title="Cluster Upgrades" href="/services/kubernetes/user-guide/cluster-upgrades">
    Upgrade your cluster to a newer Kubernetes version.
  </Card>

  <Card title="Troubleshooting" href="/services/kubernetes/user-guide/troubleshooting">
    Resolve scaling failures and node health issues.
  </Card>
</CardGroup>


# Kubernetes Troubleshooting — User Guide
Source: https://docs.xloud.tech/services/kubernetes/user-guide/troubleshooting

Resolve common Xloud K8SaaS issues — clusters stuck in CREATE_IN_PROGRESS, kubectl connection failures, nodes in NotReady state, and upgrade failures.

## Overview

This page covers the most common Kubernetes cluster issues encountered by project users,
with targeted diagnostics and resolution steps. For platform-level issues such as driver
configuration failures or quota enforcement problems, refer to the
[Kubernetes Admin Troubleshooting](/services/kubernetes/admin-guide/troubleshooting) guide.

***

## Common Issues

<AccordionGroup>
  <Accordion title="Cluster stuck in CREATE_IN_PROGRESS" icon="clock">
    **Cause**: Node provisioning is delayed — commonly due to insufficient compute quota,
    an unavailable node image, or a network configuration issue during node bootstrap.

    **Resolution**:

    ```bash title="Show cluster failure reason" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status_reason
    ```

    Check compute quota:

    ```bash title="Check project quota" theme={null}
    openstack quota show --detail
    ```

    Verify the node image exists:

    ```bash title="Verify image" theme={null}
    openstack image show fedora-coreos-39
    ```

    If resources are insufficient, request a quota increase from your administrator.
    If the image is missing, ask your administrator to upload the required image.
  </Accordion>

  <Accordion title="kubectl connection refused or timeout" icon="network">
    **Cause**: The cluster API server endpoint is unreachable. The master load balancer
    floating IP may not have been allocated, or a security group rule is blocking
    port 6443.

    **Resolution**:

    ```bash title="Show API server endpoint" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c api_address
    ```

    Verify the API address is a reachable floating IP:

    ```bash title="Test API server connectivity" theme={null}
    curl -sk https://<api-address>:6443/healthz
    ```

    Expected: `ok`

    If the endpoint is unreachable, check security groups for the master nodes:

    ```bash title="List cluster security groups" theme={null}
    openstack security group list | grep prod-cluster-01
    ```

    Ensure inbound TCP port 6443 is permitted from your management network.
  </Accordion>

  <Accordion title="Nodes in NotReady state" icon="circle-alert">
    **Cause**: The container network interface plugin has not initialized, the node is
    still bootstrapping, or the node has run out of resources.

    **Resolution**:

    ```bash title="Check node conditions" theme={null}
    kubectl describe node <node-name>
    ```

    Look for `NetworkPlugin`, `DiskPressure`, `MemoryPressure`, or `PIDPressure`
    conditions in the output.

    For CNI failures, check node logs via the instance console:

    ```bash title="Access node console" theme={null}
    openstack console url show <node-instance-id>
    ```

    Review the bootstrap logs for CNI installation errors. If the CNI plugin did not
    install correctly, the node may need to be replaced (scale down then back up).
  </Accordion>

  <Accordion title="Cluster upgrade fails mid-way" icon="circle-x">
    **Cause**: A node replacement failed during the rolling upgrade — commonly due to
    quota exhaustion or an image pull failure on the replacement node.

    **Resolution**:

    ```bash title="Check upgrade status and reason" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status -c status_reason
    ```

    Identify the failure cause from `status_reason`. Common causes:

    * **Quota exhausted**: Free up compute quota, then retry the upgrade command
    * **Image unavailable**: Verify the target template's image exists and is accessible

    After resolving the root cause, retry the upgrade:

    ```bash title="Retry upgrade" theme={null}
    openstack coe cluster upgrade prod-cluster-01 k8s-1.30-prod
    ```
  </Accordion>

  <Accordion title="Persistent volume claims not binding" icon="hard-drive">
    **Cause**: The volume driver (`cinder`) is not configured in the cluster template,
    or the storage class is missing from the cluster.

    **Resolution**:

    ```bash title="Check storage classes" theme={null}
    kubectl get storageclass
    ```

    If no storage classes exist, verify the cluster template has `--volume-driver cinder`:

    ```bash title="Show template volume driver" theme={null}
    openstack coe cluster template show k8s-1.29-prod \
      -f value -c volume_driver
    ```

    If the volume driver is missing, the cluster must be recreated from a corrected
    template. Contact your administrator to update the platform template. Your administrator can configure this through [XDeploy](/deployment).
  </Accordion>

  <Accordion title="kubectl shows certificate verification error" icon="lock">
    **Cause**: The cluster CA has been rotated since you downloaded the kubeconfig,
    or the kubeconfig references an expired certificate.

    **Resolution**: Refresh your kubeconfig:

    ```bash title="Re-download kubeconfig" theme={null}
    openstack coe cluster config prod-cluster-01 \
      --dir ~/.kube \
      --force
    ```

    ```bash title="Set kubeconfig" theme={null}
    export KUBECONFIG=~/.kube/config
    ```

    ```bash title="Verify connectivity" theme={null}
    kubectl get nodes
    ```
  </Accordion>
</AccordionGroup>

***

## Diagnostic Commands Reference

```bash title="Show cluster full detail" theme={null}
openstack coe cluster show prod-cluster-01 -f json
```

```bash title="List all clusters and their statuses" theme={null}
openstack coe cluster list
```

```bash title="Check kubectl cluster connectivity" theme={null}
kubectl cluster-info
```

```bash title="Show all system pods" theme={null}
kubectl get pods -n kube-system
```

```bash title="Describe a specific node" theme={null}
kubectl describe node <node-name>
```

***

## Next Steps

<CardGroup>
  <Card title="Deploy Cluster" href="/services/kubernetes/user-guide/deploy-cluster">
    Re-deploy a cluster after resolving provisioning issues.
  </Card>

  <Card title="Access Cluster" href="/services/kubernetes/user-guide/access-cluster">
    Reconfigure kubectl connectivity after certificate or endpoint changes.
  </Card>

  <Card title="Kubernetes Admin Troubleshooting" href="/services/kubernetes/admin-guide/troubleshooting">
    Platform-level diagnostics for driver and quota issues.
  </Card>

  <Card title="Cluster Upgrades" href="/services/kubernetes/user-guide/cluster-upgrades">
    Resume or retry failed version upgrades.
  </Card>
</CardGroup>


# Load Balancer
Source: https://docs.xloud.tech/services/load-balancer

Distribute application traffic across multiple instances with Xloud Load Balancer — layer 4/7 balancing, health monitoring, and TLS termination for.

Distribute application traffic intelligently across compute instances with built-in health monitoring and TLS termination.

<Card title="Xloud Load Balancer — Enterprise LBaaS" icon="combine" href="https://xloud.tech/xavs">
  Product details and datasheet on xloud.tech
</Card>

***

<p>Xloud Load Balancer</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/load-balancer/user-guide">
    Create load balancers, configure listeners and pools, set up health monitors, and assign floating IPs to distribute traffic across your instances.
  </Card>

  <Card title="Admin Guide" icon="shield-check" href="/services/load-balancer/admin-guide">
    Configure provider drivers, manage flavor profiles, set project quotas, and monitor the load balancing infrastructure as a platform administrator.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/load-balancer/cli-reference">
    `openstack loadbalancer` commands for managing load balancers, listeners, pools, and members from the command line.
  </Card>

  <Card title="Health Monitors" icon="heart-pulse">
    Configure TCP, HTTP, HTTPS, and PING health checks to automatically remove unhealthy members from rotation.
  </Card>
</CardGroup>

***

<p>Key Features</p>

<CardGroup>
  <Card title="Layer 4 & Layer 7 Balancing" icon="layers" href="/services/load-balancer/listeners">
    Support for TCP/UDP load balancing at layer 4 and HTTP/HTTPS routing at layer 7. Route traffic based on URL path, hostname, or header values.
  </Card>

  <Card title="Active Health Monitoring" icon="activity" href="/services/load-balancer/health-monitors">
    Continuous health checks detect member failures in seconds and automatically redirect traffic — no manual intervention required.
  </Card>

  <Card title="TLS Termination" icon="shield" href="/services/load-balancer/listeners">
    Offload TLS processing to the load balancer. Manage certificates through Xloud Key Manager for centralized certificate lifecycle control.
  </Card>

  <Card title="Session Persistence" icon="clock" href="/services/load-balancer/pools">
    SOURCE\_IP, HTTP\_COOKIE, and APP\_COOKIE persistence modes ensure session-aware applications route returning clients to the same member.
  </Card>

  <Card title="Floating IP Integration" icon="globe" href="/services/load-balancer/floating-ip">
    Associate a public floating IP directly with a load balancer VIP for external access without exposing individual backend instances.
  </Card>

  <Card title="Multiple Algorithms" icon="git-branch" href="/services/load-balancer/pools">
    Choose from ROUND\_ROBIN, LEAST\_CONNECTIONS, and SOURCE\_IP distribution algorithms. Match the balancing strategy to your workload profile.
  </Card>
</CardGroup>

***

<p>Load Balancer Components</p>

| Component      | Description                                                                                        |
| -------------- | -------------------------------------------------------------------------------------------------- |
| Load Balancer  | Top-level resource bound to a subnet. Provides the virtual IP (VIP) address for client connections |
| Listener       | Defines the protocol and port on which the load balancer accepts traffic (TCP, HTTP, HTTPS, UDP)   |
| Pool           | A group of backend members that receive traffic from a listener                                    |
| Member         | An individual backend instance registered to a pool, identified by its IP and port                 |
| Health Monitor | Periodic probes that determine member availability and control traffic routing                     |
| L7 Policy      | Layer 7 rules that redirect, reject, or forward HTTP traffic based on request properties           |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Backend instances that serve as pool members behind the load balancer
  </Card>

  <Card title="Xloud Networking" icon="network" href="/services/compute">
    Subnets, security groups, and floating IPs used by load balancer resources
  </Card>

  <Card title="Xloud Key Manager" icon="lock" href="/services/key-manager">
    TLS certificates and secrets for HTTPS listeners and end-to-end encryption
  </Card>

  <Card title="Xloud DNS" icon="globe" href="/services/dns">
    Map domain names to load balancer VIPs for application endpoint management
  </Card>

  <Card title="Xloud Identity" icon="fingerprint" href="/cli-setup">
    Authentication and RBAC policies governing load balancer resource access
  </Card>

  <Card title="Xloud Object Storage" icon="box" href="/services/object-storage">
    Store access logs and TLS certificate backups for compliance and auditing
  </Card>
</CardGroup>

***

<p>Getting Started</p>

<CardGroup>
  <Card title="Authentication" icon="key" href="/cli-setup">
    Configure Dashboard access and CLI credentials before working with Load Balancer
  </Card>

  <Card title="User Guide" icon="book-open" href="/services/load-balancer/user-guide">
    Step-by-step instructions for creating your first load balancer
  </Card>
</CardGroup>


# Load Balancer Admin Guide
Source: https://docs.xloud.tech/services/load-balancer/admin-guide

Administer Xloud Load Balancer infrastructure — configure provider drivers, flavor profiles, project quotas, monitoring, and platform security.

<p>Overview</p>

This guide covers platform-level administration of the Xloud Load Balancer service.
Administrators configure provider drivers, define flavor profiles that govern appliance
capacity, enforce per-project quotas, and monitor the health of the underlying
infrastructure.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Tabs>
  <Tab title="XDeploy" icon="server">
    The Load Balancer service is configured through the XDeploy Configuration panel:

    <Steps>
      <Step title="Open Configuration" icon="settings">
        Navigate to **XDeploy → Configuration** and select the **Load Balancer** tab.
      </Step>

      <Step title="Enable the Load Balancer service" icon="toggle-left">
        Set **Enable Load Balancer** to **Yes**.
      </Step>

      <Step title="Configure service parameters" icon="sliders">
        Set the following options as needed:

        | Setting                  | Description                                              |
        | ------------------------ | -------------------------------------------------------- |
        | **Controller Interface** | Management network interface for appliance communication |
        | **Amphora Network**      | Network and subnet used by load balancer appliances      |
        | **Flavor Sizing**        | Appliance flavor (vCPU, RAM) for capacity tiers          |
        | **Topology**             | Single or Active-Standby appliance topology              |
        | **TLS Certificates**     | Certificate configuration for HTTPS termination          |
      </Step>

      <Step title="Save and deploy" icon="rocket">
        Click **Save Configuration**, then navigate to **XDeploy → Operations** and
        run a **Deploy** or **Reconfigure** for the Load Balancer service.

        <Check>Load Balancer service is deployed and operational.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Configure the Load Balancer service by editing `octavia.conf` directly at
    `/etc/xavs/config/octavia/octavia.conf`. See the individual topic guides below
    for detailed parameters.
  </Tab>
</Tabs>

***

<CardGroup>
  <Card title="Architecture" icon="network" href="/services/load-balancer/lb-architecture">
    Controller-appliance topology, data plane, and management plane components.
  </Card>

  <Card title="Provider Drivers" icon="cpu" href="/services/load-balancer/provider-drivers">
    View and configure the load balancing provider drivers available in your deployment.
  </Card>

  <Card title="Flavor Profiles" icon="sliders" href="/services/load-balancer/flavor-profiles">
    Create capacity tiers that users select at load balancer provisioning time.
  </Card>

  <Card title="Quotas" icon="gauge" href="/services/load-balancer/lb-quotas">
    Enforce per-project resource limits for load balancers, listeners, pools, and members.
  </Card>

  <Card title="Monitoring" icon="activity" href="/services/load-balancer/lb-monitoring">
    Monitor appliance health, connection statistics, and service component status.
  </Card>

  <Card title="Security" icon="shield" href="/services/load-balancer/lb-security">
    Restrict management plane access, manage TLS certificate lifecycle, and audit quota usage.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/load-balancer/lb-troubleshooting">
    Resolve appliance provisioning failures, service outages, and high latency issues.
  </Card>
</CardGroup>

***

<p>Quick Reference</p>

| Task                            | Command                                        |
| ------------------------------- | ---------------------------------------------- |
| List all load balancers (admin) | `openstack loadbalancer list --all-projects`   |
| List all appliances             | `openstack loadbalancer amphora list`          |
| Show service providers          | `openstack loadbalancer provider list`         |
| Show default quotas             | `openstack loadbalancer quota defaults show`   |
| Failover an appliance           | `openstack loadbalancer amphora failover <id>` |
| Check service containers        | `docker ps --filter name=load-balancer`        |

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Load Balancer User Guide" icon="book-open" href="/services/load-balancer/user-guide">
    Day-to-day operations — creating load balancers, pools, and health monitors.
  </Card>

  <Card title="Identity Admin Guide" icon="fingerprint" href="/services/identity/admin-guide">
    Manage RBAC policies controlling load balancer resource access.
  </Card>
</CardGroup>


# Load Balancer CLI Reference
Source: https://docs.xloud.tech/services/load-balancer/cli-reference

Complete openstack loadbalancer CLI commands for creating and managing load balancers, listeners, pools, members, and health monitors.

## Overview

The `openstack loadbalancer` command group manages the full lifecycle of load balancers, listeners, backend pools, pool members, and health monitors.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python octaviaclient installed: `pip install python-octaviaclient`
</Note>

***

## Load Balancers

<CodeGroup>
  ```bash title="List load balancers" theme={null}
  openstack loadbalancer list
  ```

  ```bash title="Create load balancer" theme={null}
  openstack loadbalancer create \
    --name my-lb \
    --vip-subnet-id private-subnet \
    --wait
  ```

  ```bash title="Show load balancer" theme={null}
  openstack loadbalancer show my-lb
  ```

  ```bash title="Delete load balancer" theme={null}
  openstack loadbalancer delete --cascade my-lb
  ```
</CodeGroup>

***

## Listeners

<CodeGroup>
  ```bash title="List listeners" theme={null}
  openstack loadbalancer listener list
  ```

  ```bash title="Create HTTP listener" theme={null}
  openstack loadbalancer listener create \
    --name my-listener \
    --protocol HTTP \
    --protocol-port 80 \
    my-lb
  ```

  ```bash title="Create HTTPS listener with TLS" theme={null}
  openstack loadbalancer listener create \
    --name https-listener \
    --protocol TERMINATED_HTTPS \
    --protocol-port 443 \
    --default-tls-container-ref <barbican-container-ref> \
    my-lb
  ```

  ```bash title="Show listener" theme={null}
  openstack loadbalancer listener show my-listener
  ```

  ```bash title="Delete listener" theme={null}
  openstack loadbalancer listener delete my-listener
  ```
</CodeGroup>

***

## Pools

<CodeGroup>
  ```bash title="List pools" theme={null}
  openstack loadbalancer pool list
  ```

  ```bash title="Create pool (round-robin)" theme={null}
  openstack loadbalancer pool create \
    --name my-pool \
    --lb-algorithm ROUND_ROBIN \
    --listener my-listener \
    --protocol HTTP
  ```

  ```bash title="Create pool (least connections)" theme={null}
  openstack loadbalancer pool create \
    --name my-pool \
    --lb-algorithm LEAST_CONNECTIONS \
    --listener my-listener \
    --protocol HTTP
  ```

  ```bash title="Show pool" theme={null}
  openstack loadbalancer pool show my-pool
  ```

  ```bash title="Delete pool" theme={null}
  openstack loadbalancer pool delete my-pool
  ```
</CodeGroup>

***

## Members

<CodeGroup>
  ```bash title="List pool members" theme={null}
  openstack loadbalancer member list my-pool
  ```

  ```bash title="Add member" theme={null}
  openstack loadbalancer member create \
    --name web-01 \
    --address 10.0.1.10 \
    --protocol-port 80 \
    --subnet-id private-subnet \
    my-pool
  ```

  ```bash title="Show member" theme={null}
  openstack loadbalancer member show my-pool web-01
  ```

  ```bash title="Remove member" theme={null}
  openstack loadbalancer member delete my-pool web-01
  ```
</CodeGroup>

***

## Health Monitors

<CodeGroup>
  ```bash title="List health monitors" theme={null}
  openstack loadbalancer healthmonitor list
  ```

  ```bash title="Create HTTP health monitor" theme={null}
  openstack loadbalancer healthmonitor create \
    --name my-hm \
    --type HTTP \
    --delay 5 \
    --timeout 3 \
    --max-retries 3 \
    --url-path /health \
    my-pool
  ```

  ```bash title="Create TCP health monitor" theme={null}
  openstack loadbalancer healthmonitor create \
    --name tcp-hm \
    --type TCP \
    --delay 5 \
    --timeout 3 \
    --max-retries 3 \
    my-pool
  ```

  ```bash title="Show health monitor" theme={null}
  openstack loadbalancer healthmonitor show my-hm
  ```

  ```bash title="Delete health monitor" theme={null}
  openstack loadbalancer healthmonitor delete my-hm
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Create Load Balancer Guide" href="/services/load-balancer/create-lb">
    End-to-end walkthrough for creating a load balancer
  </Card>

  <Card title="Networking CLI" href="/services/networking/cli-reference">
    Floating IP and security group commands
  </Card>
</CardGroup>


# Create a Load Balancer
Source: https://docs.xloud.tech/services/load-balancer/create-lb

Provision a new Xloud Load Balancer with a listener, pool, members, and health monitor in a single guided workflow.

## Overview

Creating a load balancer in Xloud involves provisioning the top-level resource with a
virtual IP, then adding a listener (protocol and port), a pool (backend collection),
pool members (backend instances), and a health monitor. The Dashboard creation wizard
guides you through all five steps in sequence. The CLI allows each component to be
created independently.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  Load balancers must be placed on a subnet that has routing access to your backend
  instances. Verify that security group rules allow ingress on the member port from
  the load balancer's VIP subnet before adding members.
</Note>

***

## Create via Dashboard (Wizard)

<Steps>
  <Step title="Open the creation wizard" icon="plus">
    Navigate to
    **Network > Load Balancers**. Click **Create Load Balancer**.
  </Step>

  <Step title="Configure load balancer details" icon="settings">
    Complete the **Load Balancer Details** panel:

    | Field                 | Description                                                           |
    | --------------------- | --------------------------------------------------------------------- |
    | **Name**              | Display name (e.g., `prod-web-lb`)                                    |
    | **Description**       | Optional description                                                  |
    | **IP Address**        | Leave blank to auto-assign a VIP, or specify a fixed IP               |
    | **Subnet**            | The subnet hosting the VIP — must be reachable from backend instances |
    | **Availability Zone** | Fault domain for appliance placement                                  |

    <Tip>
      Use a dedicated management subnet for the VIP to isolate load balancer traffic
      from application data paths.
    </Tip>
  </Step>

  <Step title="Add a listener" icon="ear">
    Complete the **Listener Details** panel:

    | Field                | Value           | Description                            |
    | -------------------- | --------------- | -------------------------------------- |
    | **Name**             | `listener-http` | Display name                           |
    | **Protocol**         | `HTTP`          | Layer 7 protocol                       |
    | **Protocol Port**    | `80`            | Port accepting connections             |
    | **Connection Limit** | `-1`            | Unlimited; set positive integer to cap |
  </Step>

  <Step title="Configure the pool" icon="users">
    Complete the **Pool Details** panel:

    | Field                   | Value         | Description                              |
    | ----------------------- | ------------- | ---------------------------------------- |
    | **Name**                | `pool-http`   | Pool display name                        |
    | **Algorithm**           | `ROUND_ROBIN` | Traffic distribution method              |
    | **Session Persistence** | `None`        | Enable for stateful application sessions |

    **Algorithm options:**

    * `ROUND_ROBIN` — distributes requests evenly across all UP members
    * `LEAST_CONNECTIONS` — sends to the member with fewest active connections
    * `SOURCE_IP` — routes the same client IP to the same member
  </Step>

  <Step title="Add pool members" icon="user-plus">
    In the **Pool Members** panel, click **Add** next to each instance to register it.

    | Field             | Description                                                           |
    | ----------------- | --------------------------------------------------------------------- |
    | **IP Address**    | Auto-populated from the selected instance                             |
    | **Protocol Port** | Port your application listens on (e.g., `8080`)                       |
    | **Weight**        | Relative weight — higher weight receives proportionally more requests |

    <Warning>
      Members must be reachable from the load balancer's VIP subnet. Verify security
      groups allow ingress on the member port from the VIP address.
    </Warning>
  </Step>

  <Step title="Configure health monitor" icon="heart-pulse">
    Complete the **Monitor Details** panel:

    | Field              | Value     | Description                                         |
    | ------------------ | --------- | --------------------------------------------------- |
    | **Type**           | `HTTP`    | Sends a GET request and validates the response code |
    | **Delay**          | `5`       | Seconds between probes                              |
    | **Timeout**        | `3`       | Seconds to wait for a probe response                |
    | **Max Retries**    | `3`       | Failures before marking a member DOWN               |
    | **URL Path**       | `/health` | Endpoint returning HTTP 200 when healthy            |
    | **Expected Codes** | `200`     | Comma-separated acceptable HTTP status codes        |
  </Step>

  <Step title="Review and create" icon="circle-check">
    Review the summary and click **Create Load Balancer**.

    The load balancer enters **PENDING\_CREATE** status. Provisioning typically completes
    within 30–60 seconds.

    <Check>The load balancer displays status **ACTIVE** in the Load Balancers list.</Check>
  </Step>
</Steps>

***

## Create via CLI

<Steps>
  <Step title="Authenticate and create the load balancer" icon="key">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="Create load balancer" theme={null}
    openstack loadbalancer create \
      --name prod-web-lb \
      --vip-subnet-id <subnet-id>
    ```

    Wait for `ACTIVE` status:

    ```bash title="Wait for ACTIVE status" theme={null}
    openstack loadbalancer show prod-web-lb \
      -c provisioning_status -c operating_status
    ```
  </Step>

  <Step title="Create a listener" icon="ear">
    ```bash title="Create HTTP listener" theme={null}
    openstack loadbalancer listener create \
      --name listener-http \
      --protocol HTTP \
      --protocol-port 80 \
      prod-web-lb
    ```
  </Step>

  <Step title="Create a pool" icon="users">
    ```bash title="Create pool with round-robin algorithm" theme={null}
    openstack loadbalancer pool create \
      --name pool-http \
      --lb-algorithm ROUND_ROBIN \
      --listener listener-http \
      --protocol HTTP
    ```
  </Step>

  <Step title="Add members" icon="user-plus">
    ```bash title="Add backend member" theme={null}
    openstack loadbalancer member create \
      --subnet-id <instance-subnet-id> \
      --address <instance-ip> \
      --protocol-port 8080 \
      pool-http
    ```

    Repeat for each backend instance.
  </Step>

  <Step title="Create a health monitor" icon="heart-pulse">
    ```bash title="Create HTTP health monitor" theme={null}
    openstack loadbalancer healthmonitor create \
      --name hm-http \
      --delay 5 \
      --timeout 3 \
      --max-retries 3 \
      --type HTTP \
      --url-path /health \
      --expected-codes 200 \
      pool-http
    ```

    <Check>Load balancer is ACTIVE. Health monitor is probing members.</Check>
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Floating IP Assignment" href="/services/load-balancer/floating-ip">
    Expose the load balancer VIP on a public network for external access.
  </Card>

  <Card title="Listeners" href="/services/load-balancer/listeners">
    Add HTTPS, TCP, or additional HTTP listeners to the load balancer.
  </Card>

  <Card title="Health Monitors" href="/services/load-balancer/health-monitors">
    Tune health check parameters for your application's response characteristics.
  </Card>

  <Card title="Troubleshooting" href="/services/load-balancer/troubleshooting">
    Resolve PENDING\_CREATE status and member health issues.
  </Card>
</CardGroup>


# Load Balancer Flavor Profiles
Source: https://docs.xloud.tech/services/load-balancer/flavor-profiles

Create and manage capacity tier definitions that users select when provisioning Xloud Load Balancer instances.

## Overview

Flavor profiles define the compute capacity and topology configuration allocated to load
balancing appliances. Administrators create named flavors that users select at load balancer
provisioning time. Flavors abstract provider-specific settings into human-readable capacity
tiers — e.g., `standard`, `ha`, and `high-performance`.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Flavor Architecture

| Concept            | Description                                                                               |
| ------------------ | ----------------------------------------------------------------------------------------- |
| **Flavor Profile** | Provider-specific settings template (topology, compute class, etc.)                       |
| **Flavor**         | Named, user-visible tier based on a flavor profile. Users select flavors at provisioning. |

A flavor profile is always associated with a specific provider. Multiple flavors can
reference the same profile.

***

## Create a Flavor Profile

<Steps>
  <Step title="Create a flavor profile" icon="settings">
    A flavor profile wraps provider-specific settings:

    ```bash title="Create standard single-topology profile" theme={null}
    openstack loadbalancer flavorprofile create \
      --name standard-profile \
      --provider amphora \
      --flavor-data '{"loadbalancer_topology": "SINGLE"}'
    ```

    ```bash title="Create HA active-standby profile" theme={null}
    openstack loadbalancer flavorprofile create \
      --name ha-profile \
      --provider amphora \
      --flavor-data '{"loadbalancer_topology": "ACTIVE_STANDBY"}'
    ```
  </Step>

  <Step title="Create named flavors for users" icon="package">
    Expose profiles as named flavors:

    ```bash title="Create standard flavor" theme={null}
    openstack loadbalancer flavor create \
      --name standard \
      --flavorprofile standard-profile \
      --description "Standard single-instance load balancer" \
      --enable
    ```

    ```bash title="Create HA flavor" theme={null}
    openstack loadbalancer flavor create \
      --name ha \
      --flavorprofile ha-profile \
      --description "High-availability active-standby load balancer" \
      --enable
    ```
  </Step>

  <Step title="Verify flavors are available" icon="circle-check">
    ```bash title="List available flavors" theme={null}
    openstack loadbalancer flavor list
    ```

    <Check>Flavors appear in the list with status **enabled** — you can now select them when creating a load balancer.</Check>
  </Step>
</Steps>

***

## Flavor Profile Settings Reference

| Setting                 | Provider | Description                                              |
| ----------------------- | -------- | -------------------------------------------------------- |
| `loadbalancer_topology` | amphora  | `SINGLE` or `ACTIVE_STANDBY`                             |
| `compute_flavor`        | amphora  | Nova flavor for the appliance instance                   |
| `amp_image_tag`         | amphora  | Tag identifying the appliance image in the Image Service |
| `availability_zone`     | amphora  | Restrict appliance placement to a specific AZ            |

***

## Manage Flavors

<CodeGroup>
  ```bash title="List all flavors" theme={null}
  openstack loadbalancer flavor list
  ```

  ```bash title="Show flavor details" theme={null}
  openstack loadbalancer flavor show standard
  ```

  ```bash title="Disable a flavor (prevents new LBs from using it)" theme={null}
  openstack loadbalancer flavor set standard --disable
  ```

  ```bash title="Delete a flavor" theme={null}
  openstack loadbalancer flavor delete standard
  ```

  ```bash title="List flavor profiles" theme={null}
  openstack loadbalancer flavorprofile list
  ```
</CodeGroup>

<Warning>
  Disabling a flavor prevents new load balancers from using it but does not affect
  existing load balancers. Deleting a flavor profile that is referenced by a flavor
  will fail — delete the flavor first.
</Warning>

***

## Recommended Flavor Set

For most production deployments, provide at least two flavors:

| Flavor Name | Profile         | Use Case                                         |
| ----------- | --------------- | ------------------------------------------------ |
| `standard`  | Single topology | Development, testing, non-critical workloads     |
| `ha`        | Active-standby  | Production workloads requiring high availability |

***

## Next Steps

<CardGroup>
  <Card title="Provider Drivers" href="/services/load-balancer/provider-drivers">
    Understand provider-specific flavor profile settings.
  </Card>

  <Card title="Quotas" href="/services/load-balancer/lb-quotas">
    Control how many load balancers projects can provision across all flavors.
  </Card>

  <Card title="Architecture" href="/services/load-balancer/lb-architecture">
    Understand how appliance topology affects data plane resilience.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/load-balancer/lb-troubleshooting">
    Diagnose flavor-related provisioning failures.
  </Card>
</CardGroup>


# Floating IP Assignment
Source: https://docs.xloud.tech/services/load-balancer/floating-ip

Expose your Xloud Load Balancer VIP on a public network by associating a floating IP for external client access.

## Overview

A floating IP maps a public IP address from an external network to your load balancer's
virtual IP (VIP) port. This is the standard method for exposing a load balancer to
external clients. Once associated, the floating IP routes all inbound traffic to the
load balancer, which distributes it across backend members according to the listener
and pool configuration.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  Before associating a floating IP, ensure:

  * The load balancer status is **ACTIVE**
  * At least one listener is configured and **ACTIVE**
  * A floating IP has been allocated from an external network in your project
</Note>

***

## Associate a Floating IP

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Retrieve the VIP port" icon="network">
        Navigate to **Network > Load Balancers**. Open your load balancer and
        note the **VIP Port ID** from the overview panel.
      </Step>

      <Step title="Allocate a floating IP" icon="plus">
        Navigate to **Network > Floating IPs**. Click **Allocate IP to Project**
        and select your external network pool.
      </Step>

      <Step title="Associate the floating IP to the VIP" icon="link">
        Click **Associate** next to the allocated floating IP. In the **Port to be associated**
        dropdown, select the load balancer VIP port identified in step 1.

        <Check>The floating IP is now associated with the load balancer VIP. External traffic
        to the floating IP is forwarded to the load balancer.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Retrieve the VIP port ID" icon="search">
        ```bash title="Get VIP port ID" theme={null}
        openstack loadbalancer show prod-web-lb \
          -c vip_port_id -f value
        ```

        Save this value as `VIP_PORT_ID`.
      </Step>

      <Step title="Allocate a floating IP" icon="plus">
        ```bash title="Allocate floating IP from external network" theme={null}
        openstack floating ip create <external-network-name>
        ```

        Note the `floating_ip_address` from the output.
      </Step>

      <Step title="Associate floating IP with VIP port" icon="link">
        ```bash title="Associate floating IP with load balancer VIP" theme={null}
        openstack floating ip set \
          --port <VIP_PORT_ID> \
          <floating-ip-address>
        ```
      </Step>

      <Step title="Verify the association" icon="circle-check">
        ```bash title="Confirm association" theme={null}
        openstack floating ip show <floating-ip-address> \
          -c fixed_ip_address \
          -c floating_ip_address \
          -c status
        ```

        <Check>Status shows `ACTIVE` and `fixed_ip_address` matches the load balancer VIP.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Test External Access

After associating the floating IP, verify traffic reaches your backend members:

```bash title="Test HTTP access via floating IP" theme={null}
curl -v http://<floating-ip-address>/health
```

```bash title="Test HTTPS access via floating IP" theme={null}
curl -v https://<floating-ip-address>/health
```

<Check>The health endpoint responds with HTTP 200 — traffic is flowing through the load balancer to backend members.</Check>

***

## Disassociate a Floating IP

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > Floating IPs**. Click **Disassociate** next to the
    floating IP attached to your load balancer.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Disassociate floating IP" theme={null}
    openstack floating ip unset \
      --port \
      <floating-ip-address>
    ```

    The floating IP reverts to `DOWN` status and can be reassociated or released:

    ```bash title="Release floating IP back to pool" theme={null}
    openstack floating ip delete <floating-ip-address>
    ```
  </Tab>
</Tabs>

***

## DNS Configuration

Map a domain name to the floating IP for production services:

<Tabs>
  <Tab title="Xloud DNS" icon="globe">
    If using Xloud DNS, create an A record pointing to the floating IP:

    ```bash title="Create DNS A record" theme={null}
    openstack recordset create \
      --type A \
      --records <floating-ip-address> \
      --ttl 300 \
      example.com. \
      api.example.com.
    ```

    See the [Xloud DNS guide](/services/dns) for full DNS management instructions.
  </Tab>

  <Tab title="External DNS" icon="network">
    Point your external DNS provider's A record for your domain to the floating IP address.
    DNS propagation typically takes 1–5 minutes for TTLs under 5 minutes.
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Listeners" href="/services/load-balancer/listeners">
    Add HTTPS termination to your listener before exposing it publicly.
  </Card>

  <Card title="Xloud DNS" href="/services/dns">
    Create DNS records pointing to your load balancer floating IP.
  </Card>

  <Card title="Xloud Key Manager" href="/services/key-manager">
    Store TLS certificates used by HTTPS listeners.
  </Card>

  <Card title="Troubleshooting" href="/services/load-balancer/troubleshooting">
    Resolve floating IP association failures and connectivity issues.
  </Card>
</CardGroup>


# Health Monitors
Source: https://docs.xloud.tech/services/load-balancer/health-monitors

Configure TCP, HTTP, and HTTPS health checks to automatically remove unhealthy backend members from Xloud Load Balancer pools.

## Overview

Health monitors continuously probe pool members and automatically remove unhealthy
instances from rotation. When a member fails the configured number of consecutive probes,
it is marked `OFFLINE` and receives no traffic until it recovers. This guide covers all
supported monitor types, their configuration parameters, and best practices for different
application types.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Supported Monitor Types

| Type          | Protocol                          | Use Case                                          |
| ------------- | --------------------------------- | ------------------------------------------------- |
| `TCP`         | Layer 4 connection check          | Any TCP service — databases, custom protocols     |
| `HTTP`        | GET request + response code check | Web applications and REST APIs                    |
| `HTTPS`       | Encrypted GET request check       | HTTPS backends requiring secure probes            |
| `PING`        | ICMP echo                         | Basic host reachability test                      |
| `UDP-CONNECT` | UDP connection check              | UDP-based services                                |
| `TLS-HELLO`   | TLS handshake check               | Verify TLS is functional without full HTTPS probe |

***

## Configuration Parameters

| Parameter            | Description                                                    | Recommended |
| -------------------- | -------------------------------------------------------------- | ----------- |
| **Delay**            | Seconds between consecutive probes                             | `5–10`      |
| **Timeout**          | Seconds to wait for probe response before counting as failed   | `3–5`       |
| **Max Retries**      | Consecutive failures before marking member DOWN                | `3`         |
| **Max Retries Down** | Consecutive successes before restoring a DOWN member to ONLINE | `3`         |
| **URL Path**         | Path for HTTP/HTTPS monitors (must return expected code)       | `/health`   |
| **Expected Codes**   | Comma-separated HTTP status codes accepted as healthy          | `200`       |

<Tip>
  Set **Delay** to at least twice the **Timeout** value to prevent overlapping probes
  during slow response periods. For example: Delay=10, Timeout=5.
</Tip>

***

## Create a Health Monitor

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to pool health monitor" icon="heart-pulse">
        Open your load balancer, select the **Pools** tab, and click the pool name.
        Select the **Health Monitor** sub-tab. Click **Create Health Monitor**.
      </Step>

      <Step title="Select monitor type" icon="settings">
        Select the type that matches your backend service. HTTP is appropriate for web
        applications; TCP is appropriate for databases and custom protocols.
      </Step>

      <Step title="Set timing parameters" icon="clock">
        | Field                                                                                | Recommended Value |
        | ------------------------------------------------------------------------------------ | ----------------- |
        | **Delay**                                                                            | `10`              |
        | **Timeout**                                                                          | `5`               |
        | **Max Retries**                                                                      | `3`               |
        | For HTTP monitors, set **URL Path** to a dedicated health endpoint (e.g., `/health`) |                   |
        | that returns HTTP 200 only when the application is fully ready.                      |                   |
      </Step>

      <Step title="Confirm creation" icon="circle-check">
        <Check>Health monitor is ACTIVE. View member operating status to verify probes are running.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create HTTP health monitor" theme={null}
    openstack loadbalancer healthmonitor create \
      --name hm-http \
      --delay 10 \
      --timeout 5 \
      --max-retries 3 \
      --type HTTP \
      --url-path /health \
      --expected-codes 200 \
      pool-http
    ```

    ```bash title="Create TCP health monitor" theme={null}
    openstack loadbalancer healthmonitor create \
      --name hm-tcp \
      --delay 10 \
      --timeout 5 \
      --max-retries 3 \
      --type TCP \
      pool-db
    ```

    ```bash title="Create HTTPS health monitor" theme={null}
    openstack loadbalancer healthmonitor create \
      --name hm-https \
      --delay 10 \
      --timeout 5 \
      --max-retries 3 \
      --type HTTPS \
      --url-path /health \
      --expected-codes 200 \
      pool-secure
    ```
  </Tab>
</Tabs>

***

## View Member Health Status

After creating a health monitor, verify members are being probed and showing the correct
operating status:

```bash title="List member operating status" theme={null}
openstack loadbalancer member list pool-http \
  -c name -c address -c operating_status
```

Expected healthy output:

```
+--------+-------------+------------------+
| name   | address     | operating_status |
+--------+-------------+------------------+
| web-01 | 192.168.1.5 | ONLINE           |
| web-02 | 192.168.1.6 | ONLINE           |
+--------+-------------+------------------+
```

<Check>All members show `ONLINE` operating status — they are healthy and receiving traffic.</Check>

***

## Manage Health Monitors

<CodeGroup>
  ```bash title="Show health monitor details" theme={null}
  openstack loadbalancer healthmonitor show hm-http
  ```

  ```bash title="Update probe interval" theme={null}
  openstack loadbalancer healthmonitor set hm-http \
    --delay 5 \
    --timeout 3
  ```

  ```bash title="Delete health monitor" theme={null}
  openstack loadbalancer healthmonitor delete hm-http
  ```
</CodeGroup>

***

## Health Check Endpoint Design

For HTTP health monitors, implement a dedicated `/health` endpoint in your application:

<AccordionGroup>
  <Accordion title="What a health endpoint should check" icon="heart-pulse">
    A good health endpoint verifies that all critical dependencies are available:

    * Database connection is alive
    * Cache layer (Redis, Memcached) is reachable
    * Required configuration is loaded
    * Application can accept requests (not in restart/initialization state)

    Return HTTP 200 only when all checks pass. Return HTTP 503 when any dependency is unavailable.
  </Accordion>

  <Accordion title="What a health endpoint should NOT do" icon="circle-x">
    * Do NOT make the health endpoint expensive (no full DB queries)
    * Do NOT require authentication (health probes run without credentials)
    * Do NOT check external dependencies not under your control (external APIs, third-party services)
    * Do NOT return 200 during graceful shutdown — return 503 to drain traffic before stopping
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Pools" href="/services/load-balancer/pools">
    Manage the pools that health monitors are attached to.
  </Card>

  <Card title="Troubleshooting" href="/services/load-balancer/troubleshooting">
    Resolve OFFLINE member status and health probe failures.
  </Card>

  <Card title="Create Load Balancer" href="/services/load-balancer/create-lb">
    Create a complete load balancer setup with health monitoring from scratch.
  </Card>

  <Card title="Listeners" href="/services/load-balancer/listeners">
    Configure the protocol and ports that route traffic to your pools.
  </Card>
</CardGroup>


# Load Balancer Architecture
Source: https://docs.xloud.tech/services/load-balancer/lb-architecture

Understand the Xloud Load Balancer controller-appliance topology, data plane routing, and management plane components.

## Overview

The Xloud Load Balancer service uses a controller-appliance model. The controller manages
the lifecycle of load balancing appliances and translates API requests into configuration.
The appliance processes all data-plane traffic — the controller does not handle production
traffic. Understanding this architecture is essential for capacity planning, HA configuration,
and troubleshooting.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Service Topology

```mermaid theme={null}
graph TD
    Client["External Client Traffic"] --> VIP["Floating IP / VIP"]
    VIP --> AMP["Load Balancing Appliance\n(Active)"]
    AMP --> M1["Backend Member 1\n192.168.1.5:8080"]
    AMP --> M2["Backend Member 2\n192.168.1.6:8080"]
    AMP --> M3["Backend Member 3\n192.168.1.7:8080"]

    CTRL["LB Controller"] -->|Manages lifecycle| AMP
    CTRL -->|Health probes| M1
    CTRL -->|Health probes| M2
    CTRL -->|Health probes| M3

    DB[("Service DB\nMariaDB")] -->|State| CTRL
    MQ["Message Queue\nRabbitMQ"] -->|Commands| CTRL

    subgraph "Management Plane"
        CTRL
        DB
        MQ
    end

    subgraph "Data Plane"
        AMP
        M1
        M2
        M3
    end
```

***

## Component Reference

| Component              | Description                                                                                                                           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| **LB Controller**      | Translates API requests into appliance configuration. Manages appliance lifecycle. Does not handle production traffic.                |
| **Appliance**          | A virtual machine instance running the load balancing software. Processes all data-plane traffic. Created per load balancer resource. |
| **Service DB**         | MariaDB database storing load balancer resource state (listeners, pools, members, health monitors).                                   |
| **Message Queue**      | RabbitMQ queue for decoupled communication between the API and controller components.                                                 |
| **Management Network** | Dedicated network connecting the controller to appliances for configuration and health management. Isolated from tenant networks.     |

***

## Data Plane vs Management Plane

<AccordionGroup>
  <Accordion title="Data plane" icon="zap">
    The data plane carries production application traffic from clients to backend members.
    All data plane traffic flows through the appliance instance directly:

    * Client → Floating IP → Appliance VIP → Backend Member
    * The controller is **not** in the data path
    * Appliance failure = service interruption (use ACTIVE\_STANDBY topology for HA)
  </Accordion>

  <Accordion title="Management plane" icon="settings">
    The management plane carries control traffic between the controller and appliances:

    * Configuration updates (new listeners, pool changes, member additions)
    * Health probe coordination
    * Appliance certificate management and renewal
    * TLS certificates on the management network prevent unauthorized appliance access
  </Accordion>
</AccordionGroup>

***

## High Availability Topologies

| Topology         | Description                                         | Use Case                |
| ---------------- | --------------------------------------------------- | ----------------------- |
| `SINGLE`         | One appliance instance per load balancer            | Development and testing |
| `ACTIVE_STANDBY` | Active appliance + hot standby. Failover in seconds | Production workloads    |

Configure the topology via a flavor profile. See [Flavor Profiles](/services/load-balancer/flavor-profiles).

***

## Deployment Considerations

<AccordionGroup>
  <Accordion title="Management network sizing" icon="network">
    The management network must provide enough IP addresses for all appliance instances
    plus spare capacity for concurrent provisioning. Size the management network DHCP
    pool at `(max concurrent LBs) × 2 + 10` addresses.
  </Accordion>

  <Accordion title="Compute resources for appliances" icon="server">
    Each load balancer appliance is a virtual machine consuming compute resources.
    In large deployments, ensure sufficient compute capacity is reserved for appliance
    instances. Consider a dedicated host aggregate for load balancer appliances.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Provider Drivers" href="/services/load-balancer/provider-drivers">
    Configure the underlying load balancing implementation.
  </Card>

  <Card title="Flavor Profiles" href="/services/load-balancer/flavor-profiles">
    Define appliance capacity tiers including HA topology selection.
  </Card>

  <Card title="Monitoring" href="/services/load-balancer/lb-monitoring">
    Monitor appliance health and management plane connectivity.
  </Card>

  <Card title="Security" href="/services/load-balancer/lb-security">
    Secure the management network and appliance certificate lifecycle.
  </Card>
</CardGroup>


# Load Balancer Monitoring
Source: https://docs.xloud.tech/services/load-balancer/lb-monitoring

Monitor Xloud Load Balancer appliance health, connection statistics, and service component status for platform observability.

## Overview

Monitoring the load balancing infrastructure ensures production traffic is not impacted
by appliance degradation, certificate expiry, or capacity saturation. This guide covers
appliance health checks, traffic statistics, and manual failover procedures.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Appliance Health

```bash title="List all appliances with health status" theme={null}
openstack loadbalancer amphora list
```

Key status fields to monitor:

| Field             | Healthy Value | Description                               |
| ----------------- | ------------- | ----------------------------------------- |
| `status`          | `ALLOCATED`   | Appliance is assigned to a load balancer  |
| `lb_network_ip`   | Non-empty     | Management plane connectivity established |
| `cert_expiration` | Future date   | Appliance TLS certificate validity        |
| `compute_id`      | Non-empty     | Backing compute instance exists           |

```bash title="Show detailed appliance information" theme={null}
openstack loadbalancer amphora show <amphora-id>
```

***

## Traffic Statistics

<Tabs>
  <Tab title="Load balancer statistics" icon="bar-chart">
    ```bash title="Show load balancer statistics" theme={null}
    openstack loadbalancer stats show <lb-name>
    ```

    | Statistic            | Description                                            |
    | -------------------- | ------------------------------------------------------ |
    | `active_connections` | Current open connections to the load balancer          |
    | `bytes_in`           | Total bytes received from clients                      |
    | `bytes_out`          | Total bytes sent to clients                            |
    | `request_errors`     | Failed requests — useful for detecting upstream issues |
    | `total_connections`  | Lifetime connection count                              |
  </Tab>

  <Tab title="Listener statistics" icon="ear">
    ```bash title="Show per-listener statistics" theme={null}
    openstack loadbalancer listener stats show <listener-name>
    ```

    Listener statistics provide granular visibility when a load balancer has multiple
    listeners on different protocols or ports.
  </Tab>
</Tabs>

***

## Certificate Expiration Monitoring

Appliances use TLS certificates for controller-to-appliance management communication.
Monitor expiration to prevent management plane failures:

```bash title="Check certificate expiration on all appliances" theme={null}
openstack loadbalancer amphora list \
  -c id -c cert_expiration -c status \
  --sort-column cert_expiration
```

<Warning>
  Appliances with expired certificates cannot receive configuration updates from the
  controller. If an appliance certificate expires, trigger a failover to rotate the
  certificate:

  ```bash title="Rotate appliance certificate via failover" theme={null}
  openstack loadbalancer amphora failover <amphora-id>
  ```
</Warning>

***

## Manual Failover

Trigger a manual failover to replace a degraded or expired appliance:

```bash title="Failover an appliance" theme={null}
openstack loadbalancer amphora failover <amphora-id>
```

<Warning>
  Failover causes brief service interruption (typically under 30 seconds for
  ACTIVE\_STANDBY topologies) while the replacement appliance is provisioned
  and configuration is replicated.
</Warning>

Monitor failover progress:

```bash title="Monitor load balancer provisioning status during failover" theme={null}
watch -n 5 "openstack loadbalancer show <lb-name> -c provisioning_status"
```

***

## Prometheus Integration

Xloud Load Balancer exposes metrics via the Octavia Prometheus exporter when configured.
Key metrics to alert on:

| Metric                             | Alert Threshold | Description                         |
| ---------------------------------- | --------------- | ----------------------------------- |
| `octavia_loadbalancer_status`      | != 1            | Load balancer not ACTIVE            |
| `octavia_member_status`            | != 1            | Member not ONLINE                   |
| `octavia_amphora_cert_expiry_days` | \< 30           | Appliance certificate expiring soon |

***

## Next Steps

<CardGroup>
  <Card title="Security" href="/services/load-balancer/lb-security">
    Configure TLS certificate lifecycle management and management plane access controls.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/load-balancer/lb-troubleshooting">
    Use monitoring data to diagnose and resolve platform-level failures.
  </Card>

  <Card title="Flavor Profiles" href="/services/load-balancer/flavor-profiles">
    Upgrade appliance capacity when statistics show saturation.
  </Card>

  <Card title="Architecture" href="/services/load-balancer/lb-architecture">
    Understand the relationship between appliances and the management plane.
  </Card>
</CardGroup>


# Load Balancer Quotas
Source: https://docs.xloud.tech/services/load-balancer/lb-quotas

Configure per-project resource limits for Xloud Load Balancer instances, listeners, pools, and members.

## Overview

Quotas prevent individual projects from consuming excessive load balancing resources.
Default quota values are set platform-wide; administrators override them per project.
Load balancer quotas cover the full resource hierarchy — from the top-level load balancer
down to individual L7 rules.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Default Quota Reference

| Resource        | Default Limit | Description                                      |
| --------------- | ------------- | ------------------------------------------------ |
| `loadbalancer`  | 10            | Load balancer instances per project              |
| `listener`      | 50            | Listeners across all load balancers in a project |
| `pool`          | 50            | Pools across all load balancers in a project     |
| `member`        | 50            | Pool members per project                         |
| `healthmonitor` | 50            | Health monitors per project                      |
| `l7policy`      | 50            | L7 routing policies per project                  |
| `l7rule`        | 50            | L7 rules per project                             |

***

## View Quotas

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to the admin quota settings and scroll to the **Load Balancer** section
    to view platform-wide default quotas. Per-project overrides are visible in
    **Identity > Projects** (admin view) > Modify Quotas.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show platform-wide default quotas" theme={null}
    openstack loadbalancer quota defaults show
    ```

    ```bash title="Show quotas for a specific project" theme={null}
    openstack loadbalancer quota show <project-id>
    ```

    ```bash title="List quotas for all projects" theme={null}
    openstack loadbalancer quota list --all-projects
    ```
  </Tab>
</Tabs>

***

## Set Project Quotas

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Identity > Projects** (admin view). Select the project and click
    **Modify Quotas**. Scroll to the **Load Balancer** section and update the limits.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Set project-specific quotas" theme={null}
    openstack loadbalancer quota set \
      --loadbalancer 20 \
      --listener 100 \
      --pool 100 \
      --member 200 \
      --healthmonitor 50 \
      <project-id>
    ```

    ```bash title="Reset project to platform defaults" theme={null}
    openstack loadbalancer quota delete <project-id>
    ```
  </Tab>
</Tabs>

<Tip>
  Increase quotas for production projects that run multiple microservices behind
  separate load balancers. Keep development and test project quotas at defaults
  to control infrastructure costs.
</Tip>

***

## Monitor Quota Consumption

Regularly review quota consumption to identify projects approaching limits:

```bash title="List all project quotas sorted by load balancer count" theme={null}
openstack loadbalancer quota list --all-projects \
  -f json | python3 -c "
import json, sys
quotas = json.load(sys.stdin)
for q in sorted(quotas, key=lambda x: x.get('loadbalancer', 0), reverse=True)[:10]:
    print(f'{q.get(\"loadbalancer\",0):4d} LBs  {q[\"project_id\"]}')
"
```

***

## Next Steps

<CardGroup>
  <Card title="Flavor Profiles" href="/services/load-balancer/flavor-profiles">
    Create capacity tiers that users can select within their quota limits.
  </Card>

  <Card title="Monitoring" href="/services/load-balancer/lb-monitoring">
    Monitor actual resource consumption alongside quota limits.
  </Card>

  <Card title="Security" href="/services/load-balancer/lb-security">
    Audit quota usage as part of security and compliance reviews.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/load-balancer/lb-troubleshooting">
    Resolve quota exceeded errors reported by users.
  </Card>
</CardGroup>


# Load Balancer Security
Source: https://docs.xloud.tech/services/load-balancer/lb-security

Restrict management plane access, manage TLS certificate lifecycle, and audit quota usage for Xloud Load Balancer security hardening.

## Overview

Securing the Xloud Load Balancer infrastructure protects both the management plane
(controller-to-appliance communication) and the data plane (client traffic). This guide
covers network isolation of the management plane, appliance TLS certificate lifecycle,
quota auditing, and access log configuration.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Management Plane Isolation

The load balancing management network carries control traffic between the controller
and appliances. This network must be isolated from tenant and provider networks to
prevent unauthorized appliance access.

<Steps>
  <Step title="Assign a dedicated management network" icon="network">
    The management network should use a non-routable CIDR not reachable from tenant
    networks or the internet. Configure in XDeploy globals:

    ```yaml title="Load balancer management network" theme={null}
    octavia_management_network: lb-management-net
    octavia_management_subnet: lb-management-subnet
    ```
  </Step>

  <Step title="Apply security group rules" icon="shield">
    The management security group should allow only the controller IP to reach
    appliance management ports:

    * TCP 9443 — appliance API (controller → appliance only)
    * UDP 5555 — health manager heartbeat (bidirectional)

    ```bash title="Verify management security group rules" theme={null}
    openstack security group rule list octavia-management-sg
    ```
  </Step>

  <Step title="Verify network isolation" icon="circle-check">
    Confirm tenant instances cannot reach the management network CIDR:

    ```bash title="Test management network isolation" theme={null}
    # From a tenant instance — should be unreachable
    ping <management-network-cidr-first-ip>
    ```

    <Check>Ping should fail — management network is not routable from tenant networks.</Check>
  </Step>
</Steps>

***

## TLS Certificate Lifecycle

Appliances use TLS certificates for secure controller-to-appliance communication.
Monitor and rotate certificates before expiry.

```bash title="Check all appliance certificate expiration dates" theme={null}
openstack loadbalancer amphora list \
  -c id -c cert_expiration -c status \
  --sort-column cert_expiration
```

For appliances with certificates expiring within 30 days:

```bash title="Trigger certificate rotation via failover" theme={null}
openstack loadbalancer amphora failover <amphora-id>
```

<Note>
  Certificate rotation via failover replaces the appliance with a fresh instance using
  a new certificate. In ACTIVE\_STANDBY topology, this is non-disruptive. In SINGLE
  topology, expect a brief interruption.
</Note>

***

## Quota Auditing

Regularly review quota consumption to identify projects with unusual resource usage:

```bash title="Audit all project quotas" theme={null}
openstack loadbalancer quota list --all-projects
```

```bash title="Find projects using more than 80% of their load balancer quota" theme={null}
openstack loadbalancer quota list --all-projects -f json | python3 -c "
import json, sys
quotas = json.load(sys.stdin)
for q in quotas:
    lbs = q.get('in_use_loadbalancer', 0)
    limit = q.get('loadbalancer', 10)
    if limit > 0 and lbs / limit >= 0.8:
        print(f'{lbs}/{limit} LBs  {q[\"project_id\"]}')
"
```

Investigate projects consuming unusually high member or listener counts — this may
indicate misconfigured applications creating excessive resources or resource leaks.

***

## Access Log Configuration

Configure load balancer access logs to forward to your centralized logging platform.
Access logs capture source IPs, request URIs, response codes, and latency — essential
data for compliance auditing and security incident investigation.

```yaml title="Enable access logging in XDeploy globals" theme={null}
octavia_enable_access_log: "yes"
octavia_access_log_facility: LOG_LOCAL0
```

Apply:

```bash title="Apply access log configuration" theme={null}
xavs-ansible deploy --tags octavia
```

<Tip>
  Store load balancer access logs in Xloud Object Storage for long-term retention.
  A retention policy of 90 days satisfies most compliance frameworks.
</Tip>

***

## Security Checklist

<AccordionGroup>
  <Accordion title="Management network isolated" icon="network">
    Verify the management network CIDR is not reachable from tenant instances or external networks.
  </Accordion>

  <Accordion title="Appliance certificates valid" icon="shield">
    All appliances have `cert_expiration` dates more than 30 days in the future.
  </Accordion>

  <Accordion title="Quota limits configured" icon="gauge">
    All projects have explicit quota limits set — none are using unlimited defaults
    in a multi-tenant environment.
  </Accordion>

  <Accordion title="Access logs forwarded" icon="file-text">
    Access logs are configured and flowing to the centralized logging platform.
    Verify by creating a test load balancer and checking for log entries.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Monitoring" href="/services/load-balancer/lb-monitoring">
    Set up proactive alerts for certificate expiry and appliance health.
  </Card>

  <Card title="Quotas" href="/services/load-balancer/lb-quotas">
    Configure per-project resource limits to prevent over-consumption.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/load-balancer/lb-troubleshooting">
    Resolve security-related configuration and access failures.
  </Card>

  <Card title="Architecture" href="/services/load-balancer/lb-architecture">
    Review the management and data plane boundaries for security design.
  </Card>
</CardGroup>


# Load Balancer Admin Troubleshooting
Source: https://docs.xloud.tech/services/load-balancer/lb-troubleshooting

Diagnose and resolve platform-level Xloud Load Balancer issues — service outages, appliance provisioning failures, and performance degradation.

## Overview

This guide covers platform-level load balancer issues requiring administrator access —
service agent failures, appliance provisioning problems, and capacity-related performance
degradation.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  For user-facing issues (PENDING\_CREATE status, member OFFLINE, TLS errors), see the
  [Load Balancer User Troubleshooting](/services/load-balancer/troubleshooting) guide.
</Note>

***

## Service Not Responding

<AccordionGroup>
  <Accordion title="Load balancer API service unavailable" icon="server">
    **Cause**: The load balancer API service or worker containers may be stopped.

    **Diagnose**:

    ```bash title="Check all load balancer service containers" theme={null}
    docker ps --filter name=octavia
    ```

    Check container logs for errors:

    ```bash title="View API container logs" theme={null}
    docker logs octavia_api --tail 100
    ```

    ```bash title="View worker/health manager logs" theme={null}
    docker logs octavia_worker --tail 100
    docker logs octavia_health_manager --tail 100
    ```

    **Resolution**: Restart services via XDeploy:

    ```bash title="Restart load balancer services" theme={null}
    xavs-ansible deploy --tags octavia
    ```
  </Accordion>

  <Accordion title="RabbitMQ connectivity lost" icon="message-circle">
    **Cause**: The load balancer worker cannot connect to the message queue, preventing
    it from receiving provisioning commands.

    **Diagnose**:

    ```bash title="Check RabbitMQ connectivity from worker" theme={null}
    docker exec octavia_worker python3 -c "
    import pika
    conn = pika.BlockingConnection(pika.URLParameters('amqp://octavia:<password>@<host>:5672/'))
    print('RabbitMQ connection OK')
    conn.close()
    "
    ```

    **Resolution**: Verify RabbitMQ is running and the octavia user credentials are correct.
  </Accordion>
</AccordionGroup>

***

## Appliance Provisioning Failures

<AccordionGroup>
  <Accordion title="Appliance provisioning fails — DHCP pool exhausted" icon="network">
    **Cause**: The management network DHCP pool has no available IP addresses.

    **Diagnose**:

    ```bash title="Check DHCP pool usage" theme={null}
    openstack subnet show lb-management-subnet \
      -c allocation_pools -c dns_nameservers
    ```

    ```bash title="Count current appliance IPs in use" theme={null}
    openstack loadbalancer amphora list \
      -c lb_network_ip | grep -v None | wc -l
    ```

    **Resolution**: Expand the DHCP pool by updating the management subnet allocation range:

    ```bash title="Update DHCP pool range" theme={null}
    openstack subnet set \
      --allocation-pool start=<new-start>,end=<new-end> \
      lb-management-subnet
    ```
  </Accordion>

  <Accordion title="Appliance image unavailable" icon="disc">
    **Cause**: The appliance image is not registered in the Image Service, or it has been
    deactivated.

    **Diagnose**:

    ```bash title="Check appliance image availability" theme={null}
    openstack image list --tag amphora
    ```

    **Resolution**: If the image is missing, re-upload it:

    ```bash title="Upload amphora image" theme={null}
    openstack image create \
      --disk-format qcow2 \
      --container-format bare \
      --file amphora-x64-haproxy.qcow2 \
      --tag amphora \
      --public \
      amphora-image
    ```
  </Accordion>
</AccordionGroup>

***

## Performance Degradation

<AccordionGroup>
  <Accordion title="High latency through load balancer" icon="clock">
    **Cause**: Appliance compute resources are saturated, or the backend member count has
    grown beyond the flavor profile's rated capacity.

    **Diagnose**: Check appliance statistics for connection counts:

    ```bash title="Check load balancer connection statistics" theme={null}
    openstack loadbalancer stats show <lb-name>
    ```

    Compare `active_connections` against the flavor profile's rated capacity.

    **Resolution**: Create a new load balancer with a higher-capacity flavor and migrate
    listeners:

    <Warning>
      Flavor profiles cannot be changed on existing load balancers. To upgrade capacity:

      1. Create a new load balancer with the target flavor
      2. Recreate all listeners, pools, and members on the new load balancer
      3. Update DNS / floating IP to point to the new load balancer
      4. Delete the original load balancer after traffic is migrated
    </Warning>
  </Accordion>
</AccordionGroup>

***

## Service Log Reference

| Container         | Log command                                     |
| ----------------- | ----------------------------------------------- |
| Load balancer API | `docker logs octavia_api --tail 100`            |
| Worker            | `docker logs octavia_worker --tail 100`         |
| Health manager    | `docker logs octavia_health_manager --tail 100` |
| Housekeeping      | `docker logs octavia_housekeeping --tail 100`   |

***

## Next Steps

<CardGroup>
  <Card title="Monitoring" href="/services/load-balancer/lb-monitoring">
    Set up proactive monitoring to catch issues before they impact production.
  </Card>

  <Card title="Flavor Profiles" href="/services/load-balancer/flavor-profiles">
    Create higher-capacity flavor profiles for workloads experiencing saturation.
  </Card>

  <Card title="Architecture" href="/services/load-balancer/lb-architecture">
    Review the component relationships that can cause cascading failures.
  </Card>

  <Card title="Security" href="/services/load-balancer/lb-security">
    Verify management plane isolation after resolving connectivity issues.
  </Card>
</CardGroup>


# Load Balancer Listeners
Source: https://docs.xloud.tech/services/load-balancer/listeners

Add and configure HTTP, HTTPS, TCP, and UDP listeners on Xloud Load Balancers for multi-protocol traffic handling.

## Overview

A listener defines a protocol and port combination on which a load balancer accepts
inbound connections. A single load balancer supports multiple listeners simultaneously —
e.g., an HTTP listener on port 80 and an HTTPS listener on port 443 can share
the same load balancer VIP. Each listener routes traffic to its own default pool.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Supported Protocols

| Protocol           | Port          | Use Case                                                        |
| ------------------ | ------------- | --------------------------------------------------------------- |
| `HTTP`             | Typically 80  | Unencrypted web application traffic                             |
| `TERMINATED_HTTPS` | Typically 443 | TLS offloaded at the load balancer; backend receives plain HTTP |
| `HTTPS`            | Typically 443 | TLS passthrough; load balancer does not decrypt traffic         |
| `TCP`              | Any           | Any TCP service — databases, custom protocols                   |
| `UDP`              | Any           | UDP-based services — DNS, game servers                          |
| `SCTP`             | Any           | SCTP-based telecommunications traffic                           |

***

## Add a Listener

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open listener creation" icon="plus">
        Navigate to **Network > Load Balancers**, select your load balancer,
        and click the **Listeners** tab. Click **Create Listener**.
      </Step>

      <Step title="Configure the listener" icon="settings">
        | Field                     | Description                                                           |
        | ------------------------- | --------------------------------------------------------------------- |
        | **Name**                  | Display name (e.g., `listener-https`)                                 |
        | **Protocol**              | Select from supported protocols above                                 |
        | **Protocol Port**         | Port on which the listener accepts connections                        |
        | **Connection Limit**      | Maximum concurrent connections (-1 for unlimited)                     |
        | **Default TLS Container** | For TERMINATED\_HTTPS — select the certificate from Xloud Key Manager |
      </Step>

      <Step title="Associate a pool" icon="users">
        After creating the listener, associate it with a backend pool. Select an existing
        pool or create a new one from the **Pools** tab.
        <Check>Listener is ACTIVE and routing to the associated pool.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create HTTP listener" theme={null}
    openstack loadbalancer listener create \
      --name listener-http \
      --protocol HTTP \
      --protocol-port 80 \
      prod-web-lb
    ```

    ```bash title="Create HTTPS listener with TLS termination" theme={null}
    openstack loadbalancer listener create \
      --name listener-https \
      --protocol TERMINATED_HTTPS \
      --protocol-port 443 \
      --default-tls-container-ref <key-manager-container-ref> \
      prod-web-lb
    ```

    ```bash title="Create TCP listener" theme={null}
    openstack loadbalancer listener create \
      --name listener-db \
      --protocol TCP \
      --protocol-port 5432 \
      prod-db-lb
    ```
  </Tab>
</Tabs>

***

## TLS Termination (TERMINATED\_HTTPS)

TLS termination offloads certificate processing at the load balancer and forwards plain
HTTP to backend members — reducing CPU overhead on application servers.

<Steps>
  <Step title="Store certificate in Key Manager" icon="lock">
    Store your TLS certificate and private key in Xloud Key Management:

    ```bash title="Create secret container with certificate and key" theme={null}
    openstack secret store \
      --name tls-cert \
      --payload-content-type "application/pkix-cert" \
      --payload "$(cat cert.pem | base64)"

    openstack secret store \
      --name tls-key \
      --payload-content-type "application/octet-stream" \
      --payload "$(cat key.pem | base64)"

    openstack secret container create \
      --name prod-tls-container \
      --type certificate \
      --secret "certificate=$(openstack secret list --name tls-cert -c 'Secret href' -f value)" \
      --secret "private_key=$(openstack secret list --name tls-key -c 'Secret href' -f value)"
    ```
  </Step>

  <Step title="Create listener with certificate container" icon="shield">
    ```bash title="Create TERMINATED_HTTPS listener" theme={null}
    CONTAINER_REF=$(openstack secret container show prod-tls-container -c container_ref -f value)

    openstack loadbalancer listener create \
      --name listener-https \
      --protocol TERMINATED_HTTPS \
      --protocol-port 443 \
      --default-tls-container-ref $CONTAINER_REF \
      prod-web-lb
    ```

    <Check>Listener is ACTIVE and accepting encrypted connections on port 443.</Check>
  </Step>
</Steps>

***

## Manage Listeners

<CodeGroup>
  ```bash title="List listeners on a load balancer" theme={null}
  openstack loadbalancer listener list \
    --loadbalancer prod-web-lb
  ```

  ```bash title="Show listener details" theme={null}
  openstack loadbalancer listener show listener-https
  ```

  ```bash title="Update connection limit" theme={null}
  openstack loadbalancer listener set listener-http \
    --connection-limit 10000
  ```

  ```bash title="Delete a listener" theme={null}
  openstack loadbalancer listener delete listener-http
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Pools" href="/services/load-balancer/pools">
    Configure backend pools and member management for each listener.
  </Card>

  <Card title="Health Monitors" href="/services/load-balancer/health-monitors">
    Set up health checks for pools backing your listeners.
  </Card>

  <Card title="Floating IP Assignment" href="/services/load-balancer/floating-ip">
    Expose the load balancer VIP publicly after configuring listeners.
  </Card>

  <Card title="Troubleshooting" href="/services/load-balancer/troubleshooting">
    Resolve TLS handshake failures and protocol-specific issues.
  </Card>
</CardGroup>


# Load Balancer Pools
Source: https://docs.xloud.tech/services/load-balancer/pools

Create and manage backend member pools, configure traffic distribution algorithms, and handle session persistence in Xloud Load Balancer.

## Overview

A pool is a collection of backend member instances that receive traffic from a listener.
Pools define how traffic is distributed across members (algorithm), whether sessions are
sticky (persistence), and which protocol the pool uses internally. Each listener has a
default pool; additional pools can be used for L7 policy-based routing.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Distribution Algorithms

| Algorithm           | Description                                                     | Best For                                                   |
| ------------------- | --------------------------------------------------------------- | ---------------------------------------------------------- |
| `ROUND_ROBIN`       | Distributes requests evenly across all UP members               | Stateless applications with equal capacity members         |
| `LEAST_CONNECTIONS` | Sends new requests to the member with fewest active connections | Long-lived connections with variable request duration      |
| `SOURCE_IP`         | Routes requests from the same client IP to the same member      | Stateful applications without cookie-based persistence     |
| `SOURCE_IP_PORT`    | Routes based on client IP + port combination                    | Symmetric NAT environments requiring deterministic mapping |

***

## Create a Pool

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Pools tab" icon="users">
        Open your load balancer in **Network > Load Balancers** and select
        the **Pools** tab. Click **Create Pool**.
      </Step>

      <Step title="Configure the pool" icon="settings">
        | Field                   | Description                                            |
        | ----------------------- | ------------------------------------------------------ |
        | **Name**                | Pool display name                                      |
        | **Protocol**            | Must match or be compatible with the listener protocol |
        | **Algorithm**           | Traffic distribution method                            |
        | **Session Persistence** | Sticky session configuration (optional)                |
      </Step>

      <Step title="Add members" icon="user-plus">
        After pool creation, select the pool and click the **Members** sub-tab.
        Click **Add Member** to register backend instances.

        | Field                    | Description                                                   |
        | ------------------------ | ------------------------------------------------------------- |
        | **IP Address**           | Backend instance IP                                           |
        | **Protocol Port**        | Port your application listens on                              |
        | **Weight**               | Relative traffic weight (default: 1)                          |
        | **Monitor Address/Port** | Override health check target if different from member address |
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create pool linked to listener" theme={null}
    openstack loadbalancer pool create \
      --name pool-http \
      --lb-algorithm ROUND_ROBIN \
      --listener listener-http \
      --protocol HTTP
    ```

    ```bash title="Create standalone pool (for L7 routing)" theme={null}
    openstack loadbalancer pool create \
      --name pool-api \
      --lb-algorithm LEAST_CONNECTIONS \
      --loadbalancer prod-web-lb \
      --protocol HTTP
    ```
  </Tab>
</Tabs>

***

## Manage Pool Members

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to your pool and click the **Members** sub-tab. Use **Add Member** to register
    instances and **Delete** to remove members that are decommissioned.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Add a member" theme={null}
      openstack loadbalancer member create \
        --subnet-id <instance-subnet-id> \
        --address <instance-ip> \
        --protocol-port 8080 \
        pool-http
      ```

      ```bash title="List pool members with health status" theme={null}
      openstack loadbalancer member list pool-http \
        -c name -c address -c protocol_port -c operating_status
      ```

      ```bash title="Update member weight" theme={null}
      openstack loadbalancer member set pool-http <member-id> \
        --weight 2
      ```

      ```bash title="Remove a member" theme={null}
      openstack loadbalancer member delete pool-http <member-id>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Session Persistence

Session persistence routes subsequent requests from the same client to the same backend
member, enabling stateful applications to work behind the load balancer.

<Tabs>
  <Tab title="HTTP Cookie (recommended)" icon="cookie">
    The load balancer inserts a session cookie. Clients are routed to the member that
    served their first request:

    ```bash title="Create pool with HTTP cookie persistence" theme={null}
    openstack loadbalancer pool create \
      --name pool-stateful \
      --lb-algorithm ROUND_ROBIN \
      --listener listener-http \
      --protocol HTTP \
      --session-persistence type=HTTP_COOKIE
    ```
  </Tab>

  <Tab title="Source IP" icon="network">
    Routes requests from the same client IP to the same member. Works at Layer 4 but
    breaks when clients are behind NAT with shared IP addresses:

    ```bash title="Create pool with source IP persistence" theme={null}
    openstack loadbalancer pool create \
      --name pool-sourceip \
      --lb-algorithm ROUND_ROBIN \
      --listener listener-http \
      --protocol HTTP \
      --session-persistence type=SOURCE_IP
    ```
  </Tab>

  <Tab title="Application Cookie" icon="code">
    The load balancer reads an existing cookie set by your application:

    ```bash title="Create pool with application cookie persistence" theme={null}
    openstack loadbalancer pool create \
      --name pool-appcookie \
      --lb-algorithm ROUND_ROBIN \
      --listener listener-http \
      --protocol HTTP \
      --session-persistence "type=APP_COOKIE,cookie_name=JSESSIONID"
    ```
  </Tab>
</Tabs>

***

## Update Pool Settings

```bash title="Change distribution algorithm" theme={null}
openstack loadbalancer pool set pool-http \
  --lb-algorithm LEAST_CONNECTIONS
```

```bash title="Remove session persistence" theme={null}
openstack loadbalancer pool set pool-http \
  --no-session-persistence
```

***

## Next Steps

<CardGroup>
  <Card title="Health Monitors" href="/services/load-balancer/health-monitors">
    Configure health probes to automatically remove unhealthy members from pools.
  </Card>

  <Card title="Listeners" href="/services/load-balancer/listeners">
    Configure the listeners that route traffic to your pools.
  </Card>

  <Card title="Create Load Balancer" href="/services/load-balancer/create-lb">
    Full walkthrough for creating a load balancer with pool and health monitor.
  </Card>

  <Card title="Troubleshooting" href="/services/load-balancer/troubleshooting">
    Resolve member OFFLINE status and pool connectivity issues.
  </Card>
</CardGroup>


# Load Balancer Provider Drivers
Source: https://docs.xloud.tech/services/load-balancer/provider-drivers

View and configure load balancing provider drivers in Xloud to control the underlying appliance implementation.

## Overview

Provider drivers determine the underlying implementation used to create and manage load
balancing appliances. The default provider is configured during XDeploy deployment.
You can view available providers, inspect their capabilities, and allow users
to select a specific provider at provisioning time.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## View Available Providers

```bash title="List available load balancing providers" theme={null}
openstack loadbalancer provider list
```

Example output:

```
+----------+---------------------+
| name     | description         |
+----------+---------------------+
| amphora  | Amphora provider    |
| ovn      | OVN provider        |
+----------+---------------------+
```

```bash title="List provider capabilities" theme={null}
openstack loadbalancer provider capability list <provider-name>
```

The capabilities output lists supported protocols, algorithms, session persistence types,
and feature flags (ACTIVE\_STANDBY topology, L7 policies, TLS termination, etc.).

***

## Provider Comparison

| Feature            | Amphora                            | OVN                                       |
| ------------------ | ---------------------------------- | ----------------------------------------- |
| Data plane         | Instance-based appliance           | OVN logical flows (no appliance instance) |
| HA topology        | ACTIVE\_STANDBY supported          | Built-in via OVN HA                       |
| L7 routing         | Full L7 policy support             | Limited L7 support                        |
| TLS termination    | Supported                          | Supported                                 |
| Throughput         | Limited by appliance instance size | Near line-rate (kernel-based)             |
| Appliance overhead | 1 instance per load balancer       | No appliance instances                    |
| Best for           | Full-featured production LBs       | High-throughput, simpler LBs              |

***

## Configure Default Provider

The default provider is set during XDeploy deployment via globals:

```yaml title="XDeploy globals: set default load balancer provider" theme={null}
octavia_provider: "amphora"  # or "ovn"
```

Apply after changing:

```bash title="Deploy load balancer configuration" theme={null}
xavs-ansible deploy --tags octavia
```

<Warning>
  Changing the default provider affects all new load balancers. Existing load balancers
  retain their original provider — they are not automatically migrated.
</Warning>

***

## Specify Provider at Provisioning Time

You can override the default provider when creating a load balancer:

```bash title="Create load balancer with specific provider" theme={null}
openstack loadbalancer create \
  --name my-lb \
  --vip-subnet-id <subnet-id> \
  --provider ovn
```

<Note>
  Not all providers are available in all deployments. The available providers depend
  on which drivers are deployed and configured by the platform administrator.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Flavor Profiles" href="/services/load-balancer/flavor-profiles">
    Create capacity tiers that wrap provider-specific configuration for users.
  </Card>

  <Card title="Architecture" href="/services/load-balancer/lb-architecture">
    Understand the controller-appliance model for the Amphora provider.
  </Card>

  <Card title="Monitoring" href="/services/load-balancer/lb-monitoring">
    Monitor provider-specific appliance health and statistics.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/load-balancer/lb-troubleshooting">
    Resolve provider driver configuration and appliance provisioning issues.
  </Card>
</CardGroup>


# Load Balancer Troubleshooting
Source: https://docs.xloud.tech/services/load-balancer/troubleshooting

Diagnose and resolve common Xloud Load Balancer issues — provisioning failures, member health problems, and TLS errors.

## Overview

This guide covers the most common issues encountered when working with Xloud Load Balancer
— provisioning delays, member health failures, 503 errors, and TLS handshake problems.

<Note>
  For platform-level issues such as appliance provisioning failures or service agent
  outages, refer to the [Load Balancer Admin Guide — Troubleshooting](/services/load-balancer/lb-troubleshooting).
</Note>

***

## Provisioning Issues

<AccordionGroup>
  <Accordion title="Load balancer stuck in PENDING_CREATE" icon="clock">
    **Cause**: The load balancing agent did not receive or complete the provisioning request.

    **Diagnose**:

    ```bash title="Check provisioning status" theme={null}
    openstack loadbalancer show prod-web-lb \
      -c provisioning_status -c operating_status
    ```

    If status remains `PENDING_CREATE` after 5 minutes, verify the appliance was created:

    ```bash title="List appliances for this load balancer" theme={null}
    openstack loadbalancer amphora list \
      --loadbalancer prod-web-lb
    ```

    **Resolution**: Contact your platform administrator if the appliance does not appear.
    The load balancing service agent may need to be restarted.

    <Tip>
      A load balancer in `PENDING_CREATE` can be deleted and recreated. Use `--wait` to
      block until the status resolves: `openstack loadbalancer delete --wait prod-web-lb`
    </Tip>
  </Accordion>

  <Accordion title="Load balancer in ERROR status after creation" icon="circle-x">
    **Cause**: The appliance provisioning failed — common causes include exhausted
    management network DHCP pool or unavailable appliance image.

    **Diagnose**:

    ```bash title="Check provisioning status detail" theme={null}
    openstack loadbalancer show prod-web-lb -f json | python3 -m json.tool
    ```

    Review the `fault` field for a specific error message.

    **Resolution**: This is a platform-level issue. See the Admin Guide for resolution steps.
  </Accordion>
</AccordionGroup>

***

## Member Health Issues

<AccordionGroup>
  <Accordion title="Members showing OFFLINE or ERROR" icon="server">
    **Cause**: Health monitor probes are failing. Common reasons:

    * Application not running on the configured member port
    * Security group blocking probe traffic from the load balancer VIP subnet
    * Health check URL returning a non-2xx status code
    * Application is not ready (still starting up)

    **Diagnose**:

    ```bash title="Show member detail" theme={null}
    openstack loadbalancer member show pool-http <member-id>
    ```

    Test the health endpoint directly from a host on the same subnet:

    ```bash title="Test health endpoint manually" theme={null}
    curl -v http://<member-ip>:<member-port>/health
    ```

    **Resolution**: Verify that security groups allow ingress on the member port from
    the load balancer VIP address. Add a rule if missing:

    ```bash title="Allow LB probe traffic in security group" theme={null}
    openstack security group rule create \
      --ingress \
      --protocol tcp \
      --dst-port <member-port> \
      --remote-ip <lb-vip-address>/32 \
      <member-security-group>
    ```
  </Accordion>

  <Accordion title="Member flapping between ONLINE and OFFLINE" icon="activity">
    **Cause**: The health check timeout is too short, or the application has slow response
    times under load.

    **Resolution**: Increase the health monitor timeout:

    ```bash title="Increase health monitor timeout" theme={null}
    openstack loadbalancer healthmonitor set hm-http \
      --timeout 10 \
      --delay 15
    ```
  </Accordion>
</AccordionGroup>

***

## Traffic Issues

<AccordionGroup>
  <Accordion title="503 Service Unavailable from load balancer" icon="triangle-alert">
    **Cause**: All pool members are DOWN or the pool is empty.

    **Diagnose**:

    ```bash title="Check member operating status" theme={null}
    openstack loadbalancer member list pool-http \
      -c address -c protocol_port -c operating_status
    ```

    **Resolution**: Restore at least one member to `ONLINE` status by:

    1. Verifying the application is running on the member
    2. Confirming the health check URL returns HTTP 200
    3. Checking security group rules allow probe traffic
  </Accordion>

  <Accordion title="Uneven traffic distribution" icon="bar-chart">
    **Cause**: Member weights are unequal, or sticky sessions are routing traffic to
    a subset of members.

    **Diagnose**:

    ```bash title="Check member weights" theme={null}
    openstack loadbalancer member list pool-http \
      -c name -c weight -c operating_status
    ```

    **Resolution**: Reset all member weights to equal values:

    ```bash title="Normalize member weight" theme={null}
    openstack loadbalancer member set pool-http <member-id> --weight 1
    ```
  </Accordion>
</AccordionGroup>

***

## TLS Issues

<AccordionGroup>
  <Accordion title="TLS handshake failures on HTTPS listener" icon="shield-alert">
    **Cause**: Certificate container reference is invalid, the certificate has expired,
    or the private key does not match the certificate.

    **Diagnose**:

    ```bash title="Verify TLS container is accessible" theme={null}
    openstack secret container show <container-ref>
    ```

    Check certificate expiration:

    ```bash title="View certificate content" theme={null}
    openstack secret get <cert-secret-ref> --payload | \
      openssl x509 -noout -dates
    ```

    **Resolution**: Upload a valid certificate through Xloud Key Manager and update
    the listener's TLS container reference:

    ```bash title="Update listener TLS container" theme={null}
    openstack loadbalancer listener set listener-https \
      --default-tls-container-ref <new-container-ref>
    ```
  </Accordion>
</AccordionGroup>

***

## Status Reference

| Provisioning Status | Meaning                                      |
| ------------------- | -------------------------------------------- |
| `ACTIVE`            | Resource is operational                      |
| `PENDING_CREATE`    | Provisioning in progress                     |
| `PENDING_UPDATE`    | Update in progress                           |
| `PENDING_DELETE`    | Deletion in progress                         |
| `ERROR`             | Operation failed — inspect the `fault` field |

| Operating Status | Meaning                                           |
| ---------------- | ------------------------------------------------- |
| `ONLINE`         | Resource is up and passing health checks          |
| `OFFLINE`        | Resource is administratively disabled             |
| `DEGRADED`       | Some sub-resources are in error                   |
| `ERROR`          | Resource has failed health checks                 |
| `NO_MONITOR`     | No health monitor configured — traffic still sent |

***

## Next Steps

<CardGroup>
  <Card title="Health Monitors" href="/services/load-balancer/health-monitors">
    Review health monitor configuration to resolve OFFLINE member issues.
  </Card>

  <Card title="Listeners" href="/services/load-balancer/listeners">
    Verify listener protocol and TLS configuration after resolving issues.
  </Card>

  <Card title="Load Balancer Admin Troubleshooting" href="/services/load-balancer/lb-troubleshooting">
    Platform-level diagnostics for appliance and service agent failures.
  </Card>

  <Card title="Create Load Balancer" href="/services/load-balancer/create-lb">
    Start fresh with a correctly configured load balancer if needed.
  </Card>
</CardGroup>


# Load Balancer User Guide
Source: https://docs.xloud.tech/services/load-balancer/user-guide

Create and manage load balancers, listeners, pools, health monitors, and floating IP assignments in Xloud.

<p>Overview</p>

Xloud Load Balancer distributes inbound application traffic across a pool of backend
instances. Use the guides below to create load balancers, configure listeners and pools,
set up health monitors, assign public floating IPs, and troubleshoot traffic issues.

<CardGroup>
  <Card title="Create a Load Balancer" icon="plus-circle" href="/services/load-balancer/create-lb">
    Provision a new load balancer with a listener, pool, and health monitor in a single workflow.
  </Card>

  <Card title="Listeners" icon="ear" href="/services/load-balancer/listeners">
    Add HTTP, HTTPS, TCP, and UDP listeners to an existing load balancer.
  </Card>

  <Card title="Pools" icon="users" href="/services/load-balancer/pools">
    Manage backend member pools and configure traffic distribution algorithms.
  </Card>

  <Card title="Health Monitors" icon="heart-pulse" href="/services/load-balancer/health-monitors">
    Configure TCP, HTTP, and HTTPS health checks to automatically remove unhealthy members.
  </Card>

  <Card title="Floating IP Assignment" icon="globe" href="/services/load-balancer/floating-ip">
    Expose your load balancer VIP on a public network using a floating IP.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/load-balancer/troubleshooting">
    Resolve provisioning failures, member health issues, and TLS handshake errors.
  </Card>
</CardGroup>

***

<p>Core Concepts</p>

| Concept            | Description                                                                     |
| ------------------ | ------------------------------------------------------------------------------- |
| **Load Balancer**  | Top-level resource holding a virtual IP (VIP) on a chosen subnet                |
| **Listener**       | Protocol and port definition. A load balancer supports multiple listeners       |
| **Pool**           | Collection of backend members. Each listener routes to one default pool         |
| **Member**         | A backend instance IP and port registered in a pool                             |
| **Health Monitor** | Periodic probes that mark members UP or DOWN                                    |
| **L7 Policy**      | HTTP routing rules — redirect, reject, or forward based on URL path or hostname |
| **Flavor**         | Appliance capacity profile selected at provisioning time                        |

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Load Balancer Admin Guide" icon="settings" href="/services/load-balancer/admin-guide">
    Configure provider drivers, flavor profiles, quotas, and platform monitoring.
  </Card>

  <Card title="Xloud Key Manager" icon="lock" href="/services/key-manager">
    Store TLS certificates for HTTPS listener termination.
  </Card>
</CardGroup>


# XMS Admin Guide
Source: https://docs.xloud.tech/services/migration/admin-guide

Operator-facing reference for running the Xloud Migration Suite — architecture, prerequisites, credentials, network ports, storage back-ends, and capacity planning.

## Overview

The Admin Guide is for operators who install, configure, and run XMS
(Xloud Migration Suite) for their organization. It covers the pieces that sit
behind the end-user migration experience — service architecture, source
credential design, network access, storage back-ends, and capacity planning.

<Note>
  If you are looking for the workflow to actually migrate a workload, start in
  the [User Guide](/services/migration/user-guide).
</Note>

<Warning>
  XMS requires network access from the migration workers to both the source
  hypervisor API and the Xloud platform services. Review the
  [Network Ports](/services/migration/admin-guide/network-ports) page before
  onboarding a new source environment.
</Warning>

***

## Topics

<CardGroup>
  <Card title="Architecture" icon="network" href="/services/migration/admin-guide/architecture">
    How XMS is structured, how workload data flows, and how it integrates with
    the rest of the Xloud platform.
  </Card>

  <Card title="Prerequisites" icon="clipboard-check" href="/services/migration/admin-guide/prerequisites">
    Platform, project, quota, and identity prerequisites operators must meet
    before onboarding a new source.
  </Card>

  <Card title="Source Credentials" icon="key" href="/services/migration/admin-guide/source-credentials">
    How source credentials are stored, rotated, and scoped. Includes the
    recommended least-privilege role for VMware sources.
  </Card>

  <Card title="Network Ports" icon="plug" href="/services/migration/admin-guide/network-ports">
    Which ports must be reachable between XMS, source environments, Xloud
    services, and target guests.
  </Card>

  <Card title="Storage Back-ends" icon="database" href="/services/migration/admin-guide/storage-backends">
    How XMS writes migrated disk data into Xloud Block Storage and the
    supported back-end configurations.
  </Card>

  <Card title="Capacity Planning" icon="chart-bar" href="/services/migration/admin-guide/capacity-planning">
    How to size XMS for concurrent migrations, incremental sync throughput,
    and multi-wave campaigns.
  </Card>

  <Card title="Troubleshooting" icon="shield-alert" href="/services/migration/admin-guide/troubleshooting">
    Operator-side diagnostics — platform health, worker availability, and
    service-level recovery.
  </Card>
</CardGroup>

***

## Quick Links

<CardGroup>
  <Card title="User Guide" href="/services/migration/user-guide">
    End-user workflow for register, discover, preflight, migrate, and cut over
  </Card>

  <Card title="CLI Reference" href="/services/migration/cli-reference">
    Complete command-line reference for the xms CLI
  </Card>

  <Card title="Migration Overview" href="/services/migration">
    Service overview and capability summary
  </Card>
</CardGroup>


# XMS Architecture
Source: https://docs.xloud.tech/services/migration/admin-guide/architecture

How the Xloud Migration Suite is structured — control plane, migration workers, disk transport libraries, and integration with Xloud platform services.

## Overview

XMS is a service inside the Xloud platform. It has a control plane that
exposes the user-facing APIs and a pool of migration workers that execute the
actual disk transport and guest conversion work. Both run alongside the rest
of Xloud and integrate with the platform's identity, storage, compute, and
networking services.

***

## Component Map

```mermaid theme={null}
graph TB
    subgraph "Xloud Dashboard"
      UI["Migration Panel"]
    end
    subgraph "XMS Control Plane"
      API["XMS API"]
      ORCH["Job Orchestrator"]
      STATE[("Job State Store")]
    end
    subgraph "Migration Workers"
      W1["Worker 1"]
      W2["Worker 2"]
      WN["Worker N"]
    end
    subgraph "Source"
      VC["vCenter / ESXi"]
    end
    subgraph "Xloud Services"
      KS["Identity"]
      CI["Block Storage"]
      NO["Compute"]
      NE["Networking"]
    end

    UI --> API
    API --> ORCH
    ORCH <--> STATE
    ORCH --> W1
    ORCH --> W2
    ORCH --> WN
    W1 --> VC
    W2 --> VC
    WN --> VC
    W1 --> CI
    W2 --> CI
    WN --> CI
    API --> KS
    API --> CI
    API --> NO
    API --> NE
```

| Component            | Responsibility                                                                                                                 |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| **Migration Panel**  | User-facing UI in the Xloud Dashboard — sources, discovery, preflight, jobs, events                                            |
| **XMS API**          | REST API that the dashboard and CLI call. Validates requests, enforces project scoping, and hands jobs off to the orchestrator |
| **Job Orchestrator** | Assigns jobs to workers, tracks phase transitions, and publishes events                                                        |
| **Job State Store**  | Authoritative store of every job's state, progress, and event history                                                          |
| **Migration Worker** | Executes the disk transport and guest conversion pipeline for a single job                                                     |

***

## Workload Data Flow

### Cold Migration

```mermaid theme={null}
graph LR
    S["Source VM<br/>(powered off)"] -->|vSphere API| W["Migration Worker"]
    W -->|Xloud Block Storage API| V["Target Volume"]
    W -->|Xloud Compute API| I["Target Instance"]
    V --> I
```

1. Worker opens a disk transport session against the source via vSphere API
2. Worker creates an empty target volume through Xloud Block Storage
3. Disk data is streamed from the source and written into the target volume
4. Guest conversion runs against the target volume
5. The target instance is created and attached to the target volume

### Warm Migration

```mermaid theme={null}
graph LR
    S["Source VM<br/>(running)"] -->|vSphere API + CBT| W["Migration Worker"]
    W -->|Xloud Block Storage API| V["Target Volume"]
    W -.->|incremental syncs| V
    V --> I["Target Instance<br/>(at cutover)"]
```

1. Worker takes a baseline snapshot on the source
2. Full sync copies every block into the target volume
3. Between syncs, the job sits idle in the **Ready** state
4. Each incremental sync reads only the blocks that changed since the last
   baseline and writes them to the same offsets in the target volume
5. At cutover, a final incremental sync and guest conversion run before the
   target instance is launched

***

## Disk Transport Libraries

XMS uses the vSphere storage APIs and a disk transport library to read
source VM disks efficiently. The library version is selected to match the
source vSphere version and is an operator-managed component of the XMS
deployment.

<Tip>
  The transport library is the same code path for cold migrations, warm full
  syncs, and warm incremental syncs — only the set of blocks read differs
  between the three cases.
</Tip>

***

## Integration with Xloud Services

| Xloud Service     | Purpose                                                                                                                                           |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Identity**      | Project scoping, token validation, and role enforcement for every XMS API call                                                                    |
| **Block Storage** | Target volumes are created and written through the platform's block storage API, so they are indistinguishable from volumes created any other way |
| **Compute**       | The target instance is a normal Xloud compute instance — flavors, networking, and metadata all apply as usual                                     |
| **Networking**    | Target instance NICs are attached to target Xloud networks using the network mapping chosen at submit time                                        |

Because migrated volumes and instances are native Xloud objects, the standard
Xloud tooling — quotas, volume types, availability zones, security groups,
snapshots, backups — applies automatically.

***

## Multi-Source, Multi-Project

A single XMS deployment can:

* Register multiple source environments (one or more vCenters, standalone
  ESXi hosts)
* Run concurrent migration jobs across different sources
* Target different Xloud projects for different migrations

<Note>
  Project scoping is enforced at the API layer using the caller's identity
  token — users only see migrations for projects they are authorized to access.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Prerequisites" href="/services/migration/admin-guide/prerequisites">
    Platform, project, quota, and identity prerequisites
  </Card>

  <Card title="Network Ports" href="/services/migration/admin-guide/network-ports">
    Which ports must be reachable between XMS, sources, and targets
  </Card>

  <Card title="Capacity Planning" href="/services/migration/admin-guide/capacity-planning">
    Sizing XMS for concurrent migrations and multi-wave campaigns
  </Card>
</CardGroup>


# Capacity Planning
Source: https://docs.xloud.tech/services/migration/admin-guide/capacity-planning

Sizing XMS for concurrent migrations, incremental sync throughput, and multi-wave campaigns.

## Overview

Sizing XMS comes down to three questions: how many migrations do you want
to run concurrently, how much source churn will warm migrations have to
keep up with, and how long is your acceptable cutover window. This page
walks through those three questions and gives you a planning framework.

***

## Three Capacity Dimensions

<CardGroup>
  <Card title="Concurrent Migrations" icon="layers">
    How many migrations XMS can run at the same time. This is the primary
    dimension for campaigns with many small VMs.
  </Card>

  <Card title="Sustained Throughput" icon="gauge">
    How much aggregate bytes per second the migration workers can read from
    the source and write into the target. This drives how long the full sync
    phase takes.
  </Card>

  <Card title="Cutover Burst" icon="zap">
    How much short-term capacity is available for the final delta sync and
    guest conversion during a cutover. This drives how short the cutover
    window can be.
  </Card>
</CardGroup>

***

## Concurrent Migrations

Every running migration consumes:

* One disk transport session against the source
* One worker slot in the XMS control plane
* Block storage API throughput to write into the target volume
* Network bandwidth on the path between XMS and the source

Operators scale concurrent capacity by adding migration workers. The exact
ceiling depends on the deployment, but as a rule:

<Tip>
  Plan for a ceiling of concurrent migrations, not a ceiling of total
  migrations. A campaign of 200 VMs can run over several days in waves of
  10-20 concurrent migrations and finish comfortably, even on a modest
  XMS deployment.
</Tip>

### Wave Sizing

| Wave Size     | When to Use                                                                                     |
| ------------- | ----------------------------------------------------------------------------------------------- |
| **1-5 VMs**   | First wave of a campaign, high-risk workloads, or environments with restricted change windows   |
| **10-20 VMs** | Steady-state campaign waves for typical workloads                                               |
| **20-50 VMs** | High-throughput waves for small, similar VMs (for example, Windows workstation-class workloads) |
| **50+ VMs**   | Only with a dedicated XMS deployment sized for the campaign — talk to Xloud support first       |

***

## Full Sync Throughput

The full sync phase reads the entire source disk once and writes it into
the target volume. Throughput depends on:

* Source disk read speed
* Network path between XMS and the source
* Target block storage write speed
* Number of concurrent full syncs competing for the same resources

<Warning>
  The bottleneck is almost always the network path between XMS and the
  source, not the storage on either side. If full syncs are slower than
  expected, measure the raw path throughput first.
</Warning>

### Example Throughput Planning

For a wave of 10 VMs totaling 2 TB:

| Path Throughput        | Expected Full Sync Duration |
| ---------------------- | --------------------------- |
| **100 MB/s aggregate** | \~6 hours                   |
| **500 MB/s aggregate** | \~1 hour 10 minutes         |
| **1 GB/s aggregate**   | \~35 minutes                |

Aggregate throughput is the sum across all concurrent migrations, so 10
migrations at 100 MB/s each gives 1 GB/s aggregate.

***

## Warm Migration Cadence

Warm migrations trade ongoing sync bandwidth for a short cutover window.
The cadence you choose directly controls how much data a final delta has
to transfer.

| Cadence              | Bytes Per Sync (typical) | Good For                                |
| -------------------- | ------------------------ | --------------------------------------- |
| **Every 15 minutes** | Small, hundreds of MB    | Small, steady-state workloads           |
| **Hourly**           | Low GB                   | Medium workloads with predictable churn |
| **Daily**            | Tens of GB               | Large workloads or low-churn archives   |

<Tip>
  Aggressive cadences mean more frequent CBT snapshots on the source.
  Watch the source host load and the CBT change map size if you see
  slowdown on the source side.
</Tip>

***

## Cutover Window Planning

Target cutover window breaks down roughly as:

| Phase                 | Typical Duration            |
| --------------------- | --------------------------- |
| **Final delta sync**  | Seconds to a few minutes    |
| **Source power off**  | Seconds to a minute         |
| **Guest fixes**       | 30 seconds to a few minutes |
| **Finalize and boot** | Under a minute              |

For a small, healthy warm migration with low lag, the total cutover window
is typically under 5 minutes. For large or churny workloads, plan 10-20
minutes and trigger a manual **Sync Now** right before cutover to keep the
final delta small.

***

## Multi-Wave Campaign Framework

<Steps>
  <Step title="Classify workloads into waves" icon="list">
    Group source VMs by risk, churn, and cutover tolerance:

    * **Wave 0** — cold-migration lab VMs, used to validate the pipeline
    * **Wave 1** — cold migrations for workloads that tolerate downtime
    * **Wave 2** — warm migrations for production workloads with hourly sync
    * **Wave 3** — warm migrations for the highest-churn workloads with
      15-minute sync
  </Step>

  <Step title="Size XMS for the largest concurrent wave" icon="gauge">
    Count the largest number of migrations you want to run concurrently
    across all waves that overlap in time. Size workers and network path
    for that number.
  </Step>

  <Step title="Validate on Wave 0 before scaling" icon="flask-conical">
    Run Wave 0 end-to-end and measure real throughput, real cutover window,
    and real post-migration success rate. Adjust wave sizing before
    Wave 1 begins.
  </Step>

  <Step title="Schedule waves around source and target capacity" icon="calendar">
    Warm migrations occupy target volume footprint for the full sync window
    plus the cutover wait. Schedule overlapping waves so the cumulative
    target footprint fits the project quota.
  </Step>

  <Step title="Monitor and retune between waves" icon="activity">
    Use the Migration panel and platform monitoring to review wave metrics
    between waves. Retune cadence, concurrency, and network path if any
    dimension is bottlenecked.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Architecture" href="/services/migration/admin-guide/architecture">
    How XMS components are structured
  </Card>

  <Card title="Storage Back-ends" href="/services/migration/admin-guide/storage-backends">
    How XMS writes migrated data into block storage
  </Card>

  <Card title="Troubleshooting" href="/services/migration/admin-guide/troubleshooting">
    Operator-side diagnostics and recovery
  </Card>
</CardGroup>


# Network Ports
Source: https://docs.xloud.tech/services/migration/admin-guide/network-ports

Network paths and ports that must be reachable between XMS, source environments, Xloud services, and target guests.

## Overview

XMS speaks to a source hypervisor API, the platform storage and compute
APIs, and the running target instance at different phases of a migration.
Each conversation uses a specific port and needs a network path that can
sustain the traffic. This page lists the ports and paths operators need to
open.

***

## Quick Reference

| Source          | Destination         | Port | Purpose                                    | Direction |
| --------------- | ------------------- | ---- | ------------------------------------------ | --------- |
| XMS             | vCenter or ESXi     | 443  | vSphere API and disk transport negotiation | Egress    |
| XMS             | ESXi host           | 902  | Disk transport data stream                 | Egress    |
| XMS             | Xloud Identity      | 5000 | Token validation                           | Egress    |
| XMS             | Xloud Block Storage | 8776 | Volume create and write                    | Egress    |
| XMS             | Xloud Compute       | 8774 | Instance create and attach                 | Egress    |
| XMS             | Xloud Networking    | 9696 | Network mapping validation                 | Egress    |
| Dashboard / CLI | XMS API             | 443  | User-facing API                            | Egress    |

<Note>
  Port numbers shown are the defaults. Your deployment may publish XMS and
  the platform services behind a virtual IP or a TLS-terminating load
  balancer — adjust accordingly.
</Note>

***

## XMS ↔ Source

### vSphere API (port 443)

XMS opens a vSphere API session against the vCenter or ESXi endpoint to:

* Authenticate the stored credential
* Walk the inventory during discovery
* Read VM configuration, disk metadata, and CBT state during preflight
* Negotiate a disk transport session before each read

This is a TCP-over-TLS conversation and can go through an HTTPS-aware
firewall or proxy.

### Disk Transport Data Stream (port 902)

After negotiation, the actual disk data is streamed between XMS and the ESXi
host that owns the source VM's datastore. This connection is NFC
(Network File Copy) on port 902 by default.

<Warning>
  For a vCenter-managed source, open port 902 to every ESXi host that may
  own the source VM — not just to the vCenter. VMs can move between hosts,
  so the transport session may hit any host in the cluster.
</Warning>

### Bandwidth and Latency

* The disk transport session is bandwidth-bound — a full sync of a large VM
  will saturate the available path during the export phase
* Incremental warm syncs read only dirty blocks, so the steady-state
  bandwidth cost is proportional to source churn, not total disk size
* Round-trip latency over 50 ms noticeably slows the full sync phase — keep
  XMS and the source on the same LAN or metro link where possible

***

## XMS ↔ Xloud Services

XMS acts as an Xloud tenant on behalf of every migration. It makes standard
API calls against:

* **Identity** for token validation on every inbound and outbound request
* **Block Storage** to create target volumes and write disk data into them
* **Compute** to create the target instance and attach the volume
* **Networking** to validate the network mapping chosen at submit time

These calls happen over the platform's internal service network. Operators
who run XMS alongside the rest of the Xloud platform typically do not need
to make any new firewall changes for this path.

***

## Operator Access

Operators reach the XMS API through the platform's public endpoint — the
same endpoint used for the rest of the Xloud Dashboard and CLI. The
Migration panel is served by the existing Dashboard web service, and the
`xms` CLI calls the XMS API over the public endpoint.

***

## Egress From the Target Instance

After a migration completes, the target Xloud instance is a normal compute
instance. XMS does not inject any agent or open any extra port on the target
guest — standard Xloud network security group rules apply for egress,
ingress, and management access.

***

## Firewall Change Checklist

<Steps>
  <Step title="Open egress to vSphere API" icon="plug">
    Allow TCP/443 from XMS to the vCenter or ESXi endpoint.
  </Step>

  <Step title="Open egress to ESXi hosts for disk transport" icon="hard-drive">
    Allow TCP/902 from XMS to every ESXi host that may own source VM storage.
  </Step>

  <Step title="Verify internal Xloud service traffic" icon="check">
    No change required — XMS communicates with Identity, Block Storage,
    Compute, and Networking over the existing platform service network.
  </Step>

  <Step title="Verify target instance traffic" icon="shield">
    No change required — standard Xloud security group rules apply to the
    migrated instance.
  </Step>
</Steps>

<Tip>
  For vCenter-managed sources, ask the vSphere admin which ESXi hosts currently
  own the source VM's datastores. Open port 902 to all of them — VMs can
  vMotion between hosts, so the transport session may target any host.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Prerequisites" href="/services/migration/admin-guide/prerequisites">
    Platform and project prerequisites for a new source
  </Card>

  <Card title="Source Credentials" href="/services/migration/admin-guide/source-credentials">
    Build the service account and role
  </Card>

  <Card title="Capacity Planning" href="/services/migration/admin-guide/capacity-planning">
    Size XMS for concurrent migrations and multi-wave campaigns
  </Card>
</CardGroup>


# Prerequisites
Source: https://docs.xloud.tech/services/migration/admin-guide/prerequisites

Platform, identity, project, quota, and source-side prerequisites operators must satisfy before onboarding a VMware source to XMS.

## Overview

Before a new source can be registered or a new migration can run, the
operator must satisfy a small number of platform-side, project-side, and
source-side prerequisites. This page lists them in the order you should check
them.

***

## Platform Prerequisites

<Steps>
  <Step title="XMS is enabled on the Xloud platform" icon="check">
    The Migration panel must be visible in the Xloud Dashboard and the `xms`
    CLI subcommand must be installed on operator workstations. If neither is
    available, XMS is not enabled on the deployment — contact Xloud support.
  </Step>

  <Step title="Identity service is healthy" icon="key">
    XMS validates every API call against the platform identity service. If
    identity is unhealthy, source registration and job submission will fail.
    Confirm the platform health dashboard reports identity as reachable.
  </Step>

  <Step title="Block Storage and Compute are healthy" icon="database">
    XMS writes migrated data through block storage and launches target
    instances through compute. Both must be healthy before any migration can
    run.
  </Step>
</Steps>

***

## Project Prerequisites

Each target Xloud project that will receive migrated workloads needs:

| Requirement                                          | Why                                                                                                   |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Project exists and is active**                     | XMS writes volumes and instances into an existing project — it does not create new projects           |
| **Quota for compute, memory, and volume footprint**  | Quota is checked at submit time against the flavor and volume type selected                           |
| **Volume type that matches the chosen storage tier** | The operator must publish at least one volume type in the project; users pick one per migration       |
| **Target networks and subnets**                      | Every source NIC must map to a target network — the target networks must already exist in the project |
| **User role with migration permission**              | Users submitting migrations need a role that allows them to use the Migration panel in that project   |

<Tip>
  For multi-wave campaigns, create a dedicated target project per wave. This
  keeps quota accounting simple and isolates waves from each other.
</Tip>

***

## Source Environment Prerequisites

### VMware vCenter

| Requirement                            | Detail                                                                                                                               |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| **Supported vSphere version**          | vSphere 6.x, 7.x, or 8.x. Newer versions are supported as the transport library is refreshed                                         |
| **Reachable endpoint**                 | XMS must be able to reach the vCenter endpoint on the configured port (default 443)                                                  |
| **Service account**                    | A dedicated vSphere account for XMS — see [Source Credentials](/services/migration/admin-guide/source-credentials)                   |
| **Dedicated role at datacenter scope** | Create a vSphere role with the permissions listed in Source Credentials and assign it to the service account at the datacenter level |

### Standalone ESXi

Standalone ESXi is supported for lab, edge, and small-site migrations. The
same transport library and pipeline apply — the only differences are:

* Endpoint is the host address instead of a vCenter
* Username is typically `root` or a local ESXi user
* Datacenter is auto-detected as `ha-datacenter`

### Guest-Side Requirements

| Requirement                              | Why                                                                                                                          |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **VMware Tools installed (recommended)** | Exposes guest OS, hostname, and IP to discovery. Not a migration blocker when absent                                         |
| **Supported guest OS**                   | Guest must be in the Xloud guest catalog — see [Preflight Assessment](/services/migration/user-guide/preflight)              |
| **Clean boot loader**                    | XMS repairs drivers and the boot loader for platform-specific changes, but it does not fix a pre-existing broken boot loader |
| **CBT enabled (warm only)**              | Warm migration requires Changed Block Tracking on the source                                                                 |

***

## Operator Access

The operator installing or maintaining XMS typically needs:

* Administrative role on the Xloud platform (for installing, updating, and
  scaling XMS)
* Read or admin role on every target project that will receive migrations
* Credentials to register source environments (either shared service
  accounts or per-source accounts)
* Access to platform monitoring to track XMS health

***

## Pre-Onboarding Checklist

Use this checklist before onboarding a new source:

<AccordionGroup>
  <Accordion title="Platform health" icon="activity">
    * [ ] Identity service reachable and healthy
    * [ ] Block storage reachable and healthy
    * [ ] Compute reachable and healthy
    * [ ] XMS visible in the Xloud Dashboard
    * [ ] `xms` CLI installed and authenticated
  </Accordion>

  <Accordion title="Target project" icon="folder">
    * [ ] Target project exists in the correct domain
    * [ ] Compute quota covers the expected wave footprint
    * [ ] Volume quota covers the expected wave footprint (+20% buffer)
    * [ ] Volume type published for the intended storage tier
    * [ ] Target networks and subnets exist
    * [ ] Users who will submit migrations have the right role
  </Accordion>

  <Accordion title="Source environment" icon="server">
    * [ ] vSphere endpoint reachable from XMS
    * [ ] Service account created with the recommended role
    * [ ] Role assigned at the datacenter scope
    * [ ] TLS fingerprint or CA chain available if TLS verification is required
    * [ ] CBT enabled on every VM in the warm migration wave
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Source Credentials" href="/services/migration/admin-guide/source-credentials">
    Build the service account and role for XMS
  </Card>

  <Card title="Network Ports" href="/services/migration/admin-guide/network-ports">
    Open the right ports between XMS, source, and target
  </Card>

  <Card title="Register Source" href="/services/migration/user-guide/register-source">
    Add the source to XMS once prerequisites are met
  </Card>
</CardGroup>


# Source Credentials
Source: https://docs.xloud.tech/services/migration/admin-guide/source-credentials

Design, store, rotate, and scope credentials for VMware sources registered to XMS — including the recommended least-privilege vSphere role.

## Overview

Every source environment registered to XMS has a stored credential that XMS
uses for discovery, preflight, and migration. The credential is shared across
every job against that source. This page covers how operators should design
those credentials — scoping, least privilege, storage, and rotation.

***

## Principles

<CardGroup>
  <Card title="One service account per source" icon="user-cog">
    Do not reuse a personal operator account for XMS. Create a dedicated
    service account in the source directory service and use it only for
    XMS — this keeps the audit trail clean and lets you rotate without
    impacting humans.
  </Card>

  <Card title="Scope to the datacenter you migrate" icon="target">
    Assign the XMS role at the datacenter you are actually migrating — not
    at the vCenter root. This keeps blast radius small and prevents XMS
    from accidentally touching inventory in datacenters it should not see.
  </Card>

  <Card title="Separate read-only vs migration scope" icon="split">
    For cautious rollouts, register the source with a read-only account for
    discovery and preflight first. Swap to the migration-capable account
    only when you are ready to run migrations.
  </Card>

  <Card title="Rotate credentials regularly" icon="refresh-cw">
    XMS stores source credentials securely and allows in-place rotation from
    the Dashboard or CLI. Build credential rotation into your standard
    operational cadence.
  </Card>
</CardGroup>

***

## Recommended vSphere Role

Create a dedicated role on the source vCenter and assign it to the XMS
service account at the datacenter scope.

| Category                         | Privileges                                     | Used For                                                                         |
| -------------------------------- | ---------------------------------------------- | -------------------------------------------------------------------------------- |
| **System**                       | System.Anonymous, System.Read, System.View     | Session establishment, inventory read                                            |
| **Virtual Machine Config**       | DiskLease, ChangeTracking, Settings            | Enable CBT, read disk configuration                                              |
| **Virtual Machine Interact**     | PowerOff, PowerOn, Reset                       | Cold migration source power control                                              |
| **Virtual Machine State**        | CreateSnapshot, RemoveSnapshot, RevertSnapshot | Warm migration CBT anchor snapshots                                              |
| **Virtual Machine Provisioning** | DiskRandomAccess, DiskRandomRead               | Disk export over vSphere API                                                     |
| **Resource**                     | AssignVMToPool                                 | Optional — only required for cross-cluster source preparation                    |
| **Global**                       | DisableMethods, EnableMethods                  | Optional — only required to guard against concurrent modification during cutover |

<Tip>
  For discovery and preflight only, a read-only account with the **System**
  privileges above is sufficient. Add the Virtual Machine privileges only
  when you are ready to migrate.
</Tip>

***

## Credential Storage

XMS stores source credentials in the platform secret store. Credentials are:

* Encrypted at rest using the platform-managed encryption key
* Never displayed after save — the Dashboard shows a placeholder on edit
* Accessible only to the XMS control plane, which uses them to open vSphere
  API sessions on behalf of jobs

Operators cannot extract stored credentials in clear text from the platform.

***

## Credential Rotation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the source details drawer" icon="folder">
        Navigate to **Migration → Environments**, select the source, and
        click **Edit**.
      </Step>

      <Step title="Update credentials" icon="key">
        Enter the new username or password (or both) and click **Test
        Connection** to confirm the new credentials work against the source.
      </Step>

      <Step title="Save" icon="save">
        Click **Save**. XMS re-encrypts the credential and closes any open
        session that was using the old credential.

        <Check>Next discovery run uses the new credential.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash theme={null}
    # Rotate password only — read new password from stdin
    xms source update prod-vcenter --password-stdin

    # Rotate both username and password
    xms source update prod-vcenter \
      --username 'xms-service@vsphere.local' \
      --password-stdin

    # Test the updated connection without registering anything new
    xms source test prod-vcenter
    ```
  </Tab>
</Tabs>

<Warning>
  Changing the username or password of an active source invalidates any
  in-flight discovery session. Re-run discovery after rotation so the
  inventory cache refreshes against the new credential.
</Warning>

***

## Deleting a Source

Delete a source only when no active migration jobs reference it. XMS blocks
deletion of a source that is the parent of any running job. After all jobs
complete or fail, you can delete the source from the Dashboard or CLI, which
also removes the stored credential.

***

## Audit Trail

Source credential creation, update, and deletion events are captured in the
platform audit log. Operators investigating an incident can trace which
identity made credential changes and when, using the standard Xloud audit
tooling.

***

## Next Steps

<CardGroup>
  <Card title="Prerequisites" href="/services/migration/admin-guide/prerequisites">
    Platform and project prerequisites for onboarding a source
  </Card>

  <Card title="Network Ports" href="/services/migration/admin-guide/network-ports">
    Which ports must be reachable between XMS and source
  </Card>

  <Card title="Register Source" href="/services/migration/user-guide/register-source">
    End-user steps for registering a source once credentials are ready
  </Card>
</CardGroup>


# Storage Back-ends
Source: https://docs.xloud.tech/services/migration/admin-guide/storage-backends

How XMS writes migrated disk data into Xloud Block Storage and the supported back-end configurations.

## Overview

Every migration writes source disk data into Xloud Block Storage. XMS does
not manage storage directly — it uses the platform's block storage API, so
any back-end that Xloud Block Storage supports is usable as a migration
target. This page covers how the write path works and how to choose a
target back-end per migration.

***

## Write Path

At a high level, the write path is the same for cold and warm migrations:

```mermaid theme={null}
graph LR
    S[Source VM disk] --> X[Migration Worker]
    X -->|Block Storage API| BS[Xloud Block Storage]
    BS --> V[Target Volume]
```

1. The migration worker reads source blocks through the disk transport
   session
2. It creates an empty volume through the block storage API, pinned to the
   volume type the user selected at submit time
3. It writes the disk data into the new volume, and for warm migrations,
   updates the same offsets on every incremental sync
4. When guest conversion completes, the volume is marked bootable and
   attached to the target instance

Because the target volume is created through the block storage API, it
behaves exactly like any other volume — it counts against the project's
volume quota, respects the volume type's QoS settings, and can be
snapshotted, backed up, or restored by the standard Xloud tooling.

***

## Volume Type Selection

The user picks a volume type at migration submit time. The volume type
determines:

* Which back-end the target volume lives on
* What performance tier it gets (QoS, IOPS limits)
* Which availability zone it is pinned to, if the type is zone-pinned
* Which encryption key is used, if the type has encryption enabled

<Tip>
  Operators should publish a small, well-named set of volume types — for
  example, `standard-ssd`, `performance-nvme`, and `archive-hdd`. This gives
  migration users a clear choice without overwhelming them with back-end
  details.
</Tip>

***

## Multi-Back-end Clusters

If your Xloud platform has more than one storage back-end, you can direct
migrations to any of them by publishing a volume type per back-end. Common
layouts:

<CardGroup>
  <Card title="Single-Back-end" icon="hard-drive">
    One storage back-end serves every migration. The operator publishes a
    default volume type and users always pick it. Simplest to operate.
  </Card>

  <Card title="Tiered Multi-Back-end" icon="layers">
    Several back-ends at different performance and cost tiers. The operator
    publishes one volume type per tier, and users pick the tier that matches
    the workload. Common in mixed NVMe and HDD clusters.
  </Card>

  <Card title="Zone-Pinned" icon="map-pin">
    Availability-zone-pinned volume types let users pin a migration to a
    specific zone — useful for application-level affinity with existing
    Xloud workloads.
  </Card>

  <Card title="Encrypted" icon="lock">
    Encryption-enabled volume types automatically encrypt migrated volumes
    at rest using the platform-managed key. No extra action needed at
    submit time.
  </Card>
</CardGroup>

***

## Quota and Sizing

Every migration consumes quota against the target project:

| Resource                       | Impact                                                                           |
| ------------------------------ | -------------------------------------------------------------------------------- |
| **Volume count**               | One volume per source disk — multi-disk VMs use multiple volumes                 |
| **Volume size**                | Sum of source disk sizes, rounded up to the volume type granularity              |
| **Volume type-specific quota** | If your platform enforces per-type quota, make sure the chosen type has headroom |

<Warning>
  The full volume footprint is reserved at **submit time**, not at cutover.
  For warm migrations, the target volume exists for the duration of the
  sync window — plan quota accordingly.
</Warning>

***

## Incremental Sync Behavior

Warm migrations write every incremental sync to the same target volume:

* The target volume is created once, during the full sync phase
* Every incremental sync reads changed blocks from the source and writes
  them to the same offsets on the target volume
* There is no intermediate staging area — the target volume is always a
  byte-for-byte replica of the source at the time of the last sync

Because there is no staging, the volume count and volume footprint are
fixed for the life of a warm migration. Incremental syncs do not create
new volumes or new snapshots.

***

## Snapshot and Backup

Because the target volume is a native Xloud block storage volume, you can:

* Snapshot it through the standard snapshot API once the migration is done
* Back it up through Xloud Block Storage backup if your deployment supports
  cross-backend backup
* Use it as the source for Xloud Disaster Recovery protection plans

Most operators configure a post-migration snapshot automatically to give
the workload owner a rollback point immediately after cutover.

***

## Next Steps

<CardGroup>
  <Card title="Capacity Planning" href="/services/migration/admin-guide/capacity-planning">
    Size XMS for concurrent migrations and multi-wave campaigns
  </Card>

  <Card title="Block Storage" href="/services/storage/index">
    Xloud Block Storage user guide and operator reference
  </Card>

  <Card title="Disaster Recovery" href="/services/disaster-recovery">
    Add migrated volumes to disaster recovery protection plans
  </Card>
</CardGroup>


# Operator Troubleshooting
Source: https://docs.xloud.tech/services/migration/admin-guide/troubleshooting

Operator-side diagnostics for XMS — platform health, worker availability, and service-level recovery.

## Overview

This page covers the operator-facing failure modes — the ones you investigate
when multiple users report broken migrations or the Migration panel itself
is unresponsive. For end-user workflow failures (single-job symptoms), see
the [User Troubleshooting](/services/migration/user-guide/troubleshooting)
page instead.

***

## Quick Health Check

Work through these checks in order when an operator incident is reported:

<Steps>
  <Step title="Platform identity" icon="key">
    Confirm the Xloud Identity service is reachable and responding to token
    validation. XMS cannot accept any API call if identity is down.
  </Step>

  <Step title="Platform block storage and compute" icon="database">
    Confirm block storage and compute are healthy in the platform health
    dashboard. A migration cannot complete without both.
  </Step>

  <Step title="XMS API responsiveness" icon="activity">
    Call the XMS API from an operator workstation or CLI. A non-responding
    API is the single most visible symptom and usually points at control
    plane or upstream identity issues.
  </Step>

  <Step title="Worker availability" icon="server">
    Check the XMS control plane for worker availability. If zero workers are
    available, new jobs queue indefinitely.
  </Step>

  <Step title="Source reachability from XMS" icon="network">
    Confirm the XMS deployment can still reach every registered source
    endpoint. A network path change on the operator side is a common cause
    of widespread job failures.
  </Step>
</Steps>

***

## Common Operator Incidents

<AccordionGroup>
  <Accordion title="All new jobs are stuck in Queued" icon="clock">
    **Symptom**: Users submit migrations and they never leave **Queued**.

    **Cause**: No workers are available, either because workers are down or
    because an existing set of jobs is holding every worker slot.

    **Fix**:

    * Confirm worker health — are any workers in a failed state?
    * Check in-flight job count. If the job count equals the worker count,
      this is expected — jobs are waiting their turn
    * If workers are down, restart the worker pool and monitor the
      orchestrator
  </Accordion>

  <Accordion title="All sources show Disconnected" icon="plug">
    **Symptom**: Every registered source reports **Disconnected** in the
    Dashboard at the same time.

    **Cause**: A network change on the XMS side broke the outbound path to
    all sources, or the platform's outbound DNS stopped resolving.

    **Fix**:

    * Confirm DNS resolution from XMS for the source hostnames
    * Confirm outbound TCP/443 reachability from XMS to the source endpoints
    * Check for any new egress policy on the operator-managed firewall
  </Accordion>

  <Accordion title="Jobs fail in Write Volume phase" icon="database">
    **Symptom**: Multiple jobs fail during the write volume phase even
    though block storage is reported healthy.

    **Cause**: Block storage API is responsive but the target back-end is
    running out of capacity, or a specific volume type has hit its quota.

    **Fix**:

    * Review block storage capacity and volume type quota in the platform
      health dashboard
    * Confirm the target project has volume footprint headroom
    * Rebalance the campaign to a different volume type while capacity is
      provisioned
  </Accordion>

  <Accordion title="Migration panel is slow or times out" icon="gauge">
    **Symptom**: Dashboard users report the Migration panel is slow to load
    or returns a timeout.

    **Cause**: The XMS control plane is under load, the job state store is
    slow, or the upstream identity service is slow to validate tokens.

    **Fix**:

    * Measure API latency from the operator side
    * Check the job state store health
    * Check identity service latency — a slow identity service impacts
      every Xloud UI, not just XMS
  </Accordion>

  <Accordion title="Sources refresh slowly or partially" icon="search">
    **Symptom**: Discovery runs take significantly longer than previous
    runs, or return partial results.

    **Cause**: The source environment is under load, or CBT state on the
    source is being rebuilt after a host or datastore change.

    **Fix**:

    * Coordinate with the source environment owner — they may be doing
      maintenance
    * Re-run discovery during a lower-load window on the source side
  </Accordion>
</AccordionGroup>

***

## Operator Diagnostics to Collect

When escalating an incident to Xloud support, collect:

* Platform health dashboard screenshot at the time of the incident
* XMS worker availability snapshot
* A representative failed job ID and its full event stream
* Source environment type, version, and the specific endpoint affected
* Any platform-side changes in the window before the incident started
* Audit log entries for source and job operations in the incident window

Attach all of the above to the incident ticket before escalating.

***

## Recovery Operations

### Cancel a Stuck Job

Jobs that are stuck in a phase can be cancelled from the Dashboard or CLI.
Cancellation stops the phase cleanly, releases any disk transport session
and worker slot, and marks the job as **Cancelled**. Source and target
state are left unchanged — the user can re-submit or clean up manually.

### Retry After a Transient Failure

Failed jobs are not automatically retried. If a job failed due to a
transient cause (network blip, momentary source unavailability), the user
can re-submit it from the Dashboard. For warm migrations that failed after
the full sync completed, re-submission restarts from scratch — coordinate
with the user on whether a cold fallback is faster.

### Scale Worker Pool

If worker availability is the bottleneck, the operator can scale the XMS
worker pool. The exact steps depend on your deployment — typically it is a
configuration change followed by a controlled restart of the control plane.

***

## Next Steps

<CardGroup>
  <Card title="User Troubleshooting" href="/services/migration/user-guide/troubleshooting">
    Single-job failure modes and fixes
  </Card>

  <Card title="Capacity Planning" href="/services/migration/admin-guide/capacity-planning">
    Size XMS to avoid capacity-driven incidents
  </Card>

  <Card title="Architecture" href="/services/migration/admin-guide/architecture">
    Component map and workload data flow
  </Card>
</CardGroup>


# CLI Reference
Source: https://docs.xloud.tech/services/migration/cli-reference

Complete command-line reference for the xms CLI — sources, discovery, preflight, migration jobs, cutover, and events.

## Overview

The `xms` CLI is a command-line client for the Xloud Migration Suite. It
mirrors the Dashboard Migration panel: every action you can take in the UI
is available as a CLI command, and every object you can inspect in the UI is
available as a structured CLI response.

Use the CLI when you want to script multi-wave campaigns, run migrations
from CI, or integrate XMS with your own automation.

<Note>
  The CLI authenticates against the platform identity service with the same
  credentials you use for the Xloud Dashboard. Project scope is resolved
  from the authenticated token.
</Note>

***

## Global Options

These options apply to every subcommand.

| Option                            | Description                                             |
| --------------------------------- | ------------------------------------------------------- |
| `--os-cloud <name>`               | Select a named cloud from your `clouds.yaml`            |
| `--os-auth-url <url>`             | Override the identity endpoint                          |
| `--os-project-name <name>`        | Set the project scope                                   |
| `--os-project-domain-name <name>` | Set the project domain                                  |
| `--os-username <name>`            | Authenticated user                                      |
| `--os-password <password>`        | Password (prefer `--password-stdin` or env vars)        |
| `-f, --format <fmt>`              | Output format: `table` (default), `json`, `yaml`, `csv` |
| `-q, --quiet`                     | Suppress non-essential output                           |
| `-v, --verbose`                   | Show request and response detail                        |
| `--no-color`                      | Disable ANSI colors in table output                     |

<Tip>
  Store credentials in `clouds.yaml` or environment variables
  (`OS_*`) and reference the cloud with `--os-cloud`. This keeps secrets
  out of shell history.
</Tip>

***

## Sources

Manage registered migration source environments.

### xms source create

Register a new VMware source.

```bash theme={null}
xms source create \
  --name prod-vcenter \
  --platform vmware \
  --host vcenter.example.com \
  --port 443 \
  --username 'administrator@vsphere.local' \
  --password-stdin \
  --verify-ssl \
  --datacenter DC-East
```

| Flag                               | Description                                             |
| ---------------------------------- | ------------------------------------------------------- |
| `--name`                           | Friendly source name, unique per project                |
| `--platform`                       | Source platform — `vmware` for vSphere                  |
| `--host`                           | vCenter or ESXi hostname or IP                          |
| `--port`                           | vSphere API port (default 443)                          |
| `--username`                       | Source service account                                  |
| `--password-stdin`                 | Read password from stdin (preferred)                    |
| `--password`                       | Inline password — discouraged, leaves secret in history |
| `--verify-ssl` / `--no-verify-ssl` | TLS verification toggle                                 |
| `--datacenter`                     | Optional — pin discovery to a single datacenter         |

### xms source list

List registered sources.

```bash theme={null}
xms source list
xms source list -f json
```

### xms source show

Show a single source including status and detected vSphere version.

```bash theme={null}
xms source show prod-vcenter
```

### xms source update

Update fields on an existing source. Only the flags you pass are applied.

```bash theme={null}
xms source update prod-vcenter --password-stdin
xms source update prod-vcenter --host vcenter-new.example.com
xms source update prod-vcenter --verify-ssl
```

### xms source test

Re-run the connection test without changing anything.

```bash theme={null}
xms source test prod-vcenter
```

### xms source delete

Delete a source. Fails if any active job references it.

```bash theme={null}
xms source delete prod-vcenter
```

***

## Discovery

Inventory VMs and disks from a registered source.

### xms discovery start

Kick off a discovery scan.

```bash theme={null}
# Full inventory
xms discovery start --source prod-vcenter

# Scope to a datacenter
xms discovery start --source prod-vcenter --datacenter DC-East

# Scope to a folder or cluster
xms discovery start --source prod-vcenter --folder "Production/Web"
xms discovery start --source prod-vcenter --cluster Cluster-A
```

### xms discovery status

Report the current scan state.

```bash theme={null}
xms discovery status --source prod-vcenter
xms discovery status --source prod-vcenter --follow
```

`--follow` streams progress until the scan completes.

### xms discovery list

List every discovered workload for a source.

```bash theme={null}
xms discovery list --source prod-vcenter
xms discovery list --source prod-vcenter --filter 'power_state=poweredOn'
xms discovery list --source prod-vcenter --filter 'os=windows' -f json
```

### xms discovery show

Show the full discovery record for a single VM.

```bash theme={null}
xms discovery show --source prod-vcenter --vm win-ser-2022
```

***

## Preflight

Run compatibility checks against discovered workloads.

### xms preflight run

Run preflight against one or more VMs.

```bash theme={null}
# Single VM
xms preflight run --source prod-vcenter --vm win-ser-2022

# Multiple VMs from a file (one name per line)
xms preflight run --source prod-vcenter --from-file wave-1.txt

# Entire discovered inventory
xms preflight run --source prod-vcenter --all
```

### xms preflight show

Fetch the latest preflight verdict for a VM.

```bash theme={null}
xms preflight show --source prod-vcenter --vm win-ser-2022
xms preflight show --source prod-vcenter --vm win-ser-2022 -f json
```

### xms preflight list

List preflight verdicts across a source.

```bash theme={null}
xms preflight list --source prod-vcenter
xms preflight list --source prod-vcenter --filter 'verdict=block'
```

### xms preflight enable-cbt

Enable Changed Block Tracking on a source VM through the vSphere API.

```bash theme={null}
xms preflight enable-cbt --source prod-vcenter --vm db-prod-01
```

***

## Migrations

Submit, monitor, and manage migration jobs.

### xms migration submit

Submit a new migration job.

```bash theme={null}
# Cold migration
xms migration submit \
  --source prod-vcenter \
  --vm win-ser-2022 \
  --kind cold \
  --target-project infra \
  --flavor m1.large \
  --volume-type ssd \
  --availability-zone az-1 \
  --network-map 'VM Network=internal-net'

# Warm migration with a 1-hour incremental cadence
xms migration submit \
  --source prod-vcenter \
  --vm db-prod-01 \
  --kind warm \
  --target-project infra \
  --flavor m1.xlarge \
  --volume-type nvme \
  --network-map 'VM Network=internal-net' \
  --sync-cadence 1h
```

| Flag                  | Description                                          |
| --------------------- | ---------------------------------------------------- |
| `--source`            | Registered source name                               |
| `--vm`                | Source VM name                                       |
| `--kind`              | `cold` or `warm`                                     |
| `--target-project`    | Target Xloud project                                 |
| `--target-name`       | Optional — instance name in the target project       |
| `--flavor`            | Xloud flavor for the target instance                 |
| `--volume-type`       | Target volume type                                   |
| `--availability-zone` | Optional — pin the target to a zone                  |
| `--network-map`       | Source to target network mapping, can be repeated    |
| `--preserve-mac`      | Attempt to preserve the source MAC on the target NIC |
| `--sync-cadence`      | Warm only — `manual`, `15m`, `1h`, `24h`             |

### xms migration list

List migration jobs in the current project.

```bash theme={null}
xms migration list
xms migration list --filter 'status=ready'
xms migration list --filter 'kind=warm,status=syncing'
xms migration list -f json
```

### xms migration show

Show the full state of a single migration job.

```bash theme={null}
xms migration show <job-id>
xms migration show <job-id> -f json
```

### xms migration events

Stream the event history for a job.

```bash theme={null}
xms migration events --job <job-id>
xms migration events --job <job-id> --follow
xms migration events --job <job-id> --since 10m
```

### xms migration sync

Trigger an immediate incremental sync on a warm migration that is in the
**Ready** state.

```bash theme={null}
xms migration sync --job <job-id>
```

### xms migration pause

Pause scheduled incrementals without losing progress. Only applies to warm
migrations.

```bash theme={null}
xms migration pause --job <job-id>
```

### xms migration resume

Resume a paused warm migration.

```bash theme={null}
xms migration resume --job <job-id>
```

### xms migration cutover

Trigger cutover on a warm migration.

```bash theme={null}
xms migration cutover --job <job-id>
```

### xms migration cancel

Cancel a job. Releases any disk transport session and worker slot. Source
and target state are left unchanged.

```bash theme={null}
xms migration cancel --job <job-id>
```

***

## Scripting Patterns

### Wave Preflight and Submit

Run preflight on a wave file, then submit cold migrations for everything
that passes:

```bash theme={null}
SOURCE=prod-vcenter
WAVE=wave-1.txt

# Preflight the whole wave
xms preflight run --source $SOURCE --from-file $WAVE

# Submit cold migrations for Pass results
xms preflight list --source $SOURCE --filter 'verdict=pass' -f json \
  | jq -r '.[].vm' \
  | while read vm; do
      xms migration submit \
        --source $SOURCE \
        --vm "$vm" \
        --kind cold \
        --target-project infra \
        --flavor m1.large \
        --volume-type ssd \
        --network-map 'VM Network=internal-net'
    done
```

### Watch a Running Campaign

Follow every job in the current project and print a live status table:

```bash theme={null}
watch -n 10 'xms migration list --filter "status=running,status=syncing"'
```

### Cutover After All Warm Jobs Reach Ready

Wait until every warm migration in a wave is **Ready**, then cut them all
over in sequence:

```bash theme={null}
xms migration list --filter 'kind=warm' -f json \
  | jq -r '.[] | select(.status == "ready") | .id' \
  | while read job_id; do
      xms migration cutover --job $job_id
    done
```

***

## Exit Codes

The CLI exits with a non-zero code on failure so it can be used in shell
pipelines and CI:

| Exit Code | Meaning                                       |
| --------- | --------------------------------------------- |
| `0`       | Success                                       |
| `1`       | Generic error (bad flags, validation failure) |
| `2`       | Authentication or authorization failure       |
| `3`       | Source or VM not found                        |
| `4`       | Job failed or timed out while following       |
| `5`       | Network or transport error talking to XMS     |

***

## Next Steps

<CardGroup>
  <Card title="User Guide" href="/services/migration/user-guide">
    End-to-end workflow using the Dashboard or CLI
  </Card>

  <Card title="Admin Guide" href="/services/migration/admin-guide">
    Operator setup, credentials, and capacity planning
  </Card>

  <Card title="Troubleshooting" href="/services/migration/user-guide/troubleshooting">
    Diagnose job failures and post-migration issues
  </Card>
</CardGroup>


# Migration Suite (XMS)
Source: https://docs.xloud.tech/services/migration/index

Agentless workload migration from VMware vSphere and ESXi to Xloud with incremental sync, automatic driver injection, and minimal cutover downtime.

XMS (Xloud Migration Suite) moves virtual machine workloads from VMware vSphere
and standalone ESXi hosts into Xloud with no agent installed in the source guest.
Cold migrations power off the source VM and stream its disks in a single pass;
warm migrations perform an online full sync followed by incremental block-level
replication, keeping cutover downtime to minutes.

<Card title="XMS — Migration Suite" icon="external-link" href="https://xloud.tech/contact">
  Contact the Xloud team for XMS planning, sizing, and licensing
</Card>

***

## XMS Documentation

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/migration/user-guide">
    Register VMware sources, discover workloads, run preflight checks, and
    execute cold or warm migrations from the Xloud Dashboard.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/migration/admin-guide">
    Deploy the migration service, configure network access to vCenter and ESXi,
    plan storage capacity, and manage credentials for production migrations.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/migration/cli-reference">
    Available command-line workflows and the scope of Dashboard-only operations.
  </Card>

  <Card title="Block Storage" icon="hard-drive" href="/services/storage/index">
    Target volumes are provisioned through Xloud Block Storage — review volume
    types, capacity, and quotas before planning a migration wave.
  </Card>
</CardGroup>

***

## Key Capabilities

<Info>**Xloud-Developed** — XMS is developed by Xloud and ships with XAVS / XPCI.</Info>

<CardGroup>
  <Card title="Agentless Source Access" icon="server">
    Connects to vCenter or standalone ESXi over the standard vSphere API.
    No agent is installed inside the source guest and no changes are made to
    the source VM before cutover.
  </Card>

  <Card title="Cold and Warm Migration" icon="rotate-cw">
    Choose single-pass cold migration for powered-off workloads or warm
    migration with Changed Block Tracking for workloads that must stay online
    until the cutover window.
  </Card>

  <Card title="Automatic Guest Conversion" icon="cpu">
    VirtIO driver injection, boot loader repair, and removal of hypervisor-specific
    tooling are applied offline against the target volume — no manual guest
    preparation on the source side.
  </Card>

  <Card title="Preflight Assessment" icon="clipboard-check">
    Pre-migration compatibility checks for OS family, firmware type, disk
    layout, and driver support. Risks are surfaced before the job starts.
  </Card>

  <Card title="Incremental Cutover" icon="zap">
    Warm migration repeatedly syncs only changed blocks since the last sync
    using Changed Block Tracking. The final cutover replicates a minimal
    delta and switches the workload to Xloud.
  </Card>

  <Card title="End-to-End Dashboard" icon="gauge">
    Discovery, assessment, job submission, live progress, event streaming,
    and post-migration reports are all available from the Migration panel
    in the Xloud Dashboard.
  </Card>
</CardGroup>

***

## Migration Workflow

<Steps>
  <Step title="Register a VMware source" icon="plug">
    Add a vCenter or ESXi endpoint to the Migration panel with the hostname,
    port, credentials, and optional datacenter scope.
  </Step>

  <Step title="Discover workloads" icon="search">
    XMS inventories all reachable VMs, disks, networks, and guest metadata
    through the vSphere API. No agent is installed in the source guest.
  </Step>

  <Step title="Run preflight" icon="clipboard-check">
    Each candidate workload is scored for OS support, firmware type, disk
    layout, and driver availability. Blocking issues are surfaced before
    scheduling a migration.
  </Step>

  <Step title="Submit migration" icon="play">
    Choose cold (single-pass) or warm (continuous sync) mode, pick the target
    Xloud project, network, and volume type, and queue the job.
  </Step>

  <Step title="Sync and cutover" icon="rotate-cw">
    Cold migrations run end-to-end immediately. Warm migrations complete a
    full sync and then incrementally replicate until you trigger cutover.
  </Step>

  <Step title="Guest conversion and boot" icon="cpu">
    XMS injects VirtIO drivers, repairs the boot loader, and launches the
    migrated instance in the target Xloud project.
  </Step>
</Steps>

***

## Supported VMware Environments

<CardGroup>
  <Card title="vCenter Server" icon="server">
    Connect to vCenter to discover and migrate VMs across all managed clusters
    and hosts from a single source entry.
  </Card>

  <Card title="Standalone ESXi" icon="hard-drive">
    Connect directly to an individual ESXi host for smaller deployments or
    edge sites that do not run vCenter.
  </Card>

  <Card title="vSphere 6.x – 8.x" icon="check">
    Covers vSphere 6.0 through 8.0. The migration service bundles disk transport
    libraries for each supported generation.
  </Card>

  <Card title="BIOS and UEFI Guests" icon="shield">
    Both BIOS and UEFI guests migrate to Xloud without firmware-specific
    preparation. Secure Boot status is detected during discovery.
  </Card>
</CardGroup>

***

## Related Services

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Migrated workloads run as Xloud Compute instances with full lifecycle and
    scaling features.
  </Card>

  <Card title="Block Storage" icon="hard-drive" href="/services/storage/index">
    Target disks land as Xloud Block Storage volumes. Choose volume types to
    place migrated workloads on the correct storage tier.
  </Card>

  <Card title="Infrastructure Monitoring" icon="activity" href="/services/monitoring/index">
    Monitor migrated workloads with XIMP dashboards, metrics, and alerting.
  </Card>
</CardGroup>


# Migration User Guide
Source: https://docs.xloud.tech/services/migration/user-guide

Register a VMware source, discover workloads, run preflight checks, execute cold and warm migrations, and finish with cutover using the Xloud Dashboard.

## Overview

XMS (Xloud Migration Suite) moves virtual machine workloads from VMware vSphere
and standalone ESXi hosts into Xloud. This guide walks through the end-to-end
flow: register a source, discover workloads, validate compatibility, run cold
or warm migrations, and complete cutover.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Topics

<CardGroup>
  <Card title="Register a VMware Source" icon="plug" href="/services/migration/user-guide/register-source">
    Add a vCenter or ESXi endpoint to the Migration panel with credentials
    and optional datacenter scope.
  </Card>

  <Card title="Discover Workloads" icon="search" href="/services/migration/user-guide/discover-workloads">
    Inventory VMs, disks, networks, and guest metadata through the vSphere API.
  </Card>

  <Card title="Preflight Assessment" icon="clipboard-check" href="/services/migration/user-guide/preflight">
    Validate each candidate workload against Xloud compatibility rules before
    submitting a migration job.
  </Card>

  <Card title="Cold Migration" icon="power" href="/services/migration/user-guide/cold-migration">
    Power off the source VM and migrate all disks in a single pass — the
    simplest path for workloads that can tolerate a maintenance window.
  </Card>

  <Card title="Warm Migration" icon="rotate-cw" href="/services/migration/user-guide/warm-migration">
    Start a full sync while the source VM remains online, then replicate
    only changed blocks until you schedule the cutover.
  </Card>

  <Card title="Cutover" icon="zap" href="/services/migration/user-guide/cutover">
    Trigger the final sync and switch a warm migration job to its target
    Xloud instance.
  </Card>

  <Card title="Post-Migration Validation" icon="circle-check" href="/services/migration/user-guide/post-migration">
    Verify the migrated instance boots, has network connectivity, and mounts
    all original disks.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/migration/user-guide/troubleshooting">
    Diagnose discovery failures, stalled syncs, stuck cutovers, and guest
    boot issues after migration.
  </Card>
</CardGroup>

***

## Choosing Cold or Warm

<CardGroup>
  <Card title="Use Cold Migration When" icon="power">
    The workload tolerates a maintenance window, the disks are small, and
    you want the simplest possible flow — one click, one pass, done.
  </Card>

  <Card title="Use Warm Migration When" icon="rotate-cw">
    The workload must stay online until cutover, disks are large, or the
    initial sync would exceed an acceptable downtime window.
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="Admin Guide" icon="settings" href="/services/migration/admin-guide">
    Deployment, network ports, storage planning, and credential management
  </Card>

  <Card title="Block Storage" icon="hard-drive" href="/services/storage/user-guide">
    Understand volume types and quotas that apply to migrated disks
  </Card>

  <Card title="Xloud Compute" icon="server" href="/services/compute/user-guide">
    Manage migrated workloads as Xloud Compute instances
  </Card>

  <Card title="Infrastructure Monitoring" icon="activity" href="/services/monitoring/user-guide">
    Monitor migrated workloads with XIMP dashboards and alerts
  </Card>
</CardGroup>


# Cold Migration
Source: https://docs.xloud.tech/services/migration/user-guide/cold-migration

Power off a source VMware virtual machine and migrate it to Xloud in a single pass — the simplest migration path for workloads that tolerate a maintenance window.

## Overview

Cold migration is the simplest way to move a workload from VMware to Xloud.
The source VM is powered off, all disks are exported through the vSphere API,
written into Xloud Block Storage volumes, converted for Xloud drivers, and
the migrated instance is started in the target Xloud project. The source VM
remains in the source environment (powered off) until you confirm the
migration succeeded.

<Note>
  **Prerequisites**

  * A discovered workload that has passed [preflight assessment](/services/migration/user-guide/preflight)
  * A target Xloud project with sufficient quota for the compute, memory, and
    volume footprint of the migrated VM
  * A target Xloud network that can host the migrated VM's NICs
  * An acceptable maintenance window — the source VM will be powered off for
    the duration of the migration
</Note>

***

## Lifecycle

```mermaid theme={null}
graph LR
    Q[Queued] --> E[Export]
    E --> I[Inspect]
    I --> W[Write Volume]
    W --> G[Guest Fixes]
    G --> F[Finalize]
    F --> D[Completed]
    E -->|error| X[Failed]
    W -->|error| X
    G -->|error| X
    F -->|error| X
```

| Stage            | What Happens                                                                                         |
| ---------------- | ---------------------------------------------------------------------------------------------------- |
| **Queued**       | The job is accepted and waits for an available worker slot                                           |
| **Export**       | Source VM is powered off; disks are streamed over the vSphere API                                    |
| **Inspect**      | Guest OS family, firmware, and disk layout are detected                                              |
| **Write Volume** | A new Xloud Block Storage volume is created and disk data is written directly into it                |
| **Guest Fixes**  | VirtIO drivers are injected, the boot loader is repaired, and hypervisor-specific tooling is removed |
| **Finalize**     | The volume is marked bootable, metadata is attached, and the target Xloud instance is prepared       |
| **Completed**    | The target instance is created and ready to launch                                                   |

***

## Submit a Cold Migration

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Migrations tab" icon="truck">
        Navigate to **Migration → Migrations** and click **New Migration**.
        Select **Cold Migration** and pick the source environment.
      </Step>

      <Step title="Select the workload" icon="check-square">
        Pick one or more VMs from the discovered inventory. Each selected VM
        must have a Pass or Warn preflight verdict.
      </Step>

      <Step title="Choose target project and quota" icon="folder">
        | Field                 | Description                                                     |
        | --------------------- | --------------------------------------------------------------- |
        | **Target Project**    | Xloud project to receive the migrated instance                  |
        | **Instance Name**     | Name in the target project (defaults to the source VM name)     |
        | **Flavor**            | Xloud flavor that matches or exceeds the source vCPU and memory |
        | **Volume Type**       | Target storage tier for the migrated volumes                    |
        | **Availability Zone** | Optional — pin the instance to a specific zone                  |
      </Step>

      <Step title="Map networks" icon="network">
        For each source NIC, pick a target Xloud network and subnet. You can
        optionally request the same MAC address on the target network, subject
        to network policy.

        <Tip>
          Save a network mapping as a reusable profile if you plan to migrate
          many VMs from the same source into the same target environment.
        </Tip>
      </Step>

      <Step title="Review and submit" icon="play">
        Review the job summary, estimated runtime, and target resources. Click
        **Submit**. The job enters **Queued** state and is picked up by the
        next available migration worker.
      </Step>

      <Step title="Watch live progress" icon="activity">
        The **Migrations** tab updates live with per-stage progress, bytes
        transferred, and an event stream. A completed job shows a link to
        the newly created Xloud instance.

        <Check>Job status reaches **Completed** and the target instance is visible in the Xloud Dashboard.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash theme={null}
    # Submit a cold migration for a single VM
    xms migration submit \
      --source prod-vcenter \
      --vm win-ser-2022 \
      --kind cold \
      --target-project infra \
      --flavor m1.large \
      --volume-type ssd \
      --network-map 'VM Network=internal-net'

    # Watch live progress
    xms migration events --job <job-id> --follow

    # Inspect the result
    xms migration show <job-id>
    ```
  </Tab>
</Tabs>

***

## What Happens to the Source VM

<CardGroup>
  <Card title="During Migration" icon="pause">
    The source VM is powered off. All reads happen through the vSphere API,
    and the source disks are **never** written to — the migration is
    read-only on the source side.
  </Card>

  <Card title="After Migration" icon="shield">
    The source VM remains powered off in its source environment. XMS does
    not delete it — you choose when to decommission the source after you
    have validated the migrated instance on Xloud.
  </Card>
</CardGroup>

<Warning>
  Do **not** power the source VM back on after cutover if the target instance
  has already booted on Xloud. The two VMs share identity (hostname, MAC,
  disk UUIDs) and running both simultaneously can cause network and
  storage-level conflicts.
</Warning>

***

## Progress and Event Stream

Every cold migration publishes a stream of events that are visible live in
the **Migrations** tab:

| Event                 | When It Fires                                                 |
| --------------------- | ------------------------------------------------------------- |
| `source.powered_off`  | Source VM has reached `poweredOff`                            |
| `export.started`      | Disk export has opened a session with the source              |
| `export.progress`     | Emitted periodically with bytes transferred                   |
| `write.started`       | Target volume has been created and disk write has begun       |
| `write.progress`      | Emitted periodically during volume write                      |
| `guest_fixes.started` | VirtIO driver injection and boot loader repair have begun     |
| `finalize.completed`  | Target volume is bootable and attached to the target instance |
| `migration.completed` | Job is complete — target instance is ready                    |

***

## Next Steps

<CardGroup>
  <Card title="Post-Migration Validation" href="/services/migration/user-guide/post-migration">
    Verify the migrated instance boots and works as expected
  </Card>

  <Card title="Warm Migration" href="/services/migration/user-guide/warm-migration">
    Use incremental sync for workloads that can't tolerate a maintenance window
  </Card>

  <Card title="Troubleshooting" href="/services/migration/user-guide/troubleshooting">
    Diagnose stuck jobs and guest boot issues
  </Card>
</CardGroup>


# Cutover
Source: https://docs.xloud.tech/services/migration/user-guide/cutover

Execute the final sync of a warm migration, power off the source, run guest conversion, and bring the migrated workload up on Xloud.

## Overview

Cutover is the final step of a warm migration. XMS takes a last incremental
sync to bring the target volume fully up to date, powers off the source VM,
runs guest conversion against the target volume, and launches the migrated
instance on Xloud. When cutover completes, the source VM is left powered off
and the target instance is the authoritative copy.

<Note>
  **Prerequisites**

  * A warm migration job in **Ready** state (full sync completed and at least
    one successful incremental sync)
  * A maintenance window long enough for the final sync, guest conversion,
    and target boot — typically a few minutes for small workloads
  * Agreement from the workload owner that the source VM can be powered off
</Note>

***

## Lifecycle

```mermaid theme={null}
graph LR
    R[Ready] --> FD[Final Delta]
    FD --> PO[Power Off Source]
    PO --> GF[Guest Fixes]
    GF --> FI[Finalize]
    FI --> BT[Boot Target]
    BT --> D[Completed]
    FD -->|error| X[Failed]
    GF -->|error| X
    BT -->|error| X
```

| Phase                | What Happens                                                                                                      |
| -------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Final Delta**      | A last incremental sync transfers any blocks that changed since the previous sync.                                |
| **Power Off Source** | XMS issues a graceful shutdown to the source VM. If the guest does not respond in time, a hard power off is used. |
| **Guest Fixes**      | VirtIO drivers are injected, the boot loader is repaired, and hypervisor-specific tooling is removed.             |
| **Finalize**         | The target volume is marked bootable and attached to the target instance record.                                  |
| **Boot Target**      | The target Xloud instance is launched from the migrated volume.                                                   |
| **Completed**        | Cutover is complete — the target instance is running on Xloud.                                                    |

***

## Trigger Cutover

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the warm migration job" icon="folder">
        Navigate to **Migration → Warm Migration** and select the job you want
        to cut over. The job must be in **Ready** state.
      </Step>

      <Step title="Confirm the lag is acceptable" icon="clock">
        Check the **Lag** column. A low lag means the final delta sync will be
        quick. If the lag is high, trigger a **Sync Now** first and wait for
        it to finish before starting cutover.
      </Step>

      <Step title="Click Cutover" icon="play">
        Click **Cutover**. A confirmation dialog summarizes what happens next:

        * Final incremental sync runs immediately
        * Source VM is powered off
        * Guest conversion runs against the target volume
        * Target Xloud instance is launched

        Confirm to proceed. The job transitions to **Cutting Over**.
      </Step>

      <Step title="Watch live progress" icon="activity">
        The panel shows live progress for every phase. Events stream in real
        time — final delta bytes, source power off state, guest conversion
        steps, and target boot.

        <Check>Job status reaches **Completed** and the target instance is visible in the Xloud Dashboard.</Check>
      </Step>

      <Step title="Validate the target" icon="check-square">
        Click the link to the target instance, confirm it reaches an active
        power state, and proceed to
        [post-migration validation](/services/migration/user-guide/post-migration).
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash theme={null}
    # Trigger cutover on an existing warm job
    xms migration cutover --job <job-id>

    # Stream events live
    xms migration events --job <job-id> --follow

    # Confirm final status
    xms migration show <job-id>
    ```
  </Tab>
</Tabs>

***

## What Happens to the Source VM

<CardGroup>
  <Card title="During Cutover" icon="pause">
    XMS issues a graceful shutdown to the source. If the guest does not
    respond within the configured timeout, a hard power off is used to keep
    the cutover window tight.
  </Card>

  <Card title="After Cutover" icon="shield">
    The source VM stays powered off in its source environment. XMS does not
    delete the source — decommission it manually only after you have validated
    the migrated workload on Xloud.
  </Card>
</CardGroup>

<Warning>
  Do **not** power the source VM back on after cutover. The source and target
  share identity (hostname, MAC, disk UUIDs) and running both simultaneously
  can cause network and storage-level conflicts.
</Warning>

***

## Cutover Window

The cutover window is the time between the start of the final delta sync and
the target instance booting on Xloud. It determines how long the workload is
unavailable. Typical contributions:

| Phase                 | Typical Duration                                                             |
| --------------------- | ---------------------------------------------------------------------------- |
| **Final delta sync**  | Seconds to a few minutes — depends on churn since the last incremental       |
| **Source power off**  | Seconds to a minute — graceful shutdown of the guest                         |
| **Guest fixes**       | 30 seconds to a few minutes — VirtIO driver injection and boot loader repair |
| **Finalize and boot** | Under a minute — attach volume and launch target instance                    |

<Tip>
  To minimize the cutover window, run a manual **Sync Now** immediately before
  triggering cutover. This reduces the number of blocks the final delta has to
  transfer.
</Tip>

***

## Rollback

If cutover fails at any phase before the target boots, XMS leaves the source
VM powered off and the job in **Failed** state. You can:

* Power the source VM back on manually — the source data is unchanged
* Inspect the failure in the event stream and in
  [Troubleshooting](/services/migration/user-guide/troubleshooting)
* Re-trigger cutover once the underlying issue is resolved

If the target instance has already booted and you need to roll back, treat
the target as the authoritative copy and migrate back from Xloud to VMware
separately — there is no in-place revert once the target is live.

<Warning>
  A target instance that has booted on Xloud and accepted writes **cannot** be
  reverted to the source by XMS. Plan cutover timing so you have confidence in
  the migrated workload before releasing it to users.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Post-Migration Validation" href="/services/migration/user-guide/post-migration">
    Verify the migrated instance boots, networks, and behaves correctly
  </Card>

  <Card title="Troubleshooting" href="/services/migration/user-guide/troubleshooting">
    Diagnose failed cutovers, stuck guest conversion, and boot errors
  </Card>

  <Card title="Warm Migration" href="/services/migration/user-guide/warm-migration">
    Review the warm migration lifecycle and sync mechanics
  </Card>
</CardGroup>


# Discover Workloads
Source: https://docs.xloud.tech/services/migration/user-guide/discover-workloads

Inventory virtual machines, disks, networks, and guest metadata from a registered VMware source so they can be selected for migration.

## Overview

Discovery walks the inventory of a registered VMware environment and records
every candidate virtual machine along with its disks, network adapters, and
guest metadata. Discovery is safe to re-run at any time — it is read-only
against the source and does not alter the running VMs.

<Note>
  **Prerequisites**

  * A [registered VMware environment](/services/migration/user-guide/register-source)
    with status `Connected`
  * The vSphere account used for registration must have read access to the
    VMs, hosts, and datastores you plan to discover
</Note>

***

## Run Discovery

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Discover tab" icon="search">
        Navigate to **Migration → Discover** and select the VMware environment
        from the source dropdown.
      </Step>

      <Step title="Choose a scope" icon="filter">
        Optionally narrow discovery by datacenter, cluster, or folder. Leaving
        the scope empty inventories every reachable VM in the environment.
      </Step>

      <Step title="Start the scan" icon="play">
        Click **Discover**. XMS opens a vSphere API session, walks the
        inventory through a container view, and normalizes each VM into a
        platform-neutral record.

        Progress is shown live — hosts walked, VMs counted, and the currently
        inspected object.
      </Step>

      <Step title="Review the inventory" icon="list">
        When the scan completes, the inventory table lists every discovered VM
        with its key attributes. Templates are excluded by default.

        <Check>Inventory shows the expected VM count and each row has a valid OS, disk, and NIC summary.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash theme={null}
    # Kick off a discovery scan
    xms discovery start --source prod-vcenter

    # Optional — restrict to a single datacenter
    xms discovery start --source prod-vcenter --datacenter DC-East

    # Stream progress until the scan completes
    xms discovery status --source prod-vcenter --follow

    # List the discovered workloads
    xms discovery list --source prod-vcenter
    ```
  </Tab>
</Tabs>

***

## What XMS Discovers

For every VM, XMS records the following attributes and makes them available
throughout the Migration panel:

| Category       | Attributes                                                                                         |
| -------------- | -------------------------------------------------------------------------------------------------- |
| **Identity**   | Name, instance UUID, managed object reference, power state                                         |
| **Compute**    | vCPU count, memory, hardware version, firmware (BIOS or UEFI), Secure Boot                         |
| **Guest**      | Reported OS family, guest hostname and IP addresses (when guest tools are installed), tools status |
| **Disks**      | Count, size, controller type, thin or thick provisioning, datastore path                           |
| **Networks**   | Adapter count, MAC addresses, port group names, adapter type                                       |
| **Protection** | Changed Block Tracking state, snapshot count, template flag                                        |

<Tip>
  Guest hostname and IP addresses come from VMware Tools. If tools are not
  installed on a source VM, the discovery record still contains all hypervisor
  attributes — it has no guest-level fields. This is not a migration
  blocker.
</Tip>

***

## Inventory Columns

The **Discover** inventory table exposes the most important fields for
triage and selection:

| Column              | Meaning                                           |
| ------------------- | ------------------------------------------------- |
| **Name**            | VM display name                                   |
| **Power State**     | `poweredOn`, `poweredOff`, or `suspended`         |
| **Guest OS**        | OS family reported by guest tools or guest ID     |
| **Firmware**        | `bios` or `efi`                                   |
| **vCPU / Memory**   | Current resource allocation                       |
| **Disks**           | Total count and aggregate size                    |
| **CBT**             | Whether Changed Block Tracking is already enabled |
| **Last Discovered** | Timestamp of the most recent scan                 |

Use the column filters to narrow the list — for example, show only powered-off
Windows VMs under 100 GB for an initial cold-migration wave.

***

## Re-running Discovery

Discovery is idempotent. Re-running it refreshes the inventory cache and
reflects any changes made in the source:

* VMs added, deleted, renamed, or moved between clusters
* Disk layouts changed on the source side
* Power-state transitions since the last scan

<Warning>
  Re-running discovery does not affect in-progress migration jobs. Warm
  migrations keep their original baseline — they do not pick up new disks
  that appear after the job started. Add new disks by submitting a new job.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Preflight Assessment" href="/services/migration/user-guide/preflight">
    Score each workload for compatibility before migrating
  </Card>

  <Card title="Cold Migration" href="/services/migration/user-guide/cold-migration">
    Migrate powered-off VMs in a single pass
  </Card>

  <Card title="Warm Migration" href="/services/migration/user-guide/warm-migration">
    Migrate running VMs with incremental sync
  </Card>
</CardGroup>


# Post-Migration Validation
Source: https://docs.xloud.tech/services/migration/user-guide/post-migration

Verify a migrated workload boots, networks, stores, and behaves correctly on Xloud before retiring the source VM.

## Overview

After a cold migration completes or a warm migration cuts over, you should
validate the migrated workload end-to-end before retiring the source VM.
Post-migration validation covers boot, network, storage, guest services, and
application-level smoke tests. Only after these checks pass should you
decommission the source.

<Note>
  **Prerequisites**

  * A completed [cold migration](/services/migration/user-guide/cold-migration)
    or [warm migration cutover](/services/migration/user-guide/cutover)
  * Xloud Dashboard access to the target project
  * Guest-level credentials or a console login path for the migrated instance
</Note>

***

## Validation Checklist

<Steps>
  <Step title="Target instance reaches active power state" icon="power">
    In the Xloud Dashboard, open **Compute → Instances** in the target project
    and confirm the migrated instance shows an active power state. A status
    of `ERROR` or `BUILD` stuck for more than a few minutes indicates a
    platform-level issue — see
    [Troubleshooting](/services/migration/user-guide/troubleshooting).

    <Check>Instance is active and the volume is attached.</Check>
  </Step>

  <Step title="Boot completed and console is reachable" icon="monitor">
    Open the instance console from the Xloud Dashboard and confirm:

    * The guest reached a login prompt or desktop session
    * No unexpected disk-check prompts or recovery-mode banners
    * Boot loader messages match the expected OS (GRUB for Linux, Windows
      Boot Manager for Windows)

    <Check>Guest reaches normal login prompt without dropping to recovery mode.</Check>
  </Step>

  <Step title="Network is reachable" icon="network">
    Confirm the instance responds on its configured network:

    * Instance has the expected IP addresses assigned
    * Ping or SSH or RDP from an allowed source reaches the instance
    * Guest-level interface count matches the source (every source NIC was mapped)
    * DNS resolution works from inside the guest

    <Check>You can reach the guest using the connection method used on the source.</Check>
  </Step>

  <Step title="Storage is intact" icon="hard-drive">
    From inside the guest, verify:

    * Every disk from the source is present and mounted
    * Filesystems mount read-write as expected
    * Disk sizes match the source
    * No filesystem errors appear in dmesg or the Windows Event Viewer

    <Check>All source disks are present, mounted, and report no filesystem errors.</Check>
  </Step>

  <Step title="Guest services are running" icon="play">
    Log in to the guest and verify the services that run on the source are
    running on the target:

    * Linux: `systemctl list-units --state=running` matches the expected set
    * Windows: Services MMC shows the expected services in Running state
    * Application-specific daemons (databases, web servers, custom services)
      are up and responding

    <Check>Every service that runs on the source is running on the target.</Check>
  </Step>

  <Step title="Application-level smoke test" icon="flask-conical">
    Run the workload's application-level health check:

    * Databases: connect and run a trivial query
    * Web apps: load the home page and a backend route
    * Custom services: run the test suite or health endpoint

    <Check>Application smoke test returns the same result on the target as on the source.</Check>
  </Step>
</Steps>

***

## Common Post-Migration Symptoms

| Symptom                                               | Likely Cause                                                             | Fix                                                                                                               |
| ----------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- |
| **Guest boots to recovery mode (Linux)**              | GRUB was pointing at old device names                                    | Boot from recovery, regenerate GRUB config, update initramfs                                                      |
| **Windows BSOD on boot (INACCESSIBLE\_BOOT\_DEVICE)** | VirtIO storage driver was not loaded at early boot                       | Attach the target volume to a rescue instance and enable the `viostor` service at boot                            |
| **Network interface missing**                         | Guest tied the interface to a specific MAC or driver                     | Remove the stale interface record (for example, NetworkManager keyfile or udev persistent-net rule) and re-detect |
| **Hostname resolves but services not reachable**      | Firewall inside the guest blocks the new IP range                        | Update the guest firewall rules to allow the target Xloud network range                                           |
| **High CPU usage at idle**                            | Leftover source-side tools (for example, VMware Tools) running in a loop | Uninstall source-side hypervisor tools from the guest                                                             |
| **Clock drift**                                       | Guest time source is still pointing at source NTP                        | Reconfigure the guest time source for the target network                                                          |

<Tip>
  Most post-migration issues are guest-side side effects, not migration
  failures. The migration writes the source disks faithfully — the guest just
  needs to re-learn its new environment.
</Tip>

***

## Decommissioning the Source

Only after every checklist item passes and the workload owner has confirmed
the migrated instance is authoritative should you retire the source VM.

<Warning>
  Do **not** delete the source VM immediately. Keep it powered off for a
  rollback window (typically 7-30 days depending on your change management
  policy) before deleting it permanently. If a post-migration issue appears
  later, the source is still available as a reference.
</Warning>

### Recommended Decommission Sequence

<Steps>
  <Step title="Confirm validation passed" icon="check-square">
    All checklist items above are green and the workload owner has signed off.
  </Step>

  <Step title="Keep source powered off" icon="pause">
    Leave the source VM in the VMware inventory, powered off. Do not delete
    it yet.
  </Step>

  <Step title="Wait out the rollback window" icon="clock">
    Wait the number of days your change management policy requires. During
    this window the source remains available as a reference or rollback target.
  </Step>

  <Step title="Delete the source VM" icon="trash">
    Once the rollback window has elapsed, delete the source VM through vSphere.
    XMS does not delete source VMs — this is always a manual step by the source
    environment owner.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Troubleshooting" href="/services/migration/user-guide/troubleshooting">
    Diagnose boot, network, and guest-level issues after migration
  </Card>

  <Card title="Xloud Compute" href="/services/compute">
    Manage the migrated instance lifecycle on Xloud
  </Card>

  <Card title="Infrastructure Monitoring" href="/services/monitoring">
    Add the migrated instance to monitoring and alerting
  </Card>
</CardGroup>


# Preflight Assessment
Source: https://docs.xloud.tech/services/migration/user-guide/preflight

Validate a discovered workload against the Xloud compatibility rules before submitting a migration — OS support, firmware, disk layout, and driver readiness.

## Overview

Preflight assessment runs a set of read-only checks against a discovered
workload and returns a compatibility verdict before you submit a migration
job. The goal is to catch the issues that would cause a migration to fail or
a post-migration boot problem — OS not supported, firmware mismatch, disk
layout too complex — while you still have time to fix them on the source.

<Note>
  **Prerequisites**

  * A successful [discovery scan](/services/migration/user-guide/discover-workloads)
    for the source environment
  * Credentials on the registered source with read access to VM configuration
    and disk metadata
</Note>

***

## Run Preflight

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Select candidate workloads" icon="check-square">
        In **Migration → Discover**, tick the VMs you plan to migrate and click
        **Preflight Selected**. You can preflight a single VM or an entire wave.
      </Step>

      <Step title="Start the assessment" icon="clipboard-check">
        XMS inspects each VM through the vSphere API and applies the Xloud
        compatibility ruleset. The scan is read-only and completes in seconds
        for most workloads.
      </Step>

      <Step title="Review the verdict" icon="list">
        Each VM receives one of three outcomes:

        | Verdict   | Meaning                                                           |
        | --------- | ----------------------------------------------------------------- |
        | **Pass**  | All checks passed — safe to migrate                               |
        | **Warn**  | Migration can proceed but with caveats — review the warnings      |
        | **Block** | One or more checks failed — resolve before submitting a migration |

        Expand a row to see every individual check, its status, and the
        suggested remediation.

        <Check>All target workloads reach **Pass** or **Warn** before you submit a migration.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash theme={null}
    # Run preflight against one or more VMs
    xms preflight run --source prod-vcenter --vm win-ser-2022

    # Preflight an entire wave from a file
    xms preflight run --source prod-vcenter --from-file wave-1.txt

    # Fetch the latest verdict
    xms preflight show --source prod-vcenter --vm win-ser-2022
    ```
  </Tab>
</Tabs>

***

## Check Categories

Preflight runs checks across five categories. Each category produces one or
more individual checks that roll up into a category score.

### Compute

| Check                | Description                                                                                                   |
| -------------------- | ------------------------------------------------------------------------------------------------------------- |
| **Firmware type**    | BIOS and UEFI are both supported. Secure Boot state is recorded for post-migration validation.                |
| **Hardware version** | vSphere hardware versions are checked against the supported range for the installed disk transport libraries. |
| **Guest OS family**  | The reported guest OS is matched against the supported Xloud guest catalog.                                   |
| **Resource shape**   | vCPU count and memory are sanity-checked against project quotas in the target Xloud project.                  |

### Storage

| Check                   | Description                                                                                                                      |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| **Disk count and size** | Total disk size is compared to available quota and to the target volume type's maximum size.                                     |
| **Disk layout**         | Complex layouts (raw device mappings, shared disks, independent disks) are flagged. Standard thin and thick disks are supported. |
| **Snapshot state**      | Active snapshots at migration time may slow down export. XMS flags VMs with long snapshot chains.                                |

### Network

| Check                | Description                                                                                                                        |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| **Adapter count**    | Each source adapter must map to a target Xloud network.                                                                            |
| **Port group names** | Port group names are captured for the later network mapping step.                                                                  |
| **MAC preservation** | The option to preserve MACs is flagged as informational — a network mapping with conflicting MACs will be rejected at submit time. |

### Guest

| Check                | Description                                                                                                 |
| -------------------- | ----------------------------------------------------------------------------------------------------------- |
| **Driver readiness** | The OS is matched against the VirtIO driver set that will be injected during guest conversion.              |
| **Boot loader**      | GRUB (Linux) and the Windows BCD are confirmed present. Missing boot loaders are a hard block.              |
| **Filesystem type**  | Supported filesystems include ext2/3/4, XFS, Btrfs, NTFS, FAT32, ReFS, and LVM with these underlying types. |

### Protection

| Check                | Description                                                                                                                    |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| **CBT enabled**      | Warm migration requires Changed Block Tracking. XMS surfaces this as a warning and offers to enable it in-place.               |
| **Snapshot support** | Warm migration needs the ability to take a snapshot at sync start. Hosts with disabled snapshot support are blocked from warm. |

***

## Common Blockers and Remediation

| Blocker                    | Root Cause                                                              | Fix                                                                                       |
| -------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| **Unsupported guest OS**   | The guest ID returned by vSphere does not match the Xloud guest catalog | Upgrade or reinstall the guest OS to a supported version before migrating                 |
| **Raw Device Mappings**    | A disk is mapped directly to a LUN on the source                        | Convert the RDM to a VMFS-backed disk before migration                                    |
| **Shared disks**           | Two VMs share the same VMDK                                             | Un-share or take an online copy before migration                                          |
| **CBT not enabled (warm)** | Change tracking has never been turned on                                | Enable CBT from preflight — XMS will write the config change through the vSphere API      |
| **Missing boot loader**    | Damaged GRUB or BCD                                                     | Repair the boot loader on the source first; migration cannot recover a broken boot loader |

<Tip>
  Most warnings are safe to ignore for a first pass. Focus on blockers and
  come back to the warnings if a post-migration boot issue appears.
</Tip>

***

## Preflight vs Runtime Checks

Preflight is a **read-only** prediction. XMS still performs runtime validation
at every stage of the migration pipeline — disk reads, transport negotiation,
guest conversion, and volume finalize each have their own checks. A preflight
pass does not guarantee a successful migration, but a preflight block does
guarantee a migration would fail.

***

## Next Steps

<CardGroup>
  <Card title="Cold Migration" href="/services/migration/user-guide/cold-migration">
    Run a single-pass migration against a powered-off workload
  </Card>

  <Card title="Warm Migration" href="/services/migration/user-guide/warm-migration">
    Start a continuous sync against a running workload
  </Card>

  <Card title="Troubleshooting" href="/services/migration/user-guide/troubleshooting">
    Diagnose preflight blockers and discovery errors
  </Card>
</CardGroup>


# Register a VMware Source
Source: https://docs.xloud.tech/services/migration/user-guide/register-source

Add a vCenter Server or standalone ESXi host to XMS as a migration source so its virtual machines become discoverable from the Xloud Dashboard.

## Overview

Before XMS can discover or migrate workloads, you must register the source
vSphere environment as an **environment** in the Migration panel. An environment
stores the endpoint URL, credentials, and optional scope (datacenter) used for
every subsequent discovery and migration job against that source.

<Tip>
  In the Dashboard, source VMware endpoints are called **Environments**. In the
  CLI, the same objects are called **sources** (`xms source create`, `xms source list`).
  Both refer to the same registered endpoints.
</Tip>

<Note>
  **Prerequisites**

  * An active Xloud account with permission to use the Migration panel
  * A reachable vCenter Server or standalone ESXi host
  * A vSphere account with at least read permission for inventory discovery
    and snapshot, disk read, and Changed Block Tracking permissions for
    migration (see the **Required Permissions** table below)
  * The fingerprint or CA chain of the source if you choose to verify TLS
</Note>

***

## Register from the Dashboard

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Migration panel" icon="truck">
        Sign in to the **Xloud Dashboard** and navigate to **Migration → Environments**.
        Click **Add Environment** and choose **VMware vSphere** as the platform.
      </Step>

      <Step title="Enter connection details" icon="plug">
        | Field                | Description                                                                                   |
        | -------------------- | --------------------------------------------------------------------------------------------- |
        | **Environment Name** | Friendly name for the source (e.g., `prod-vcenter`, `edge-esxi-01`)                           |
        | **Host**             | Hostname or IP of the vCenter Server or standalone ESXi host                                  |
        | **Port**             | `443` (default)                                                                               |
        | **Username**         | vSphere account (e.g., `administrator@vsphere.local` for vCenter, `root` for standalone ESXi) |
        | **Password**         | vSphere account password                                                                      |
        | **Datacenter**       | Optional — restrict discovery to a single datacenter                                          |
        | **Verify TLS**       | Enable for production; disable only for lab or self-signed certificates                       |
      </Step>

      <Step title="Test the connection" icon="flask-conical">
        Click **Test Connection**. XMS opens a vSphere API session, fetches the
        product name and build number, and closes the session. A green check
        confirms the endpoint, credentials, and TLS options are valid.

        <Check>Connection test returns the remote product name and build.</Check>
      </Step>

      <Step title="Save the environment" icon="save">
        Click **Save**. The environment appears in the **Environments** list
        with status `Connected` and the detected vSphere version.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Register a VMware source with the `xms` CLI:

    ```bash theme={null}
    xms source create \
      --name prod-vcenter \
      --platform vmware \
      --host vcenter.example.com \
      --port 443 \
      --username 'administrator@vsphere.local' \
      --password-stdin \
      --verify-ssl
    ```

    Pass the password through `stdin` to keep it out of shell history. Use
    `--datacenter <name>` to scope discovery and `--no-verify-ssl` only when
    connecting to self-signed lab hosts.
  </Tab>
</Tabs>

***

## Required Permissions

XMS needs the following vSphere privileges on the target inventory objects.
Create a dedicated role and assign it to the XMS service account at the
datacenter level for the cleanest setup.

| Category                        | Privilege                                      | Used By                                                                     |
| ------------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------- |
| **System**                      | System.Anonymous, System.Read, System.View     | Session establishment and inventory read                                    |
| **VirtualMachine.Config**       | DiskLease, ChangeTracking, Settings            | Enabling CBT and reading disk configuration                                 |
| **VirtualMachine.Interact**     | PowerOff, PowerOn, Reset                       | Cold migration source power control                                         |
| **VirtualMachine.State**        | CreateSnapshot, RemoveSnapshot, RevertSnapshot | Warm migration CBT anchor snapshots                                         |
| **VirtualMachine.Provisioning** | DiskRandomAccess, DiskRandomRead               | Disk export over vSphere API                                                |
| **Resource**                    | AssignVMToPool (target moves only)             | Optional — only required for cross-cluster preparation                      |
| **Global**                      | DisableMethods, EnableMethods                  | Optional — only required to guard against concurrent changes during cutover |

<Tip>
  A read-only account is enough to register the source and run discovery.
  You can assign the higher-privilege role closer to the migration window
  to limit blast radius.
</Tip>

***

## Standalone ESXi vs vCenter

<CardGroup>
  <Card title="vCenter" icon="server">
    Preferred for production — one environment entry covers all clusters and
    hosts, role-based access control is centralized, and vMotion and snapshots
    are coordinated through vCenter.
  </Card>

  <Card title="Standalone ESXi" icon="hard-drive">
    Supported for smaller sites and edge deployments. Use the host IP or FQDN
    as the endpoint and `root` (or a local ESXi account) as the username.
    Datacenter is auto-detected as `ha-datacenter`.
  </Card>
</CardGroup>

***

## Editing and Deleting Environments

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    From **Migration → Environments**, select a row to see the details drawer.
    Available actions:

    * **Edit** — update endpoint, credentials, or TLS verification
    * **Test Connection** — re-run the connectivity check at any time
    * **Delete** — remove the environment (only permitted when no active
      migration jobs reference it)
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash theme={null}
    xms source list
    xms source show prod-vcenter
    xms source update prod-vcenter --password-stdin
    xms source delete prod-vcenter
    ```
  </Tab>
</Tabs>

<Warning>
  Changing the username or password of an active source invalidates any
  in-flight discovery. Re-run discovery after updating credentials so the
  inventory cache refreshes.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Discover Workloads" href="/services/migration/user-guide/discover-workloads">
    Inventory VMs and disks from the newly registered source
  </Card>

  <Card title="Preflight Assessment" href="/services/migration/user-guide/preflight">
    Validate compatibility before submitting a migration
  </Card>

  <Card title="Admin — Source Credentials" href="/services/migration/admin-guide/source-credentials">
    Credential storage, rotation, and least-privilege setup
  </Card>
</CardGroup>


# Migration Troubleshooting
Source: https://docs.xloud.tech/services/migration/user-guide/troubleshooting

Diagnose and recover from common failures across source registration, discovery, preflight, migration, cutover, and post-migration.

## Overview

Migration is a pipeline that touches a source hypervisor API, a storage
back-end, a guest operating system, and an Xloud project. A failure at any
stage produces an event on the failed job. This page maps the most common
symptoms to the underlying cause and the fix.

<Tip>
  Every XMS migration job exposes a live event stream. Start every
  investigation by reading the events — most failures report the root cause
  as a typed event before the job transitions to `Failed`.
</Tip>

***

## Source Registration

<AccordionGroup>
  <Accordion title="Test connection fails with TLS error" icon="shield-alert">
    **Symptom**: **Test Connection** returns an SSL or TLS error when saving a
    new vSphere source.

    **Cause**: The source uses a self-signed certificate or a certificate
    chain that is not trusted by XMS.

    **Fix**:

    * For production: install the source's CA chain into the XMS trust store,
      then re-test
    * For lab use only: disable **Verify TLS** on the source — not recommended
      for production environments
  </Accordion>

  <Accordion title="Test connection fails with authentication error" icon="key">
    **Symptom**: Endpoint reachable, but the returned error says the
    credentials are invalid.

    **Cause**: The username format is wrong, the password is wrong, or the
    account is locked in the source directory service.

    **Fix**:

    * Use `administrator@vsphere.local` (or your SSO domain) for vCenter
    * Use `root` or a local ESXi account for standalone ESXi
    * Re-enter the password — XMS does not echo a stored password on edit
    * Check the source directory service for account lockout
  </Accordion>

  <Accordion title="Save succeeds but the environment stays Disconnected" icon="plug">
    **Symptom**: The environment saves but the status is **Disconnected** in
    the list.

    **Cause**: XMS cannot reach the endpoint from the path used by the
    background refresh. The save-time test uses the same network path.

    **Fix**:

    * Confirm the source endpoint is reachable from the XMS service
    * Check any firewall between XMS and the source — vSphere API is 443
    * Re-run **Test Connection** from the environment details drawer
  </Accordion>
</AccordionGroup>

***

## Discovery

<AccordionGroup>
  <Accordion title="Discovery returns zero VMs" icon="circle-slash">
    **Symptom**: Discovery completes but no VMs appear in the inventory.

    **Cause**: The vSphere account registered on the source has no read
    permission on any object inside the datacenter scope.

    **Fix**:

    * Confirm the account has at least read permission at the datacenter
      level (or whichever scope you pinned the environment to)
    * Re-run discovery and watch the progress counter
  </Accordion>

  <Accordion title="Discovery hangs on a specific host" icon="clock">
    **Symptom**: Progress stalls when XMS reaches a specific ESXi host.

    **Cause**: The host is in a degraded state or has an open vSphere API
    session limit.

    **Fix**:

    * From vSphere, confirm the host is connected and not in maintenance mode
      for an unexpected reason
    * Cancel discovery, wait for the stale session to clear, then re-run
  </Accordion>

  <Accordion title="Inventory missing fields (OS, IP, hostname)" icon="triangle-alert">
    **Symptom**: Discovered VMs have no guest OS, hostname, or IP addresses.

    **Cause**: VMware Tools is not installed on the source guest, or the tools
    daemon is not running.

    **Fix**:

    * Install or start VMware Tools on the source guest
    * Re-run discovery — the hypervisor fields (firmware, disks, NICs) are
      always populated, only guest-level fields need tools
  </Accordion>
</AccordionGroup>

***

## Preflight

<AccordionGroup>
  <Accordion title="Unsupported guest OS" icon="triangle-alert">
    **Symptom**: Preflight blocks a VM with an "unsupported guest OS" check.

    **Cause**: The guest identifier returned by vSphere does not match the
    Xloud guest catalog — either the guest is truly unsupported or the guest
    identifier has not been updated after an OS upgrade on the source.

    **Fix**:

    * Upgrade or reinstall the guest to a supported version, or
    * On the source, update the VM's guest OS identifier to match the
      installed guest, then re-run preflight
  </Accordion>

  <Accordion title="Raw Device Mapping detected" icon="triangle-alert">
    **Symptom**: Preflight blocks a VM with an RDM disk.

    **Cause**: XMS migrates virtual disks, not LUN-level device mappings.

    **Fix**: Convert the RDM to a standard VMFS-backed disk on the source
    before migrating.
  </Accordion>

  <Accordion title="CBT not enabled for warm migration" icon="shield-alert">
    **Symptom**: Warm migration is blocked because Changed Block Tracking is
    not enabled on the source.

    **Cause**: CBT was never turned on for this VM.

    **Fix**: Use the **Enable CBT** action on the preflight result. XMS writes
    the configuration change through the vSphere API. Re-run preflight.
  </Accordion>
</AccordionGroup>

***

## Cold Migration

<AccordionGroup>
  <Accordion title="Export phase fails with connection reset" icon="network">
    **Symptom**: The export phase starts, transfers some data, then fails with
    a connection reset or transport error.

    **Cause**: The network path between XMS and the source dropped mid-transfer,
    or the source closed the disk transport session.

    **Fix**:

    * Confirm the network path between XMS and the source is stable
    * Re-submit the migration — cold migration restarts from the beginning of
      the export
  </Accordion>

  <Accordion title="Inspect phase fails to detect guest" icon="eye-off">
    **Symptom**: Inspect phase reports an unknown or unsupported guest layout.

    **Cause**: The guest has a non-standard partition layout, a damaged
    partition table, or an encrypted root that XMS cannot unlock.

    **Fix**:

    * Repair the partition table on the source before migrating
    * For encrypted roots, remove the encryption or provide the key through
      the guest fixes configuration
  </Accordion>

  <Accordion title="Guest fixes phase fails" icon="wrench">
    **Symptom**: VirtIO driver injection or boot loader repair fails.

    **Cause**: The guest filesystem is corrupted, or the boot loader on the
    source was already broken before migration.

    **Fix**:

    * Repair the source boot loader first, then re-submit migration
    * For Windows, confirm the boot configuration database is present
    * For Linux, confirm GRUB is installed and readable
  </Accordion>
</AccordionGroup>

***

## Warm Migration

<AccordionGroup>
  <Accordion title="Incremental sync transfers too much data" icon="trending-up">
    **Symptom**: Incremental syncs are running long and transferring more data
    than expected for the sync interval.

    **Cause**: CBT was reset on the source (snapshot removal, host reboot,
    storage vMotion), forcing a full re-read on the next sync.

    **Fix**:

    * This is self-healing — one incremental sync will transfer everything
      and subsequent syncs will return to normal
    * Avoid taking or removing snapshots on the source mid-migration
  </Accordion>

  <Accordion title="Warm job stuck in Syncing" icon="clock">
    **Symptom**: A warm job has been in **Syncing** for far longer than the
    previous syncs took.

    **Cause**: Network path congestion, slow source disks, or an open disk
    transport session that has not closed.

    **Fix**:

    * Let the sync finish if the progress counter is still advancing
    * If progress is fully stalled, pause and resume the job — XMS re-opens
      the disk transport session
  </Accordion>

  <Accordion title="Lag keeps growing" icon="triangle-alert">
    **Symptom**: The **Lag** column keeps growing and the scheduled
    incrementals do not keep up with source churn.

    **Cause**: The cadence is too slow for the churn rate, or the network path
    cannot sustain the incremental throughput.

    **Fix**:

    * Increase sync cadence (daily → hourly → every 15 minutes)
    * Trigger **Sync Now** manually to catch up
    * If network path is the bottleneck, move the migration to a window with
      more available bandwidth
  </Accordion>
</AccordionGroup>

***

## Cutover

<AccordionGroup>
  <Accordion title="Source refuses graceful shutdown" icon="power">
    **Symptom**: Cutover waits on the source power-off phase longer than
    expected.

    **Cause**: The guest is not responding to ACPI shutdown, typically because
    a process is holding the shutdown or guest tools are not running.

    **Fix**:

    * XMS falls back to hard power off after the configured timeout
    * To avoid the wait, log in to the source before cutover and quiesce
      in-flight workloads
  </Accordion>

  <Accordion title="Guest fixes fails on cutover" icon="wrench">
    **Symptom**: Guest conversion fails after the final delta sync.

    **Cause**: The source filesystem was modified in a way that breaks guest
    conversion — typically a partial update applied mid-migration.

    **Fix**:

    * Do not apply operating-system updates on the source during an active
      warm migration
    * If the fix path is unclear, submit a cold migration from the same
      source to replace the warm target volume
  </Accordion>

  <Accordion title="Target instance boots to error" icon="triangle-alert">
    **Symptom**: Target instance reaches power-on but the guest fails to boot
    cleanly.

    **Cause**: A driver, boot loader, or filesystem issue that did not surface
    during guest fixes.

    **Fix**: See the Post-Migration section below — target-side boot issues
    are handled the same way regardless of whether the job was cold or warm.
  </Accordion>
</AccordionGroup>

***

## Post-Migration

<AccordionGroup>
  <Accordion title="Linux guest drops to recovery mode" icon="terminal">
    **Symptom**: Guest reaches a `(initramfs)` or rescue shell instead of a
    normal login.

    **Cause**: GRUB is looking for old device names or the initramfs is
    missing required VirtIO modules.

    **Fix**:

    1. Boot into the guest rescue shell (from the instance console)
    2. Mount the root filesystem
    3. Regenerate the initramfs to include VirtIO modules
    4. Update the GRUB configuration
    5. Reboot
  </Accordion>

  <Accordion title="Windows guest BSOD with INACCESSIBLE_BOOT_DEVICE" icon="monitor">
    **Symptom**: Windows bluescreens early in boot with
    `INACCESSIBLE_BOOT_DEVICE`.

    **Cause**: The `viostor` driver was installed but the boot-start service
    is not enabled, so Windows cannot load the driver before mounting the
    system disk.

    **Fix**:

    1. Attach the target volume to a rescue Windows instance
    2. Enable the `viostor` service at boot (registry service start type = 0)
    3. Detach and boot the original target instance
  </Accordion>

  <Accordion title="Guest network interface missing" icon="network">
    **Symptom**: The guest boots, but the expected interface is absent.

    **Cause**: The guest persisted the old interface name or MAC and is
    confused by the new virtual hardware.

    **Fix**:

    * Linux: remove persistent network rules (for example, NetworkManager
      keyfiles pinned to the old MAC), then reboot
    * Windows: remove the hidden ghost adapter from Device Manager and
      re-detect
  </Accordion>
</AccordionGroup>

***

## Collecting Diagnostics

When a failure is not covered by this guide, collect the following before
opening a support request:

<Steps>
  <Step title="Job ID and timestamps" icon="clipboard">
    Copy the job ID, creation time, failure time, and failed phase from the
    job details panel.
  </Step>

  <Step title="Full event stream" icon="list">
    Export or copy the full event stream from the job details panel. The
    stream shows every phase transition and the typed error event.
  </Step>

  <Step title="Source environment snapshot" icon="server">
    Record the source environment type (vCenter or standalone ESXi), version,
    build, and the guest OS as reported by discovery.
  </Step>

  <Step title="Target project and quota" icon="folder">
    Record the target project, flavor, volume type, and any quota warnings
    visible in the Xloud Dashboard at the time of failure.
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="User Guide" href="/services/migration/user-guide">
    Return to the migration user guide hub
  </Card>

  <Card title="Admin Guide" href="/services/migration/admin-guide">
    Operator-facing setup, prerequisites, and capacity planning
  </Card>

  <Card title="CLI Reference" href="/services/migration/cli-reference">
    Full command-line reference for the xms CLI
  </Card>
</CardGroup>


# Warm Migration
Source: https://docs.xloud.tech/services/migration/user-guide/warm-migration

Replicate a running VMware virtual machine to Xloud with a full sync followed by continuous incremental block-level replication — minimal downtime at cutover.

## Overview

Warm migration moves a workload from VMware to Xloud while the source VM
continues to run. XMS takes a snapshot on the source, copies the full disk
contents into a target Xloud Block Storage volume, and then replicates only
the blocks that change on the source between syncs. When you trigger cutover,
XMS replicates the final delta, powers the source off, runs guest conversion,
and starts the target instance on Xloud — typically within minutes.

<Note>
  **Prerequisites**

  * A discovered workload that has passed [preflight assessment](/services/migration/user-guide/preflight)
  * Changed Block Tracking (CBT) available on the source host
  * A target Xloud project with enough quota for the compute, memory, and
    volume footprint of the migrated VM
  * A stable network path between the migration service and the source
    environment that can sustain the full sync and incremental syncs
</Note>

***

## Lifecycle

```mermaid theme={null}
graph LR
    I[Initializing] --> F[Full Sync]
    F --> R[Ready]
    R --> S[Syncing]
    S --> R
    R --> C[Cutting Over]
    C --> D[Completed]
    I -->|error| X[Failed]
    F -->|error| X
    S -->|error| X
    C -->|error| X
```

| Phase            | Meaning                                                                                               |
| ---------------- | ----------------------------------------------------------------------------------------------------- |
| **Initializing** | XMS enables CBT on the source if needed and records a change tracking baseline                        |
| **Full Sync**    | The entire disk contents are copied to the target volume. Source VM keeps running the whole time.     |
| **Ready**        | Initial copy is complete. The job sits idle between scheduled syncs.                                  |
| **Syncing**      | An incremental sync is in progress — only blocks changed since the last baseline are read and applied |
| **Cutting Over** | Final delta sync, source power off, guest conversion, and target boot                                 |
| **Completed**    | Target instance is running on Xloud                                                                   |

***

## Submit a Warm Migration

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Start a new migration" icon="rotate-cw">
        Navigate to **Migration → Migrations** and click **New Migration**.
        Select **Warm Migration** as the migration type and pick the source
        environment.
      </Step>

      <Step title="Pick source workload" icon="check-square">
        Select the VM from the discovered inventory. Only VMs with a Pass or
        Warn preflight verdict are eligible.
      </Step>

      <Step title="Configure target" icon="folder">
        | Field                 | Description                                                           |
        | --------------------- | --------------------------------------------------------------------- |
        | **Target Project**    | Xloud project that will host the migrated instance                    |
        | **Instance Name**     | Name the target instance will use after cutover                       |
        | **Flavor**            | Xloud flavor to match or exceed the source shape                      |
        | **Volume Type**       | Storage tier for the target volume (chosen once, used for every sync) |
        | **Availability Zone** | Optional — pin the target instance to a zone                          |
      </Step>

      <Step title="Map networks and set sync schedule" icon="network">
        Map every source NIC to a target Xloud network and subnet. Choose a
        sync cadence:

        | Cadence          | Typical Use                                  |
        | ---------------- | -------------------------------------------- |
        | **Manual**       | You trigger each incremental sync explicitly |
        | **Every 15 min** | Small, steady-state workloads                |
        | **Hourly**       | Medium workloads with predictable churn      |
        | **Daily**        | Large workloads or low-churn archives        |

        <Tip>
          You can change the cadence at any time. Scheduling sync only
          controls how often new syncs start — it does not affect cutover.
        </Tip>
      </Step>

      <Step title="Start full sync" icon="play">
        Click **Start Full Sync**. XMS takes the baseline snapshot on the
        source, starts streaming disk data, and writes it into the target
        volume. The panel shows live progress — bytes read, bytes written,
        and estimated completion time.
      </Step>

      <Step title="Monitor ready state" icon="activity">
        When the full sync completes, the job transitions to **Ready**. From
        here, you can let the scheduler drive incrementals or trigger
        **Sync Now** manually.

        The **Last Sync** and **Lag** columns in the job list show how far
        behind the target is from the source.

        <Check>Job status reaches **Ready** and per-sync lag stays within your tolerance.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash theme={null}
    # Submit a warm migration
    xms migration submit \
      --source prod-vcenter \
      --vm db-prod-01 \
      --kind warm \
      --target-project infra \
      --flavor m1.xlarge \
      --volume-type ssd \
      --network-map 'VM Network=internal-net' \
      --sync-cadence 1h

    # Trigger an incremental sync manually
    xms migration sync --job <job-id>

    # Watch the lag
    xms migration show <job-id>
    ```
  </Tab>
</Tabs>

***

## How Incremental Sync Works

XMS uses vSphere Changed Block Tracking (CBT) to read only the blocks that
have changed since the previous sync. Each sync runs through the same disk
transport library used for the full sync, so the data path is identical — the
only difference is that incremental syncs read the change map first and then
stream only the dirty regions.

```mermaid theme={null}
graph LR
    A["CBT change map<br/>since last baseline"] --> B["Read only<br/>dirty blocks"]
    B --> C["Write to target<br/>volume at same offsets"]
    C --> D["Advance baseline<br/>in the job record"]
```

Because every write targets the same offsets on the same Xloud Block Storage
volume, there is no intermediate copy and the target volume is always a
byte-for-byte current replica of the source at the time of the last sync.

***

## Ready, Waiting for Cutover

Once the full sync has completed, the job stays in **Ready** state indefinitely.
You can:

* Let the scheduler run incrementals on the cadence you configured
* Trigger **Sync Now** manually to force an incremental at any time
* **Pause** the job to stop incrementals without losing progress
* Proceed to [cutover](/services/migration/user-guide/cutover) when you are
  ready to switch the workload to Xloud

<Warning>
  The job retains CBT baselines as long as it is active. If you pause a warm
  migration for an extended period, re-enable CBT tracking and run a fresh
  sync before cutover to keep the delta small.
</Warning>

***

## Sync Metrics

Each warm migration exposes live sync metrics in the Warm Migration tab:

| Metric           | Description                                                 |
| ---------------- | ----------------------------------------------------------- |
| **Last Sync**    | Timestamp of the most recent completed sync                 |
| **Lag**          | Time elapsed since the last successful sync                 |
| **Bytes (Full)** | Total bytes read during the initial full sync               |
| **Bytes (Incr)** | Total bytes transferred across all incremental syncs so far |
| **Sync Count**   | Number of successful incremental syncs performed            |
| **Rate**         | Current sync throughput (bytes/s during active sync)        |

***

## Next Steps

<CardGroup>
  <Card title="Cutover" href="/services/migration/user-guide/cutover">
    Execute the final sync and switch the workload to Xloud
  </Card>

  <Card title="Post-Migration Validation" href="/services/migration/user-guide/post-migration">
    Verify the migrated instance boots and behaves correctly
  </Card>

  <Card title="Troubleshooting" href="/services/migration/user-guide/troubleshooting">
    Diagnose stalled syncs, CBT resets, and connectivity errors
  </Card>
</CardGroup>


# Agent Configuration
Source: https://docs.xloud.tech/services/monitoring/admin-guide/agent-config

Deploy XIMP monitoring agents via XDeploy or manually, configure custom scrape targets, and manage per-node agent authentication tokens.

## Overview

Monitoring agents run on every managed node and are responsible for collecting host-level
metrics, forwarding logs, and reporting health status to the XIMP platform. Agents are
deployed automatically when a node is registered in XDeploy, or manually for nodes
outside XDeploy management.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Access to **XDeploy** (the **Xloud Dashboard**) for node management
  * SSH access to target nodes for manual agent installation
</Note>

***

## Deploy via XDeploy

<Steps>
  <Step title="Register the node" icon="plus">
    Navigate to **XDeploy → Infrastructure → Nodes → Add Node**. Provide the node
    IP, hostname, and role. XDeploy installs and configures the XIMP agent as part
    of the node onboarding process.
  </Step>

  <Step title="Verify agent registration" icon="circle-check">
    Navigate to **Monitor Center > Monitoring** (Agents, admin view). The new node appears
    with status **Active** within 2–3 minutes of registration.

    ```bash title="Check agent status via CLI" theme={null}
    ximp agent list --status all
    ```

    <Check>Agent shows `ACTIVE` and last-seen timestamp is within the scrape interval.</Check>
  </Step>

  <Step title="Configure custom scrape targets" icon="settings">
    If the node runs application-specific metric endpoints (e.g., a database exporter),
    add them as additional scrape targets:

    Navigate to **Monitor Center > Monitoring** (Add Scrape Target, admin view) and
    provide the endpoint URL and scrape interval.

    <Tip>
      Metric endpoints must be accessible from the XIMP collector nodes. Ensure
      security group rules permit inbound connections on the target port from the
      collector IP range.
    </Tip>
  </Step>
</Steps>

***

## Manual Agent Installation

For nodes not managed by XDeploy, install and configure the agent manually.

<Steps>
  <Step title="Install the agent package" icon="download">
    ```bash title="Install XIMP agent" theme={null}
    apt install ximp-agent
    ```
  </Step>

  <Step title="Retrieve an agent token" icon="key">
    Generate an agent authentication token from the XIMP portal:

    ```bash title="Create agent token" theme={null}
    ximp agent token create \
      --node <HOSTNAME> \
      --expires 365d
    ```

    Store the token securely — it cannot be retrieved after the command completes.
  </Step>

  <Step title="Configure the agent" icon="file-text">
    Edit `/etc/ximp/agent.yaml`:

    ```yaml title="/etc/ximp/agent.yaml" theme={null}
    server:
      endpoint: https://ximp.xloud.internal:9090
      auth_token: <AGENT_AUTH_TOKEN>

    node:
      hostname: compute-node-04
      labels:
        role: compute
        az: zone-a

    scrape_interval: 30s
    log_paths:
      - /var/log/syslog
      - /var/log/nova/nova-compute.log
    ```
  </Step>

  <Step title="Start and enable the agent" icon="play">
    ```bash title="Enable and start XIMP agent" theme={null}
    systemctl enable --now ximp-agent
    ```

    ```bash title="Verify agent is running" theme={null}
    systemctl status ximp-agent
    ```

    <Check>Agent service is `active (running)` and registers with XIMP within 30 seconds.</Check>
  </Step>
</Steps>

***

## Managing Agent Tokens

<Tabs>
  <Tab title="Token Operations" icon="key">
    ```bash title="List all agent tokens" theme={null}
    ximp agent token list
    ```

    ```bash title="Create a new token" theme={null}
    ximp agent token create --node compute-node-04 --expires 365d
    ```

    ```bash title="Revoke a token (e.g., for decommissioned nodes)" theme={null}
    ximp agent token revoke <TOKEN_ID>
    ```
  </Tab>

  <Tab title="Token Rotation" icon="refresh-cw">
    Rotate agent tokens periodically (recommended: every 365 days) or immediately
    if a node is decommissioned or suspected compromised.

    <Steps>
      <Step title="Generate a new token" icon="plus">
        ```bash title="Create replacement token" theme={null}
        ximp agent token create --node compute-node-04 --expires 365d
        ```
      </Step>

      <Step title="Update agent configuration" icon="settings">
        Update `/etc/ximp/agent.yaml` on the node with the new token, then restart:

        ```bash title="Restart agent with new token" theme={null}
        systemctl restart ximp-agent
        ```
      </Step>

      <Step title="Revoke the old token" icon="trash">
        ```bash title="Revoke old token" theme={null}
        ximp agent token revoke <OLD_TOKEN_ID>
        ```

        <Check>Old token is revoked and agent authenticates successfully with the new token.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Troubleshooting Agents

| Symptom               | Diagnostic                       | Resolution                                     |
| --------------------- | -------------------------------- | ---------------------------------------------- |
| Agent shows `offline` | `systemctl status ximp-agent`    | Restart service; check token validity          |
| Agent not in list     | `ximp agent list --status all`   | Verify token; re-register node                 |
| High scrape latency   | `ximp agent stats --node <HOST>` | Reduce scrape targets; scale up node resources |
| Authentication errors | Check `/var/log/ximp/agent.log`  | Rotate token; verify endpoint URL              |

***

## Next Steps

<CardGroup>
  <Card title="Metric Endpoints" href="/services/monitoring/admin-guide/metric-endpoints">
    Add custom application scrape targets to agents
  </Card>

  <Card title="Log Collection" href="/services/monitoring/admin-guide/log-collection">
    Configure which log files are forwarded by each agent
  </Card>

  <Card title="Security" href="/services/monitoring/admin-guide/security">
    Agent token management and dashboard access control
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/admin-guide/troubleshooting">
    Diagnose agent connectivity and metric collection issues
  </Card>
</CardGroup>


# Alert Channels
Source: https://docs.xloud.tech/services/monitoring/admin-guide/alert-channels

Configure XIMP notification channels — email, webhook, PagerDuty, and Slack — and set up multi-tier escalation policies for critical infrastructure alerts.

## Overview

Alert channels define how XIMP delivers notifications when alert rules fire. Channels
must be configured before creating alert rules that reference them. This page covers
creating and testing all supported channel types, and configuring escalation policies
for critical alerts.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Tabs>
  <Tab title="XDeploy" icon="server">
    SMTP email alerts and webhook notifications can be configured directly in XDeploy:

    <Steps>
      <Step title="Open Configuration" icon="settings">
        Navigate to **XDeploy → Configuration** and select the **Monitoring** tab.
      </Step>

      <Step title="Configure alert channels" icon="bell">
        Scroll to the **Alert Configuration** section and set the notification channels:

        | Setting               | Description                                                 |
        | --------------------- | ----------------------------------------------------------- |
        | **SMTP Server**       | Mail server hostname and port                               |
        | **SMTP From Address** | Sender email address for alert notifications                |
        | **SMTP Recipients**   | Comma-separated recipient email addresses                   |
        | **SMTP TLS Mode**     | TLS encryption mode (STARTTLS or SSL)                       |
        | **Webhook URL**       | Endpoint URL for Slack, Microsoft Teams, or custom webhooks |
      </Step>

      <Step title="Test the channels" icon="flask-conical">
        Use the **Test Email** and **Test Webhook** buttons to verify delivery before
        saving. This sends a test notification to confirm connectivity and credentials.

        <Tip>
          Always test channels immediately after configuration. Discovering a broken
          channel during a real incident delays response significantly.
        </Tip>
      </Step>

      <Step title="Save and deploy" icon="rocket">
        Click **Save Configuration**, then navigate to **XDeploy → Operations** and
        run a **Reconfigure** for the monitoring services.

        <Check>Alert channels are configured and tested.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Alert channels can also be configured through the XIMP CLI and the Monitoring
    Dashboard. See the sections below for channel-specific configuration.
  </Tab>
</Tabs>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * SMTP, webhook, PagerDuty, or Slack credentials depending on the channel type
</Note>

***

## Creating Alert Channels

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Alert Channels" icon="bell">
        Navigate to **Monitor Center > Monitoring** (Add Channel, admin view).
      </Step>

      <Step title="Configure the channel" icon="settings">
        Select the channel type and complete the required fields:

        | Channel Type  | Required Fields                                            |
        | ------------- | ---------------------------------------------------------- |
        | **Email**     | SMTP server, port, from address, recipient list, TLS mode  |
        | **Webhook**   | URL, HTTP method, optional Authorization header            |
        | **PagerDuty** | Integration key (from PagerDuty service), severity mapping |
        | **Slack**     | Incoming webhook URL, target channel name                  |
        | **Teams**     | Incoming webhook URL                                       |
      </Step>

      <Step title="Test the channel" icon="flask-conical">
        Click **Test** after saving to send a test notification. Verify delivery before
        assigning the channel to production alert rules.

        <Tip>
          Use the **Test** button immediately after creating a channel. Discovering a
          broken channel during a real incident delays response significantly.
        </Tip>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List configured channels" theme={null}
    ximp alert channel list
    ```

    ```bash title="Add email channel" theme={null}
    ximp alert channel create \
      --name ops-email \
      --type email \
      --to ops-team@example.com \
      --smtp smtp.example.com:587
    ```

    ```bash title="Add webhook channel" theme={null}
    ximp alert channel create \
      --name ops-webhook \
      --type webhook \
      --url https://hooks.example.com/ximp \
      --method POST
    ```

    ```bash title="Test a channel" theme={null}
    ximp alert channel test ops-webhook
    ```

    ```bash title="Delete a channel" theme={null}
    ximp alert channel delete ops-webhook
    ```
  </Tab>
</Tabs>

***

## Escalation Policies

Configure multi-tier escalation for critical infrastructure alerts:

<Steps>
  <Step title="Navigate to Escalation Policies" icon="arrow-up">
    Navigate to **Monitor Center > Monitoring** (Escalation Policies, admin view).
  </Step>

  <Step title="Define escalation tiers" icon="layers">
    | Tier       | Trigger              | Channel              | Delay      |
    | ---------- | -------------------- | -------------------- | ---------- |
    | **Tier 1** | Alert fires          | PagerDuty on-call    | Immediate  |
    | **Tier 2** | Not acknowledged     | Slack ops channel    | 5 minutes  |
    | **Tier 3** | Still unacknowledged | Page on-call manager | 15 minutes |

    <Warning>
      Escalation policies only apply to alert rules with the policy assigned. Verify
      every critical alert rule has an escalation policy — a rule without one stops
      at Tier 1 and never escalates.
    </Warning>
  </Step>

  <Step title="Assign to alert rules" icon="link">
    Open the target alert rule and set the **Escalation Policy** field. The policy
    applies to all future alert events for that rule.

    <Check>Alert rule references the escalation policy. Future alert events will escalate per the configured tiers.</Check>
  </Step>
</Steps>

***

## Channel Configuration Reference

<AccordionGroup>
  <Accordion title="Email (SMTP)" icon="mail">
    ```yaml title="Email channel configuration" theme={null}
    name: ops-email
    type: email
    smtp:
      server: smtp.example.com
      port: 587
      username: ximp@example.com
      password: <SMTP_PASSWORD>
      tls: starttls
    from: ximp@example.com
    recipients:
      - ops-team@example.com
      - oncall@example.com
    ```
  </Accordion>

  <Accordion title="PagerDuty" icon="bell">
    ```yaml title="PagerDuty channel configuration" theme={null}
    name: pagerduty-oncall
    type: pagerduty
    integration_key: <PAGERDUTY_INTEGRATION_KEY>
    severity_mapping:
      critical: critical
      warning: warning
      info: info
    ```

    Generate the integration key in PagerDuty under **Services → Integrations → Add Integration → Events API v2**.
  </Accordion>

  <Accordion title="Slack" icon="message-square">
    ```yaml title="Slack channel configuration" theme={null}
    name: slack-ops
    type: slack
    webhook_url: https://hooks.slack.com/services/T.../B.../...
    channel: "#ops-alerts"
    username: "XIMP Alerts"
    icon_emoji: ":alert:"
    ```

    Generate the webhook URL in Slack under **Apps → Incoming Webhooks → Add to Slack**.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Metrics & Alerts (User Guide)" href="/services/monitoring/user-guide/metrics-alerts">
    How users create alert rules that reference these channels
  </Card>

  <Card title="Alert Rules (Advanced)" href="/services/monitoring/user-guide/alert-rules">
    Silence rules, inhibition, and escalation policy usage
  </Card>

  <Card title="Security" href="/services/monitoring/admin-guide/security">
    Dashboard access control and credential management
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/admin-guide/troubleshooting">
    Diagnose channel delivery failures and SMTP connectivity issues
  </Card>
</CardGroup>


# XIMP Architecture
Source: https://docs.xloud.tech/services/monitoring/admin-guide/architecture

Understand the XIMP service layers — collection, storage, and serving — and how metrics, logs, and flow data move from infrastructure sources to operators.

## Overview

XIMP is composed of multiple service layers that collect, transport, store, and serve
observability data to operators and automation systems. Understanding the architecture
helps administrators plan deployments, troubleshoot ingestion issues, and optimize
resource allocation for the monitoring platform itself.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Tabs>
  <Tab title="XDeploy" icon="server">
    XIMP services are enabled and configured through the XDeploy Configuration panel:

    <Steps>
      <Step title="Open Configuration" icon="settings">
        Navigate to **XDeploy → Configuration** and select the **Monitoring** tab.
      </Step>

      <Step title="Enable monitoring components" icon="toggle-left">
        Toggle the monitoring services your deployment requires:

        | Setting                    | Description                                                 |
        | -------------------------- | ----------------------------------------------------------- |
        | **Enable Prometheus**      | Metric collection and time-series storage                   |
        | **Enable Grafana**         | Visualization dashboards and metric exploration             |
        | **Enable Central Logging** | Log collection, indexing, and search (OpenSearch + Fluentd) |
        | **CIS Compliance Level**   | Security compliance scanning tier                           |
        | **Scan Schedule**          | Automated compliance scan frequency                         |
      </Step>

      <Step title="Save and deploy" icon="rocket">
        Click **Save Configuration**, then navigate to **XDeploy → Operations** and
        run a **Deploy** or **Reconfigure** for the monitoring services.

        <Check>XIMP monitoring stack is deployed and collecting data.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Configure monitoring services by editing configuration files directly at
    `/etc/xavs/config/prometheus/`, `/etc/xavs/config/grafana/`, and
    `/etc/xavs/config/opensearch/`. See the individual component guides for
    detailed parameters.
  </Tab>
</Tabs>

***

## Architecture Diagram

```mermaid theme={null}
graph TD
    subgraph Sources
        A["Infrastructure Nodes\n(Compute, Storage, Network)"]
        B["Virtual Machines\n(XAVS Guest Agent)"]
        C["Applications\n(Metrics Endpoints)"]
        D["Network Devices\n(Flow Export)"]
    end

    subgraph Collection
        E["Metric Agents\n(Node Exporters)"]
        F["Log Collectors\n(File and Syslog)"]
        G["Flow Collector\n(NetFlow / sFlow)"]
    end

    subgraph Storage
        H["Metric Store\n(Time-Series DB)"]
        I["Log Index\n(Search Engine)"]
        J["Flow Store\n(Flow DB)"]
    end

    subgraph Serving
        K["Query API"]
        L["Alert Engine"]
        M["XIMP Dashboard"]
    end

    A --> E & F
    B --> E & F
    C --> E
    D --> G
    E --> H
    F --> I
    G --> J
    H & I & J --> K
    K --> L & M
    L --> N["Notification Channels\n(Email / Webhook / PagerDuty)"]
```

***

## Service Components

| Layer          | Component      | Role                                                                               |
| -------------- | -------------- | ---------------------------------------------------------------------------------- |
| **Collection** | Metric Agent   | Runs on each node; scrapes metrics from local services and exports to metric store |
| **Collection** | Log Collector  | Tails log files and forwards structured log events to the log index                |
| **Collection** | Flow Collector | Receives NetFlow/sFlow exports from network devices for traffic analysis           |
| **Storage**    | Metric Store   | High-performance time-series database for metric retention and query               |
| **Storage**    | Log Index      | Full-text search engine for log data with configurable retention                   |
| **Storage**    | Flow Store     | Database optimized for network flow record storage and aggregation                 |
| **Serving**    | Query API      | Unified query interface for metrics, logs, and flow data                           |
| **Serving**    | Alert Engine   | Evaluates rules against live metric and log streams; fires notifications           |
| **Serving**    | Dashboard      | Web interface for visualization, exploration, and alert management                 |

***

## Component Deep Dive

<AccordionGroup>
  <Accordion title="Metric Agent" icon="activity">
    The Metric Agent runs as a systemd service (`ximp-agent`) on every managed node.
    It scrapes metrics from:

    * Local node exporters (CPU, memory, disk, network)
    * Service-specific exporters registered as scrape targets
    * Application endpoints exposing metrics in the standard format

    Agents authenticate to the XIMP collector using per-node tokens. Tokens are
    generated during node registration and rotated on a configurable schedule.

    Default scrape interval: **30 seconds**
  </Accordion>

  <Accordion title="Log Collector" icon="file-text">
    The Log Collector tails configured log file paths and forwards events to the
    Log Index. It handles:

    * Multi-line log entries (stack traces, long SQL queries)
    * JSON-structured log parsing for service logs
    * Syslog reception for services that write to syslog instead of files

    Log collector configuration is defined in `/etc/ximp/log-sources.yaml` on each
    managed node and managed by XDeploy.
  </Accordion>

  <Accordion title="Alert Engine" icon="bell">
    The Alert Engine evaluates all active alert rules against the metric and log
    streams on each collection cycle. When a rule's condition is met for the
    full evaluation period:

    1. An alert event is created and stored
    2. Notifications are sent to all configured channels
    3. The alert remains active until the condition is no longer met (resolution event)

    The engine supports inhibition rules and silence matching to reduce notification
    noise during major incidents or maintenance windows.
  </Accordion>
</AccordionGroup>

***

## Deployment Topology

<Tabs>
  <Tab title="Standard (Single-Node XIMP)" icon="server">
    For environments up to \~50 monitored nodes, all XIMP services can run on a
    single dedicated node:

    * Metric Store, Log Index, Flow Store co-located
    * Query API and Dashboard on the same node
    * Alert Engine evaluates all rules
    * Estimated resources: 8 vCPU, 32 GB RAM, 2 TB SSD storage
  </Tab>

  <Tab title="Scaled (Multi-Node XIMP)" icon="layers">
    For larger environments (50+ nodes) or high-cardinality metric workloads,
    distribute XIMP services:

    * Metric Store: dedicated node with NVMe storage for write throughput
    * Log Index: dedicated node with SSD storage for index performance
    * Multiple collector nodes for horizontal scraping scale
    * Shared Query API and Dashboard behind load balancer

    XDeploy manages the multi-node XIMP deployment. Navigate to
    **XDeploy → Configuration → Monitoring** to configure the topology.
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Agent Configuration" href="/services/monitoring/admin-guide/agent-config">
    Deploy and configure monitoring agents on managed nodes
  </Card>

  <Card title="Metric Endpoints" href="/services/monitoring/admin-guide/metric-endpoints">
    Configure scrape targets and metric namespaces
  </Card>

  <Card title="Log Collection" href="/services/monitoring/admin-guide/log-collection">
    Set up log source paths and syslog forwarding
  </Card>

  <Card title="Retention Policies" href="/services/monitoring/admin-guide/retention">
    Configure how long metric and log data is retained
  </Card>
</CardGroup>


# DDoS Protection
Source: https://docs.xloud.tech/services/monitoring/admin-guide/ddos-protection

Configure XIMP DDoS detection thresholds, mitigation policies, whitelist exemptions, and review attack events and false-positive unblocking procedures.

## Overview

XIMP's DDoS prevention module analyzes traffic patterns and automatically mitigates
volumetric and application-layer attacks before they reach protected workloads.
The module operates in two modes: Monitor (detection only) and Mitigate (automatic
blocking).

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Network flow collection configured (see [Agent Configuration](/services/monitoring/admin-guide/agent-config))
  * Baseline traffic patterns established (minimum 72 hours of Monitor mode data recommended)
</Note>

***

## Configuring DDoS Protection Policy

<Steps>
  <Step title="Navigate to DDoS Protection" icon="shield">
    Navigate to **Monitor Center > Monitoring** (DDoS Policies, admin view).
  </Step>

  <Step title="Configure detection settings" icon="settings">
    | Setting                    | Description                                                        | Recommended                     |
    | -------------------------- | ------------------------------------------------------------------ | ------------------------------- |
    | **Detection Mode**         | `Monitor` (alerts only) or `Mitigate` (automatic blocking)         | Start with `Monitor`            |
    | **Threshold — Volumetric** | Inbound packet rate (pps) or bandwidth (Mbps) triggering detection | 2× baseline peak                |
    | **Threshold — SYN Flood**  | New TCP connections/second before SYN-cookie protection activates  | 10,000 conn/s                   |
    | **Block Duration**         | How long a detected source is blocked before reassessment          | 5 minutes                       |
    | **Whitelist**              | IP ranges exempt from DDoS mitigation                              | Monitoring systems, partner IPs |

    <Warning>
      Start with **Monitor** mode to baseline normal traffic patterns for at least
      72 hours before switching to **Mitigate** mode. Aggressive thresholds in
      mitigation mode may block legitimate traffic, causing customer-facing outages.
    </Warning>
  </Step>

  <Step title="Add whitelist entries" icon="list">
    Add IP ranges that should never be blocked regardless of traffic volume:

    * XIMP monitoring system IPs (prevent self-blocking)
    * Partner or customer IP ranges with legitimate high-volume traffic
    * Internal automation systems

    Navigate to **Monitor Center > Monitoring** (DDoS Whitelist, admin view).
  </Step>

  <Step title="Switch to Mitigate mode" icon="shield">
    After at least 72 hours of Monitor mode with no false positives:

    1. Review the alert history for any false positive detections
    2. Add any flagged legitimate sources to the whitelist
    3. Switch the policy to **Mitigate** mode

    <Check>Policy shows Mitigate mode active. Check the DDoS Events feed to confirm no legitimate traffic is being blocked.</Check>
  </Step>
</Steps>

***

## Reviewing DDoS Events

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Monitor Center > Monitoring** (DDoS Events, admin view) to review detected and
    mitigated attacks:

    | Column           | Description                                                  |
    | ---------------- | ------------------------------------------------------------ |
    | **Time**         | When the detection occurred                                  |
    | **Source IP**    | Originating attack IP or range                               |
    | **Type**         | `volumetric`, `syn-flood`, `application-layer`, or `anomaly` |
    | **Peak Rate**    | Maximum observed attack bandwidth or packet rate             |
    | **Status**       | `Active`, `Mitigated`, or `Expired`                          |
    | **Action Taken** | `Alert only` (Monitor mode) or `Blocked` (Mitigate mode)     |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List recent DDoS events" theme={null}
    ximp security ddos events --last 24h
    ```

    ```bash title="View details of a specific event" theme={null}
    ximp security ddos event show <EVENT_ID>
    ```

    ```bash title="List currently blocked sources" theme={null}
    ximp security ddos blocklist
    ```
  </Tab>
</Tabs>

***

## Handling False Positives

If a legitimate source is incorrectly blocked:

<Steps>
  <Step title="Identify the blocked source" icon="search">
    ```bash title="Check if a specific IP is blocked" theme={null}
    ximp security ddos blocklist | grep <IP_ADDRESS>
    ```
  </Step>

  <Step title="Unblock the source" icon="unlock">
    ```bash title="Unblock a specific source" theme={null}
    ximp security ddos unblock --source 203.0.113.50
    ```
  </Step>

  <Step title="Add to whitelist to prevent future blocks" icon="list">
    Navigate to **Monitor Center > Monitoring** (DDoS Whitelist, admin view) and add the IP range
    of the legitimate source with a descriptive comment.

    <Check>Source is unblocked and whitelist entry prevents future false positives.</Check>
  </Step>
</Steps>

***

## Next Steps

<CardGroup>
  <Card title="Network Monitoring (User Guide)" href="/services/monitoring/user-guide/network-monitoring">
    User-level network traffic analysis for attack investigation
  </Card>

  <Card title="Alert Channels" href="/services/monitoring/admin-guide/alert-channels">
    Configure notification channels for DDoS detection events
  </Card>

  <Card title="Security" href="/services/monitoring/admin-guide/security">
    Overall XIMP security configuration including access control
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/admin-guide/troubleshooting">
    Diagnose false positive blocks and detection threshold tuning
  </Card>
</CardGroup>


# Log Collection
Source: https://docs.xloud.tech/services/monitoring/admin-guide/log-collection

Configure XIMP log source paths, syslog forwarding, and log format parsers for centralized log ingestion across all managed infrastructure nodes.

## Overview

XIMP collects logs from all registered nodes via file-based collection and syslog
forwarding. Centralized log data flows into the log index for full-text search and
alert-based detection. This page covers configuring log sources and troubleshooting
ingestion issues.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * XIMP agents deployed on target nodes (see [Agent Configuration](/services/monitoring/admin-guide/agent-config))
</Note>

***

## Configuring Log Sources

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="View configured log sources" icon="list">
        Navigate to **Monitor Center > Logging** (Log Sources, admin view). Each log source
        defines a file path pattern, the node scope it applies to, and the expected
        format.
      </Step>

      <Step title="Add a log source" icon="plus">
        Click **Add Log Source** and provide:

        | Field             | Description                                                            |
        | ----------------- | ---------------------------------------------------------------------- |
        | **Path Pattern**  | Glob pattern for the log file (e.g., `/var/log/nova/*.log`)            |
        | **Node Selector** | Applies this source to nodes matching the label (e.g., `role=compute`) |
        | **Format**        | `json`, `plain`, or `multiline` (for stack traces)                     |
        | **Service Label** | Tag applied to all ingested events for filtering                       |
      </Step>

      <Step title="Verify ingestion" icon="circle-check">
        After saving, navigate to **Monitor Center > Logging** (admin view) and filter
        by the new service label. Entries should appear within the configured scrape interval.

        <Check>Log events appear in Log Explorer with correct service label and timestamp.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Configuration File" icon="file-text">
    Log sources can also be defined in `/etc/ximp/log-sources.yaml` on the managed node:

    ```yaml title="/etc/ximp/log-sources.yaml" theme={null}
    sources:
      - path: /var/log/nova/*.log
        format: plain
        labels:
          service: nova
          role: compute

      - path: /var/log/cinder/*.log
        format: plain
        labels:
          service: cinder

      - path: /var/log/docker/containers/**/*.log
        format: json
        labels:
          service: docker

      - path: /var/log/syslog
        format: plain
        labels:
          service: syslog
    ```

    Reload the agent after updating the configuration:

    ```bash title="Reload agent configuration" theme={null}
    systemctl reload ximp-agent
    ```
  </Tab>
</Tabs>

***

## Syslog Forwarding

Services that write to syslog rather than log files can forward directly to XIMP's
syslog receiver.

<Steps>
  <Step title="Configure rsyslog forwarding" icon="send">
    Add a forwarding rule to `/etc/rsyslog.d/99-ximp.conf` on the source node:

    ```bash title="/etc/rsyslog.d/99-ximp.conf" theme={null}
    # Forward all facility/severity combinations to XIMP via TCP
    *.* @@ximp.xloud.internal:5140
    ```

    <Note>
      Use TCP forwarding (`@@`) for reliable delivery. UDP forwarding (`@`) may drop
      messages under high log volume and is not recommended for production.
    </Note>
  </Step>

  <Step title="Reload rsyslog" icon="refresh-cw">
    ```bash title="Reload rsyslog" theme={null}
    systemctl reload rsyslog
    ```
  </Step>

  <Step title="Verify ingestion" icon="circle-check">
    Navigate to **Monitor Center > Logging** (admin view) and filter by `service:syslog`
    and the source hostname. Entries should appear within 60 seconds.

    <Check>Syslog events appear in Log Explorer with correct host and timestamp.</Check>
  </Step>
</Steps>

***

## Log Format Parsers

Configure parsers for structured log formats to enable field-level filtering in
Log Analytics:

| Format      | Configuration    | Notes                                                      |
| ----------- | ---------------- | ---------------------------------------------------------- |
| `json`      | Automatic        | Fields extracted automatically from JSON keys              |
| `plain`     | Default          | Full-text search only; no structured field extraction      |
| `multiline` | Requires pattern | Stack traces and multi-line entries joined before indexing |

For multiline logs (Java stack traces, Python tracebacks):

```yaml title="Multiline log source configuration" theme={null}
sources:
  - path: /var/log/app/*.log
    format: multiline
    multiline:
      start_pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
      negate: false
      match: after
    labels:
      service: app-service
```

***

## Next Steps

<CardGroup>
  <Card title="Agent Configuration" href="/services/monitoring/admin-guide/agent-config">
    Deploy agents that run the log collection defined here
  </Card>

  <Card title="Retention Policies" href="/services/monitoring/admin-guide/retention">
    Configure how long collected log data is retained
  </Card>

  <Card title="Log Analytics (User Guide)" href="/services/monitoring/user-guide/log-analytics">
    How tenants query and create alerts from the logs you've configured
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/admin-guide/troubleshooting">
    Diagnose log ingestion backlogs and missing log data
  </Card>
</CardGroup>


# Metric Endpoints
Source: https://docs.xloud.tech/services/monitoring/admin-guide/metric-endpoints

Configure XIMP metric scrape targets, manage endpoint discovery, and organize metrics by namespace for alert rules and dashboards.

## Overview

XIMP scrapes metrics from standardized metric endpoints exposed by services across
the infrastructure. Most infrastructure services are auto-detected via agent registration.
This page covers configuring additional endpoints for application-specific exporters
and managing the namespace organization of collected metrics.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Target endpoints must be reachable from the XIMP collector nodes
</Note>

***

## Viewing Configured Targets

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Monitor Center > Monitoring** (Scrape Targets, admin view). Each entry shows:

    * **URL**: The endpoint being scraped
    * **Status**: `UP` (healthy) or `DOWN` (not reachable)
    * **Last Scrape**: Timestamp of the most recent successful scrape
    * **Labels**: Metadata tags applied to all metrics from this target
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all scrape targets" theme={null}
    ximp target list
    ```

    ```bash title="Check target health with details" theme={null}
    ximp target health --verbose
    ```

    Targets with status `DOWN` are not being scraped. Review connectivity and
    authentication configuration for affected targets.
  </Tab>
</Tabs>

***

## Adding Custom Metric Endpoints

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Monitor Center > Monitoring** (Add Scrape Target, admin view):

    | Field               | Description                                                           |
    | ------------------- | --------------------------------------------------------------------- |
    | **URL**             | Full endpoint URL (e.g., `http://10.0.1.71:9399/metrics`)             |
    | **Scrape Interval** | How often to collect (default: `30s`)                                 |
    | **Labels**          | Key-value metadata tags for the target (e.g., `service=xloud-filefs`) |
    | **TLS Config**      | CA certificate path for HTTPS endpoints                               |
    | **Auth**            | Basic auth credentials or bearer token for secured endpoints          |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Add scrape target" theme={null}
    ximp target add \
      --url http://10.0.1.71:9399/metrics \
      --interval 30s \
      --label service=xloud-filefs \
      --label host=xd1
    ```

    ```bash title="Add target with basic auth" theme={null}
    ximp target add \
      --url https://10.0.1.71:9291/metrics \
      --interval 60s \
      --label service=storage-cluster \
      --auth-user admin \
      --auth-password <PASSWORD>
    ```

    ```bash title="Remove a target" theme={null}
    ximp target remove <TARGET_ID>
    ```
  </Tab>
</Tabs>

***

## Metric Namespaces

XIMP organizes metrics by namespace. Use the namespace prefix when writing alert
rules and dashboard queries:

| Namespace       | Source                  | Example Metrics                            |
| --------------- | ----------------------- | ------------------------------------------ |
| `xloud_compute` | Compute node agents     | `cpu_utilization`, `memory_used_bytes`     |
| `xloud_storage` | XSDS cluster            | `pool_used_bytes`, `osd_latency_ms`        |
| `xloud_network` | Network agents          | `interface_rx_bytes`, `packet_loss_pct`    |
| `xloud_vm`      | XAVS Guest Agent        | `vm_cpu_usage`, `vm_memory_rss`            |
| `xloud_filefs`  | File FS service         | `requests_total`, `guestfs_handles_active` |
| `xdr`           | Disaster recovery agent | `replication_lag_seconds`, `site_health`   |

***

## Firewall Requirements

Metric endpoints must be accessible from the XIMP collector nodes. Configure
security groups or firewall rules to permit:

| Source            | Destination                  | Port                   | Protocol |
| ----------------- | ---------------------------- | ---------------------- | -------- |
| XIMP collector IP | Target node                  | Configured scrape port | TCP      |
| XIMP collector IP | 9100 (node exporter default) | All nodes              | TCP      |
| XIMP collector IP | 9283 (XSDS metrics)          | Storage nodes          | TCP      |

<Note>
  XIMP collector IPs are listed under **Monitor Center > Monitoring** (Collectors, admin view).
  Update security group rules to include all collector IPs when adding new collectors
  during scale-out.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Agent Configuration" href="/services/monitoring/admin-guide/agent-config">
    Configure agents and their default scrape targets on managed nodes
  </Card>

  <Card title="Alert Channels" href="/services/monitoring/admin-guide/alert-channels">
    Configure notification channels using the metric namespaces defined here
  </Card>

  <Card title="Retention Policies" href="/services/monitoring/admin-guide/retention">
    Configure how long metric data from each namespace is retained
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/admin-guide/troubleshooting">
    Diagnose targets in DOWN state and missing metric series
  </Card>
</CardGroup>


# Retention Policies
Source: https://docs.xloud.tech/services/monitoring/admin-guide/retention

Configure XIMP metric and log retention periods to balance historical depth against storage costs for raw, downsampled, and log data.

## Overview

Retention policies control how long XIMP stores metric, log, and flow data. Longer
retention enables deeper historical analysis and incident review; shorter retention
reduces storage costs. Downsampled metric retention provides long-term trend data
at a fraction of the storage cost of raw metrics.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Confirmation from compliance and operations teams on required retention periods before reducing existing values
</Note>

***

## Recommended Retention Settings

| Data Type                     | Recommended Retention | Storage Impact              | Notes                                |
| ----------------------------- | --------------------- | --------------------------- | ------------------------------------ |
| Metrics (raw, 30s resolution) | 30 days               | \~2 GB per node per month   | Full resolution for recent incidents |
| Metrics (5-min downsampled)   | 1 year                | \~200 MB per node per month | Medium-term trend analysis           |
| Metrics (1-hour downsampled)  | 3 years               | \~20 MB per node per month  | Long-term capacity planning          |
| Logs                          | 90 days               | Varies by log verbosity     | Incident review and audit            |
| Flow data                     | 30 days               | \~5 GB per Gbps per month   | Network forensics                    |

***

## Configuring Retention

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Monitor Center > Monitoring** (Retention, admin view) and configure
    retention for each data type.

    1. Select the data type (Metrics Raw, Metrics Downsampled, Logs, Flows)
    2. Enter the retention duration (e.g., `30d`, `90d`, `1y`)
    3. Click **Apply**

    <Warning>
      Reducing retention periods deletes historical data immediately and irreversibly.
      Confirm with compliance and operations teams before shortening any retention window.
      Data deleted by retention policy cannot be recovered.
    </Warning>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="View current retention policies" theme={null}
    ximp retention list
    ```

    ```bash title="Set raw metric retention" theme={null}
    ximp retention set \
      --type metrics-raw \
      --duration 30d
    ```

    ```bash title="Set log retention" theme={null}
    ximp retention set \
      --type logs \
      --duration 90d
    ```

    ```bash title="Set flow data retention" theme={null}
    ximp retention set \
      --type flows \
      --duration 30d
    ```
  </Tab>
</Tabs>

***

## Downsampling Configuration

Downsampling aggregates raw metric points into lower-resolution summaries at configurable
intervals. This enables long-term retention at a fraction of the storage cost.

| Downsampling Level | Resolution | Applied After | Storage vs Raw   |
| ------------------ | ---------- | ------------- | ---------------- |
| Level 1            | 5 minutes  | 7 days        | \~10× reduction  |
| Level 2            | 1 hour     | 30 days       | \~120× reduction |

<Note>
  Downsampled data retains statistical aggregates: min, max, sum, count, and average.
  Exact per-second values are not recoverable after the raw data retention period expires.
  Ensure raw retention is long enough for the typical incident investigation window.
</Note>

Configure downsampling rules in **Monitor Center > Monitoring** (Downsampling, admin view).

***

## Compliance Retention Requirements

For regulatory compliance, consult the following minimum retention guidelines:

| Regulation    | Minimum Log Retention    | Notes                                  |
| ------------- | ------------------------ | -------------------------------------- |
| ISO 27001     | 1 year                   | Security events and access logs        |
| SOC 2 Type II | 1 year                   | Covers audit period plus review buffer |
| PCI DSS       | 1 year (3 months online) | Transaction-related system logs        |
| HIPAA         | 6 years                  | Healthcare system access logs          |

<Tip>
  Set XIMP log retention to match your most demanding regulatory requirement. Storage
  cost for log data is typically dominated by verbosity — reduce log levels to `WARNING`
  on non-critical services to reduce volume without losing important events.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Log Collection" href="/services/monitoring/admin-guide/log-collection">
    Control log ingestion volume by configuring which services are collected
  </Card>

  <Card title="Architecture" href="/services/monitoring/admin-guide/architecture">
    Understand storage layer sizing for your retention requirements
  </Card>

  <Card title="Security" href="/services/monitoring/admin-guide/security">
    Access controls for retention policy management
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/admin-guide/troubleshooting">
    Diagnose storage pressure caused by high-cardinality metrics or high log volume
  </Card>
</CardGroup>


# Security
Source: https://docs.xloud.tech/services/monitoring/admin-guide/security

Secure XIMP with agent token management, dashboard role-based access control via Xloud identity, and TLS certificate lifecycle management.

## Overview

XIMP security encompasses agent authentication via per-node tokens, dashboard access
control through Xloud identity roles, and TLS certificate lifecycle management for
all platform communications.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Agent Authentication

Each XIMP agent authenticates to the collector using a unique per-node token.
Rotate tokens periodically and immediately when a node is decommissioned or
suspected compromised.

<Tabs>
  <Tab title="Token Lifecycle" icon="key">
    ```bash title="Generate a new agent token" theme={null}
    ximp agent token create --node compute-node-04 --expires 365d
    ```

    ```bash title="List all agent tokens with expiry" theme={null}
    ximp agent token list
    ```

    ```bash title="Revoke an agent token" theme={null}
    ximp agent token revoke <TOKEN_ID>
    ```

    <Warning>
      Revoking a token immediately disconnects the associated agent. Ensure the
      replacement token is deployed to the agent configuration before revoking
      the old one, or the node will stop reporting metrics.
    </Warning>
  </Tab>

  <Tab title="Token Rotation Procedure" icon="refresh-cw">
    <Steps>
      <Step title="Generate new token" icon="plus">
        ```bash title="Create replacement token" theme={null}
        ximp agent token create --node <HOSTNAME> --expires 365d
        ```
      </Step>

      <Step title="Update agent configuration" icon="file-text">
        Update `/etc/ximp/agent.yaml` on the node with the new token value:

        ```yaml title="/etc/ximp/agent.yaml" theme={null}
        server:
          auth_token: <NEW_TOKEN>
        ```
      </Step>

      <Step title="Restart agent and verify" icon="circle-check">
        ```bash title="Restart agent" theme={null}
        systemctl restart ximp-agent
        ```

        ```bash title="Verify agent reconnected" theme={null}
        ximp agent list --node <HOSTNAME>
        ```

        <Check>Agent shows `ACTIVE` with a recent last-seen timestamp.</Check>
      </Step>

      <Step title="Revoke old token" icon="trash">
        ```bash title="Revoke old token" theme={null}
        ximp agent token revoke <OLD_TOKEN_ID>
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Dashboard Access Control

XIMP dashboard access is controlled through Xloud identity roles. Assign roles
based on job function to enforce least-privilege access.

| Role                | Access Level                                       | Typical Assignees        |
| ------------------- | -------------------------------------------------- | ------------------------ |
| `monitoring-viewer` | Read-only — dashboards and alert history           | Developers, stakeholders |
| `monitoring-editor` | Create and edit dashboards, rules, and channels    | Operations engineers     |
| `monitoring-admin`  | Full access — agents, retention, security settings | Platform administrators  |

Assign roles through **XDeploy → Identity → Role Assignments**.

<Tip>
  Use the `monitoring-viewer` role for application teams who need to observe their
  service metrics without the ability to modify alert rules that affect other teams.
</Tip>

***

## TLS Configuration

All XIMP communication uses TLS:

* Agent-to-collector: TLS 1.3 minimum
* Dashboard and API: TLS 1.3 minimum, HSTS enabled
* Internal service communication: mTLS for collector-to-store traffic

Certificates are managed by XDeploy and renewed automatically 30 days before expiry.

<Tabs>
  <Tab title="Certificate Status" icon="shield">
    ```bash title="Check all certificate expiry dates" theme={null}
    ximp tls status
    ```

    Expected output shows certificate subjects, expiry dates, and days remaining.
    Certificates within 30 days of expiry trigger an automatic renewal.
  </Tab>

  <Tab title="Manual Renewal" icon="refresh-cw">
    If automatic renewal fails (e.g., due to DNS misconfiguration):

    ```bash title="Manually renew all certificates" theme={null}
    ximp tls renew --all
    ```

    ```bash title="Renew a specific certificate" theme={null}
    ximp tls renew --component collector
    ```

    After renewal, verify the new certificate is in effect:

    ```bash title="Verify renewed certificate" theme={null}
    ximp tls status --component collector
    ```

    <Check>Certificate expiry date is at least 90 days in the future.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Agent Configuration" href="/services/monitoring/admin-guide/agent-config">
    Deploy and configure agents whose tokens are managed here
  </Card>

  <Card title="Alert Channels" href="/services/monitoring/admin-guide/alert-channels">
    Secure notification channel credentials and SMTP authentication
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/admin-guide/troubleshooting">
    Diagnose TLS and authentication errors
  </Card>

  <Card title="Xloud Identity" href="/services/identity/admin-guide">
    Manage the Xloud identity roles used for XIMP access control
  </Card>
</CardGroup>


# Troubleshooting
Source: https://docs.xloud.tech/services/monitoring/admin-guide/troubleshooting

Diagnose XIMP administrative issues — high cardinality metric performance problems, log ingestion backlogs, missing dashboard data, and scrape target failures.

## Overview

This page covers administrator-level XIMP troubleshooting. For user-facing issues
such as alert delivery failures and missing metrics on dashboards, see the
[XIMP User Guide Troubleshooting](/services/monitoring/user-guide/troubleshooting) page.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Access to XIMP CLI and management interfaces
</Note>

***

## Common Issues

<AccordionGroup>
  <Accordion title="High cardinality causing metric store performance issues" icon="gauge">
    **Cause**: Metric labels with unbounded values (e.g., request IDs, user IDs, or
    ephemeral container names) create millions of unique metric series, degrading
    query performance and consuming excessive storage.

    **Diagnosis**:

    ```bash title="List highest-cardinality metric series" theme={null}
    ximp metric cardinality top --limit 20
    ```

    **Resolution**:
    Drop or relabel high-cardinality labels in the scrape configuration:

    ```yaml title="Relabel config — drop high-cardinality label" theme={null}
    relabel_configs:
      - source_labels: [request_id]
        action: drop
    ```

    Apply via:

    ```bash title="Apply relabel configuration" theme={null}
    ximp target update <TARGET_ID> --relabel-file relabel.yaml
    ```

    <Warning>
      Dropping a label is irreversible for historical data. The label will be absent
      from future ingested metrics. Consider using `labelmap` to replace high-cardinality
      values with aggregate labels instead of dropping them entirely.
    </Warning>
  </Accordion>

  <Accordion title="Log ingestion backlog" icon="clock">
    **Cause**: Log volume exceeds the collector's processing capacity, causing a write
    backlog and delayed delivery to the search index.

    **Diagnosis**:

    ```bash title="Check ingestion queue depth" theme={null}
    ximp log ingest-status
    ```

    **Resolution**:

    * Reduce log verbosity on high-volume services (set log level to `WARNING` instead of `DEBUG`):
      ```bash title="Example: reduce Nova log level" theme={null}
      docker exec nova_api crudini --set /etc/nova/nova.conf DEFAULT debug false
      ```
    * Increase log collector worker count in the XIMP configuration:
      Navigate to **Monitor Center > Logging** (Collector Settings, admin view)
    * Add a second log collector node through XDeploy for horizontal scaling

    <Tip>
      A single high-verbosity service at DEBUG level can generate more log volume than
      100 services at INFO. Identify the top log emitters:
      `ximp log stats top-emitters --last 1h`
    </Tip>
  </Accordion>

  <Accordion title="Scrape target in DOWN state" icon="activity">
    **Cause**: The scrape target is unreachable — firewall blocking, service down,
    or authentication failure.

    **Diagnosis**:

    ```bash title="Check specific target health" theme={null}
    ximp target health --target <URL> --verbose
    ```

    Common causes:

    | Symptom                 | Cause                              | Resolution                                     |
    | ----------------------- | ---------------------------------- | ---------------------------------------------- |
    | Connection refused      | Service not running on target port | Verify service is running; check port          |
    | Timeout                 | Firewall blocking                  | Add inbound rule for XIMP collector IP         |
    | 401 Unauthorized        | Invalid auth credentials           | Update auth config in target definition        |
    | 503 Service Unavailable | Service overloaded                 | Review service health; reduce scrape frequency |
  </Accordion>

  <Accordion title="XIMP metric store disk full" icon="hard-drive">
    **Cause**: Metric volume has exceeded the allocated storage for the metric store.
    This can occur from high cardinality, insufficient retention management, or
    unexpected metric bursts.

    **Diagnosis**:

    ```bash title="Check metric store disk usage" theme={null}
    ximp storage status
    ```

    **Resolution** (in order of preference):

    1. Reduce raw metric retention to free space immediately:
       ```bash title="Reduce raw retention to 15 days (emergency)" theme={null}
       ximp retention set --type metrics-raw --duration 15d
       ```
    2. Identify and drop high-cardinality series (see above)
    3. Expand storage on the metric store node through XDeploy
    4. Add a second metric store node for horizontal capacity
  </Accordion>

  <Accordion title="Dashboard shows 'No data' for a metric" icon="gauge">
    **Cause**: The scrape target is down, the agent is offline, or the metric name has
    changed after a software update.

    **Diagnosis**:

    1. Check target health: `ximp target health --target <URL>`
    2. Verify agent is active: `ximp agent list --node <HOSTNAME>`
    3. Search for the metric by prefix to find renamed metrics:
       ```bash title="Search metrics by prefix" theme={null}
       ximp metric search --prefix xloud_compute_cpu
       ```

    If the metric was renamed in a recent software update, update dashboard queries
    and alert rules to use the new metric name.
  </Accordion>
</AccordionGroup>

***

## Diagnostics Reference

| Issue            | Diagnostic Command                       |
| ---------------- | ---------------------------------------- |
| Cardinality      | `ximp metric cardinality top --limit 20` |
| Log backlog      | `ximp log ingest-status`                 |
| Target DOWN      | `ximp target health --verbose`           |
| Storage usage    | `ximp storage status`                    |
| Agent offline    | `ximp agent list --status offline`       |
| Top log emitters | `ximp log stats top-emitters --last 1h`  |

***

## Next Steps

<CardGroup>
  <Card title="Agent Configuration" href="/services/monitoring/admin-guide/agent-config">
    Review and fix agent configuration that may be causing issues
  </Card>

  <Card title="Retention Policies" href="/services/monitoring/admin-guide/retention">
    Adjust retention settings to address storage pressure
  </Card>

  <Card title="Metric Endpoints" href="/services/monitoring/admin-guide/metric-endpoints">
    Review and fix scrape target configurations
  </Card>

  <Card title="User Guide Troubleshooting" href="/services/monitoring/user-guide/troubleshooting">
    User-facing issues — alerts not firing, log delays
  </Card>
</CardGroup>


# Infrastructure Monitoring
Source: https://docs.xloud.tech/services/monitoring/index

End-to-end observability for infrastructure, applications, networks, and security with the Xloud Infrastructure Monitoring Platform (XIMP).

The Xloud Infrastructure Monitoring Platform (XIMP) delivers comprehensive, end-to-end
observability across your entire environment — from bare-metal hosts and virtual machines
to applications, networks, and security events. Proactive alerting, unified dashboards,
and intelligent analytics give operations teams the insight needed to detect, diagnose,
and resolve issues before they impact workloads.

<Card title="XIMP — Infrastructure Monitoring Platform" icon="external-link" href="https://xloud.tech/XIMP">
  Full platform overview, module descriptions, and datasheet on xloud.tech
</Card>

***

<p>XIMP Documentation</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/monitoring/user-guide">
    Navigate dashboards, configure alert rules, analyze logs, and monitor network traffic.
    Step-by-step workflows for operations teams and application owners.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/monitoring/admin-guide">
    Deploy and configure monitoring agents, set up metric endpoints, configure log
    collection pipelines, alert channels, and retention policies.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/monitoring/cli-reference">
    Query metrics with PromQL, manage alerts via Alertmanager API, and retrieve
    Ceilometer resource metrics from the command line.
  </Card>

  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Monitor virtual machine instances and hypervisor-level metrics through XIMP.
  </Card>
</CardGroup>

***

<p>Platform Modules</p>

<CardGroup>
  <Card title="Metrics Collection" icon="chart-bar" href="/services/monitoring/admin-guide/metric-endpoints">
    Collect time-series metrics from infrastructure hosts, virtual machines, containers,
    and application endpoints with configurable scrape intervals.
  </Card>

  <Card title="Application Observability" icon="activity" href="/services/monitoring/user-guide/dashboards">
    Trace application performance, measure latency, and correlate errors across distributed
    services with end-to-end request visibility.
  </Card>

  <Card title="Log Analytics" icon="file-text" href="/services/monitoring/user-guide/log-analytics">
    Centralize, index, and search log streams from all components. Detect anomalies and
    correlate events across the entire infrastructure stack.
  </Card>

  <Card title="Network Traffic Monitoring" icon="network" href="/services/monitoring/user-guide/network-monitoring">
    Deep packet inspection and flow analysis to identify bandwidth consumers, detect
    anomalous traffic patterns, and baseline normal network behavior.
  </Card>

  <Card title="Security & IDS" icon="shield" href="/services/monitoring/admin-guide/security">
    Intrusion detection, behavioral analysis, and security event correlation to surface
    threats and unauthorized access attempts in real time.
  </Card>

  <Card title="DDoS Prevention" icon="shield-alert" href="/services/monitoring/admin-guide/ddos-protection">
    Automated traffic analysis and mitigation to detect and block volumetric and
    application-layer denial-of-service attacks before they reach workloads.
  </Card>

  <Card title="Alerting & Notifications" icon="bell" href="/services/monitoring/user-guide/alert-rules">
    Rule-based alerting with configurable thresholds, severity levels, and notification
    channels including email, webhook, and PagerDuty.
  </Card>

  <Card title="Configuration Management" icon="settings" href="/services/monitoring/admin-guide/agent-config">
    Track configuration changes across all managed nodes with drift detection and
    compliance reporting against defined baselines.
  </Card>
</CardGroup>

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Monitor instance health, resource utilization, and live performance metrics
  </Card>

  <Card title="Disaster Recovery" icon="rotate-ccw" href="/services/disaster-recovery/index">
    Combine monitoring insights with DR automation for resilient workload protection
  </Card>

  <Card title="Xloud Networking" icon="network" href="/services/networking/index">
    Network traffic monitoring and security event correlation for virtual networks
  </Card>
</CardGroup>


# Activity Log
Source: https://docs.xloud.tech/services/monitoring/user-guide/activity-log

See every action taken across your Xloud Platform — who did what, when, and with what outcome — from a single searchable timeline in Monitor Center.

## Overview

Activity Log shows a live timeline of every action taken across your Xloud Platform. Each
entry tells you **who** did it, **what** they did, **which resource** was affected, and
**whether it succeeded** — so you can audit activity, investigate changes, and spot
problems at a glance.

<Warning>
  Activity Log is available in the **administrator view only**, under
  **Monitor Center → Activity Log**. If the page shows an empty state asking
  you to enable Central Logging, ask your administrator to turn it on before
  you can use this feature.
</Warning>

***

## Video Walkthrough

<iframe title="How to Audit Activity on Xloud" />

***

## Opening Activity Log

<Steps>
  <Step title="Switch to the admin view" icon="shield">
    Sign in to the Xloud Dashboard with an administrator account.
  </Step>

  <Step title="Open Monitor Center → Activity Log" icon="clipboard-list">
    From the left navigation, expand **Monitor Center** and click **Activity Log**.

    <Check>The page loads with summary cards at the top, a filter bar below, and a live event table.</Check>
  </Step>
</Steps>

***

## What You See at the Top

Four summary cards show a rolling count of events matching your current filters:

| Card               | Shows                                                                         |
| ------------------ | ----------------------------------------------------------------------------- |
| **Total Events**   | Every event in the current view                                               |
| **Completed**      | Actions that succeeded                                                        |
| **Warnings (4xx)** | Actions that were rejected — for example, missing permission or a bad request |
| **Failed (5xx)**   | Actions that hit a server-side error                                          |

The counts update live as new events arrive.

***

## Filtering the Timeline

Use the filter bar to narrow the timeline down to the events you need. Filters stack,
so you can combine them freely.

| Filter              | What it does                                                                                               |
| ------------------- | ---------------------------------------------------------------------------------------------------------- |
| **Service**         | Show only events from one service — Compute, Storage, Network, Identity, Image, DNS, Key Manager, and more |
| **Action**          | Show only a specific type of action — create, delete, update, reboot, migrate, snapshot, and so on         |
| **Resource Type**   | Show only actions against a specific kind of resource (instance, volume, network, image, etc.)             |
| **Date Range**      | Scope the timeline to a start and end date-time                                                            |
| **Search URL / ID** | Text search across request URLs and resource IDs — press **Enter** to apply                                |

<Tip>
  To find out who deleted a specific resource, set **Action** to `delete` and paste the
  resource ID into the **Search URL / ID** box.
</Tip>

***

## Event Table

Each row in the table is one action. At a glance you can see:

| Column        | What it tells you                                                   |
| ------------- | ------------------------------------------------------------------- |
| **Activity**  | The service tag and a short description of the action               |
| **Target**    | The resource that was affected (hover for the full ID)              |
| **Status**    | Completed, Failed, Running, or Queued                               |
| **Initiator** | Where the request came from — Dashboard, CLI, or API                |
| **User**      | The person or service account that made the request                 |
| **Project**   | The project the action was scoped to                                |
| **Started**   | When the action happened — `4s ago`, `2m ago`, hover for exact time |
| **Duration**  | How long the action took                                            |

***

## Live Mode

A **Live** switch on the right side of the filter bar (paired with a spinning sync
icon) controls auto-refresh. When it's on, the timeline updates every few seconds and
new rows briefly highlight green as they arrive. Turn it off when you want a stable
snapshot, or use the refresh icon next to it for a one-off update.

<Info>
  Live updates pause automatically when you switch to a different browser tab or open
  the event details — so the page doesn't waste bandwidth in the background.
</Info>

***

## Inspecting an Event

Click any row to open the **Detail** panel on the right. It shows the full context for
that action:

* Activity name, service, and action type
* User and project (names, with UUIDs visible on hover)
* **Source IP** the request came from — useful for security reviews
* Exact timestamp and duration
* The request URL that was called
* A unique Request ID you can share with support
* If the action failed, an **Error Details** section with the exact error message

***

## Exporting Activity

The **Export** button on the filter bar opens a dropdown with four choices, so you can
pick exactly what you need:

| Scope            | Format      | When to use                                                         |
| ---------------- | ----------- | ------------------------------------------------------------------- |
| **Current page** | CSV or JSON | Quick export of just the rows visible right now                     |
| **All filtered** | CSV or JSON | Export every event matching the current filters (up to 10,000 rows) |

Use CSV for spreadsheets and ticket attachments, JSON for pipelines and log tools.

<Tip>
  Narrow the filters first — Service plus Action plus a tight Date Range usually gives
  you a small, readable export. "All filtered" caps at 10,000 rows, so unfiltered wide
  time ranges may be truncated.
</Tip>

***

## Common Tasks

<AccordionGroup>
  <Accordion title="Find out who deleted a resource" icon="trash">
    1. Set **Action** to `delete`
    2. Paste the resource ID into **Search URL / ID** and press Enter

    The matching row shows the user, project, and source IP behind the delete.
  </Accordion>

  <Accordion title="Track failed logins" icon="lock">
    1. Set **Service** to Identity
    2. Set **Action** to `authenticate`
    3. Look at the **Warnings (4xx)** and **Failed (5xx)** summary cards for counts

    Open any row for the source IP and the exact error message.
  </Accordion>

  <Accordion title="Watch activity in real time" icon="clock">
    1. Set the **Date Range** to the last hour (or leave it empty)
    2. Turn on the **Live** toggle

    New rows highlight green as they arrive. Click any row to pause updates and inspect it.
  </Accordion>

  <Accordion title="Audit everything a project did today" icon="folder">
    1. Paste the project ID into **Search URL / ID** and press Enter
    2. Set **Date Range** to today

    Use **Export → All filtered → CSV** to attach the result to a change record or ticket.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Log Analytics" href="/services/monitoring/user-guide/log-analytics">
    Search infrastructure log streams alongside activity events
  </Card>

  <Card title="Metrics & Alerts" href="/services/monitoring/user-guide/metrics-alerts">
    Track infrastructure health with metric thresholds and notifications
  </Card>

  <Card title="Dashboards" href="/services/monitoring/user-guide/dashboards">
    Visualize infrastructure health in the XIMP overview dashboard
  </Card>
</CardGroup>


# Alert Rules
Source: https://docs.xloud.tech/services/monitoring/user-guide/alert-rules

Advanced XIMP alert rule configuration — compound conditions, silences, inhibition rules, on-call escalation, and GitOps-based rule management.

## Overview

This page covers advanced alert rule configuration beyond basic threshold alerts —
including compound multi-condition rules, silencing active alerts during maintenance
windows, inhibition rules that suppress lower-severity alerts when a critical one
is already firing, and escalation policies.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * At least one notification channel configured
    (see [XIMP Admin — Alert Channels](/services/monitoring/admin-guide/alert-channels))
</Note>

***

## Compound Alert Rules

Combine multiple conditions in a single rule using AND/OR logic:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Monitor Center > Monitoring** (Create Alert Rule, admin view) and
    click **Advanced Mode**.

    In Advanced Mode, add multiple conditions:

    | Field           | Value                                |
    | --------------- | ------------------------------------ |
    | **Condition A** | `xloud_compute_cpu_utilization > 85` |
    | **Operator**    | `AND`                                |
    | **Condition B** | `xloud_compute_memory_free_pct < 15` |

    This rule fires only when both CPU is above 85% AND available memory is below
    15% — reducing false positives from transient CPU spikes.
  </Tab>

  <Tab title="Rule File" icon="code">
    ```yaml title="alert-resource-pressure.yaml" theme={null}
    name: resource-pressure-critical
    severity: critical
    evaluation_period: 5m
    conditions:
      - metric: xloud_compute_cpu_utilization
        condition: ">"
        threshold: 85
      - metric: xloud_compute_memory_free_pct
        condition: "<"
        threshold: 15
    logic: AND
    notification_channels:
      - ops-pagerduty
    ```

    ```bash title="Create compound alert rule" theme={null}
    ximp alert rule create --file alert-resource-pressure.yaml
    ```
  </Tab>
</Tabs>

***

## Silencing Alerts

Silences temporarily suppress alert notifications during planned maintenance.
The alert rule continues to evaluate — only notifications are suppressed.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create a silence" icon="bell-slash">
        Navigate to **Monitor Center > Monitoring** (Create Silence, admin view).

        | Field        | Description                                                                     |
        | ------------ | ------------------------------------------------------------------------------- |
        | **Matchers** | Label selectors that match the alerts to silence (e.g., `host=compute-node-03`) |
        | **Duration** | How long the silence is active (e.g., `2h`)                                     |
        | **Comment**  | Reason for the silence (required — links to change ticket)                      |
        | **Creator**  | Your username (auto-populated)                                                  |
      </Step>

      <Step title="Verify silence is active" icon="circle-check">
        Navigate to the Silences section in **Monitor Center > Monitoring**. Active silences show their matcher,
        creator, and expiry time.

        <Check>Any alerts matching the silence matchers show status `Silenced` instead of firing notifications.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a 2-hour silence for a host" theme={null}
    ximp alert silence create \
      --matcher 'host=compute-node-03' \
      --duration 2h \
      --comment "Scheduled maintenance window - firmware update"
    ```

    ```bash title="List active silences" theme={null}
    ximp alert silence list --status active
    ```

    ```bash title="Expire a silence early" theme={null}
    ximp alert silence expire <SILENCE_ID>
    ```
  </Tab>
</Tabs>

***

## Inhibition Rules

Inhibition rules suppress lower-severity alerts when a higher-severity alert is
already active for the same source. This prevents alert storms during major incidents.

```yaml title="Example: Suppress warnings when critical is firing" theme={null}
inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal:
      - host
```

This rule suppresses all `warning` alerts for a host when a `critical` alert is
already firing for that same host — reducing notification noise during a major outage.

Configure inhibition rules via **Monitor Center > Monitoring** (Inhibition Rules, admin view).

***

## Escalation Policies

Configure multi-tier escalation for critical alerts:

<Steps>
  <Step title="Define escalation tiers" icon="arrow-up">
    Navigate to **Monitor Center > Monitoring** (Escalation Policies, admin view).

    | Tier   | Channel              | Delay      | Condition            |
    | ------ | -------------------- | ---------- | -------------------- |
    | Tier 1 | PagerDuty on-call    | Immediate  | Alert fires          |
    | Tier 2 | Slack ops channel    | 5 minutes  | Not acknowledged     |
    | Tier 3 | Page on-call manager | 15 minutes | Still unacknowledged |
  </Step>

  <Step title="Assign to alert rules" icon="link">
    Open an existing alert rule and set the **Escalation Policy** field to the
    policy you created. The policy applies to all future alert events for that rule.

    <Tip>
      Create separate escalation policies for different severity levels — critical
      infrastructure alerts may warrant a 3-tier escalation while informational
      alerts can go to a single team Slack channel with no escalation.
    </Tip>
  </Step>
</Steps>

***

## GitOps-Based Rule Management

Manage alert rules as code for version-controlled, auditable configurations:

```bash title="Export all current alert rules" theme={null}
ximp alert rule export --format yaml --output ./alert-rules/
```

```bash title="Import rules from directory" theme={null}
ximp alert rule import --dir ./alert-rules/ --apply
```

Store rule files in your infrastructure repository and apply changes through
your CI/CD pipeline. This enables peer review of alert rule changes and
automatic rollback if a rule causes issues.

***

## Next Steps

<CardGroup>
  <Card title="Metrics & Alerts" href="/services/monitoring/user-guide/metrics-alerts">
    Basic alert rule creation for metric thresholds
  </Card>

  <Card title="XIMP Admin — Alert Channels" href="/services/monitoring/admin-guide/alert-channels">
    Configure the notification channels referenced by alert rules
  </Card>

  <Card title="Dashboards" href="/services/monitoring/user-guide/dashboards">
    Visualize metrics alongside alert thresholds
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/user-guide/troubleshooting">
    Diagnose alert rules that are not firing or delivering notifications
  </Card>
</CardGroup>


# Dashboards
Source: https://docs.xloud.tech/services/monitoring/user-guide/dashboards

Navigate XIMP infrastructure dashboards, create custom views, and drill down into compute, storage, and network metrics.

## Overview

XIMP dashboards provide real-time and historical views of your environment across
infrastructure, application, and network layers. Built-in dashboards cover the most
common monitoring needs; custom dashboards let you build focused views for specific
teams or services.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * XIMP accessible from your Dashboard account (**Monitor Center** (admin view))
</Note>

<Warning>
  Monitoring dashboards are accessed through the admin panel. You must have administrator privileges to view infrastructure-level metrics described in this section.
</Warning>

***

## Built-In Dashboards

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Access the monitoring portal" icon="activity">
        Navigate to
        **Monitor Center** (admin view) to open the XIMP portal.
      </Step>

      <Step title="Select a dashboard" icon="layout-grid">
        The **Home** view displays the default infrastructure overview dashboard. Use
        the left panel to switch between:

        | Dashboard                   | Shows                                                        |
        | --------------------------- | ------------------------------------------------------------ |
        | **Infrastructure Overview** | CPU, memory, disk, and network utilization per node          |
        | **Compute Instances**       | Per-instance metrics — vCPU usage, memory pressure, disk I/O |
        | **Network**                 | Interface throughput, packet loss, connection counts         |
        | **Storage**                 | IOPS, throughput, latency, and capacity utilization by pool  |
        | **Applications**            | Service-level latency, error rate, and throughput            |
        | **Security Events**         | IDS alerts, failed auth attempts, anomaly detections         |
      </Step>

      <Step title="Filter and zoom" icon="zoom-in">
        Use the time range picker in the top-right corner to select a preset range
        (Last 1h, Last 24h, Last 7d) or define a custom range. Click any panel to
        drill down into more granular metrics.

        <Tip>
          Pin frequently-used dashboards to your home screen using the star icon in
          the dashboard header. Pinned dashboards appear in the Quick Access bar.
        </Tip>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    List available dashboards programmatically for automation or scripting:

    ```bash title="List available dashboards" theme={null}
    ximp dashboard list
    ```

    ```bash title="Export a dashboard as JSON" theme={null}
    ximp dashboard export --name "Infrastructure Overview" \
      --output infra-overview.json
    ```
  </Tab>
</Tabs>

***

## Creating Custom Dashboards

<Steps>
  <Step title="Create a new dashboard" icon="plus">
    Navigate to **Monitor Center > Monitoring** (admin view) and click **New Dashboard**. Enter a
    name and optional description.
  </Step>

  <Step title="Add panels" icon="layout-grid">
    Click **Add Panel** and select a visualization type:

    | Type            | Best For                                              |
    | --------------- | ----------------------------------------------------- |
    | **Time Series** | CPU, memory, network metrics over time                |
    | **Gauge**       | Current utilization percentage                        |
    | **Bar Chart**   | Comparing metrics across multiple hosts               |
    | **Table**       | Tabular metric data with sorting and filtering        |
    | **Stat**        | Single prominent value (e.g., uptime, total alerts)   |
    | **Heatmap**     | Distribution of values over time                      |
    | **Logs**        | Embedded log panel for dashboard-level log visibility |
  </Step>

  <Step title="Configure the metric query" icon="search">
    Define the data source and query for the panel. XIMP supports the native
    metric query language for filtering by host, service, tag, or time range.

    Example query for CPU utilization:

    ```
    xloud_compute_cpu_utilization{host="compute-node-01"}
    ```

    <Note>
      Queries are not executed against external systems — all data flows through
      XIMP's internal collection pipeline. No external API keys are required.
    </Note>
  </Step>

  <Step title="Save and share" icon="share">
    Click **Save Dashboard**. Use the **Share** button to generate a read-only
    link for stakeholders who do not have monitoring portal access.

    <Check>Dashboard is saved and visible to all project members in the Dashboards list.</Check>
  </Step>
</Steps>

***

## Dashboard Best Practices

<CardGroup>
  <Card title="Group by context" icon="layout">
    Create separate dashboards for different audiences: an operations NOC dashboard
    with health-at-a-glance panels, and a detailed engineering dashboard with deep
    per-component metrics.
  </Card>

  <Card title="Use template variables" icon="code">
    Add template variables (dropdown filters) to dashboards so a single dashboard
    covers all hosts or services. Use `$host` or `$service` as query variables.
  </Card>

  <Card title="Set alert thresholds visually" icon="bell">
    Add threshold lines to time-series panels to make alert levels visually obvious.
    This helps operators recognize whether current values are within or outside
    acceptable ranges at a glance.
  </Card>

  <Card title="Link to related dashboards" icon="link">
    Add panel links that navigate to more detailed dashboards when an anomaly is
    spotted. Click a high-CPU panel → open the detailed compute node dashboard.
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="Metrics & Alerts" href="/services/monitoring/user-guide/metrics-alerts">
    Create alert rules that fire when dashboard metrics breach thresholds
  </Card>

  <Card title="Log Analytics" href="/services/monitoring/user-guide/log-analytics">
    Add log panels to dashboards for correlated event visibility
  </Card>

  <Card title="Network Monitoring" href="/services/monitoring/user-guide/network-monitoring">
    Explore network traffic panels and flow-based visualizations
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/user-guide/troubleshooting">
    Diagnose missing metrics and dashboard data issues
  </Card>
</CardGroup>


# Log Analytics
Source: https://docs.xloud.tech/services/monitoring/user-guide/log-analytics

Search, filter, and analyze centralized log streams in XIMP. Create log-based alert rules for real-time event detection across your infrastructure.

## Overview

XIMP aggregates log streams from all registered infrastructure nodes and services
into a centralized, searchable index. Log Analytics lets you query events across
your entire environment with full-text search, structured field filtering, and
anomaly detection — from a single interface.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * Log collection configured for the services you want to query
    (see [XIMP Admin — Log Collection](/services/monitoring/admin-guide/log-collection))
</Note>

***

## Searching Logs

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open Log Explorer" icon="search">
        Navigate to **Monitor Center > Logging** (admin view).
      </Step>

      <Step title="Search and filter" icon="filter">
        Use the query bar at the top to filter log entries:

        | Filter      | Syntax Example                       |
        | ----------- | ------------------------------------ |
        | By host     | `host:compute-node-01`               |
        | By service  | `service:nova-compute`               |
        | By severity | `level:ERROR`                        |
        | Full-text   | `"connection refused"`               |
        | Combined    | `host:xd1 level:ERROR service:nova*` |

        <Tip>
          Use the **Add Filter** panel on the left to build queries visually. The
          query bar updates automatically as filters are applied.
        </Tip>
      </Step>

      <Step title="Adjust the time range" icon="clock">
        Use the time picker to scope your search. For incident investigation, set
        an exact range spanning the incident window to avoid scrolling through
        unrelated events.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Search logs from the last hour" theme={null}
    ximp log search \
      --query 'level:ERROR service:nova*' \
      --from now-1h \
      --limit 100
    ```

    ```bash title="Export logs to a file" theme={null}
    ximp log export \
      --query 'host:xd1' \
      --from 2026-03-01 \
      --to 2026-03-15 \
      --output /tmp/xd1-logs-march.jsonl
    ```

    ```bash title="Tail live log stream" theme={null}
    ximp log tail --query 'level:ERROR'
    ```
  </Tab>
</Tabs>

***

## Log-Based Alert Rules

Create alerts that fire whenever a log entry matching a query appears.

<Steps>
  <Step title="Open Log Explorer" icon="search">
    Navigate to **Monitor Center > Logging** (admin view) and build the query that
    should trigger an alert.
  </Step>

  <Step title="Create alert from query" icon="bell">
    Click **Create Alert** in the Log Explorer toolbar.

    | Field         | Description                                         |
    | ------------- | --------------------------------------------------- |
    | **Name**      | Descriptive alert name                              |
    | **Query**     | The log search query (pre-filled from Log Explorer) |
    | **Condition** | `at least N occurrences within M minutes`           |
    | **Severity**  | `Critical`, `Warning`, or `Info`                    |
    | **Channels**  | Notification channels to alert                      |

    <Note>
      Log-based alerts have a minimum evaluation interval of 1 minute. For
      near-real-time security event detection, use the Security and IDS module
      which processes events with sub-minute latency.
    </Note>
  </Step>

  <Step title="Save and verify" icon="circle-check">
    Click **Save**. The rule activates and evaluates the log query on each
    collection cycle.

    <Check>Alert rule appears in **Monitor Center > Monitoring** (Alerting section, admin view) with type `Log`.</Check>
  </Step>
</Steps>

***

## Useful Query Patterns

<AccordionGroup>
  <Accordion title="Find all errors in the last hour" icon="circle-x">
    ```
    level:ERROR
    ```

    Set time range to "Last 1h" in the time picker.
  </Accordion>

  <Accordion title="Track authentication failures" icon="lock">
    ```
    service:keystone "authentication failed"
    ```

    Use this for security auditing and failed login detection.
  </Accordion>

  <Accordion title="Find OOM (out of memory) events" icon="server">
    ```
    "Out of memory" OR "oom-kill" OR "kernel: Killed process"
    ```

    Identifies instances where the kernel killed processes due to memory pressure.
  </Accordion>

  <Accordion title="Disk I/O errors" icon="hard-drive">
    ```
    "I/O error" OR "EXT4-fs error" OR "blk_update_request" level:ERROR
    ```

    Surfaces disk-level errors that may indicate failing storage devices.
  </Accordion>

  <Accordion title="Network connection refused" icon="network">
    ```
    "Connection refused" OR "ECONNREFUSED" level:ERROR
    ```

    Identifies services that are failing to connect to their dependencies.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Metrics & Alerts" href="/services/monitoring/user-guide/metrics-alerts">
    Combine log-based alerts with metric thresholds for comprehensive coverage
  </Card>

  <Card title="XIMP Admin — Log Collection" href="/services/monitoring/admin-guide/log-collection">
    Configure log source paths and syslog forwarding (administrator)
  </Card>

  <Card title="Network Monitoring" href="/services/monitoring/user-guide/network-monitoring">
    Analyze network traffic alongside log events for incident correlation
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/user-guide/troubleshooting">
    Diagnose missing or delayed log ingestion
  </Card>
</CardGroup>


# Metrics & Alerts
Source: https://docs.xloud.tech/services/monitoring/user-guide/metrics-alerts

Create and manage XIMP alert rules, define notification channels, and monitor alert history for your infrastructure and applications.

## Overview

XIMP evaluates metric-based alert rules continuously against live time-series data.
When a rule's condition is met for the configured evaluation period, XIMP fires an
alert to the configured notification channels. This page covers creating, managing,
and troubleshooting alert rules.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * At least one notification channel configured (see [Alert Rules Advanced](/services/monitoring/user-guide/alert-rules))
</Note>

***

## Creating Alert Rules

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Alert Rules" icon="bell">
        Navigate to **Monitor Center > Monitoring** (Alerting section, admin view) and click **New Alert Rule**.
      </Step>

      <Step title="Define the condition" icon="settings">
        Configure the alert trigger:

        | Field                 | Description                                                                |
        | --------------------- | -------------------------------------------------------------------------- |
        | **Name**              | Descriptive label for the rule (e.g., `high-cpu-utilization`)              |
        | **Metric**            | The time-series metric to evaluate (e.g., `xloud_compute_cpu_utilization`) |
        | **Condition**         | Threshold operator: `>`, `<`, `>=`, `<=`, or `== NaN`                      |
        | **Threshold**         | Numeric value that triggers the alert                                      |
        | **Evaluation Period** | Duration the condition must persist before firing (e.g., 5 minutes)        |
        | **Severity**          | `Critical`, `Warning`, or `Info`                                           |
      </Step>

      <Step title="Assign a notification channel" icon="send">
        Under **Notifications**, select one or more configured channels (email, webhook,
        or on-call integration). Multiple channels can be assigned per rule.

        <Warning>
          Alert rules with no notification channel assigned are evaluated but never
          delivered to operators. Always assign at least one channel for production rules.
        </Warning>
      </Step>

      <Step title="Save and activate" icon="circle-check">
        Click **Save and Enable**. The rule enters the **Active** state and begins
        evaluating on the next collection cycle.

        <Check>Alert rule appears in the Active Rules list with state **Evaluating**.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create from a definition file" icon="plus">
        ```bash title="Create alert rule from file" theme={null}
        ximp alert rule create --file alert-cpu-high.yaml
        ```

        ```yaml title="alert-cpu-high.yaml" theme={null}
        name: high-cpu-utilization
        metric: xloud_compute_cpu_utilization
        condition: ">"
        threshold: 90
        evaluation_period: 5m
        severity: warning
        notification_channels:
          - ops-email
          - pagerduty-oncall
        ```
      </Step>

      <Step title="List existing rules" icon="list">
        ```bash title="List all alert rules" theme={null}
        ximp alert rule list
        ```

        ```bash title="Show specific rule details" theme={null}
        ximp alert rule show high-cpu-utilization
        ```
      </Step>

      <Step title="Update or delete a rule" icon="settings">
        ```bash title="Update rule threshold" theme={null}
        ximp alert rule update high-cpu-utilization --threshold 85
        ```

        ```bash title="Delete a rule" theme={null}
        ximp alert rule delete high-cpu-utilization
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Viewing Alert History

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Monitor Center > Monitoring** (Alert History, admin view) to see a timestamped
    feed of all alert fire and resolution events.

    Filter by:

    * **Rule name** — view history for a specific alert rule
    * **Severity** — show only Critical or Warning events
    * **Time range** — focus on a specific incident window
    * **Status** — Active (currently firing) or Resolved
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="View alert history for a rule" theme={null}
    ximp alert history --rule high-cpu-utilization --last 24h
    ```

    ```bash title="View all active (currently firing) alerts" theme={null}
    ximp alert list --status active
    ```

    ```bash title="Acknowledge an active alert" theme={null}
    ximp alert acknowledge <ALERT_ID> --comment "Investigating high CPU on compute-node-03"
    ```
  </Tab>
</Tabs>

***

## Common Alert Rules Reference

| Alert                 | Metric                               | Condition | Threshold  | Evaluation Period |
| --------------------- | ------------------------------------ | --------- | ---------- | ----------------- |
| High CPU              | `xloud_compute_cpu_utilization`      | `>`       | 90%        | 5m                |
| Low memory            | `xloud_compute_memory_free_pct`      | `<`       | 10%        | 5m                |
| High disk I/O latency | `xloud_storage_osd_apply_latency_ms` | `>`       | 20ms       | 10m               |
| Pool capacity warning | `xloud_storage_pool_used_pct`        | `>`       | 70%        | 15m               |
| Host unreachable      | `up{job="node_exporter"}`            | `==`      | 0          | 2m                |
| Replication lag       | `xdr_replication_lag_seconds`        | `>`       | RPO target | 5m                |

***

## Next Steps

<CardGroup>
  <Card title="Alert Rules (Advanced)" href="/services/monitoring/user-guide/alert-rules">
    Compound conditions, silences, inhibition rules, and escalation policies
  </Card>

  <Card title="XIMP Admin — Alert Channels" href="/services/monitoring/admin-guide/alert-channels">
    Configure email, webhook, PagerDuty, and Slack notification channels
  </Card>

  <Card title="Dashboards" href="/services/monitoring/user-guide/dashboards">
    Visualize the metrics your alert rules monitor
  </Card>

  <Card title="Troubleshooting" href="/services/monitoring/user-guide/troubleshooting">
    Diagnose alert rules that are not firing as expected
  </Card>
</CardGroup>


# Network Monitoring
Source: https://docs.xloud.tech/services/monitoring/user-guide/network-monitoring

Analyze traffic flows, detect anomalous patterns, and monitor protocol distribution across virtual and physical networks.

## Overview

The XIMP Network Traffic Monitoring module provides deep visibility into traffic flows,
bandwidth consumption, and protocol distribution across your virtual and physical networks.
Operations teams use it to identify top bandwidth consumers, investigate anomalous patterns,
and baseline normal network behavior.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * Network flow collection configured by your administrator
    (see [XIMP Admin — Agent Configuration](/services/monitoring/admin-guide/agent-config))
</Note>

***

## Network Traffic Views

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Monitor Center > Monitoring** (Network section, admin view) to access network monitoring views.

    | View                  | Shows                                                              |
    | --------------------- | ------------------------------------------------------------------ |
    | **Traffic Overview**  | Total inbound/outbound traffic per interface over time             |
    | **Top Talkers**       | Highest-bandwidth source/destination pairs                         |
    | **Protocol Analysis** | Traffic breakdown by protocol (TCP, UDP, ICMP, application layer)  |
    | **Flow Table**        | Individual network flows with source, destination, port, and bytes |
    | **Anomaly Detection** | Unusual traffic patterns flagged by behavioral analysis            |

    <Tip>
      Set the **Scope** filter to a specific project, network, or subnet to isolate
      traffic for a particular project or application tier.
    </Tip>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Get top bandwidth consumers (last 1h)" theme={null}
    ximp network top-talkers --period 1h --limit 10
    ```

    ```bash title="Show traffic for a specific host" theme={null}
    ximp network flows --host 10.0.1.50 --from now-24h
    ```

    ```bash title="Export flow data" theme={null}
    ximp network export \
      --network prod-tenant-net \
      --from 2026-03-17T00:00 \
      --to 2026-03-17T23:59 \
      --output flows-2026-03-17.csv
    ```
  </Tab>
</Tabs>

***

## Analyzing Traffic Anomalies

<Steps>
  <Step title="Review the Anomaly Detection panel" icon="circle-x">
    Navigate to **Monitor Center > Monitoring** (Network Anomaly, admin view). XIMP uses behavioral
    baselines to flag traffic patterns that deviate significantly from historical norms.

    Each anomaly entry shows:

    * Detection time and duration
    * Affected host or network segment
    * Anomaly type (volumetric, port scan, protocol violation, etc.)
    * Confidence score
  </Step>

  <Step title="Drill into suspicious flows" icon="search">
    Click on an anomaly event to view the associated flow records. Use the Flow Table
    to examine individual connections:

    | Column             | Description                         |
    | ------------------ | ----------------------------------- |
    | **Source IP**      | Originating IP address              |
    | **Destination IP** | Target IP address                   |
    | **Port**           | Destination port                    |
    | **Protocol**       | TCP, UDP, ICMP                      |
    | **Bytes**          | Total data transferred in this flow |
    | **Duration**       | Flow lifetime in seconds            |
  </Step>

  <Step title="Correlate with logs" icon="file-text">
    Cross-reference suspicious traffic with log events in the Log Explorer:

    ```
    host:<SUSPICIOUS_IP> level:ERROR
    ```

    Combined network flow data and log events often confirm whether an anomaly
    is malicious or benign (e.g., a legitimate backup job generating unusual
    burst traffic).

    <Tip>
      Use XIMP's **Linked Panels** feature to open the Log Explorer pre-filtered
      to the host and time range of a network anomaly with a single click.
    </Tip>
  </Step>
</Steps>

***

## Setting Network Traffic Alerts

Alert on network conditions that indicate problems or security events:

<Tabs>
  <Tab title="Bandwidth Threshold Alert" icon="gauge">
    Navigate to **Monitor Center > Monitoring** (Create Alert Rule, admin view):

    | Field                 | Value                               |
    | --------------------- | ----------------------------------- |
    | **Metric**            | `xloud_network_interface_rx_bytes`  |
    | **Condition**         | `>`                                 |
    | **Threshold**         | 900000000 (900 MB/s — 90% of 1 GbE) |
    | **Evaluation Period** | 5 minutes                           |
    | **Severity**          | Warning                             |
  </Tab>

  <Tab title="Packet Loss Alert" icon="circle-x">
    ```yaml title="alert-packet-loss.yaml" theme={null}
    name: high-packet-loss
    metric: xloud_network_packet_loss_pct
    condition: ">"
    threshold: 1
    evaluation_period: 5m
    severity: critical
    notification_channels:
      - ops-email
    ```

    ```bash title="Create alert" theme={null}
    ximp alert rule create --file alert-packet-loss.yaml
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Metrics & Alerts" href="/services/monitoring/user-guide/metrics-alerts">
    Create bandwidth and packet loss alert rules
  </Card>

  <Card title="Log Analytics" href="/services/monitoring/user-guide/log-analytics">
    Correlate network anomalies with log events from the same time window
  </Card>

  <Card title="Alert Rules (Advanced)" href="/services/monitoring/user-guide/alert-rules">
    Configure compound alert conditions and escalation for network events
  </Card>

  <Card title="XIMP Admin — DDoS Protection" href="/services/monitoring/admin-guide/ddos-protection">
    Configure automatic DDoS mitigation policies (administrator)
  </Card>
</CardGroup>


# Troubleshooting
Source: https://docs.xloud.tech/services/monitoring/user-guide/troubleshooting

Diagnose common XIMP user-facing issues — alerts not firing, missing metrics, log ingestion delays, and dashboard data problems.

## Overview

This page covers the most common issues encountered when using XIMP — from alert
rules that fail to fire, to dashboards showing no data, to missing or delayed logs.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * For agent and infrastructure-level issues, contact your monitoring administrator. Your administrator can configure this through [XDeploy](/deployment).
</Note>

***

## Common Issues

<AccordionGroup>
  <Accordion title="Alert not firing despite threshold being exceeded" icon="bell">
    **Cause**: The evaluation period has not elapsed, the notification channel is
    misconfigured, or the alert rule is in a silenced state.

    **Resolution**:

    1. Verify the rule's evaluation period — the condition must persist for the full
       duration before the alert fires:
       ```bash title="Check alert rule configuration" theme={null}
       ximp alert rule show <RULE_NAME>
       ```
    2. Check **Monitor Center > Monitoring** (Alert Channels, admin view) to confirm the notification channel
       is active and credentials are valid
    3. Check **Monitor Center > Monitoring** (Silences, admin view) — confirm no active silence covers
       the alert:
       ```bash title="List active silences" theme={null}
       ximp alert silence list --status active
       ```
    4. Verify the metric has data in the time window — navigate to the Dashboards and
       check whether the metric panel shows values above the threshold

    <Warning>
      An alert rule evaluating `xloud_compute_cpu_utilization > 90` will only fire
      if the metric is above 90% for the ENTIRE evaluation period. Brief spikes that
      resolve within the period will not trigger the alert.
    </Warning>
  </Accordion>

  <Accordion title="Metrics not appearing in dashboards" icon="activity">
    **Cause**: The monitoring agent on the target host is not running, or the host
    is not registered with XIMP.

    **Resolution**:

    ```bash title="Check agent registration" theme={null}
    ximp agent list --status all
    ```

    Look for hosts with status `offline` or `unknown`. If metrics are missing, contact your administrator. They can verify the monitoring agent status through [XDeploy](/deployment/operations).
  </Accordion>

  <Accordion title="Log ingestion delayed or missing" icon="file-text">
    **Cause**: Log collector is not configured for the service, the log file path
    has changed, or the collector is experiencing a backlog.

    **Resolution**:

    1. Navigate to **Monitor Center > Logging** (Log Sources, admin view) and verify the
       log source configuration for the affected service
    2. Confirm the file path pattern matches the current log file location
    3. Check the collector queue depth:
       ```bash title="Check ingestion queue depth" theme={null}
       ximp log ingest-status
       ```

    <Note>
      Log ingestion uses file-based collection. If a service rotates logs to a new
      path after an update, the collector configuration must be updated to match.
      Contact your monitoring administrator to update log source configurations. Your administrator can configure this through [XDeploy](/deployment).
    </Note>
  </Accordion>

  <Accordion title="Dashboard shows 'No data' for a metric" icon="gauge">
    **Cause**: The scrape target is down, the agent is offline, or the metric name
    has changed after a software update.

    **Resolution**:

    1. Check the target health: navigate to **Monitor Center > Monitoring** (Scrape Targets, admin view)
       and look for targets in `DOWN` state
    2. Verify the agent is active for that host:
       ```bash title="Check agent status" theme={null}
       ximp agent list --node <HOSTNAME>
       ```
    3. Search for the metric to verify it exists and find the correct name:
       ```bash title="Search metrics by prefix" theme={null}
       ximp metric search --prefix xloud_compute_cpu
       ```
  </Accordion>

  <Accordion title="Alert notifications not delivered" icon="send">
    **Cause**: The notification channel configuration is invalid, credentials have
    expired, or the destination is temporarily unreachable.

    **Resolution**:

    1. Navigate to **Monitor Center > Monitoring** (Alert Channels, admin view) and use the **Test** button
       to send a test notification
    2. If the test fails, review the channel configuration:
       ```bash title="Check channel configuration" theme={null}
       ximp alert channel show <CHANNEL_NAME>
       ```
    3. For email channels: verify SMTP credentials and server reachability
    4. For webhook channels: verify the URL is accessible from the XIMP server
    5. For PagerDuty: verify the integration key has not been rotated

    <Tip>
      Send a test notification immediately after creating or modifying a channel.
      Do not rely on a real alert event to discover that a channel is broken.
    </Tip>
  </Accordion>
</AccordionGroup>

***

## Diagnostics Reference

| Issue                 | First Step                                                                              |
| --------------------- | --------------------------------------------------------------------------------------- |
| Alert not firing      | `ximp alert rule show <RULE_NAME>`                                                      |
| Agent offline         | Contact your administrator to verify agent status via [XDeploy](/deployment/operations) |
| Missing metric        | `ximp metric search --prefix <METRIC_PREFIX>`                                           |
| Log ingestion backlog | `ximp log ingest-status`                                                                |
| Channel test          | Use **Test** button in Dashboard or `ximp alert channel test <NAME>`                    |

***

## When to Contact Your Administrator

Contact your monitoring administrator if any of the following persist. Your administrator can configure this through [XDeploy](/deployment).

* A host does not appear in `ximp agent list` after restarting the agent service
* All metrics are missing for multiple hosts simultaneously
* Log ingestion queue depth has been growing for more than 1 hour
* TLS certificate errors prevent agent communication

See the [XIMP Admin Guide](/services/monitoring/admin-guide) for administrator-level
diagnostics and configuration.

***

## Next Steps

<CardGroup>
  <Card title="XIMP Admin Guide" href="/services/monitoring/admin-guide">
    Infrastructure-level XIMP administration and agent configuration
  </Card>

  <Card title="Metrics & Alerts" href="/services/monitoring/user-guide/metrics-alerts">
    Review and adjust alert rule configurations
  </Card>

  <Card title="Dashboards" href="/services/monitoring/user-guide/dashboards">
    Verify metric availability in dashboard panels
  </Card>

  <Card title="Support" href="mailto:support@xloud.tech">
    Contact Xloud support for issues requiring platform-level investigation
  </Card>
</CardGroup>


# Networking Administration
Source: https://docs.xloud.tech/services/networking/admin-guide

Configure SDN agents, provider networks, VLAN and VXLAN segmentation, QoS policies, network quotas, and security hardening for Xloud Networking.

<p>Overview</p>

Xloud Networking administration covers the full lifecycle of the SDN fabric — from
deploying and monitoring the agents that drive the virtual switching layer to defining
provider networks, enforcing QoS policies, and hardening the network plane against
misconfiguration and spoofing attacks. Use this guide to maintain the networking infrastructure across your cluster.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Tabs>
  <Tab title="XDeploy" icon="server">
    Network configuration is managed through the XDeploy Configuration panel:

    <Steps>
      <Step title="Open Configuration" icon="settings">
        Navigate to **XDeploy → Configuration** and select the **Network** tab.
      </Step>

      <Step title="Configure network settings" icon="network">
        Set the following options as needed:

        | Setting                | Description                                            |
        | ---------------------- | ------------------------------------------------------ |
        | **External Interface** | Physical interface for provider network traffic        |
        | **VPNaaS**             | Toggle to enable VPN-as-a-Service                      |
        | **QoS**                | Toggle to enable Quality of Service bandwidth policies |
        | **VLAN Trunking**      | Toggle to enable VLAN trunk port support               |
        | **Agent HA**           | Toggle to enable L3/DHCP agent high availability       |
        | **SR-IOV**             | Toggle to enable Single Root I/O Virtualization        |
      </Step>

      <Step title="Save and deploy" icon="rocket">
        Click **Save Configuration**, then navigate to **XDeploy → Operations** and
        run a **Deploy** or **Reconfigure** for the Networking service.

        <Check>Network configuration is applied across all nodes.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Configure networking by editing `neutron.conf`, `ml2_conf.ini`, and agent
    configuration files directly at `/etc/xavs/config/neutron/`. See the individual
    topic guides below for detailed parameters.
  </Tab>
</Tabs>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * `openstack` CLI installed and configured
  * XDeploy access for cluster-level configuration changes
  * All networking agents running and healthy on all nodes
</Note>

***

<p>Administration Topics</p>

<CardGroup>
  <Card title="Service Architecture" icon="network" href="/services/networking/architecture">
    Distributed SDN agent model — API server, message bus, L2/L3 agents, DHCP, and metadata
  </Card>

  <Card title="Provider Networks" icon="layers" href="/services/networking/provider-networks">
    Configure VLAN, flat, and VXLAN provider networks and physical interface mappings
  </Card>

  <Card title="Network Agent Management" icon="activity" href="/services/networking/network-agents">
    Monitor agent health, enable or disable agents for maintenance, and recover failed agents
  </Card>

  <Card title="DHCP Configuration" icon="server" href="/services/networking/dhcp">
    Manage DHCP agents, network assignments, and high-availability DHCP
  </Card>

  <Card title="L3 Router Configuration" icon="route" href="/services/networking/l3-routing">
    Enable HA routers with VRRP failover and distributed virtual routing for scale
  </Card>

  <Card title="Quality of Service" icon="gauge" href="/services/networking/qos">
    Apply bandwidth limits and burst controls to ports and networks
  </Card>

  <Card title="Network Quotas" icon="shield" href="/services/networking/quotas">
    Set per-project limits for networks, subnets, routers, floating IPs, and security groups
  </Card>

  <Card title="Security Hardening" icon="lock" href="/services/networking/security">
    Port security, anti-spoofing enforcement, allowed address pairs, and default group hardening
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/networking/admin-troubleshooting">
    Diagnose agent failures, VXLAN tunnel issues, HA router failover, and MTU problems
  </Card>
</CardGroup>

***

<p>Architecture Overview</p>

```mermaid theme={null}
graph TD
    U([Dashboard / CLI]) --> API[Networking API<br/>:9696]
    API --> DB[(Database)]
    API --> MQ[Message Bus]
    MQ --> L2A1[L2 Agent<br/>Compute Node 1]
    MQ --> L2A2[L2 Agent<br/>Compute Node 2]
    MQ --> L3A[L3 Agent<br/>Network Node]
    MQ --> DHCP[DHCP Agent<br/>Network Node]
    L3A --> GW([External Gateway])
    L2A1 --> VS1[Virtual Switch<br/>Host 1]
    L2A2 --> VS2[Virtual Switch<br/>Host 2]
    style API fill:#197560,color:#fff
    style L3A fill:#197560,color:#fff
    style DHCP fill:#3F8F7E,color:#fff
```

***

<p>Agent Health Quick Check</p>

```bash title="Check all networking agent status" theme={null}
openstack network agent list
```

All agents should show `Alive: True` and `Admin State: UP`. If any agent shows
`Alive: False`, see [Network Agent Management](/services/networking/network-agents)
for recovery procedures.

***

<p>Related Resources</p>

<CardGroup>
  <Card title="Networking User Guide" icon="book-open" href="/services/networking/user-guide">
    User-facing workflows for creating networks, routers, floating IPs, and security groups
  </Card>

  <Card title="Compute Admin Guide" icon="server" href="/services/compute/admin-guide">
    Manage compute hosts, flavors, and quotas alongside networking configuration
  </Card>

  <Card title="Authentication" icon="key" href="/cli-setup">
    Configure admin credentials and project-scoped access for networking operations
  </Card>

  <Card title="CLI Setup" icon="terminal" href="/cli-setup">
    Install and configure the `openstack` CLI for full administrative control
  </Card>
</CardGroup>


# Networking Admin Troubleshooting
Source: https://docs.xloud.tech/services/networking/admin-troubleshooting

Diagnose and resolve Xloud Networking infrastructure issues — downed agents, VXLAN tunnel failures, HA router failover problems, and MTU mismatches.

## Overview

This guide covers infrastructure-level networking issues that require administrator
access to diagnose and resolve — agent failures, VXLAN tunnel connectivity, HA router
failover, and MTU configuration across the physical underlay.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * SSH access to compute and network nodes
  * XDeploy access for agent restarts
</Note>

***

## Diagnostic Quick Reference

```bash title="Overview agent health across the cluster" theme={null}
openstack network agent list --long
```

```bash title="List agents by host" theme={null}
openstack network agent list -f value -c Host -c Binary -c Alive | sort
```

```bash title="Show all routers with HA state" theme={null}
openstack router list --all-projects -f json | grep -E '"id"|"ha"|"status"'
```

***

## Common Issues

<AccordionGroup>
  <Accordion title="Agent is down or showing stale heartbeat" icon="activity">
    **Cause**: The agent process has crashed, the host is unreachable, or the message
    bus is not delivering heartbeats.

    **Resolution**:

    1. Identify the affected host:
       ```bash title="List agents with heartbeat timestamps" theme={null}
       openstack network agent list --long
       ```
    2. SSH to the affected host and check the agent service:
       ```bash title="Check L2 agent (Linux bridge)" theme={null}
       sudo systemctl status neutron-linuxbridge-agent
       ```
       ```bash title="Check agent container logs" theme={null}
       sudo docker logs neutron_openvswitch_agent --tail 100
       ```
    3. Restart the agent via XDeploy:
       ```bash title="Redeploy networking agents" theme={null}
       xavs-ansible deploy --tags neutron
       ```

    <Tip>
      After restarting, allow up to 30 seconds for the agent to re-register and send
      a heartbeat. Verify with `openstack network agent list --long`.
    </Tip>
  </Accordion>

  <Accordion title="VXLAN tunnel failures between compute nodes" icon="layers">
    **Cause**: MTU mismatch, firewall blocking UDP 4789, or misconfigured tunnel
    endpoint IPs.

    **Resolution**:

    1. Verify UDP 4789 (VXLAN) is reachable between compute nodes:
       ```bash title="Test VXLAN port reachability" theme={null}
       nc -uvz <remote-compute-ip> 4789
       ```
    2. Confirm tunnel endpoint IPs:
       ```bash title="Show L2 agent configuration including tunnel IP" theme={null}
       openstack network agent list --agent-type ovs --long
       ```
       The `Configuration` field shows `tunnel_types` and `local_ip`.
    3. Verify the physical interface MTU accommodates VXLAN overhead:
       ```bash title="Check physical interface MTU" theme={null}
       ip link show eth0 | grep mtu
       ```
       For VXLAN, the physical MTU must be at least `1550` to carry 1500-byte tenant
       frames with 50-byte encapsulation overhead.
  </Accordion>

  <Accordion title="MTU mismatch causing application-level failures" icon="split">
    **Cause**: Tenant network MTU exceeds the physical network capacity after
    VXLAN encapsulation overhead.

    **Resolution**:

    1. Set the correct MTU on the affected tenant network:
       ```bash title="Update network MTU" theme={null}
       openstack network set app-network --mtu 1450
       ```
    2. The DHCP agent automatically pushes the updated MTU to new instances via
       DHCP option 26. Existing instances need a manual update or DHCP renewal:
       ```bash title="Set MTU on Linux guest" theme={null}
       ip link set eth0 mtu 1450
       ```

    <Info>
      MTU recommendations: VXLAN networks = `1450`, VLAN networks = `1500`,
      jumbo-frame VLAN = up to `9000` (requires switch support end-to-end).
    </Info>
  </Accordion>

  <Accordion title="HA router not recovering after L3 agent failure" icon="route">
    **Cause**: VRRP failover completed but the new master has not programmed
    floating IP NAT rules, or the failover did not complete.

    **Resolution**:

    1. Check the HA state across L3 agents:
       ```bash title="Show router HA status" theme={null}
       openstack router show ha-router -f json | grep -E "ha|status"
       ```
    2. List L3 agents for the router — confirm one is `active`:
       ```bash title="List L3 agents for the router" theme={null}
       openstack network agent list --router ha-router
       ```
    3. If stuck, trigger rescheduling by toggling admin state:
       ```bash title="Reschedule the HA router" theme={null}
       openstack router set ha-router --disable
       openstack router set ha-router --enable
       ```
    4. Check L3 agent logs on network nodes for VRRP negotiation errors:
       ```bash title="View L3 agent logs" theme={null}
       sudo docker logs neutron_l3_agent --tail 200
       ```

    <Tip>
      A long VRRP keepalive timeout (default \~3 seconds, dead interval \~10 seconds)
      can cause a 10–30 second outage before the standby takes over. Tune the VRRP
      timers in XDeploy if faster failover is required.
    </Tip>
  </Accordion>

  <Accordion title="Provider network ports not binding" icon="plug">
    **Cause**: Physical network mapping misconfiguration or the L2 agent on the
    compute node does not have the bridge mapped.

    **Resolution**:

    1. Verify the bridge mapping on the affected compute node:
       ```bash title="Check bridge mapping in L2 agent config" theme={null}
       openstack network agent show <l2-agent-id> -f json | grep bridge_mappings
       ```
    2. Confirm the bridge exists on the host:
       ```bash title="Check bridge on compute node" theme={null}
       ssh xloud@<compute-node> "ip link show br-ex"
       ```
    3. If the bridge is missing, redeploy the networking configuration:
       ```bash title="Redeploy networking" theme={null}
       xavs-ansible deploy --tags neutron
       ```
  </Accordion>
</AccordionGroup>

***

## Log Locations

| Service        | Log Location                            |
| -------------- | --------------------------------------- |
| Networking API | `docker logs neutron_server`            |
| L2 Agent (SDN) | `docker logs neutron_openvswitch_agent` |
| L3 Agent       | `docker logs neutron_l3_agent`          |
| DHCP Agent     | `docker logs neutron_dhcp_agent`        |
| Metadata Agent | `docker logs neutron_metadata_agent`    |

***

## Next Steps

<CardGroup>
  <Card title="Network Agent Management" href="/services/networking/network-agents">
    Manage agent enable/disable state and monitor health
  </Card>

  <Card title="L3 Router Configuration" href="/services/networking/l3-routing">
    Configure HA and DVR to prevent the issues described in this guide
  </Card>

  <Card title="Provider Networks" href="/services/networking/provider-networks">
    Verify provider network configuration if port bindings are failing
  </Card>

  <Card title="User Troubleshooting" href="/services/networking/troubleshooting">
    Tenant-facing connectivity and floating IP troubleshooting
  </Card>
</CardGroup>


# Networking Service Architecture
Source: https://docs.xloud.tech/services/networking/architecture

Understand the distributed SDN agent model powering Xloud Networking. API server, message bus, L2/L3 agents, DHCP, and metadata service components.

## Overview

Xloud Networking follows a distributed agent model. A central API and database tier manages
resource state, while per-node agents program the virtual switching fabric in real time.
Understanding this architecture helps administrators diagnose failures, plan capacity, and
maintain the networking plane across the cluster.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Architecture Diagram

```mermaid theme={null}
graph TD
    U([Dashboard / CLI]) --> API[Networking API<br/>:9696]
    API --> DB[(Database)]
    API --> MQ[Message Bus<br/>:5672]
    MQ --> L2A1[L2 Agent<br/>Compute Node 1]
    MQ --> L2A2[L2 Agent<br/>Compute Node 2]
    MQ --> L2A3[L2 Agent<br/>Compute Node N]
    MQ --> L3A[L3 Agent<br/>Network Node]
    MQ --> DHCP[DHCP Agent<br/>Network Node]
    MQ --> META[Metadata Agent<br/>Network Node]
    L3A --> GW([External Gateway<br/>Provider Network])
    L2A1 --> VS1[Virtual Switch<br/>Host 1]
    L2A2 --> VS2[Virtual Switch<br/>Host 2]
    L2A3 --> VS3[Virtual Switch<br/>Host N]
    style API fill:#197560,color:#fff
    style L3A fill:#197560,color:#fff
    style DHCP fill:#3F8F7E,color:#fff
    style META fill:#3F8F7E,color:#fff
```

***

## Agent Components

| Agent           | Default Port        | Role                                                                   |
| --------------- | ------------------- | ---------------------------------------------------------------------- |
| Networking API  | 9696                | RESTful endpoint for all network resource management                   |
| L2 Agent        | N/A (message bus)   | Programs virtual switching and port bindings on each compute node      |
| L3 Agent        | N/A (message bus)   | Manages routers, NAT rules, and floating IP translation                |
| DHCP Agent      | 67/68 (DHCP)        | Provides IP address assignment for tenant subnets                      |
| Metadata Agent  | 80 (internal proxy) | Forwards instance metadata requests from the network namespace         |
| RPC Message Bus | 5672 (AMQP)         | Carries control messages between the API server and distributed agents |

***

## Request Flow: Create Network and Create Instance

```mermaid theme={null}
sequenceDiagram
    participant User
    participant API as Networking API
    participant DB as Database
    participant MQ as Message Bus
    participant L2 as L2 Agent
    participant DHCP as DHCP Agent

    User->>API: POST /v2.0/networks
    API->>DB: Store network record
    API-->>User: 201 Created (network-id)

    User->>API: POST /v2.0/subnets
    API->>DB: Store subnet record
    API->>MQ: Notify DHCP agent
    MQ->>DHCP: Configure DHCP namespace
    API-->>User: 201 Created (subnet-id)

    User->>API: POST /v2.0/ports (via Nova)
    API->>DB: Store port record
    API->>MQ: Bind port to host
    MQ->>L2: Program virtual switch port
    L2-->>MQ: Port bound
    API-->>User: 200 OK (port active)
```

***

## Data Plane: Virtual Switching

Each compute node runs an L2 agent that programs the local virtual switch to enforce:

* **Port bindings** — connect instance virtual NICs to the correct network segment
* **VLAN or VXLAN segmentation** — isolate tenant traffic from other projects
* **Security group rules** — iptables/nftables rules enforced per port
* **Anti-spoofing** — MAC and IP address binding prevents address impersonation

<Info>
  The specific virtual switch backend (Linux bridge, Open vSwitch, or OVN) is configured
  during cluster deployment via XDeploy. The choice of backend affects performance
  characteristics and advanced features like DVR and hardware offload.
</Info>

***

## High Availability Considerations

| Component      | HA Mechanism                          | Impact of Failure                             |
| -------------- | ------------------------------------- | --------------------------------------------- |
| Networking API | Multiple API instances behind HAProxy | Single instance loss: no impact               |
| Database       | Galera cluster (3+ nodes)             | Partial loss: degraded writes                 |
| Message Bus    | RabbitMQ cluster (3+ nodes)           | Partial loss: agent messaging delayed         |
| L2 Agent       | One per compute node                  | Agent loss: no new port bindings on that host |
| L3 Agent       | VRRP failover (HA routers)            | Active agent loss: standby takes over         |
| DHCP Agent     | Multiple agents per network           | Agent loss: secondary agent serves leases     |

***

## Next Steps

<CardGroup>
  <Card title="Network Agent Management" href="/services/networking/network-agents">
    Monitor agent health and manage agent lifecycle
  </Card>

  <Card title="Provider Networks" href="/services/networking/provider-networks">
    Configure physical network mappings and segmentation types
  </Card>

  <Card title="L3 Router Configuration" href="/services/networking/l3-routing">
    Enable HA and distributed routing for production deployments
  </Card>

  <Card title="DHCP Configuration" href="/services/networking/dhcp">
    Manage DHCP agents and subnet assignments
  </Card>
</CardGroup>


# Networking CLI Reference
Source: https://docs.xloud.tech/services/networking/cli-reference

Complete openstack network CLI commands for managing Xloud Networking — networks, subnets, routers, floating IPs, security groups, and ports.

## Overview

The `openstack network` command group manages software-defined networks, subnets, routers, ports, floating IPs, and security group rules.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python neutronclient installed: `pip install python-neutronclient`
</Note>

***

## Networks

<CodeGroup>
  ```bash title="List networks" theme={null}
  openstack network list
  openstack network list --internal
  openstack network list --external
  ```

  ```bash title="Create private network" theme={null}
  openstack network create private-net
  ```

  ```bash title="Create with provider type (admin)" theme={null}
  openstack network create \
    --provider-network-type vlan \
    --provider-physical-network physnet1 \
    --provider-segment 100 \
    vlan-100
  ```

  ```bash title="Show network" theme={null}
  openstack network show private-net
  ```

  ```bash title="Delete network" theme={null}
  openstack network delete private-net
  ```
</CodeGroup>

***

## Subnets

<CodeGroup>
  ```bash title="List subnets" theme={null}
  openstack subnet list
  ```

  ```bash title="Create subnet" theme={null}
  openstack subnet create \
    --network private-net \
    --subnet-range 10.0.1.0/24 \
    --dns-nameserver 8.8.8.8 \
    private-subnet
  ```

  ```bash title="Create with DHCP disabled" theme={null}
  openstack subnet create \
    --network private-net \
    --subnet-range 10.0.2.0/24 \
    --no-dhcp \
    static-subnet
  ```

  ```bash title="Show subnet" theme={null}
  openstack subnet show private-subnet
  ```

  ```bash title="Delete subnet" theme={null}
  openstack subnet delete private-subnet
  ```
</CodeGroup>

***

## Routers

<CodeGroup>
  ```bash title="List routers" theme={null}
  openstack router list
  ```

  ```bash title="Create router with external gateway" theme={null}
  openstack router create my-router
  openstack router set --external-gateway external my-router
  ```

  ```bash title="Add subnet interface" theme={null}
  openstack router add subnet my-router private-subnet
  ```

  ```bash title="Remove subnet interface" theme={null}
  openstack router remove subnet my-router private-subnet
  ```

  ```bash title="Show router details" theme={null}
  openstack router show my-router
  ```

  ```bash title="Delete router" theme={null}
  openstack router delete my-router
  ```
</CodeGroup>

***

## Floating IPs

<CodeGroup>
  ```bash title="List floating IPs" theme={null}
  openstack floating ip list
  ```

  ```bash title="Allocate floating IP" theme={null}
  openstack floating ip create external
  ```

  ```bash title="Associate to instance" theme={null}
  openstack server add floating ip <server> <floating-ip>
  ```

  ```bash title="Disassociate" theme={null}
  openstack server remove floating ip <server> <floating-ip>
  ```

  ```bash title="Release floating IP" theme={null}
  openstack floating ip delete <floating-ip>
  ```
</CodeGroup>

***

## Security Groups

<CodeGroup>
  ```bash title="List security groups" theme={null}
  openstack security group list
  ```

  ```bash title="Create security group" theme={null}
  openstack security group create web-servers \
    --description "Allow HTTP and SSH"
  ```

  ```bash title="Add inbound rules" theme={null}
  openstack security group rule create \
    --protocol tcp --dst-port 22 --remote-ip 0.0.0.0/0 \
    web-servers

  openstack security group rule create \
    --protocol tcp --dst-port 80 --remote-ip 0.0.0.0/0 \
    web-servers

  openstack security group rule create \
    --protocol tcp --dst-port 443 --remote-ip 0.0.0.0/0 \
    web-servers
  ```

  ```bash title="Allow ICMP (ping)" theme={null}
  openstack security group rule create \
    --protocol icmp \
    web-servers
  ```

  ```bash title="List rules in a group" theme={null}
  openstack security group rule list web-servers
  ```

  ```bash title="Delete rule" theme={null}
  openstack security group rule delete <rule-id>
  ```
</CodeGroup>

***

## Ports

<CodeGroup>
  ```bash title="List ports" theme={null}
  openstack port list
  openstack port list --network private-net
  ```

  ```bash title="Create port with fixed IP" theme={null}
  openstack port create \
    --network private-net \
    --fixed-ip subnet=private-subnet,ip-address=10.0.1.50 \
    my-port
  ```

  ```bash title="Show port" theme={null}
  openstack port show <port>
  ```

  ```bash title="Disable port security" theme={null}
  openstack port set --disable-port-security <port>
  ```

  ```bash title="Delete port" theme={null}
  openstack port delete <port>
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Network User Guide" href="/services/networking/create-network">
    Step-by-step network and subnet creation walkthrough
  </Card>

  <Card title="Compute CLI" href="/services/compute/cli-reference">
    Attach floating IPs and security groups to instances
  </Card>
</CardGroup>


# Create a Network
Source: https://docs.xloud.tech/services/networking/create-network

Provision an isolated tenant network and subnet in Xloud. Configure IP space, DHCP, DNS, and port security using the Dashboard or CLI.

## Overview

Every project in Xloud Networking starts with a network — an isolated Layer 2 broadcast
domain that instances attach to via virtual ports. Paired with a subnet, it defines the
IP address space, DHCP assignment, and routing gateway for your workload tier.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Create a Network and Subnet

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Networks">
        Navigate to **Network > Networks** in the sidebar. Click **Create Network**.
      </Step>

      <Step title="Configure the network">
        The create dialog shows the following fields:

        | Field                     | Type      | Required | Description                                        |
        | ------------------------- | --------- | -------- | -------------------------------------------------- |
        | **Network Name**          | Text      | Yes      | Network display name                               |
        | **Description**           | Text area | No       | Optional notes                                     |
        | **Available Zone**        | Dropdown  | No       | Pin to a specific zone                             |
        | **MTU**                   | Number    | No       | Maximum transmission unit (68-9000, default: 1500) |
        | **Create Subnet**         | Checkbox  | No       | Toggle to create a subnet with the network         |
        | **Port Security Enabled** | Switch    | No       | Enable/disable port security (default: enabled)    |

        <Note>
          Administrators see additional fields: **Shared** (make visible to all projects),
          **External Network** (mark as router external gateway), **Project** selector,
          and **Provider Network Type** (vxlan, flat, vlan, gre) with segmentation ID.
        </Note>
      </Step>

      <Step title="Configure the subnet (if enabled)">
        When **Create Subnet** is checked, additional fields appear:

        | Field           | Type     | Required | Description                                     |
        | --------------- | -------- | -------- | ----------------------------------------------- |
        | **Subnet Name** | Text     | Yes      | Subnet display name                             |
        | **IP Version**  | Dropdown | Yes      | IPv4 or IPv6                                    |
        | **CIDR**        | Text     | Yes      | Network address block (e.g., `192.168.10.0/24`) |

        For IPv6, additional fields appear: **RA Mode** and **Address Mode** with options
        `dhcpv6-stateful`, `dhcpv6-stateless`, `slaac`.

        **Advanced options** (click to expand):

        | Field                | Type      | Description                                          |
        | -------------------- | --------- | ---------------------------------------------------- |
        | **Disable Gateway**  | Checkbox  | Remove the default gateway from the subnet           |
        | **Gateway IP**       | IP input  | Custom gateway address (auto-assigned if blank)      |
        | **Enable DHCP**      | Radio     | Enable or disable DHCP for this subnet               |
        | **Allocation Pools** | Text area | IP range pairs (e.g., `192.168.10.2,192.168.10.200`) |
        | **DNS**              | Text area | One DNS server per line                              |
        | **Host Routes**      | Text area | Static routes (e.g., `192.168.200.0/24,10.56.1.254`) |

        <Warning>
          Choose a CIDR that does not overlap with existing subnets in your project.
          Overlapping subnets cause routing failures that are difficult to diagnose.
        </Warning>
      </Step>

      <Step title="Create the network">
        Click **Confirm**. The network appears in the list with status **Active**.

        <Check>Network shows status **Active** with the configured subnet.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <Steps>
      <Step title="Create the network">
        ```bash title="Create network" theme={null}
        openstack network create app-network
        ```
      </Step>

      <Step title="Create a subnet">
        ```bash title="Create subnet with DHCP and DNS" theme={null}
        openstack subnet create app-subnet \
          --network app-network \
          --subnet-range 192.168.10.0/24 \
          --gateway 192.168.10.1 \
          --dns-nameserver 8.8.8.8 \
          --dns-nameserver 8.8.4.4
        ```
      </Step>

      <Step title="Verify">
        <CodeGroup>
          ```bash title="List networks" theme={null}
          openstack network list
          ```

          ```bash title="Show subnet details" theme={null}
          openstack subnet show app-subnet
          ```
        </CodeGroup>

        <Check>Network shows status `ACTIVE` and the subnet lists the correct CIDR.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## View Networks

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > Networks**. The list shows networks in tabs:

    | Tab                          | Shows                                      |
    | ---------------------------- | ------------------------------------------ |
    | **Current Project Networks** | Networks owned by your current project     |
    | **Shared Networks**          | Networks shared across projects            |
    | **External Networks**        | Networks with external gateway capability  |
    | **All Networks**             | All visible networks (admin role required) |

    List columns:

    | Column                 | Description                                            |
    | ---------------------- | ------------------------------------------------------ |
    | **ID/Name**            | Network identifier (clickable to view details)         |
    | **Is Current Project** | Whether the network belongs to your project            |
    | **External**           | Whether this is an external gateway network (Yes/No)   |
    | **Shared**             | Whether the network is shared across projects (Yes/No) |
    | **Status**             | Active, Build, Down, or Error                          |
    | **Subnet Count**       | Number of subnets (with popover showing details)       |
    | **Created At**         | Creation timestamp                                     |

    Filter by **Name**, **Shared**, **External**, or **Project Range**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List all networks" theme={null}
      openstack network list
      ```

      ```bash title="List external networks" theme={null}
      openstack network list --external
      ```

      ```bash title="Show network details" theme={null}
      openstack network show app-network
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Network Detail

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a network name to open the detail page. Three tabs are available:

    * **Detail** — Network configuration summary
    * **Subnets** — All subnets in this network with CIDR, DHCP status, gateway
    * **Ports** — All ports attached to this network (instances, routers, DHCP)
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List subnets in a network" theme={null}
      openstack subnet list --network app-network
      ```

      ```bash title="List ports in a network" theme={null}
      openstack port list --network app-network
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Subnets" href="/services/networking/subnets">
    Add additional subnets to your network for multi-tier isolation
  </Card>

  <Card title="Routers" href="/services/networking/routers">
    Connect your network to the external internet with an L3 router
  </Card>

  <Card title="Security Groups" href="/services/networking/security-groups">
    Define firewall rules to control traffic to your instances
  </Card>

  <Card title="Floating IPs" href="/services/networking/floating-ips">
    Allocate public IPs and associate them with your instances
  </Card>
</CardGroup>


# DHCP Configuration
Source: https://docs.xloud.tech/services/networking/dhcp

Manage DHCP agents and network assignments in Xloud Networking. Configure high-availability DHCP and control IP address assignment for tenant subnets.

## Overview

The DHCP agent provides IP address assignment, DNS resolver delivery, and host route
injection for tenant subnets. Multiple DHCP agents can serve the same network for high
availability — when the active agent fails, a standby takes over without disrupting
existing DHCP leases.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * At least one running DHCP agent (verify with `openstack network agent list --agent-type dhcp`)
</Note>

***

## View Networks Served by an Agent

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > Network Agents** (admin view). Click the agent ID of a DHCP agent.
    The **Networks** tab lists all networks this agent is currently serving.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List networks served by a specific DHCP agent" theme={null}
    openstack network list --agent <dhcp-agent-id>
    ```
  </Tab>
</Tabs>

***

## Schedule a Network to a DHCP Agent

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > Network Agents** (admin view), click a DHCP agent, then click
    **Add Network** in the **Networks** tab. Select the network from the list.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="List available DHCP agents" icon="list">
        ```bash title="List DHCP agents with host names" theme={null}
        openstack network agent list --agent-type dhcp
        ```

        Note the agent IDs for the agents you want to use.
      </Step>

      <Step title="Assign network to primary agent" icon="plus">
        ```bash title="Schedule network to DHCP agent" theme={null}
        openstack network agent add network --dhcp <agent-id> <network-id>
        ```
      </Step>

      <Step title="Assign to a second agent for HA" icon="shield">
        ```bash title="Add network to second DHCP agent for redundancy" theme={null}
        openstack network agent add network --dhcp <second-agent-id> <network-id>
        ```

        Both agents will serve DHCP requests for the network. The agents elect a primary
        using a distributed locking mechanism — the other acts as standby.

        <Check>The network now has two DHCP agents for high availability.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Remove a Network from an Agent

```bash title="Remove network from a specific DHCP agent" theme={null}
openstack network agent remove network --dhcp <agent-id> <network-id>
```

<Warning>
  Remove the network from a DHCP agent only after confirming another healthy agent is
  serving it. Run `openstack network list --agent <standby-agent-id>` to verify the
  standby agent shows the network before removing the primary.
</Warning>

***

## DHCP Agent HA Architecture

```mermaid theme={null}
graph TD
    SUBNET[Tenant Subnet] --> AGENT1[DHCP Agent 1<br/>Network Node 1<br/>Primary]
    SUBNET --> AGENT2[DHCP Agent 2<br/>Network Node 2<br/>Standby]
    INST1[Instance 1] -->|DHCP Request| SUBNET
    INST2[Instance 2] -->|DHCP Request| SUBNET
    AGENT1 -.->|Failover| AGENT2
    style AGENT1 fill:#197560,color:#fff
    style AGENT2 fill:#3F8F7E,color:#fff
```

***

## Troubleshoot DHCP Issues

<AccordionGroup>
  <Accordion title="Instances not receiving IP addresses" icon="server">
    **Cause**: DHCP agent is down, subnet DHCP is disabled, or allocation pool is exhausted.

    **Resolution**:

    1. Confirm DHCP is enabled on the subnet:
       ```bash title="Check subnet DHCP status" theme={null}
       openstack subnet show <subnet-name> -f json | grep enable_dhcp
       ```
    2. Verify the DHCP agent is alive:
       ```bash title="Check DHCP agent health" theme={null}
       openstack network agent list --agent-type dhcp
       ```
    3. Check the allocation pool size vs. current port count:
       ```bash title="Count ports on network" theme={null}
       openstack port list --network <network-name> | wc -l
       ```
  </Accordion>

  <Accordion title="Stale DNS or gateway options on instances" icon="settings">
    **Cause**: Subnet DNS or gateway was updated but running instances have stale DHCP leases.

    **Resolution**:
    Force DHCP renewal on the affected instance:

    ```bash title="Renew DHCP lease on Linux guest" theme={null}
    sudo dhclient -r eth0 && sudo dhclient eth0
    ```

    Alternatively, restart the network interface or reboot the instance.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Network Agent Management" href="/services/networking/network-agents">
    Monitor and manage the full set of SDN agents across your cluster
  </Card>

  <Card title="Subnets" href="/services/networking/subnets">
    Configure DHCP allocation pools, DNS, and host routes at the subnet level
  </Card>

  <Card title="DNS Configuration" href="/services/networking/dns-config">
    Control DNS resolver delivery to instances through DHCP
  </Card>

  <Card title="Admin Troubleshooting" href="/services/networking/admin-troubleshooting">
    Resolve agent and connectivity issues with diagnostic commands
  </Card>
</CardGroup>


# DNS Configuration
Source: https://docs.xloud.tech/services/networking/dns-config

Configure DNS name servers and hostname resolution for Xloud tenant subnets. Push resolvers to instances via DHCP for private and public name resolution.

## Overview

DNS configuration in Xloud Networking controls which resolvers instances use for hostname
resolution. Name servers are set at the subnet level and pushed to instances via DHCP at
boot time. You can configure private internal resolvers, public fallback resolvers, or a
combination — giving instances access to both internal service hostnames and public domain
names.

<Note>
  **Prerequisites**

  * An existing subnet with DHCP enabled ([Create and Manage Subnets](/services/networking/subnets))
  * Dashboard access or CLI configured with valid credentials
</Note>

***

## Configure DNS on a Subnet

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the subnet for editing" icon="compass">
        Navigate to
        **Network > Networks**. Click the network name, open the **Subnets** tab,
        and click **Edit Subnet** on the target subnet.
      </Step>

      <Step title="Update DNS">
        In the **Advanced Options** section, update the **DNS** field.
        Enter one resolver per line.

        | Resolver Type     | Example     | Use Case                                        |
        | ----------------- | ----------- | ----------------------------------------------- |
        | Internal resolver | `10.0.0.53` | Resolve private hostnames (e.g., `db.internal`) |
        | Public fallback   | `8.8.8.8`   | Resolve public internet domains                 |
        | Secondary public  | `8.8.4.4`   | Redundant public resolver                       |

        <Tip>
          Place your internal resolver first. Instances try resolvers in the order they
          are listed — putting the internal resolver first speeds up private hostname lookups.
        </Tip>
      </Step>

      <Step title="Save and verify" icon="circle-check">
        Click **Save**. Existing instances pick up the new resolvers on their next DHCP renewal.

        <Check>New instances launched on this subnet receive the updated DNS resolvers automatically.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Set DNS resolvers on a subnet" icon="settings">
        ```bash title="Update DNS name servers" theme={null}
        openstack subnet set app-subnet \
          --dns-nameserver 10.0.0.53 \
          --dns-nameserver 8.8.8.8
        ```

        Repeat `--dns-nameserver` for each resolver. The order of the flags determines
        the resolver priority.
      </Step>

      <Step title="Verify the configuration" icon="circle-check">
        ```bash title="Show subnet DNS config" theme={null}
        openstack subnet show app-subnet -f json | grep dns_nameservers
        ```

        <Check>Output lists the configured resolvers in priority order.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Apply DNS Changes to Running Instances

DNS resolvers are delivered via DHCP at instance boot. Running instances retain their
current resolvers until their DHCP lease renews or you force a renewal.

<Tabs>
  <Tab title="Linux instances" icon="terminal">
    ```bash title="Force DHCP renewal (systemd-networkd)" theme={null}
    sudo networkctl renew eth0
    ```

    ```bash title="Force DHCP renewal (NetworkManager)" theme={null}
    sudo nmcli device reapply eth0
    ```

    ```bash title="Force DHCP renewal (dhclient)" theme={null}
    sudo dhclient -r eth0 && sudo dhclient eth0
    ```
  </Tab>

  <Tab title="Verify resolution" icon="circle-check">
    After renewal, confirm the instance is using the updated resolvers:

    ```bash title="Check /etc/resolv.conf" theme={null}
    cat /etc/resolv.conf
    ```

    ```bash title="Test internal resolution" theme={null}
    nslookup db.internal 10.0.0.53
    ```

    ```bash title="Test public resolution" theme={null}
    nslookup google.com 8.8.8.8
    ```

    <Check>Both internal and public hostnames resolve correctly.</Check>
  </Tab>
</Tabs>

***

## Remove DNS Resolvers

```bash title="Clear DNS name servers from subnet" theme={null}
openstack subnet set app-subnet --no-dns-nameservers
```

<Warning>
  Removing all DNS resolvers means instances will have no resolvers after their next
  DHCP renewal. They will be unable to resolve any hostnames, including internal ones.
  Always configure at least one resolver.
</Warning>

***

## DNS Reference

| Resolver           | Address              | Notes                                          |
| ------------------ | -------------------- | ---------------------------------------------- |
| Google Public DNS  | `8.8.8.8`, `8.8.4.4` | Low-latency, widely available                  |
| Cloudflare DNS     | `1.1.1.1`, `1.0.0.1` | Privacy-focused, fast                          |
| Internal (example) | `10.0.0.53`          | Resolves private hostnames within your cluster |

<Info>
  For deployments with a dedicated DNS service, Xloud Networking supports the optional
  DNS service (Xloud DNS) which provides zone management, record lifecycle, and floating
  IP-to-hostname associations. Contact your administrator to enable Xloud DNS.
</Info>

***

## Next Steps

<CardGroup>
  <Card title="Create and Manage Subnets" href="/services/networking/subnets">
    Manage all subnet settings including DHCP, allocation pools, and host routes
  </Card>

  <Card title="Network Troubleshooting" href="/services/networking/troubleshooting">
    Diagnose DHCP and name resolution failures on your instances
  </Card>

  <Card title="Floating IP Addresses" href="/services/networking/floating-ips">
    Assign public IPs to instances with DNS-resolvable hostnames
  </Card>

  <Card title="Network Topologies" href="/services/networking/network-topology">
    Design multi-tier topologies with DNS-aware subnets
  </Card>
</CardGroup>


# Floating IP Addresses
Source: https://docs.xloud.tech/services/networking/floating-ips

Allocate, associate, and release floating IPs for external access to Xloud Compute instances using the Dashboard or CLI.

## Overview

Floating IPs provide external access to instances running on private networks. A floating
IP maps a publicly routable address to an instance's private port via NAT on the router.
You can associate, disassociate, and reassign floating IPs without downtime.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  A router with an external gateway must be connected to the instance's network before
  floating IPs can be associated. See [Routers](/services/networking/routers).
</Note>

***

## Allocate a Floating IP

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Floating IPs">
        Navigate to **Network > Floating IPs** in the sidebar. Click **Allocate IP**.
      </Step>

      <Step title="Configure the allocation">
        | Field                   | Type      | Required         | Description                                                          |
        | ----------------------- | --------- | ---------------- | -------------------------------------------------------------------- |
        | **Network**             | Dropdown  | Yes              | Select the external network to allocate from                         |
        | **Owned Subnet**        | Dropdown  | No               | Optionally select a specific subnet within the external network      |
        | **Floating IP Address** | Text      | No               | Request a specific IP (shown after subnet selection, when not batch) |
        | **Description**         | Text area | No               | Label the IP's intended use                                          |
        | **Batch Allocate**      | Checkbox  | No               | Toggle to allocate multiple IPs at once                              |
        | **Count**               | Number    | If batch enabled | Number of IPs to allocate (default: 2)                               |

        <Note>
          Administrators see an additional **Project** selector to allocate IPs
          for other projects. If QoS is enabled on the platform, a **QoS Policy**
          selector also appears for all users.
        </Note>
      </Step>

      <Step title="Allocate">
        Click **Confirm**. The floating IP appears in the list with status **Down**
        (not yet associated with an instance).

        <Check>Floating IP allocated and ready for association.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    ```bash title="Allocate floating IP from external network" theme={null}
    openstack floating ip create public
    ```

    Replace `public` with the name of your external network.
  </Tab>
</Tabs>

***

## View Floating IPs

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > Floating IPs**. The list shows:

    | Column                  | Description                                              |
    | ----------------------- | -------------------------------------------------------- |
    | **ID/Floating IP**      | The allocated public address (clickable to view details) |
    | **Description**         | Optional description text                                |
    | **Associated Resource** | Instance or port bound to this IP (if associated)        |
    | **Status**              | Available, Pending, Active, Error, or Down               |
    | **Created At**          | Allocation timestamp                                     |
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all floating IPs" theme={null}
    openstack floating ip list
    ```
  </Tab>
</Tabs>

***

## Associate with an Instance

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Associate dialog">
        On the floating IP row, click **Associate** (available when status is
        `Down` or `Available`).
      </Step>

      <Step title="Select the target">
        Select the instance port to associate with from the fixed IP table.
        Ports that are not routable to the external network show a reason label.

        Click **Confirm**.

        <Check>The floating IP status changes to **Active** and shows the associated instance.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Associate floating IP to instance" theme={null}
      openstack server add floating ip <INSTANCE> <FLOATING_IP>
      ```

      ```bash title="Associate to specific fixed IP" theme={null}
      openstack server add floating ip \
        --fixed-ip-address <FIXED_IP> \
        <INSTANCE> <FLOATING_IP>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Disassociate a Floating IP

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    On the floating IP row, click **Disassociate** (available when status is `Active`).
    Confirm the action. The IP returns to `Down` status and can be reassigned.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Remove floating IP from instance" theme={null}
    openstack server remove floating ip <INSTANCE> <FLOATING_IP>
    ```
  </Tab>
</Tabs>

***

## Release a Floating IP

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click the **More** dropdown on the floating IP row and select **Release**.
    The IP is returned to the external network pool.

    <Danger>
      Releasing a floating IP is permanent. If you release a static IP that DNS
      records or external systems reference, those references will break immediately.
    </Danger>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Release floating IP" theme={null}
    openstack floating ip delete <FLOATING_IP_ID>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Routers" href="/services/networking/routers">
    Set up the external gateway required for floating IP routing
  </Card>

  <Card title="Security Groups" href="/services/networking/security-groups">
    Allow inbound traffic on the floating IP's port
  </Card>

  <Card title="Create a Network" href="/services/networking/create-network">
    Provision the internal network for your instances
  </Card>

  <Card title="Launch an Instance" href="/services/compute/launch-instance">
    Create an instance to associate the floating IP with
  </Card>
</CardGroup>


# Networking
Source: https://docs.xloud.tech/services/networking/index

Software-defined networking for Xloud Cloud Platform. Virtual networks, subnets, routers, floating IPs, security groups, and advanced SDN policies.

Flexible, software-defined networking that connects your cloud workloads — from simple tenant
networks to complex multi-tier topologies with distributed routing and policy enforcement.

<Card title="XAVS — Advanced Virtualization Solutions" icon="external-link" href="https://xloud.tech/xavs">
  Product details on xloud.tech
</Card>

***

<p>Xloud Networking</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/networking/user-guide">
    Create networks, subnets, routers, and security groups. Allocate floating IPs and
    build tenant network topologies from the Dashboard or CLI.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/networking/admin-guide">
    Configure provider networks, VLAN and VXLAN segmentation, SDN agents, QoS policies,
    and advanced routing for your cluster.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/networking/cli-reference">
    `openstack network`, `openstack router`, and `openstack security group` commands for
    managing all networking resources from the command line.
  </Card>

  <Card title="Related Services" icon="layers">
    Xloud Networking integrates tightly with Compute, Block Storage, and Load Balancer
    to deliver end-to-end connectivity for your workloads.
  </Card>
</CardGroup>

***

<p>Key Features</p>

<CardGroup>
  <Card title="Software-Defined Networking" icon="network" href="/services/networking/architecture">
    Fully programmable virtual network fabric. Provision isolated tenant networks,
    shared provider networks, and routed topologies — all through a single API.
  </Card>

  <Card title="Distributed Routing" icon="route" href="/services/networking/l3-routing">
    Distribute L3 routing across compute nodes to eliminate centralized bottlenecks
    and deliver high-throughput east-west connectivity for modern application architectures.
  </Card>

  <Card title="Floating IPs" icon="globe" href="/services/networking/floating-ips">
    Map public IP addresses dynamically to private instances. Associate, disassociate,
    and reassign without downtime — ideal for blue-green deployments and failover.
  </Card>

  <Card title="Security Groups" icon="shield" href="/services/networking/security-groups">
    Stateful, per-port firewall rules enforced at the hypervisor level. Define ingress
    and egress policies with protocol, port, and CIDR granularity.
  </Card>

  <Card title="QoS Policies" icon="gauge" href="/services/networking/qos">
    Apply bandwidth limits and burst controls to individual ports or networks. Enforce
    service-level guarantees and prevent noisy-neighbour interference.
  </Card>

  <Card title="VLAN and VXLAN Segmentation" icon="layers" href="/services/networking/provider-networks">
    Segment workloads with hardware-backed VLAN isolation or scale-out VXLAN overlays
    that support tens of thousands of isolated tenant networks.
  </Card>

  <Card title="VPN as a Service" icon="lock">
    IPsec site-to-site VPN tunnels for secure inter-site connectivity. Establish encrypted
    tunnels between Xloud tenant networks and remote sites or on-premises data centers.
    Available with XPCI.
  </Card>
</CardGroup>

***

<p>Networking Components</p>

| Component        | Description                                                                                       |
| ---------------- | ------------------------------------------------------------------------------------------------- |
| SDN API          | RESTful endpoint for all network operations — networks, subnets, ports, routers, and policies     |
| L2 Agent         | Manages virtual switching on each compute node — programs virtual ports and enforces segmentation |
| L3 Agent         | Handles routing between subnets, NAT for floating IPs, and external gateway connectivity          |
| DHCP Agent       | Provides IP address assignment and DNS configuration for tenant subnets                           |
| Metadata Agent   | Delivers instance metadata over the network to instances at boot time                             |
| Security Service | Enforces stateful firewall rules and anti-spoofing policies per port                              |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Virtual machine instances that connect to tenant networks and floating IPs
  </Card>

  <Card title="Xloud Block Storage" icon="hard-drive">
    Persistent volumes that mount inside compute instances on your networks
  </Card>

  <Card title="Xloud Load Balancer" icon="scale">
    Distribute traffic across instances using networking infrastructure
  </Card>

  <Card title="Xloud Identity" icon="fingerprint" href="/cli-setup">
    Project-scoped authentication and RBAC for networking resources
  </Card>

  <Card title="Xloud High Availability" icon="heart-pulse">
    HA routers and redundant DHCP agents for network resilience
  </Card>

  <Card title="XDeploy" icon="rocket">
    Configure provider networks, VLAN ranges, and SDN agents during deployment
  </Card>
</CardGroup>

***

<p>Getting Started</p>

<CardGroup>
  <Card title="Authentication" icon="key" href="/cli-setup">
    Configure Dashboard access and CLI credentials before managing networking resources
  </Card>

  <Card title="Compute User Guide" icon="server" href="/services/compute/user-guide">
    Launch instances and connect them to networks using security groups and floating IPs
  </Card>
</CardGroup>


# IP Address Management (IPAM)
Source: https://docs.xloud.tech/services/networking/ipam

Configure built-in and external IPAM drivers for automated IP allocation. Integrates with Infoblox, Bluecat, and NetBox.

## Overview

Xloud Networking manages IP address allocation through a pluggable IPAM (IP Address Management) driver. The default built-in driver handles IP allocation from subnets defined within the platform. For organizations with existing enterprise IPAM infrastructure — such as Infoblox, Bluecat, or NetBox — an external driver can delegate IP assignment and tracking to that system, ensuring consistency across the entire network environment.

When an external IPAM driver is active, every Neutron port creation calls the IPAM API to reserve an IP before the port is assigned. On port deletion, the IP is released back to the external system. This integration is transparent to tenants — they create ports and subnets through the standard Dashboard or CLI interface.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Network connectivity from all Neutron agent nodes to the IPAM system management API
  * Service account credentials and appropriate permissions on the IPAM system
  * The IPAM driver package must be installed on all Neutron API and agent nodes
</Note>

***

## Built-In IPAM vs External IPAM

| Attribute                    | Built-In IPAM                      | External IPAM                            |
| ---------------------------- | ---------------------------------- | ---------------------------------------- |
| **IP allocation source**     | Neutron subnet pool                | External IPAM database                   |
| **Conflict prevention**      | Within platform only               | Across entire enterprise network         |
| **DNS integration**          | Manual or via Designate            | Automatic via IPAM system                |
| **Visibility**               | Platform-only                      | Enterprise-wide IP tracking              |
| **Audit trail**              | Neutron database                   | IPAM system audit logs                   |
| **Configuration complexity** | None                               | Requires IPAM API integration            |
| **Recommended for**          | Isolated or greenfield deployments | Enterprise with existing IPAM governance |

***

## Supported IPAM Drivers

| Driver                      | Integration Type          | Protocol     | Notes                                              |
| --------------------------- | ------------------------- | ------------ | -------------------------------------------------- |
| **Internal (built-in)**     | Native Neutron allocation | Internal SQL | Default — no external system required              |
| **Infoblox**                | WAPI REST API             | HTTPS        | DNS, DHCP, and IP reservation; enterprise standard |
| **Bluecat Address Manager** | REST API                  | HTTPS        | Supports IPAM, DNS, and DHCP workflows             |
| **NetBox**                  | REST API (via middleware) | HTTPS        | Community-supported; requires adapter plugin       |
| **phpIPAM**                 | REST API (via middleware) | HTTPS        | Open-source IPAM, adapter available                |
| **SolarWinds IPAM**         | REST API (via middleware) | HTTPS        | Enterprise monitoring + IPAM combined              |

***

## Configure the IPAM Driver

The IPAM driver is set in `neutron.conf`. All Neutron API nodes must use the same driver configuration to ensure consistent IP allocation.

### Internal Driver (Default)

```ini title="neutron.conf — built-in IPAM driver" theme={null}
[DEFAULT]
ipam_driver = internal
```

No additional configuration is required. Neutron manages IP allocation from subnet allocation pools defined in the platform.

### Infoblox Driver

The Infoblox driver uses the Infoblox WAPI (Web API) to reserve and release IP addresses. It supports DNS host record creation and metadata synchronization alongside IP allocation.

```ini title="neutron.conf — Infoblox IPAM driver" theme={null}
[DEFAULT]
ipam_driver = infoblox

[infoblox]
cloud_data_center_id = 1
ipam_agent_workers = 2
wapi_url = https://10.0.10.7/wapi/v2.10
wapi_username = neutron-svc
wapi_password = <password>
wapi_version = 2.10
ssl_verify = true
network_template = default
admin_network_deletion = false
wapi_max_results = 1000
```

**Required Infoblox configuration:**

The service account (`neutron-svc`) must have IPAM Admin rights on the Infoblox Grid. Create a Network View in Infoblox to map to the Xloud environment, and configure member assignments as needed.

### Bluecat Address Manager Driver

Bluecat integration requires the `networking-bluecatnetworks` driver package. Contact Bluecat for the Neutron driver compatible with your BAM version.

```ini title="neutron.conf — Bluecat IPAM driver" theme={null}
[DEFAULT]
ipam_driver = bluecatnetworks

[bluecatnetworks]
bcn_bam_address = 10.0.10.8
bcn_bam_user = neutron-api
bcn_bam_password = <password>
bcn_bam_configuration = default
bcn_bam_ip_offset = 0
bcn_dns_deploy_on_every_action = false
bcn_bam_updatemodifyhost = true
```

### NetBox Driver (via Middleware)

NetBox does not ship with a native Neutron IPAM driver. Integration is achieved through a middleware adapter that intercepts IPAM allocation calls and forwards them to the NetBox REST API.

```ini title="neutron.conf — NetBox IPAM driver (adapter)" theme={null}
[DEFAULT]
ipam_driver = netbox_neutron_driver

[netbox]
netbox_url = https://netbox.internal.example.com
netbox_token = <api-token>
netbox_vrf = default
netbox_site = datacenter-1
```

<Note>
  The NetBox adapter is a community-maintained plugin. Verify driver compatibility with your deployed Neutron version before deployment. Contact [support@xloud.tech](mailto:support@xloud.tech) for guidance on driver selection.
</Note>

***

## Apply the Configuration

In XAVS deployments, the IPAM driver configuration is managed through XDeploy and Ansible:

```bash title="Apply Neutron configuration changes" theme={null}
xavs-ansible deploy --tags neutron
```

After deployment, verify Neutron is running with the configured driver:

```bash title="Check Neutron server configuration" theme={null}
docker exec neutron_server grep ipam_driver /etc/neutron/neutron.conf
```

***

## Subnet Allocation with External IPAM

When an external IPAM driver is active, subnet and port creation behaves differently from the built-in driver. The IPAM system must have the IP range pre-defined before Neutron subnets are created from it.

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    <Steps>
      <Step title="Pre-configure the network range in IPAM" icon="settings">
        In your external IPAM system (Infoblox, Bluecat, etc.), create the network container
        for the IP range you intend to use in Xloud (e.g., `10.50.0.0/24`).

        Assign the range to the appropriate view or zone before creating the Neutron subnet.
      </Step>

      <Step title="Create the subnet in Xloud" icon="plus">
        Navigate to **Network > Networks**, select the network, and click
        **Create Subnet**.

        Enter the CIDR range that matches the pre-configured range in your IPAM system.
        The IPAM driver validates that the range is available before creating the subnet.
      </Step>

      <Step title="Verify IP reservation in IPAM" icon="circle-check">
        After launching an instance or creating a port on this subnet, verify in your IPAM
        dashboard that the IP appears as reserved with the correct hostname and metadata.

        <Check>IP allocation confirmed — Neutron port creation is synchronized with the external IPAM system.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Source your credentials file to authenticate with the Xloud platform:

    ```bash title="Load credentials" theme={null}
    source openrc.sh
    ```

    <Tip>
      Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
    </Tip>

    ```bash title="Create a network" theme={null}
    openstack network create internal-production
    ```

    ```bash title="Create a subnet mapped to IPAM range" theme={null}
    openstack subnet create \
      --network internal-production \
      --subnet-range 10.50.0.0/24 \
      --gateway 10.50.0.1 \
      --dns-nameserver 10.0.10.7 \
      production-subnet-01
    ```

    ```bash title="Create a port and verify IP assignment" theme={null}
    openstack port create \
      --network internal-production \
      --fixed-ip subnet=production-subnet-01 \
      test-port-01

    openstack port show test-port-01 -c fixed_ips
    ```

    Verify in your IPAM system that `10.50.0.x` shows as reserved with the port's MAC address and hostname.
  </Tab>
</Tabs>

***

## IP Release and Lifecycle

IP addresses are released back to the IPAM system when ports are deleted. This happens automatically during instance termination, port deletion, or subnet removal.

```bash title="Delete a port and release the IP" theme={null}
openstack port delete <port-id>
```

After deletion, verify in the IPAM system that the address is no longer reserved. If the IP is not released (network error during deletion), use the IPAM system's manual release procedure to reclaim the address.

<Warning>
  If the external IPAM system is unreachable during port creation, Neutron will fail the port creation request. Ensure the IPAM system has high availability or that a fallback configuration is in place before enabling external IPAM in production.
</Warning>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Port creation fails with 'IPAM driver error'" icon="triangle-alert">
    **Cause**: The IPAM system is unreachable or authentication failed.

    **Resolution**:

    * Test API connectivity: `curl -k -u neutron-svc:<password> https://10.0.10.7/wapi/v2.10`
    * Verify credentials in `neutron.conf`
    * Check Neutron server logs: `docker logs neutron_server | grep -i ipam`
    * Confirm the IP range is defined in the external IPAM system before subnet creation
  </Accordion>

  <Accordion title="IP not released after port deletion" icon="x-circle">
    **Cause**: Network error during the IPAM release call, or the IPAM system rejected the release.

    **Resolution**:

    * Check Neutron server logs for IPAM release errors
    * Manually release the IP in the IPAM system UI
    * Verify the service account has delete/release permissions in the IPAM system
  </Accordion>

  <Accordion title="Subnet creation rejected by IPAM driver" icon="ban">
    **Cause**: The requested IP range is not pre-configured in the external IPAM system, or it conflicts with an existing reservation.

    **Resolution**:

    * Create the network container in the IPAM system before creating the Neutron subnet
    * Verify the CIDR does not overlap with existing networks in the IPAM view
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Subnets" href="/services/networking/subnets">
    Create and configure subnets, allocation pools, and DNS settings
  </Card>

  <Card title="DNS Configuration" href="/services/networking/dns-config">
    Configure per-subnet DNS resolvers and domain assignments
  </Card>

  <Card title="External DNS Providers" href="/services/dns/external-providers">
    Integrate Designate with Infoblox, BIND9, and other external DNS systems
  </Card>

  <Card title="Networking Architecture" href="/services/networking/architecture">
    Understand the Networking service topology and plugin architecture
  </Card>
</CardGroup>


# L3 Router Configuration
Source: https://docs.xloud.tech/services/networking/l3-routing

Configure HA routers and distributed virtual routing. Enable VRRP failover and DVR for scalable traffic.

## Overview

Xloud Networking supports two advanced router modes for production deployments: High
Availability (HA) routers using VRRP for automatic failover between L3 agents, and
Distributed Virtual Routing (DVR) that moves L3 forwarding to each compute node to
eliminate centralized bottlenecks. This guide covers enabling and validating both modes.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * At least two L3 agents running for HA routers
  * XDeploy access to enable DVR cluster-wide
</Note>

***

## High-Availability Routers

HA routers use VRRP to provide automatic failover between L3 agent instances. When the
active L3 agent fails, a standby agent takes ownership of the router's namespace and
floating IP NAT rules within seconds.

```mermaid theme={null}
graph LR
    INST([Instances]) --> HA_R{HA Router<br/>VIP}
    HA_R -->|Active| L3_1[L3 Agent<br/>Network Node 1]
    HA_R -.->|Standby| L3_2[L3 Agent<br/>Network Node 2]
    L3_1 --> EXT([External Network])
    L3_2 -.->|Failover path| EXT
    style HA_R fill:#197560,color:#fff
    style L3_1 fill:#3F8F7E,color:#fff
```

### Create an HA Router

<Tabs>
  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create HA router with external gateway" icon="plus">
        ```bash title="Create HA router" theme={null}
        openstack router create ha-router \
          --ha \
          --external-gateway public
        ```

        The `--ha` flag schedules the router across all available L3 agents automatically.
      </Step>

      <Step title="Verify HA state" icon="circle-check">
        ```bash title="Show HA router fields" theme={null}
        openstack router show ha-router -f json | grep ha
        ```

        Confirm `ha: true` and `status: ACTIVE` in the output.

        ```bash title="List L3 agents handling the router" theme={null}
        openstack network agent list --router ha-router
        ```

        <Check>At least two L3 agents appear, one `active` and one `standby`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

<Info>
  HA routers require at least two L3 agents running in the cluster. Xloud Networking
  automatically schedules the router across all available L3 agents. Verify with
  `openstack network agent list --agent-type l3` before creating HA routers.
</Info>

***

## Distributed Virtual Routing (DVR)

DVR moves L3 forwarding from a centralized agent to each compute node, eliminating the
network node as a bottleneck for east-west and north-south traffic.

| Mode           | Traffic Path                                                              | Best For                                  |
| -------------- | ------------------------------------------------------------------------- | ----------------------------------------- |
| Centralized L3 | All traffic through network node                                          | Simple deployments, ≤ 10 compute nodes    |
| DVR            | East-west direct between compute nodes; north-south via dedicated gateway | High-throughput workloads, large clusters |

### Enable DVR

<Tabs>
  <Tab title="XDeploy" icon="rocket">
    Enable DVR cluster-wide in XDeploy under **Configuration → Networking**:

    | Parameter            | Value  | Description                               |
    | -------------------- | ------ | ----------------------------------------- |
    | `neutron_l3_ha`      | `True` | Enable VRRP HA for centralized L3 agents  |
    | `enable_neutron_dvr` | `True` | Distribute L3 forwarding to compute nodes |

    Click **Save and Deploy**. XDeploy applies the configuration via xavs-ansible.

    <Warning>
      Enabling DVR on an existing cluster requires a rolling restart of all L3 and
      L2 agents. Plan a maintenance window and verify floating IP connectivity after
      the change. Test with a non-production network first.
    </Warning>
  </Tab>
</Tabs>

### Create a Distributed Router

<Tabs>
  <Tab title="CLI" icon="terminal">
    ```bash title="Create distributed router" theme={null}
    openstack router create distributed-router \
      --distributed \
      --external-gateway public
    ```

    ```bash title="Verify distributed routing is enabled" theme={null}
    openstack router show distributed-router -f json | grep distributed
    ```

    Confirm `distributed: true` in the output.
  </Tab>
</Tabs>

***

## Router Administration Reference

| Operation               | CLI Command                                                   |
| ----------------------- | ------------------------------------------------------------- |
| List all routers        | `openstack router list`                                       |
| Show router detail      | `openstack router show <name>`                                |
| Enable router           | `openstack router set <name> --enable`                        |
| Disable router          | `openstack router set <name> --disable`                       |
| Set external gateway    | `openstack router set <name> --external-gateway <network>`    |
| Remove external gateway | `openstack router unset <name> --external-gateway`            |
| Add subnet interface    | `openstack router add subnet <router> <subnet>`               |
| Remove subnet interface | `openstack router remove subnet <router> <subnet>`            |
| Add static route        | `openstack router set <name> --route destination=X,gateway=Y` |
| Delete router           | `openstack router delete <name>`                              |

***

## Next Steps

<CardGroup>
  <Card title="Network Agent Management" href="/services/networking/network-agents">
    Verify L3 agents are healthy across all network nodes
  </Card>

  <Card title="Provider Networks" href="/services/networking/provider-networks">
    Configure external networks that routers use as gateways
  </Card>

  <Card title="Routers and Gateways" href="/services/networking/routers">
    User guide for creating and managing routers
  </Card>

  <Card title="Admin Troubleshooting" href="/services/networking/admin-troubleshooting">
    Diagnose HA router failover and DVR issues
  </Card>
</CardGroup>


# Network Agent Management
Source: https://docs.xloud.tech/services/networking/network-agents

Monitor and manage Xloud SDN agents across your cluster. Check agent health, enable or disable agents for maintenance, and schedule networks to DHCP agents.

## Overview

Xloud Networking distributes work across multiple agents running on compute and network
nodes. Monitoring agent health is a core administrative responsibility — a downed agent
can silently prevent new port bindings, DHCP assignments, or routing updates. This guide
covers how to inspect, manage, and recover agents across your cluster.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * `openstack` CLI installed and configured
</Note>

***

## Inspect Agent Health

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Network Agents" icon="compass">
        Navigate to **Network > Network Agents** (admin view).
      </Step>

      <Step title="Review agent status" icon="activity">
        Each row represents one agent instance. Review the following columns:

        | Column             | Healthy Value            | Action If Unhealthy                                    |
        | ------------------ | ------------------------ | ------------------------------------------------------ |
        | **Alive**          | Yes (green)              | Restart the service on the affected host               |
        | **Admin State**    | Up                       | Enable via CLI: `openstack network agent set --enable` |
        | **Binary**         | Agent process name       | Check system service logs on the host                  |
        | **Host**           | Fully-qualified hostname | Confirm the host is reachable on the network           |
        | **Last Heartbeat** | Recent timestamp         | Investigate if stale by more than 30 seconds           |
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="List all agents" icon="list">
        ```bash title="List all networking agents" theme={null}
        openstack network agent list
        ```

        All agents should show `Alive: True` and `Admin State: UP`.
      </Step>

      <Step title="Filter by agent type" icon="filter">
        ```bash title="List DHCP agents" theme={null}
        openstack network agent list --agent-type dhcp
        ```

        ```bash title="List L3 agents" theme={null}
        openstack network agent list --agent-type l3
        ```

        ```bash title="List L2 agents on a specific host" theme={null}
        openstack network agent list --host compute-node-1
        ```
      </Step>

      <Step title="Show detailed agent information" icon="info">
        ```bash title="Show agent configuration details" theme={null}
        openstack network agent show <agent-id>
        ```

        The `configuration` field shows the agent's live settings including tunnel type
        and local endpoint IP.
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Enable and Disable Agents

Disable an agent before performing maintenance on its host to prevent the scheduler
from assigning new work to it. Re-enable after maintenance completes.

<Tabs>
  <Tab title="CLI" icon="terminal">
    ```bash title="Disable agent for maintenance" theme={null}
    openstack network agent set <agent-id> --disable
    ```

    ```bash title="Re-enable agent after maintenance" theme={null}
    openstack network agent set <agent-id> --enable
    ```
  </Tab>
</Tabs>

<Warning>
  Disabling an L3 or DHCP agent causes affected routers and subnets to lose that agent's
  services. Ensure redundant agents are running before disabling any agent. Verify with
  `openstack network agent list` that at least one healthy agent of the same type remains.
</Warning>

***

## Agent Type Reference

| Agent Binary                | Type         | Runs On       | Responsibilities                         |
| --------------------------- | ------------ | ------------- | ---------------------------------------- |
| `neutron-dhcp-agent`        | DHCP         | Network nodes | IP assignment, DNS, host routes via DHCP |
| `neutron-l3-agent`          | L3           | Network nodes | Router NAT, floating IPs, VRRP           |
| `neutron-openvswitch-agent` | L2 (SDN)     | Compute nodes | SDN-based L2 switching and port bindings |
| `neutron-linuxbridge-agent` | Linux bridge | Compute nodes | Linux bridge-based L2 switching          |
| `neutron-metadata-agent`    | Metadata     | Network nodes | Instance metadata proxy                  |
| `neutron-metering-agent`    | Metering     | Network nodes | Traffic metering for billing             |

***

## Restart an Agent

When an agent shows `Alive: False`, restart the service on the affected host. Agents
running inside Docker containers are managed by XDeploy:

```bash title="Restart networking agents via XDeploy" theme={null}
xavs-ansible deploy --tags neutron
```

After restarting, allow up to 30 seconds for the agent to re-register and send a
heartbeat. Verify recovery:

```bash title="Confirm agent is alive" theme={null}
openstack network agent list --long | grep <host-name>
```

<Check>Agent shows `Alive: True` and a recent heartbeat timestamp.</Check>

***

## Next Steps

<CardGroup>
  <Card title="DHCP Configuration" href="/services/networking/dhcp">
    Schedule networks to DHCP agents and configure HA for high availability
  </Card>

  <Card title="L3 Router Configuration" href="/services/networking/l3-routing">
    Configure HA and distributed routing across L3 agents
  </Card>

  <Card title="Service Architecture" href="/services/networking/architecture">
    Understand the distributed agent model and message bus communication
  </Card>

  <Card title="Admin Troubleshooting" href="/services/networking/admin-troubleshooting">
    Diagnose and resolve agent failures and VXLAN connectivity issues
  </Card>
</CardGroup>


# Network Topologies
Source: https://docs.xloud.tech/services/networking/network-topology

Design and visualize multi-tier network topologies in Xloud Cloud Platform. Reference architectures for web, app, and database tier isolation with SDN.

## Overview

Xloud Networking's SDN fabric supports a wide range of topology patterns — from a single
flat network for development environments to fully isolated multi-tier architectures for
production workloads. This page describes common reference topologies, their component
requirements, and how security groups enforce trust boundaries between tiers.

<Note>
  **Prerequisites**

  * Familiarity with [networks](/services/networking/create-network), [subnets](/services/networking/subnets), [routers](/services/networking/routers), and [security groups](/services/networking/security-groups)
  * At least one external or provider network available in your cluster
</Note>

***

## Standard Three-Tier Topology

The recommended topology for most production applications. Each application tier is
isolated on its own subnet, all tiers route through a shared router, and only the
web tier exposes floating IPs to the internet.

```mermaid theme={null}
graph TD
    EXT([External Network<br/>203.0.113.0/24]) --> R[Router<br/>main-router]
    R --> WEB[Web Subnet<br/>192.168.10.0/24]
    R --> APP[App Subnet<br/>192.168.20.0/24]
    R --> DB[DB Subnet<br/>192.168.30.0/24]
    WEB --> W1[Web Instance 1]
    WEB --> W2[Web Instance 2]
    APP --> A1[App Instance 1]
    APP --> A2[App Instance 2]
    DB --> D1[DB Instance 1]
    DB --> D2[DB Instance 2]
    W1 -.->|Floating IP| EXT
    W2 -.->|Floating IP| EXT
    style R fill:#197560,color:#fff
    style EXT fill:#3F8F7E,color:#fff
```

### Component Checklist

| Resource                                       | Purpose                                                       |
| ---------------------------------------------- | ------------------------------------------------------------- |
| `web-network` / `web-subnet` (192.168.10.0/24) | Hosts web-tier instances with floating IPs                    |
| `app-network` / `app-subnet` (192.168.20.0/24) | Internal app tier — no floating IPs                           |
| `db-network` / `db-subnet` (192.168.30.0/24)   | Database tier — no floating IPs, restricted access            |
| `main-router`                                  | Routes all subnets, external gateway for NAT                  |
| `web-sg`                                       | Allows TCP 80, 443 from internet; TCP 22 from management CIDR |
| `app-sg`                                       | Allows traffic from `web-sg` only                             |
| `db-sg`                                        | Allows database port from `app-sg` only                       |

<Info>
  Security groups enforce the trust boundary between tiers. Apply a strict group to
  the DB subnet that only allows connections from the App subnet's security group —
  not from `0.0.0.0/0`.
</Info>

***

## Isolated Development Topology

A minimal topology for development and testing environments. All instances share a single
network and subnet. One floating IP provides external access for the developer.

```mermaid theme={null}
graph LR
    EXT([External Network]) --> R[Router]
    R --> DEV[Dev Network<br/>10.0.1.0/24]
    DEV --> I1[Instance 1]
    DEV --> I2[Instance 2]
    DEV --> I3[Instance 3]
    I1 -.->|Floating IP| EXT
    style R fill:#197560,color:#fff
    style EXT fill:#3F8F7E,color:#fff
```

This topology is appropriate for individual developer sandboxes, CI/CD test environments,
and proof-of-concept workloads. It minimizes resource consumption while providing full
internet egress via NAT.

***

## Shared Services Topology

A multi-project topology where shared infrastructure services (monitoring, logging, secrets)
run on a dedicated network accessible to all application projects via router peering.

```mermaid theme={null}
graph TD
    EXT([External Network]) --> CORE_R[Core Router]
    CORE_R --> SHARED[Shared Services Subnet<br/>172.16.0.0/24]
    CORE_R --> APP1[Project A Subnet<br/>192.168.1.0/24]
    CORE_R --> APP2[Project B Subnet<br/>192.168.2.0/24]
    SHARED --> MON[Monitoring]
    SHARED --> LOG[Logging]
    SHARED --> VAULT[Key Management]
    APP1 --> A1[App A Instances]
    APP2 --> A2[App B Instances]
    style CORE_R fill:#197560,color:#fff
    style EXT fill:#3F8F7E,color:#fff
```

***

## High Availability Topology

A topology designed for production availability requirements. Redundant instances in each
tier are distributed across the router's subnet interfaces, with HA floating IPs that can
be reassigned during failover events.

```mermaid theme={null}
graph TD
    EXT([External Network]) --> HA_R[HA Router<br/>VRRP Active/Standby]
    HA_R --> WEB_SUBNET[Web Subnet<br/>192.168.10.0/24]
    WEB_SUBNET --> LB[Load Balancer VIP]
    LB --> W1[Web Instance AZ-1]
    LB --> W2[Web Instance AZ-2]
    W1 --> APP_SUBNET[App Subnet<br/>192.168.20.0/24]
    W2 --> APP_SUBNET
    APP_SUBNET --> A1[App Instance AZ-1]
    APP_SUBNET --> A2[App Instance AZ-2]
    style HA_R fill:#197560,color:#fff
    style EXT fill:#3F8F7E,color:#fff
    style LB fill:#3F8F7E,color:#fff
```

<Tip>
  Enable HA routers (`--ha` flag) for production deployments to protect against L3 agent
  failures. See the [L3 Router Configuration](/services/networking/l3-routing) guide for
  HA and DVR setup.
</Tip>

***

## MTU Considerations

Different network types require different MTU settings to avoid packet fragmentation.

| Network Type           | Recommended MTU | Reason                               |
| ---------------------- | --------------- | ------------------------------------ |
| VXLAN tenant networks  | 1450            | 50-byte VXLAN encapsulation overhead |
| VLAN provider networks | 1500            | No encapsulation overhead            |
| Jumbo-frame VLAN       | Up to 9000      | Requires switch support end-to-end   |

```bash title="Set network MTU for VXLAN" theme={null}
openstack network set app-network --mtu 1450
```

***

## Next Steps

<CardGroup>
  <Card title="Create a Network" href="/services/networking/create-network">
    Provision the networks required for your chosen topology
  </Card>

  <Card title="Routers and Gateways" href="/services/networking/routers">
    Connect your subnets and configure the external gateway
  </Card>

  <Card title="Network Security Groups" href="/services/networking/security-groups">
    Define trust boundaries between tiers with stateful firewall rules
  </Card>

  <Card title="L3 Router Configuration" href="/services/networking/l3-routing">
    Enable HA routers and distributed virtual routing for production deployments
  </Card>
</CardGroup>


# Provider Networks
Source: https://docs.xloud.tech/services/networking/provider-networks

Configure VLAN, flat, and VXLAN provider networks in Xloud. Map physical network interfaces to tenant segmentation types for data center integration.

## Overview

Provider networks connect tenant virtual networks directly to the physical underlay.
Administrators define the physical interface mapping and segmentation type during
deployment or reconfiguration. Tenants cannot modify provider network attributes —
they select from the options an administrator has made available.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * XDeploy access for cluster-level interface configuration
  * Physical network interfaces identified and mapped in XDeploy
</Note>

***

## Network Type Comparison

| Type      | Segmentation            | Scale                         | Use Case                               |
| --------- | ----------------------- | ----------------------------- | -------------------------------------- |
| **Flat**  | None                    | 1 per physical interface      | Untagged management or public networks |
| **VLAN**  | 802.1Q VLAN ID (1–4094) | \~4000 per physical interface | Traditional data center integration    |
| **VXLAN** | VNI (1–16 million)      | Virtually unlimited           | Large-scale multi-tenant clouds        |

***

## Configure Physical Interface Mappings

Provider network parameters are set during initial cluster deployment or reconfiguration
via XDeploy. These settings apply to all cluster nodes.

<Tabs>
  <Tab title="XDeploy" icon="rocket">
    <Steps>
      <Step title="Open Networking configuration" icon="compass">
        In XDeploy, navigate to **Configuration → Networking**.
      </Step>

      <Step title="Configure physical interface mapping" icon="settings">
        Set the interface-to-bridge mapping for each network type:

        | Parameter                     | Example Value      | Description                               |
        | ----------------------------- | ------------------ | ----------------------------------------- |
        | `neutron_bridge_mappings`     | `physnet1:br-ex`   | Maps a physical network name to a bridge  |
        | `neutron_flat_networks`       | `physnet1`         | Networks carrying untagged (flat) traffic |
        | `neutron_network_vlan_ranges` | `physnet1:100:200` | VLAN ID range allocated to tenants        |
        | `neutron_tunnel_types`        | `vxlan`            | Overlay type for tenant networks          |
      </Step>

      <Step title="Apply the configuration" icon="play">
        Click **Save and Deploy**. XDeploy applies the settings to all cluster nodes
        via xavs-ansible.

        <Warning>
          Modifying bridge mappings on a running cluster briefly interrupts networking
          on affected nodes. Schedule this change during a maintenance window and notify
          tenants in advance.
        </Warning>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Create Provider Networks

<Tabs>
  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate as admin" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create a VLAN provider network" icon="layers">
        ```bash title="Create VLAN provider network" theme={null}
        openstack network create provider-vlan100 \
          --provider-network-type vlan \
          --provider-physical-network physnet1 \
          --provider-segment 100 \
          --share
        ```

        | Option                        | Description                                  |
        | ----------------------------- | -------------------------------------------- |
        | `--provider-network-type`     | `vlan`, `flat`, or `vxlan`                   |
        | `--provider-physical-network` | Name matching `neutron_bridge_mappings` key  |
        | `--provider-segment`          | VLAN ID (VLAN type) or VNI (VXLAN type)      |
        | `--share`                     | Makes the network accessible to all projects |
      </Step>

      <Step title="Create a flat provider network" icon="minus">
        ```bash title="Create flat provider network" theme={null}
        openstack network create external-flat \
          --provider-network-type flat \
          --provider-physical-network physnet1 \
          --external \
          --share
        ```

        Use `--external` to mark the network as a valid target for router external gateways.
      </Step>

      <Step title="Create a VXLAN network" icon="layers">
        ```bash title="Create VXLAN tenant network" theme={null}
        openstack network create overlay-vxlan \
          --provider-network-type vxlan \
          --provider-segment 10001
        ```

        VXLAN networks use a VNI as the segment identifier. Omit `--share` to scope
        the network to a specific project.
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Provider Network Administration

### List Provider Networks

```bash title="List all networks with provider attributes" theme={null}
openstack network list --long
```

### Update a Provider Network

```bash title="Update provider network description" theme={null}
openstack network set provider-vlan100 --description "VLAN 100 for production tier"
```

### Delete a Provider Network

```bash title="Delete provider network" theme={null}
openstack network delete provider-vlan100
```

<Warning>
  Deleting a provider network disconnects all instances attached to it. Confirm that
  no instances, routers, or floating IPs are using the network before deleting.
</Warning>

***

## Segmentation Reference

| Network Type | Segment Identifier | Range                          | Notes                                        |
| ------------ | ------------------ | ------------------------------ | -------------------------------------------- |
| VLAN         | VLAN ID            | 1–4094                         | Requires upstream switch trunk configuration |
| VXLAN        | VNI                | 1–16,777,215                   | Software overlay — no switch config required |
| Flat         | None               | N/A — 1 per physical interface | Untagged — use only for management networks  |

<Info>
  VXLAN tenant networks are the default type in Xloud deployments. They require no
  switch configuration and scale to tens of thousands of isolated tenant networks.
  VLAN is recommended when direct integration with physical data center switching is required.
</Info>

***

## Next Steps

<CardGroup>
  <Card title="Network Agent Management" href="/services/networking/network-agents">
    Verify L2 agents are healthy on all compute nodes after changing bridge mappings
  </Card>

  <Card title="L3 Router Configuration" href="/services/networking/l3-routing">
    Configure routers to use the provider networks you just created as external gateways
  </Card>

  <Card title="Service Architecture" href="/services/networking/architecture">
    Understand how L2 agents program provider network attachments
  </Card>

  <Card title="Admin Troubleshooting" href="/services/networking/admin-troubleshooting">
    Resolve VXLAN tunnel and provider network connectivity issues
  </Card>
</CardGroup>


# Network Quality of Service
Source: https://docs.xloud.tech/services/networking/qos

Apply bandwidth limits, DSCP marking, and minimum bandwidth guarantees to network ports. Define and manage QoS policies.

## Overview

Network QoS (Quality of Service) policies enforce traffic controls at the virtual switch level on each hypervisor. Policies apply bandwidth limits, burst allowances, DSCP markings, and minimum bandwidth guarantees to individual ports or entire networks. Controls are enforced by the L2 agent — no guest OS configuration is required.

Administrators create and optionally share policies; project users apply them to their ports and networks.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * QoS service plugin enabled in your cluster (configured via XDeploy under **Networking → Plugins**)
  * Open vSwitch or Linux Bridge L2 agent running on all compute nodes
</Note>

***

## QoS Rule Types

Xloud Networking supports four QoS rule types. Multiple rules of different types can be combined within a single policy.

| Rule Type               | CLI `--type`          | Direction        | Hardware Required | Description                                                   |
| ----------------------- | --------------------- | ---------------- | ----------------- | ------------------------------------------------------------- |
| **Bandwidth Limit**     | `bandwidth-limit`     | Egress / Ingress | No                | Cap maximum throughput with optional burst allowance          |
| **DSCP Marking**        | `dscp-marking`        | Egress only      | No                | Mark outgoing packets with a DSCP value for QoS-aware routing |
| **Minimum Bandwidth**   | `minimum-bandwidth`   | Egress / Ingress | Yes (SR-IOV)      | Guarantee a minimum throughput floor                          |
| **Minimum Packet Rate** | `minimum-packet-rate` | Egress / Ingress | Yes (SR-IOV)      | Guarantee a minimum packet-per-second rate                    |

<Info>
  Minimum bandwidth and minimum packet rate guarantees require SR-IOV hardware and a compatible network backend. Standard OVS-based ports support bandwidth limits and DSCP marking only.
</Info>

***

## Create a QoS Policy

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to QoS Policies" icon="compass">
        Navigate to **Network > QoS Policies** (admin view). Click **Create Policy**.
      </Step>

      <Step title="Define the policy" icon="settings">
        | Field       | Value                | Description                                     |
        | ----------- | -------------------- | ----------------------------------------------- |
        | **Name**    | e.g., `10mbps-limit` | Descriptive label for the policy                |
        | **Shared**  | Enabled              | Makes the policy available to all projects      |
        | **Default** | Optional             | Applies this policy to all new ports by default |
      </Step>

      <Step title="Add a bandwidth limit rule" icon="sliders">
        Open the newly created policy and click **Add Bandwidth Limit Rule**:

        | Field              | Value   | Description                             |
        | ------------------ | ------- | --------------------------------------- |
        | **Max Kbps**       | `10240` | Maximum sustained throughput (10 Mbps)  |
        | **Max Burst Kbps** | `20480` | Allowed burst above the limit (20 Mbps) |
        | **Direction**      | Egress  | Traffic direction to limit              |

        <Check>Click **Save** to activate the rule.</Check>
      </Step>

      <Step title="Apply to a port or network" icon="link">
        To apply to a specific port: navigate to **Network > Ports** (admin view), select the
        port, click **Edit Port**, and set the **QoS Policy** field.

        To apply to an entire network: navigate to **Network > Networks** (admin view), select
        the network, click **Edit Network**, and set the **QoS Policy** field.

        <Info>
          Applying a QoS policy to a network sets a default for all **new** ports on that
          network. Existing ports are not retroactively updated.
        </Info>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the policy" icon="plus">
        ```bash title="Create shared QoS policy" theme={null}
        openstack network qos policy create 10mbps-limit --share
        ```
      </Step>

      <Step title="Add a bandwidth limit rule" icon="sliders">
        ```bash title="Add egress bandwidth limit (10 Mbps, 20 Mbps burst)" theme={null}
        openstack network qos rule create 10mbps-limit \
          --type bandwidth-limit \
          --max-kbps 10240 \
          --max-burst-kbps 20480 \
          --egress
        ```

        ```bash title="Add ingress bandwidth limit" theme={null}
        openstack network qos rule create 10mbps-limit \
          --type bandwidth-limit \
          --max-kbps 10240 \
          --max-burst-kbps 20480 \
          --ingress
        ```
      </Step>

      <Step title="Apply to a port" icon="link">
        ```bash title="Apply QoS policy to a specific port" theme={null}
        openstack port set <port-id> --qos-policy 10mbps-limit
        ```
      </Step>

      <Step title="Apply to a network" icon="network">
        ```bash title="Apply QoS policy to an entire network" theme={null}
        openstack network set app-network --qos-policy 10mbps-limit
        ```

        <Check>All new ports on `app-network` will inherit the 10 Mbps limit automatically.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## DSCP Marking

DSCP (Differentiated Services Code Point) marking tags outgoing packets so that upstream routers and switches can prioritize traffic flows. This is essential for real-time workloads such as VoIP, video conferencing, and database replication traffic.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open a QoS policy" icon="compass">
        Navigate to **Network > QoS Policies** (admin view) and open an existing policy or
        create a new one.
      </Step>

      <Step title="Add DSCP marking rule" icon="tag">
        Click **Add DSCP Marking Rule** and set the DSCP value:

        | Traffic Class         | DSCP Value | Use Case                  |
        | --------------------- | ---------- | ------------------------- |
        | Best Effort (BE)      | `0`        | Default traffic           |
        | Assured Forwarding 11 | `10`       | Standard business traffic |
        | Assured Forwarding 21 | `18`       | Priority business traffic |
        | Expedited Forwarding  | `46`       | VoIP, real-time traffic   |
        | Class Selector 3      | `24`       | Database / transactional  |

        <Check>Click **Save**. The rule applies immediately to new packet flows on assigned ports.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create QoS policy with DSCP marking for real-time traffic" theme={null}
    openstack network qos policy create realtime-voip --share
    ```

    ```bash title="Add DSCP EF marking (value 46 — Expedited Forwarding)" theme={null}
    openstack network qos rule create realtime-voip \
      --type dscp-marking \
      --dscp-mark 46
    ```

    ```bash title="Apply to a VoIP application port" theme={null}
    openstack port set <voip-port-id> --qos-policy realtime-voip
    ```

    <Tip>
      Combine DSCP marking with a bandwidth limit rule in the same policy to both classify
      and constrain VoIP traffic simultaneously.
    </Tip>
  </Tab>
</Tabs>

***

## Guaranteed Minimum Bandwidth

Minimum bandwidth rules provide a throughput floor — a guarantee that the port will always receive at least the specified bandwidth even under network congestion. This requires SR-IOV hardware and a compatible network backend.

<Warning>
  Minimum bandwidth guarantees are only supported on SR-IOV virtual functions (VFs) with a compatible hardware NIC. Standard OVS ports do not enforce minimum bandwidth floors — only maximum limits.
</Warning>

<Tabs>
  <Tab title="CLI" icon="terminal">
    ```bash title="Create minimum bandwidth policy for guaranteed throughput" theme={null}
    openstack network qos policy create guaranteed-1gbps --share
    ```

    ```bash title="Add 1 Gbps minimum bandwidth guarantee (egress)" theme={null}
    openstack network qos rule create guaranteed-1gbps \
      --type minimum-bandwidth \
      --min-kbps 1048576 \
      --egress
    ```

    ```bash title="Apply to an SR-IOV port" theme={null}
    openstack port set <sriov-port-id> --qos-policy guaranteed-1gbps
    ```
  </Tab>

  <Tab title="Minimum Packet Rate" icon="activity">
    Minimum packet rate rules guarantee a floor in packets per second (pps) — important for workloads that generate many small packets (e.g., DNS, financial trading systems).

    ```bash title="Create minimum packet rate policy" theme={null}
    openstack network qos policy create min-pps-policy --share
    ```

    ```bash title="Guarantee 500,000 packets per second minimum" theme={null}
    openstack network qos rule create min-pps-policy \
      --type minimum-packet-rate \
      --min-kpps 500 \
      --egress
    ```

    ```bash title="Apply to trading application port" theme={null}
    openstack port set <trading-port-id> --qos-policy min-pps-policy
    ```
  </Tab>
</Tabs>

***

## Manage QoS Policies

<Tabs>
  <Tab title="List and Inspect" icon="list">
    ```bash title="List all QoS policies" theme={null}
    openstack network qos policy list
    ```

    ```bash title="Show policy details" theme={null}
    openstack network qos policy show 10mbps-limit
    ```

    ```bash title="List rules in a policy" theme={null}
    openstack network qos rule list 10mbps-limit
    ```
  </Tab>

  <Tab title="Update Rules" icon="settings">
    ```bash title="Update bandwidth limit rule" theme={null}
    openstack network qos rule set 10mbps-limit <rule-id> \
      --max-kbps 20480 \
      --max-burst-kbps 40960
    ```

    ```bash title="Update DSCP mark" theme={null}
    openstack network qos rule set realtime-voip <rule-id> \
      --dscp-mark 24
    ```
  </Tab>

  <Tab title="Remove and Delete" icon="trash">
    ```bash title="Remove QoS policy from a port" theme={null}
    openstack port unset <port-id> --qos-policy
    ```

    ```bash title="Remove QoS policy from a network" theme={null}
    openstack network set app-network --no-qos-policy
    ```

    ```bash title="Delete a QoS rule from a policy" theme={null}
    openstack network qos rule delete 10mbps-limit <rule-id>
    ```

    ```bash title="Delete a QoS policy" theme={null}
    openstack network qos policy delete 10mbps-limit
    ```

    <Warning>
      Deleting a QoS policy that is still assigned to ports or networks will fail. Remove
      all assignments before deleting the policy.
    </Warning>
  </Tab>
</Tabs>

***

## Per-Port vs. Per-Network QoS

| Scope           | Application                                  | Use Case                                                                         |
| --------------- | -------------------------------------------- | -------------------------------------------------------------------------------- |
| **Per-port**    | Direct assignment to a specific port         | Granular control per instance NIC — ideal for mixed workload environments        |
| **Per-network** | Applied to a network; inherited by new ports | Consistent SLA enforcement across all instances on a tenant network              |
| **Combined**    | Network policy + port override               | Set a default at network level; override for specific ports that need exceptions |

<Tip>
  Apply QoS at the network level for consistent defaults, then override individual ports for exceptions (e.g., a database port that needs a higher limit than the general application tier).
</Tip>

***

## Validation

Confirm QoS enforcement is active after applying a policy:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > Ports** (admin view) and select the port. The **QoS Policy** field
    should display the assigned policy name. Navigate to the instance and run a bandwidth
    test from inside the guest to confirm enforcement.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Confirm QoS policy applied to port" theme={null}
    openstack port show <port-id> -c qos_policy_id
    ```

    ```bash title="Verify policy rules are active" theme={null}
    openstack network qos rule list <policy-name-or-id>
    ```

    ```bash title="Test bandwidth from inside the guest (requires iperf3)" theme={null}
    # On a target host:
    iperf3 -s
    # On the guest port with the limit applied:
    iperf3 -c <target-ip> -t 30
    ```

    <Check>The measured throughput should not exceed the configured `--max-kbps` value during sustained transfer.</Check>
  </Tab>
</Tabs>

***

## Best Practices

<CardGroup>
  <Card title="Tiered Policy Names" icon="layers">
    Create named tiers — `bronze-10mbps`, `silver-100mbps`, `gold-1gbps` — and share them
    across projects for consistent, predictable service levels.
  </Card>

  <Card title="Apply at Network Level" icon="network">
    Apply QoS to networks for automatic inheritance. Override at the port level only
    for exceptions that need different treatment from the network default.
  </Card>

  <Card title="Burst Sizing" icon="gauge">
    Set burst to 2× the sustained limit. Short traffic spikes are absorbed by the burst
    allowance without impacting the sustained throughput commitment.
  </Card>

  <Card title="Monitor with XIMP" icon="activity">
    Use Xloud XIMP to track port-level throughput in real time and confirm QoS policies
    are enforcing expected limits under production load.
  </Card>
</CardGroup>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="QoS policy not visible in Dashboard" icon="eye">
    **Cause**: The QoS service plugin is not enabled in the Networking configuration.

    **Resolution**: In XDeploy, navigate to **Networking → Plugins** and enable the QoS plugin. Redeploy the networking service:

    ```bash title="Redeploy networking service" theme={null}
    xavs-ansible deploy -t neutron
    ```

    Restart the L2 agent on all compute nodes after the plugin is enabled.
  </Accordion>

  <Accordion title="Bandwidth limit not enforced" icon="sliders">
    **Cause**: The L2 agent on the compute node hosting the instance may not have the QoS extension loaded.

    **Resolution**: Verify the agent has the `qos` extension active:

    ```bash title="Check L2 agent capabilities" theme={null}
    openstack network agent show <agent-id> -c configurations
    ```

    Look for `qos` in the `extensions` list. If absent, restart the L2 agent on the affected compute node.
  </Accordion>

  <Accordion title="Cannot delete QoS policy" icon="trash">
    **Cause**: The policy is still assigned to one or more ports or networks.

    **Resolution**: Find and clear all assignments:

    ```bash title="Find ports using the policy" theme={null}
    openstack port list --long -c ID -c "QoS Policy ID"
    ```

    Remove assignments from each port, then retry the delete.
  </Accordion>

  <Accordion title="Minimum bandwidth rule creation fails" icon="circle-x">
    **Cause**: Minimum bandwidth rules require SR-IOV hardware and a compatible backend (OVS-DPDK or SR-IOV). Standard OVS does not support minimum bandwidth enforcement.

    **Resolution**: Use bandwidth limit rules for maximum throughput control on standard OVS ports. Deploy SR-IOV ports for guaranteed minimum bandwidth requirements.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Network Quotas" href="/services/networking/quotas">
    Limit the number of networking resources per project alongside QoS controls
  </Card>

  <Card title="Provider Networks" href="/services/networking/provider-networks">
    Configure the physical network that QoS policies apply to at the hypervisor level
  </Card>

  <Card title="Service Architecture" href="/services/networking/architecture">
    Understand how L2 agents enforce QoS rules at the virtual switch level
  </Card>

  <Card title="Admin Troubleshooting" href="/services/networking/admin-troubleshooting">
    Diagnose QoS enforcement issues and L2 agent configuration problems
  </Card>
</CardGroup>


# Network Quotas
Source: https://docs.xloud.tech/services/networking/quotas

Manage networking resource quotas per project in Xloud Cloud Platform. Set limits for networks, subnets, routers, floating IPs, and security groups.

## Overview

Quotas limit the number of networking resources a project can consume. Default quotas
are set cluster-wide during deployment. You can override them per project to
accommodate workloads that require higher limits or to restrict projects that should
operate within a strict budget.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * The target project ID or name
</Note>

***

## Default Quota Reference

| Resource             | Default Limit | Notes       |
| -------------------- | ------------- | ----------- |
| Networks             | 10            | Per project |
| Subnets              | 10            | Per project |
| Ports                | 50            | Per project |
| Routers              | 10            | Per project |
| Floating IPs         | 50            | Per project |
| Security Groups      | 10            | Per project |
| Security Group Rules | 100           | Per project |

***

## View Current Quotas

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Identity > Projects** (admin view). Select the project, then click
    **Edit Quota**. Scroll to the **Network** section to review and update networking
    quotas.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Show all quotas for a project" icon="list">
        ```bash title="Show project quotas" theme={null}
        openstack quota show <project-id>
        ```

        The output includes all service quotas — filter with `grep` for networking
        resources:

        ```bash title="Filter networking quotas" theme={null}
        openstack quota show <project-id> | grep -E "network|subnet|port|router|floating|secgroup"
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Update Quotas for a Project

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Identity > Projects** (admin view), select the project, click
    **Edit Quota**, update the values in the **Network** section, and click **Save**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Update networking quotas for a project" theme={null}
    openstack quota set \
      --networks 20 \
      --subnets 20 \
      --ports 100 \
      --routers 10 \
      --floating-ips 100 \
      --secgroups 20 \
      --secgroup-rules 200 \
      <project-id>
    ```

    <Tip>
      Set a quota to `0` to completely block a resource type for a project. Set to
      `-1` for unlimited — use with caution in multi-tenant environments as unlimited
      quotas allow a single project to exhaust shared IP address pools.
    </Tip>
  </Tab>
</Tabs>

***

## Reset Quotas to Defaults

```bash title="Reset project quotas to cluster defaults" theme={null}
openstack quota delete <project-id>
```

This removes all per-project quota overrides. The project reverts to the cluster-wide
default quotas.

***

## View Quota Usage

```bash title="Show quota usage for a project" theme={null}
openstack quota show --usage <project-id>
```

The `--usage` flag adds `In Use` and `Reserved` columns alongside the quota limit, making
it easy to identify projects approaching their limits.

***

## Floating IP Pool Management

Floating IPs are allocated from the external network's address pool. Monitor pool
exhaustion at the cluster level:

```bash title="List all floating IPs across all projects" theme={null}
openstack floating ip list --all-projects
```

```bash title="Count floating IPs by status" theme={null}
openstack floating ip list --all-projects -f value -c Status | sort | uniq -c
```

<Warning>
  If the external network's address pool is exhausted, no new floating IPs can be
  allocated regardless of per-project quota limits. Coordinate with your network
  administrator to expand the provider network's address range.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Quality of Service" href="/services/networking/qos">
    Control bandwidth alongside quota management for fair resource sharing
  </Card>

  <Card title="Security Hardening" href="/services/networking/security">
    Harden network security policies across all projects
  </Card>

  <Card title="Provider Networks" href="/services/networking/provider-networks">
    Expand the external network pool to support more floating IP allocations
  </Card>

  <Card title="Admin Troubleshooting" href="/services/networking/admin-troubleshooting">
    Diagnose quota exhaustion and resource allocation failures
  </Card>
</CardGroup>


# Routers and Gateways
Source: https://docs.xloud.tech/services/networking/routers

Create L3 routers to connect subnets and provide external gateway access. Manage interfaces, SNAT, and floating IP routing.

## Overview

Routers connect subnets within your project and provide external gateway access for
floating IP NAT and outbound internet connectivity. Each router can connect multiple
subnets and optionally attach to an external network for public access.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Create a Router

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Routers">
        Navigate to **Network > Routers** in the sidebar. Click **Create Router**.
      </Step>

      <Step title="Configure the router">
        | Field                              | Type               | Required                    | Description                                            |
        | ---------------------------------- | ------------------ | --------------------------- | ------------------------------------------------------ |
        | **Name**                           | Text               | Yes                         | Router display name                                    |
        | **Description**                    | Text area          | No                          | Optional notes                                         |
        | **Availability Zone Hints**        | Multi-select table | No                          | Pin to specific availability zones                     |
        | **Options: Open External Gateway** | Checkbox           | No                          | Toggle to attach an external gateway                   |
        | **External Gateway**               | Select table       | Required if gateway enabled | Choose the provider/public network for internet access |

        <Tip>
          Selecting an external network sets the router's default gateway and enables
          NAT for floating IP allocation. Leave unchecked for purely internal routing
          between project subnets.
        </Tip>
      </Step>

      <Step title="Create the router">
        Click **Confirm**. The router appears in the list.

        <Check>Router shows status **Active** with the external gateway (if configured).</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Create router with external gateway" theme={null}
      openstack router create main-router \
        --external-gateway public
      ```

      ```bash title="Create internal-only router" theme={null}
      openstack router create internal-router
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## View Routers

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > Routers**. The list shows:

    | Column                    | Description                                      |
    | ------------------------- | ------------------------------------------------ |
    | **ID/Name**               | Router identifier (clickable to view details)    |
    | **Status**                | Active or Error                                  |
    | **Open External Gateway** | Whether an external gateway is attached (Yes/No) |
    | **External Network**      | Name of the external network (with link)         |
    | **External Fixed IP**     | IP address on the external network               |
    | **Created At**            | Creation timestamp                               |

    Filter by **Name** or **Status**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List routers" theme={null}
      openstack router list
      ```

      ```bash title="Show router details" theme={null}
      openstack router show main-router
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Manage Router Interfaces

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a router name to open the detail page. The **Interfaces** tab shows all
    connected subnets and ports.

    | Column            | Description                       |
    | ----------------- | --------------------------------- |
    | **Port ID**       | Interface port identifier         |
    | **Bind Resource** | Instance link (if applicable)     |
    | **Owned Network** | Network this interface belongs to |
    | **MAC Address**   | Hardware address                  |
    | **Status**        | Active or Down                    |

    **Connect a Subnet**: Click the **More** dropdown on the router row and select
    **Connect Subnet**. Choose the subnet to attach.

    **Disconnect a Subnet**: Click the **More** dropdown and select
    **Disconnect Subnet**. Choose the subnet to remove.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Add subnet interface to router" theme={null}
      openstack router add subnet main-router app-subnet
      ```

      ```bash title="Remove subnet interface" theme={null}
      openstack router remove subnet main-router app-subnet
      ```

      ```bash title="List router ports" theme={null}
      openstack port list --router main-router
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Manage External Gateway

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    From the router row's **More** dropdown:

    | Action            | When Available                  | Description                            |
    | ----------------- | ------------------------------- | -------------------------------------- |
    | **Set Gateway**   | No external gateway attached    | Attach an external network             |
    | **Close Gateway** | External gateway attached       | Remove the external gateway            |
    | **Enable SNAT**   | Gateway attached, SNAT disabled | Enable source NAT for outbound traffic |
    | **Disable SNAT**  | Gateway attached, SNAT enabled  | Disable source NAT                     |
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Set external gateway" theme={null}
      openstack router set --external-gateway public main-router
      ```

      ```bash title="Remove external gateway" theme={null}
      openstack router unset --external-gateway main-router
      ```

      ```bash title="Enable SNAT" theme={null}
      openstack router set --enable-snat main-router
      ```

      ```bash title="Disable SNAT" theme={null}
      openstack router set --disable-snat main-router
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Delete a Router

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click the **More** dropdown on the router row and select **Delete**.

    <Danger>
      All subnet interfaces must be disconnected before a router can be deleted.
      Disconnect all subnets first, then delete the router.
    </Danger>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Remove interfaces then delete" theme={null}
    openstack router remove subnet main-router app-subnet
    openstack router delete main-router
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Floating IPs" href="/services/networking/floating-ips">
    Allocate public IPs and associate them with instances via this router
  </Card>

  <Card title="Create a Network" href="/services/networking/create-network">
    Provision networks and subnets to connect to this router
  </Card>

  <Card title="Security Groups" href="/services/networking/security-groups">
    Control traffic with per-port firewall rules
  </Card>

  <Card title="VPN" href="/services/networking/vpn">
    Create IPsec site-to-site VPN tunnels through this router
  </Card>
</CardGroup>


# Network Security Hardening
Source: https://docs.xloud.tech/services/networking/security

Harden Xloud Networking with port security, anti-spoofing, allowed address pairs, and default security group hardening for production environments.

## Overview

Xloud Networking enforces several layers of security at the virtual port level — MAC and
IP anti-spoofing, stateful security groups, and port security policies. This guide covers
administrator-level hardening steps to strengthen these controls for production
deployments.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * Familiarity with [security groups](/services/networking/security-groups) and
    [provider networks](/services/networking/provider-networks)
</Note>

***

## Port Security and Anti-Spoofing

Port security enforces MAC and IP anti-spoofing rules on every virtual port. It is
enabled by default on all networks. Disabling it is a security exception that should
be documented and reviewed.

<Tabs>
  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Verify port security is enabled" icon="shield">
        ```bash title="Check port security on network" theme={null}
        openstack network show app-network -f json | grep port_security_enabled
        ```

        Output should show `"port_security_enabled": true`.
      </Step>

      <Step title="Check per-port security" icon="search">
        ```bash title="Check port security on specific port" theme={null}
        openstack port show <port-id> -f json | grep port_security_enabled
        ```
      </Step>

      <Step title="Disable port security for network appliances (exceptional)" icon="settings">
        In rare cases, network appliances that use multiple source IPs (virtual firewalls,
        load balancers, NAT devices) require port security to be disabled on their specific port:

        ```bash title="Disable port security on a single port" theme={null}
        openstack port set <port-id> \
          --no-security-group \
          --disable-port-security
        ```

        <Danger>
          Disabling port security removes all anti-spoofing enforcement on that port.
          Apply this only to ports owned by trusted, administratively controlled devices.
          Document the exception and review it quarterly.
        </Danger>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Allowed Address Pairs

Allowed address pairs permit a port to send and receive traffic using additional IP or
MAC addresses beyond its primary assignment. Required for virtual IP scenarios such as
keepalived, VRRP, and CARP.

```bash title="Add allowed address pair to port" theme={null}
openstack port set <port-id> \
  --allowed-address ip-address=192.168.10.200,mac-address=fa:16:3e:xx:xx:xx
```

```bash title="Add IP-only allowed address pair (any MAC)" theme={null}
openstack port set <port-id> \
  --allowed-address ip-address=192.168.10.200
```

```bash title="List current allowed address pairs" theme={null}
openstack port show <port-id> -f json | grep allowed_address_pairs
```

<Tip>
  For keepalived VIPs, add the virtual IP as an allowed address pair on all instances
  that participate in the VRRP group. The active node uses the VIP; the standby holds
  it in readiness without generating anti-spoofing violations.
</Tip>

***

## Default Security Group Hardening

The default security group Xloud Networking creates for each project allows all egress
traffic and all inbound traffic from members of the same group. For production
environments, harden this group by removing the permissive inbound rule.

<Tabs>
  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="List rules in the default group" icon="list">
        ```bash title="List default security group rules" theme={null}
        openstack security group rule list default --ingress
        ```
      </Step>

      <Step title="Identify the permissive same-group rule" icon="search">
        Look for a rule with `Remote Security Group: default` and no protocol restriction.
        This rule allows all traffic from instances in the same group.
      </Step>

      <Step title="Remove the permissive rule" icon="trash-2">
        ```bash title="Delete the permissive rule" theme={null}
        openstack security group rule delete <rule-id>
        ```

        <Warning>
          Modifying the default security group affects all instances in the project that
          have not been assigned an explicit security group. Test in a non-production
          project first, and verify that application-to-application traffic that relied
          on this rule has an explicit rule in place.
        </Warning>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Network-Level Security Checklist

<AccordionGroup>
  <Accordion title="Port security enabled on all networks" icon="shield">
    ```bash title="Audit port security across all networks" theme={null}
    openstack network list -f value -c ID | xargs -I{} openstack network show {} -f json | grep -E '"id"|port_security_enabled'
    ```

    Any network showing `"port_security_enabled": false` should be reviewed.
  </Accordion>

  <Accordion title="No wildcard SSH rules in production" icon="lock">
    ```bash title="Find security group rules allowing SSH from any IP" theme={null}
    openstack security group rule list --all-projects --protocol tcp --dst-port 22 --ingress | grep "0.0.0.0/0"
    ```

    Each result is a potential security risk. Work with project owners to restrict
    these rules to management CIDRs or bastion host addresses.
  </Accordion>

  <Accordion title="Unused floating IPs released" icon="globe">
    ```bash title="List DOWN (unassociated) floating IPs" theme={null}
    openstack floating ip list --all-projects --status DOWN
    ```

    Unused floating IPs consume addresses from the external pool. Coordinate with
    project owners to release IPs that are no longer needed.
  </Accordion>

  <Accordion title="Router external gateways reviewed" icon="route">
    ```bash title="List all routers with external gateways" theme={null}
    openstack router list --all-projects -f json | grep external_gateway_info
    ```

    Confirm each router's external gateway is intentional. Unintended gateways can
    expose tenant networks to external routing.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Network Security Groups" href="/services/networking/security-groups">
    User guide for creating and managing per-port firewall rules
  </Card>

  <Card title="Network Quotas" href="/services/networking/quotas">
    Limit resource consumption to reduce attack surface
  </Card>

  <Card title="Provider Networks" href="/services/networking/provider-networks">
    Control physical network access at the provider layer
  </Card>

  <Card title="Admin Troubleshooting" href="/services/networking/admin-troubleshooting">
    Diagnose port security and anti-spoofing configuration issues
  </Card>
</CardGroup>


# Network Security Groups
Source: https://docs.xloud.tech/services/networking/security-groups

Create and manage stateful firewall rules for Xloud instances. Control ingress and egress traffic by protocol, port, and CIDR with security groups.

## Overview

Security groups are stateful, per-port firewall rulesets enforced at the hypervisor level.
Every instance begins with a default security group that blocks all inbound traffic. Add
rules to permit the specific protocols your workload requires — changes take effect
immediately without a restart or interface bounce.

<Note>
  **Prerequisites**

  * An active Xloud project with at least one running instance
  * Dashboard access or CLI configured with valid credentials
</Note>

***

## Create a Security Group

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Security Groups" icon="compass">
        Navigate to
        **Network > Security Groups**. Click **Create Security Group**.
      </Step>

      <Step title="Name the group" icon="settings">
        | Field           | Description                                |
        | --------------- | ------------------------------------------ |
        | **Name**        | Short, descriptive name, e.g., `web-sg`    |
        | **Description** | Optional — e.g., "HTTP/HTTPS for web tier" |

        <Info>
          Each new security group automatically includes two egress rules that allow all
          outbound IPv4 and IPv6 traffic. Add ingress rules for the specific ports your
          workload exposes.
        </Info>
      </Step>

      <Step title="Add ingress rules">
        Click the security group name to open the detail page, then click **Add Rule**.

        The Add Rule form has these fields:

        | Field                | Type      | Required    | Options                           |
        | -------------------- | --------- | ----------- | --------------------------------- |
        | **Direction**        | Radio     | Yes         | Ingress, Egress                   |
        | **Ether Type**       | Radio     | Yes         | IPv4, IPv6                        |
        | **Protocol**         | Dropdown  | Yes         | TCP, UDP, ICMP, ANY               |
        | **Port Range Min**   | Number    | Conditional | Shown for TCP/UDP only            |
        | **Port Range Max**   | Number    | Conditional | Shown for TCP/UDP only            |
        | **Remote IP Prefix** | Text      | No          | CIDR notation (e.g., `0.0.0.0/0`) |
        | **Description**      | Text area | No          | Rule description                  |

        Common rules to add:

        | Direction | Protocol | Port Range | Remote               | Purpose              |
        | --------- | -------- | ---------- | -------------------- | -------------------- |
        | Ingress   | TCP      | 80         | 0.0.0.0/0            | HTTP                 |
        | Ingress   | TCP      | 443        | 0.0.0.0/0            | HTTPS                |
        | Ingress   | TCP      | 22         | `<your-office-CIDR>` | SSH management       |
        | Ingress   | ICMP     | Any        | 0.0.0.0/0            | Ping and diagnostics |

        <Warning>
          Avoid rules with remote `0.0.0.0/0` for SSH (port 22) in production.
          Restrict to your management CIDR or route SSH through a bastion host.
        </Warning>
      </Step>

      <Step title="Assign to an instance" icon="link">
        Navigate to **Compute > Instances**, click the **More** dropdown on the instance
        row, then select **Manage Security Group** under **Related Resources**. Select
        a port and add `web-sg` to the assigned security groups.

        <Check>The rule takes effect immediately — no restart required.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the security group" icon="plus">
        ```bash title="Create security group" theme={null}
        openstack security group create web-sg \
          --description "HTTP/HTTPS for web tier"
        ```
      </Step>

      <Step title="Add ingress rules" icon="shield">
        ```bash title="Allow HTTP" theme={null}
        openstack security group rule create web-sg \
          --protocol tcp --dst-port 80 --ingress --remote-ip 0.0.0.0/0
        ```

        ```bash title="Allow HTTPS" theme={null}
        openstack security group rule create web-sg \
          --protocol tcp --dst-port 443 --ingress --remote-ip 0.0.0.0/0
        ```

        ```bash title="Allow SSH from management network" theme={null}
        openstack security group rule create web-sg \
          --protocol tcp --dst-port 22 --ingress --remote-ip 10.0.0.0/8
        ```

        ```bash title="Allow ICMP" theme={null}
        openstack security group rule create web-sg \
          --protocol icmp --ingress --remote-ip 0.0.0.0/0
        ```
      </Step>

      <Step title="Assign to an instance" icon="link">
        ```bash title="Add security group to instance" theme={null}
        openstack server add security group my-instance web-sg
        ```

        <Check>Rules apply immediately without a reboot or interface restart.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Common Rules Reference

| Use Case           | Direction | Protocol | Port | Remote          |
| ------------------ | --------- | -------- | ---- | --------------- |
| HTTP web traffic   | Ingress   | TCP      | 80   | 0.0.0.0/0       |
| HTTPS web traffic  | Ingress   | TCP      | 443  | 0.0.0.0/0       |
| SSH access         | Ingress   | TCP      | 22   | Management CIDR |
| ICMP ping          | Ingress   | ICMP     | Any  | 0.0.0.0/0       |
| MySQL / MariaDB    | Ingress   | TCP      | 3306 | App tier CIDR   |
| PostgreSQL         | Ingress   | TCP      | 5432 | App tier CIDR   |
| Redis              | Ingress   | TCP      | 6379 | App tier CIDR   |
| Custom UDP service | Ingress   | UDP      | 1194 | 0.0.0.0/0       |
| All outbound       | Egress    | Any      | Any  | 0.0.0.0/0       |

***

## Source Security Group Rules

Rules can reference another security group as the remote source instead of a CIDR.
This allows traffic from any instance assigned the referenced group, regardless of IP.

```bash title="Allow traffic from app tier security group" theme={null}
openstack security group rule create db-sg \
  --protocol tcp \
  --dst-port 5432 \
  --ingress \
  --remote-group app-sg
```

<Tip>
  Security group references are more maintainable than CIDR-based rules in dynamic
  environments — you add or remove instances from the source group rather than updating
  IP ranges in rules.
</Tip>

***

## Manage Rules and Groups

### Remove a Rule

```bash title="List rules in a group" theme={null}
openstack security group rule list web-sg
```

```bash title="Delete a specific rule" theme={null}
openstack security group rule delete <rule-id>
```

### Remove a Security Group from an Instance

```bash title="Remove security group from instance" theme={null}
openstack server remove security group my-instance web-sg
```

### Delete a Security Group

```bash title="Delete security group" theme={null}
openstack security group delete web-sg
```

<Warning>
  Deleting a security group that is still assigned to instances will fail. Remove
  all instance assignments before deleting the group.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Floating IP Addresses" href="/services/networking/floating-ips">
    Associate public IPs with instances — ensure your security group allows inbound traffic first
  </Card>

  <Card title="Security Hardening" href="/services/networking/security">
    Administrator guide for port security, anti-spoofing, and default group hardening
  </Card>

  <Card title="Network Troubleshooting" href="/services/networking/troubleshooting">
    Diagnose security group and connectivity issues
  </Card>

  <Card title="Create a Network" href="/services/networking/create-network">
    Set up the network your secured instances attach to
  </Card>
</CardGroup>


# Create and Manage Subnets
Source: https://docs.xloud.tech/services/networking/subnets

Add, configure, and manage subnets within Xloud tenant networks. Control DHCP, allocation pools, DNS, and host routes from the Dashboard or CLI.

## Overview

Subnets define the IP address space within a network. Each subnet specifies a CIDR block,
optional DHCP assignment, a gateway IP, and DNS resolvers that are pushed to instances at
boot. A single network can host multiple subnets — e.g., one IPv4 and one IPv6
range — or separate subnets for different application tiers.

<Note>
  **Prerequisites**

  * An existing Xloud tenant network (see [Create a Network](/services/networking/create-network))
  * Dashboard access or CLI configured with valid credentials
</Note>

***

## Add a Subnet to an Existing Network

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the network" icon="compass">
        Navigate to **Network > Networks** and click the network name to open it.
      </Step>

      <Step title="Add a subnet" icon="plus">
        Click the **Subnets** tab, then click **Create Subnet**.
      </Step>

      <Step title="Configure CIDR and gateway" icon="settings">
        | Field           | Description                                                          |
        | --------------- | -------------------------------------------------------------------- |
        | **Subnet Name** | Descriptive name, e.g., `db-subnet`                                  |
        | **CIDR**        | CIDR block, e.g., `10.0.2.0/24`                                      |
        | **IP Version**  | IPv4 or IPv6                                                         |
        | **Gateway IP**  | Optional — leave blank for auto-assign or enter `0.0.0.0` to disable |
      </Step>

      <Step title="Configure DHCP and allocation options" icon="sliders">
        | Option               | Description                                                     |
        | -------------------- | --------------------------------------------------------------- |
        | **Enable DHCP**      | Provide automatic IP assignment for instances on this subnet    |
        | **Allocation Pools** | Restrict the DHCP-assigned range, e.g., `10.0.2.10,10.0.2.200`  |
        | **DNS**              | Comma-separated list of resolvers                               |
        | **Host Routes**      | Static routes injected via DHCP in `destination,nexthop` format |

        <Tip>
          Use allocation pools to reserve address ranges for manually configured hosts
          such as database nodes or network appliances with static IPs.
        </Tip>
      </Step>

      <Step title="Create the subnet" icon="circle-check">
        Click **Create** to save the subnet.

        <Check>The subnet appears in the network's Subnets tab with status **Active**.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Add subnet with DHCP" icon="plus">
        ```bash title="Create subnet with allocation pool" theme={null}
        openstack subnet create db-subnet \
          --network app-network \
          --subnet-range 10.0.2.0/24 \
          --gateway 10.0.2.1 \
          --allocation-pool start=10.0.2.10,end=10.0.2.200 \
          --dns-nameserver 8.8.8.8
        ```
      </Step>

      <Step title="Add subnet without DHCP (static IPs)" icon="minus">
        ```bash title="Create subnet with DHCP disabled" theme={null}
        openstack subnet create db-subnet-static \
          --network app-network \
          --subnet-range 10.0.2.0/24 \
          --no-dhcp
        ```

        Use `--no-dhcp` for subnets where IP assignments are managed manually — e.g.,
        database tiers where each node is pre-configured with a fixed address.
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="Show subnet details" theme={null}
        openstack subnet show db-subnet
        ```

        <Check>Confirm the CIDR, gateway, and DHCP settings match the intended configuration.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Subnet Management Operations

### Update DNS Resolvers

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > Networks**, open the subnet row, and click **Edit Subnet**.
    Update the **DNS** field in the **Advanced Options**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Update DNS on a subnet" theme={null}
    openstack subnet set app-subnet \
      --dns-nameserver 10.0.0.53 \
      --dns-nameserver 8.8.8.8
    ```

    Existing instances pick up the new resolvers on their next DHCP renewal. To apply
    immediately inside the guest OS, run `dhclient -r && dhclient` on the instance.
  </Tab>
</Tabs>

### Add Host Routes

Host routes are injected via DHCP and add static routing entries on the guest OS. Use
them to direct traffic for specific CIDRs through a dedicated gateway.

```bash title="Add host route to subnet" theme={null}
openstack subnet set app-subnet \
  --host-route destination=172.16.0.0/12,gateway=192.168.10.254
```

### Disable or Enable DHCP

```bash title="Disable DHCP on a subnet" theme={null}
openstack subnet set app-subnet --no-dhcp
```

```bash title="Re-enable DHCP on a subnet" theme={null}
openstack subnet set app-subnet --dhcp
```

<Warning>
  Disabling DHCP on a subnet that has instances running will cause those instances to
  fail IP renewal on the next lease expiry. Ensure all instances on the subnet are
  configured with static IPs before disabling DHCP.
</Warning>

***

## Subnet Configuration Reference

| Parameter       | CLI Flag               | Description                                         |
| --------------- | ---------------------- | --------------------------------------------------- |
| CIDR            | `--subnet-range`       | IP address range in CIDR notation                   |
| Gateway         | `--gateway`            | Default gateway IP for DHCP clients                 |
| DHCP            | `--dhcp` / `--no-dhcp` | Enable or disable automatic IP assignment           |
| Allocation Pool | `--allocation-pool`    | Start/end range for DHCP-assigned addresses         |
| DNS Servers     | `--dns-nameserver`     | Resolvers pushed to instances (repeat for multiple) |
| Host Routes     | `--host-route`         | Static routes injected via DHCP                     |
| IP Version      | `--ip-version`         | `4` for IPv4, `6` for IPv6                          |

***

## Next Steps

<CardGroup>
  <Card title="Routers and Gateways" href="/services/networking/routers">
    Attach subnets to a router to enable inter-subnet routing and internet access
  </Card>

  <Card title="DNS Configuration" href="/services/networking/dns-config">
    Configure DNS name servers and hostname resolution for your subnets
  </Card>

  <Card title="Create a Network" href="/services/networking/create-network">
    Create the network that hosts your subnets
  </Card>

  <Card title="Network Troubleshooting" href="/services/networking/troubleshooting">
    Diagnose and resolve subnet connectivity and DHCP issues
  </Card>
</CardGroup>


# Networking Troubleshooting
Source: https://docs.xloud.tech/services/networking/troubleshooting

Diagnose and resolve common networking issues in Xloud Cloud Platform — connectivity failures, floating IP problems, DHCP errors, and MTU mismatches.

## Overview

This guide covers the most common networking issues encountered in Xloud Cloud Platform
and provides step-by-step resolution procedures. Each scenario includes diagnostic
commands and remediation steps that you can apply from the Dashboard or CLI.

<Note>
  **Prerequisites**

  * CLI configured with valid credentials
  * Access to the **Xloud Dashboard** for instance console access
</Note>

***

## Diagnostic Quick Reference

Before working through individual scenarios, run these commands to gather a complete
picture of your networking state:

```bash title="Check overall network resource status" theme={null}
openstack network list
openstack subnet list
openstack router list
openstack network agent list
```

```bash title="Check instance network attachment" theme={null}
openstack server show my-instance -f json | grep -E "addresses|security_groups"
```

***

## Common Issues

<AccordionGroup>
  <Accordion title="Instance has no network connectivity" icon="wifi-off">
    **Cause**: Missing router interface, DHCP failure, or security group blocking traffic.

    **Resolution**:

    1. Verify the instance received a DHCP-assigned IP:
       ```bash title="Check instance IP assignments" theme={null}
       openstack server show my-instance -f json | grep addresses
       ```
    2. Confirm the subnet has a router interface:
       ```bash title="List router interfaces" theme={null}
       openstack router port list main-router
       ```
    3. Check the security group allows the expected traffic:
       ```bash title="List ingress rules for security group" theme={null}
       openstack security group rule list --ingress web-sg
       ```
    4. Verify the DHCP agent is alive:
       ```bash title="Check DHCP agent status" theme={null}
       openstack network agent list --agent-type dhcp
       ```

    <Tip>
      Use the Dashboard console (VNC) under **Compute > Instances** and use the Console action to
      check whether the instance received an IP from inside the guest OS. A valid IP confirms
      DHCP is working and the issue is at the network or security group layer.
    </Tip>
  </Accordion>

  <Accordion title="Floating IP is unreachable" icon="globe">
    **Cause**: Missing association, router has no external gateway, or the security group
    is missing an ingress rule for the required port.

    **Resolution**:

    1. Confirm the floating IP is associated:
       ```bash title="Show floating IP status" theme={null}
       openstack floating ip show 203.0.113.45
       ```
       The `port_id` field must be non-empty and `status` must be `ACTIVE`.
    2. Verify the router has an external gateway:
       ```bash title="Show router gateway" theme={null}
       openstack router show main-router -f json | grep external_gateway_info
       ```
    3. Check the security group assigned to the instance allows the port you are
       connecting on (e.g., TCP 22 for SSH, TCP 80 for HTTP):
       ```bash title="List security group rules" theme={null}
       openstack security group rule list --ingress <sg-name>
       ```
    4. Ping the floating IP from outside to confirm basic reachability:
       ```bash title="Test ICMP reachability" theme={null}
       ping -c 4 203.0.113.45
       ```
  </Accordion>

  <Accordion title="DHCP not assigning addresses" icon="server">
    **Cause**: DHCP agent is down, DHCP is disabled on the subnet, or the allocation
    pool is exhausted.

    **Resolution**:

    1. Confirm DHCP is enabled on the subnet:
       ```bash title="Check subnet DHCP status" theme={null}
       openstack subnet show app-subnet -f json | grep enable_dhcp
       ```
    2. Check the allocation pool is not exhausted:
       ```bash title="List ports consuming addresses" theme={null}
       openstack port list --network app-network
       ```
       Compare the count against your pool size (a `/24` provides 253 usable addresses).
    3. Verify the DHCP agent is alive:
       ```bash title="List DHCP agents" theme={null}
       openstack network agent list --agent-type dhcp
       ```
       If the agent shows `Alive: False`, contact your administrator to restart the
       agent service. See the [Network Agent Management](/services/networking/network-agents) guide.
  </Accordion>

  <Accordion title="MTU mismatch causing packet fragmentation" icon="split">
    **Cause**: VXLAN encapsulation adds a 50-byte overhead. If instances use the default
    MTU of 1500, large packets are fragmented silently, degrading throughput and causing
    application-level timeouts.

    **Resolution**:

    Update the network MTU to account for encapsulation overhead:

    ```bash title="Set VXLAN-appropriate MTU on network" theme={null}
    openstack network set app-network --mtu 1450
    ```

    Instances receive the updated MTU via DHCP option 26 on next renewal. For
    immediate effect, set the MTU manually on the guest OS:

    ```bash title="Set MTU on Linux guest" theme={null}
    ip link set eth0 mtu 1450
    ```

    <Info>
      For VXLAN networks, Xloud recommends an MTU of `1450`. For VLAN networks backed
      by jumbo-frame-capable switches, you may use up to `9000`.
    </Info>
  </Accordion>

  <Accordion title="Security group rules not taking effect" icon="shield">
    **Cause**: The security group is not assigned to the instance, or the rule was
    added to the wrong group.

    **Resolution**:

    1. List security groups assigned to the instance:
       ```bash title="Show instance security groups" theme={null}
       openstack server show my-instance -f json | grep security_groups
       ```
    2. List rules in the expected group:
       ```bash title="Show all rules in group" theme={null}
       openstack security group rule list web-sg
       ```
    3. If the group is not assigned, add it:
       ```bash title="Assign security group to instance" theme={null}
       openstack server add security group my-instance web-sg
       ```

    <Check>Rules take effect immediately without a restart — re-test connectivity after adding the group.</Check>
  </Accordion>

  <Accordion title="Router not forwarding traffic after failover" icon="route">
    **Cause**: HA router VRRP failover completed but the new master has not programmed
    floating IP NAT rules correctly.

    **Resolution**:

    1. Check the router HA state:
       ```bash title="Show router HA fields" theme={null}
       openstack router show ha-router -f json | grep -E "ha|status"
       ```
    2. List L3 agents for the router and confirm exactly one is active:
       ```bash title="List L3 agents handling the router" theme={null}
       openstack network agent list --router ha-router
       ```
    3. Trigger rescheduling by toggling the router admin state:
       ```bash title="Reschedule router" theme={null}
       openstack router set ha-router --disable
       openstack router set ha-router --enable
       ```
  </Accordion>
</AccordionGroup>

***

## Useful Diagnostic Commands

```bash title="Show all ports on a network" theme={null}
openstack port list --network app-network
```

```bash title="Show port details including fixed IP and security groups" theme={null}
openstack port show <port-id>
```

```bash title="List all floating IPs in the project" theme={null}
openstack floating ip list
```

```bash title="Show network agent heartbeat timestamps" theme={null}
openstack network agent list --long
```

***

## Next Steps

<CardGroup>
  <Card title="Network Agent Management" href="/services/networking/network-agents">
    Monitor and manage SDN agents when diagnostic commands show agent failures
  </Card>

  <Card title="Network Security Groups" href="/services/networking/security-groups">
    Review and update firewall rules to resolve connectivity issues
  </Card>

  <Card title="Routers and Gateways" href="/services/networking/routers">
    Verify and correct router configuration for floating IP and NAT issues
  </Card>

  <Card title="DHCP Configuration" href="/services/networking/dhcp">
    Administer DHCP agents for IP assignment troubleshooting
  </Card>
</CardGroup>


# Networking User Guide
Source: https://docs.xloud.tech/services/networking/user-guide

Configure networks, subnets, routers, floating IPs, and security groups in Xloud Cloud Platform. Step-by-step workflows for the Dashboard and CLI.

<p>Overview</p>

Xloud Networking delivers software-defined connectivity for your cloud workloads. Every
project receives its own isolated network plane — you define the topology, address space,
routing policy, and access controls. Project networks remain fully isolated from one another
and from the physical underlay unless you explicitly connect them through a router with an
external gateway.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

<p>Key Concepts</p>

| Resource           | Description                                                                                          |
| ------------------ | ---------------------------------------------------------------------------------------------------- |
| **Network**        | An isolated L2 broadcast domain. Instances attach to networks via virtual ports.                     |
| **Subnet**         | An IP address range assigned to a network, with DHCP, gateway, and DNS configuration.                |
| **Router**         | An L3 device that routes traffic between subnets and provides external gateway connectivity via NAT. |
| **Port**           | A virtual network interface connecting an instance or router to a network.                           |
| **Floating IP**    | A publicly routable address that maps to a port's private IP via NAT on the router.                  |
| **Security Group** | A stateful, per-port firewall ruleset controlling ingress and egress traffic.                        |

<Tip>
  Start with the simplest topology that meets your requirements: one network, one subnet,
  one router with an external gateway, and a security group. Expand from there as your
  workload grows.
</Tip>

***

<p>Networking Topics</p>

<CardGroup>
  <Card title="Create a Network" icon="network" href="/services/networking/create-network">
    Provision an isolated project network and configure its IP address space with a subnet
  </Card>

  <Card title="Create and Manage Subnets" icon="layers" href="/services/networking/subnets">
    Add subnets, configure DHCP allocation pools, DNS resolvers, and host routes
  </Card>

  <Card title="Routers and Gateways" icon="route" href="/services/networking/routers">
    Connect project subnets to the internet with L3 routers and external gateways
  </Card>

  <Card title="Floating IP Addresses" icon="globe" href="/services/networking/floating-ips">
    Allocate public IPs from the external pool and associate them with instances
  </Card>

  <Card title="Network Security Groups" icon="shield" href="/services/networking/security-groups">
    Define stateful firewall rules controlling ingress and egress traffic per instance port
  </Card>

  <Card title="DNS Configuration" icon="server" href="/services/networking/dns-config">
    Configure DNS name servers pushed to instances via DHCP for hostname resolution
  </Card>

  <Card title="Network Topologies" icon="layout-grid" href="/services/networking/network-topology">
    Reference architectures for three-tier, HA, and shared-services network designs
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/networking/troubleshooting">
    Diagnose and resolve connectivity, floating IP, and DHCP issues
  </Card>
</CardGroup>

***

<p>Getting Started Workflow</p>

```mermaid theme={null}
graph LR
    A[Create Network] --> B[Create Subnet]
    B --> C[Create Router]
    C --> D[Add Router Interface]
    D --> E[Create Security Group]
    E --> F[Create Instance]
    F --> G[Allocate Floating IP]
    G --> H[Associate Floating IP]
    style A fill:#197560,color:#fff
    style H fill:#3F8F7E,color:#fff
```

***

<p>Related Resources</p>

<CardGroup>
  <Card title="Networking Admin Guide" icon="settings" href="/services/networking/admin-guide">
    Provider networks, QoS, quotas, agent management, and security hardening
  </Card>

  <Card title="Compute User Guide" icon="server" href="/services/compute/user-guide">
    Launch instances and attach them to the networks you create here
  </Card>

  <Card title="Authentication" icon="key" href="/cli-setup">
    Configure project credentials and CLI access for networking operations
  </Card>

  <Card title="CLI Setup" icon="terminal" href="/cli-setup">
    Install and configure the `openstack` CLI for networking management
  </Card>
</CardGroup>


# VPN as a Service
Source: https://docs.xloud.tech/services/networking/vpn

Create IPsec site-to-site VPN tunnels for secure inter-site connectivity between your Xloud environment and remote networks.

## Overview

VPN as a Service (VPNaaS) provides IPsec-based site-to-site tunnel connectivity, enabling secure communication between your Xloud private cloud and remote data centers, branch offices, or other cloud environments. VPNaaS is available with XPCI deployments.

<Note>
  **Prerequisites**

  * Active project with `member` role or higher
  * At least one router with an external gateway configured
  * Remote site VPN endpoint details (peer IP, subnets, pre-shared key)
  * VPNaaS enabled by your administrator (`enable_neutron_vpnaas: "yes"`)
</Note>

***

## Key Concepts

| Concept                   | Description                                                                                                                                                                             |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **IKE Policy**            | Defines the Internet Key Exchange parameters used during Phase 1 negotiation — authentication algorithm, encryption algorithm, IKE version (v1 or v2), and key lifetime                 |
| **IPsec Policy**          | Defines the Phase 2 parameters for the data channel — encryption algorithm, authentication algorithm, encapsulation mode (tunnel or transport), and Perfect Forward Secrecy (PFS) group |
| **VPN Service**           | Associates a VPN with a specific router and subnet in your project                                                                                                                      |
| **Endpoint Group**        | Defines the local or remote subnets that participate in the VPN tunnel                                                                                                                  |
| **IPsec Site Connection** | Combines the IKE policy, IPsec policy, VPN service, and endpoint groups into an active tunnel to a remote peer                                                                          |

***

## Supported Algorithms

<Tabs>
  <Tab title="IKE Policy" icon="key">
    | Parameter                    | Supported Values                | Default |
    | ---------------------------- | ------------------------------- | ------- |
    | **IKE Version**              | v1, v2                          | v1      |
    | **Auth Algorithm**           | sha1, sha256, sha384, sha512    | sha1    |
    | **Encryption Algorithm**     | aes-128, aes-192, aes-256, 3des | aes-128 |
    | **Phase 1 Negotiation Mode** | main                            | main    |
    | **Lifetime Value**           | 60 -- 86400 seconds             | 3600    |
    | **PFS**                      | group2, group5, group14         | group5  |

    <Tip>
      Use IKEv2 with AES-256 and SHA-256 for production deployments. IKEv1 is supported for backward compatibility with legacy equipment.
    </Tip>
  </Tab>

  <Tab title="IPsec Policy" icon="shield">
    | Parameter                | Supported Values                | Default |
    | ------------------------ | ------------------------------- | ------- |
    | **Transform Protocol**   | esp, ah, ah-esp                 | esp     |
    | **Auth Algorithm**       | sha1, sha256, sha384, sha512    | sha1    |
    | **Encryption Algorithm** | aes-128, aes-192, aes-256, 3des | aes-128 |
    | **Encapsulation Mode**   | tunnel, transport               | tunnel  |
    | **PFS**                  | group2, group5, group14         | group5  |
    | **Lifetime Value**       | 60 -- 86400 seconds             | 3600    |

    <Tip>
      Enable Perfect Forward Secrecy (PFS) with at least group14 (2048-bit DH) for production tunnels.
    </Tip>
  </Tab>
</Tabs>

***

## Create a VPN Connection

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create an IKE Policy" icon="key">
        Navigate to **Network > VPNs** (IKE Policies tab) and click **Create**.

        | Field                    | Value                                          |
        | ------------------------ | ---------------------------------------------- |
        | **Name**                 | A descriptive name (e.g., `ike-aes256-sha256`) |
        | **IKE Version**          | v2                                             |
        | **Encryption Algorithm** | aes-256                                        |
        | **Auth Algorithm**       | sha256                                         |
        | **PFS**                  | group14                                        |
        | **Lifetime Value**       | 3600                                           |
      </Step>

      <Step title="Create an IPsec Policy" icon="shield">
        Navigate to **Network > VPNs** (IPsec Policies tab) and click **Create**.

        | Field                    | Value                                            |
        | ------------------------ | ------------------------------------------------ |
        | **Name**                 | A descriptive name (e.g., `ipsec-aes256-sha256`) |
        | **Transform Protocol**   | esp                                              |
        | **Encryption Algorithm** | aes-256                                          |
        | **Auth Algorithm**       | sha256                                           |
        | **Encapsulation Mode**   | tunnel                                           |
        | **PFS**                  | group14                                          |
      </Step>

      <Step title="Create a VPN Gateway">
        Navigate to **Network > VPNs** (VPN Gateways tab) and click **Create**.

        | Field      | Value                                                |
        | ---------- | ---------------------------------------------------- |
        | **Name**   | A descriptive name (e.g., `vpn-to-branch-office`)    |
        | **Router** | Select the router with an external gateway           |
        | **Subnet** | Select the local subnet to expose through the tunnel |
      </Step>

      <Step title="Create Endpoint Groups" icon="circle-dot">
        Navigate to **Network > VPNs** (VPN EndPoint Groups tab) and create two endpoint groups:

        **Local Endpoint Group:**

        | Field         | Value                               |
        | ------------- | ----------------------------------- |
        | **Name**      | `local-subnets`                     |
        | **Type**      | subnet                              |
        | **Endpoints** | Select your local project subnet(s) |

        **Remote Endpoint Group:**

        | Field         | Value                                                  |
        | ------------- | ------------------------------------------------------ |
        | **Name**      | `remote-subnets`                                       |
        | **Type**      | cidr                                                   |
        | **Endpoints** | Enter the remote subnet CIDRs (e.g., `192.168.1.0/24`) |
      </Step>

      <Step title="Create the IPsec Site Connection" icon="link">
        Navigate to **Network > VPNs** (IPsec Site Connections tab) and click **Create**.

        | Field                           | Value                                                  |
        | ------------------------------- | ------------------------------------------------------ |
        | **Name**                        | `connection-to-branch`                                 |
        | **VPN Service**                 | Select the VPN service created above                   |
        | **IKE Policy**                  | Select the IKE policy created above                    |
        | **IPsec Policy**                | Select the IPsec policy created above                  |
        | **Local Endpoint Group**        | `local-subnets`                                        |
        | **Peer Endpoint Group**         | `remote-subnets`                                       |
        | **Peer Gateway Public Address** | Public IP of the remote VPN device                     |
        | **Peer ID**                     | Remote peer identifier (typically the peer gateway IP) |
        | **Pre-Shared Key(PSK) String**  | Shared secret agreed upon with the remote site         |

        Click **Create** to establish the tunnel.

        <Check>The connection appears in the list with status **Active** once both sides negotiate successfully.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create an IKE Policy" icon="key">
        ```bash title="Create IKE policy" theme={null}
        openstack vpn ike policy create ike-aes256-sha256 \
          --ike-version v2 \
          --auth-algorithm sha256 \
          --encryption-algorithm aes-256 \
          --pfs group14 \
          --lifetime units=seconds,value=3600
        ```
      </Step>

      <Step title="Create an IPsec Policy" icon="shield">
        ```bash title="Create IPsec policy" theme={null}
        openstack vpn ipsec policy create ipsec-aes256-sha256 \
          --transform-protocol esp \
          --auth-algorithm sha256 \
          --encryption-algorithm aes-256 \
          --pfs group14 \
          --encapsulation-mode tunnel
        ```
      </Step>

      <Step title="Create a VPN Service" icon="server">
        ```bash title="Create VPN service" theme={null}
        openstack vpn service create vpn-to-branch-office \
          --router my-router \
          --subnet my-local-subnet
        ```
      </Step>

      <Step title="Create Endpoint Groups" icon="circle-dot">
        ```bash title="Create local endpoint group" theme={null}
        openstack vpn endpoint group create local-subnets \
          --type subnet \
          --value my-local-subnet
        ```

        ```bash title="Create remote endpoint group" theme={null}
        openstack vpn endpoint group create remote-subnets \
          --type cidr \
          --value 192.168.1.0/24
        ```
      </Step>

      <Step title="Create the IPsec Site Connection" icon="link">
        ```bash title="Create IPsec site connection" theme={null}
        openstack vpn ipsec site connection create connection-to-branch \
          --vpnservice vpn-to-branch-office \
          --ikepolicy ike-aes256-sha256 \
          --ipsecpolicy ipsec-aes256-sha256 \
          --local-endpoint-group local-subnets \
          --peer-endpoint-group remote-subnets \
          --peer-address 203.0.113.50 \
          --peer-id 203.0.113.50 \
          --psk "your-pre-shared-key"
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Validation

Confirm the VPN tunnel is established and operational:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Network > VPNs** (IPsec Site Connections tab). The connection status should display **Active**.

    | Status             | Meaning                                           |
    | ------------------ | ------------------------------------------------- |
    | **Active**         | Tunnel is established and passing traffic         |
    | **Down**           | Tunnel negotiation failed or peer is unreachable  |
    | **Pending Create** | Connection is being provisioned                   |
    | **Error**          | Configuration error — review IKE/IPsec parameters |

    <Check>Connection status is **Active** — the tunnel is operational.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Check connection status" theme={null}
    openstack vpn ipsec site connection show connection-to-branch -c status -c id
    ```

    Expected output:

    ```text title="Expected response" theme={null}
    +--------+--------------------------------------+
    | Field  | Value                                |
    +--------+--------------------------------------+
    | id     | a1b2c3d4-e5f6-7890-abcd-ef1234567890 |
    | status | ACTIVE                               |
    +--------+--------------------------------------+
    ```

    <Check>Status shows `ACTIVE` — the VPN tunnel is operational.</Check>
  </Tab>
</Tabs>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Connection stuck in DOWN status" icon="circle-x">
    **Cause**: The remote peer is unreachable or IKE/IPsec parameters do not match.

    **Resolution**:

    * Verify the peer gateway IP is reachable from the router's external network
    * Confirm that IKE version, encryption, authentication, and PFS settings match on both sides
    * Check that the pre-shared key is identical on both endpoints
    * Verify security group rules allow UDP ports 500 and 4500 (IKE/NAT-T) and IP protocol 50 (ESP)
  </Accordion>

  <Accordion title="Tunnel established but no traffic flows" icon="wifi-off">
    **Cause**: Endpoint group CIDRs do not match between the local and remote configurations.

    **Resolution**:

    * Verify the local endpoint group subnets match what the remote side expects as "remote" subnets
    * Verify the remote endpoint group CIDRs match the actual subnets behind the remote peer
    * Check routing tables on both sides to ensure traffic is directed through the tunnel
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Security Groups" href="/services/networking/security-groups">
    Configure firewall rules to control traffic flow through the VPN tunnel
  </Card>

  <Card title="Routers" href="/services/networking/routers">
    Manage the routers that anchor your VPN services
  </Card>

  <Card title="Network Topology" href="/services/networking/network-topology">
    Visualize your network layout including VPN connections
  </Card>

  <Card title="Key Manager" href="/services/key-manager">
    Store and manage VPN pre-shared keys and certificates securely
  </Card>
</CardGroup>


# Object Storage
Source: https://docs.xloud.tech/services/object-storage

Scalable, durable object storage for Xloud private cloud — store and retrieve data at any scale with container-based organization, ACL controls, and large.

Massively scalable object storage for your private cloud — organize data in containers, control access with ACLs, and store objects of any size.

<Card title="Xloud Object Storage — Scale-Out Storage" icon="box" href="https://xloud.tech/xavs">
  Product details and datasheet on xloud.tech
</Card>

***

<p>Xloud Object Storage</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/object-storage/user-guide">
    Create containers, upload objects, manage ACLs, enable versioning, and work with large objects using the Xloud Dashboard or CLI.
  </Card>

  <Card title="Admin Guide" icon="shield-check" href="/services/object-storage/admin-guide">
    Configure storage policies, manage rings, control replication, enforce quotas, and maintain the object storage cluster as a platform administrator.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/object-storage/cli-reference">
    `openstack object` and `openstack container` commands for managing object storage resources from the command line.
  </Card>

  <Card title="Access Control" icon="lock">
    Container-level and account-level ACLs control which users and services can read or write object data across your project boundaries.
  </Card>
</CardGroup>

***

<p>Key Features</p>

<CardGroup>
  <Card title="Unlimited Scale" icon="infinity" href="/services/object-storage/architecture">
    Store billions of objects across petabytes of capacity. The distributed ring architecture scales horizontally — add storage nodes without downtime.
  </Card>

  <Card title="Built-In Redundancy" icon="shield-check" href="/services/object-storage/replication">
    Configurable replication factors ensure multiple copies of every object are maintained across different zones and drives, protecting against hardware failure.
  </Card>

  <Card title="Object Versioning" icon="history" href="/services/object-storage/versioning">
    Enable versioning on containers to retain all previous versions of every object. Recover from accidental overwrites or deletions without external backup tools.
  </Card>

  <Card title="Large Object Support" icon="file-archive" href="/services/object-storage/large-objects">
    Store objects larger than 5 GB using Static Large Objects (SLO) or Dynamic Large Objects (DLO). Assemble multi-part uploads into a single addressable object.
  </Card>

  <Card title="Temporary URLs" icon="link" href="/services/object-storage/temp-urls">
    Generate time-limited, cryptographically signed URLs for sharing objects without granting permanent access — ideal for file distribution and download links.
  </Card>

  <Card title="Server-Side Encryption" icon="lock" href="/services/object-storage/security">
    Encrypt object data at rest using keys managed through Xloud Key Manager. Per-container or per-object encryption policies protect sensitive data automatically.
  </Card>
</CardGroup>

***

<p>Object Storage Components</p>

| Component      | Description                                                                                             |
| -------------- | ------------------------------------------------------------------------------------------------------- |
| Account        | Top-level namespace for a project's object storage. Holds containers and account-level metadata         |
| Container      | A logical bucket within an account. Holds objects and defines access policies and storage configuration |
| Object         | An individual data item stored in a container. Includes the payload and user-defined metadata           |
| Metadata       | Key-value pairs attached to accounts, containers, or objects for organization and workflow integration  |
| Storage Policy | A named configuration specifying replication factor, placement rules, and erasure coding for containers |
| Ring           | The consistent hash ring mapping objects to physical storage locations across the cluster               |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Key Manager" icon="lock" href="/services/key-manager">
    Manage encryption keys for server-side object encryption
  </Card>

  <Card title="Xloud Identity" icon="fingerprint" href="/cli-setup">
    Authentication tokens and RBAC governing container access
  </Card>

  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Instance backup storage, disk image exports, and application data archiving
  </Card>

  <Card title="Xloud Image Service" icon="layers" href="/services/compute">
    Store virtual machine images backed by object storage containers
  </Card>

  <Card title="Xloud DNS" icon="globe" href="/services/dns">
    Map custom domain names to object storage container endpoints
  </Card>

  <Card title="Xloud Load Balancer" icon="combine" href="/services/load-balancer">
    Distribute object storage API requests across multiple proxy nodes
  </Card>
</CardGroup>

***

<p>Getting Started</p>

<CardGroup>
  <Card title="Authentication" icon="key" href="/cli-setup">
    Configure Dashboard access and CLI credentials before working with Object Storage
  </Card>

  <Card title="User Guide" icon="book-open" href="/services/object-storage/user-guide">
    Step-by-step instructions for creating your first container and uploading objects
  </Card>
</CardGroup>


# Object Storage Access Control
Source: https://docs.xloud.tech/services/object-storage/access-control

Configure read and write ACLs on Xloud Object Storage containers to control cross-project access and enable public or restricted sharing.

## Overview

Container ACLs control which users and projects can read from or write to a container.
By default, containers are private — only the owning project has access. ACLs are set
as container metadata headers and can grant access to specific users, entire projects,
or the public.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## ACL Format Reference

| Value                    | Meaning                             |
| ------------------------ | ----------------------------------- |
| `<project-id>:<user-id>` | Specific user in a specific project |
| `<project-id>:*`         | All users in a specific project     |
| `.r:*`                   | Public anonymous read access        |
| `.r:*,.rlistings`        | Public read AND directory listing   |

***

## Configure ACLs

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to the container and click the **Edit** icon. In the **Access Control**
    section:

    * **Read ACL**: Comma-separated list of `<project-id>:<user-id>` pairs, or `.r:*` for
      public read access
    * **Write ACL**: Comma-separated list of `<project-id>:<user-id>` pairs controlling
      write access

    <Warning>
      ACL values of `.r:*` or `.r:*,.rlistings` grant public anonymous read access.
      Verify this is intentional before saving — objects in publicly accessible containers
      are reachable by anyone with the URL.
    </Warning>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Grant read access to a specific user" theme={null}
      openstack container set \
        --property "X-Container-Read=<project-id>:<user-id>" \
        app-backups
      ```

      ```bash title="Grant read access to all users in a project" theme={null}
      openstack container set \
        --property "X-Container-Read=<project-id>:*" \
        shared-data
      ```

      ```bash title="Grant public read access" theme={null}
      openstack container set \
        --property "X-Container-Read=.r:*,.rlistings" \
        public-assets
      ```

      ```bash title="Grant write access to another project" theme={null}
      openstack container set \
        --property "X-Container-Write=<other-project-id>:*" \
        shared-uploads
      ```

      ```bash title="Remove all ACLs (make private)" theme={null}
      openstack container set \
        --property "X-Container-Read=" \
        --property "X-Container-Write=" \
        app-backups
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## View Current ACLs

```bash title="Show container ACLs" theme={null}
openstack container show app-backups | grep -i "read\|write"
```

***

## Account-Level Access Control

The object store account (project) supports an additional read ACL at the account level:

```bash title="Show account metadata including ACLs" theme={null}
openstack object store account show
```

```bash title="Set account-level read ACL" theme={null}
openstack object store account set \
  --property X-Account-Meta-Access-Control-Allow-Origin="https://app.example.com"
```

***

## Best Practices

<AccordionGroup>
  <Accordion title="Principle of least privilege" icon="shield">
    * Grant the minimum required access — prefer `<project-id>:<user-id>` over `<project-id>:*`
    * Only use `.r:*` for containers explicitly intended for public access
    * Use `.rlistings` only when directory browsing is intentionally public
  </Accordion>

  <Accordion title="Regular ACL audit" icon="file-text">
    Review containers with non-empty read or write ACLs quarterly. Revoke access for
    decommissioned projects and users immediately:

    ```bash title="Check all container ACLs in your project" theme={null}
    for c in $(openstack container list -f value -c Name); do
      echo "=== $c ==="; openstack container show "$c" | grep -i "read\|write"
    done
    ```
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Upload Objects" href="/services/object-storage/upload-objects">
    Upload objects to your access-controlled container
  </Card>

  <Card title="Versioning" href="/services/object-storage/versioning">
    Enable version retention for objects in the container
  </Card>

  <Card title="Troubleshooting" href="/services/object-storage/troubleshooting">
    Resolve 403 access errors on containers
  </Card>

  <Card title="Object Storage Admin Guide" href="/services/object-storage/admin-guide">
    Platform-wide security and ACL governance
  </Card>
</CardGroup>


# Object Storage Admin Guide
Source: https://docs.xloud.tech/services/object-storage/admin-guide

Administer Xloud Object Storage — configure storage policies, manage rings, control replication, enforce quotas, secure the cluster, and monitor storage health.

<p>Overview</p>

This guide covers platform-level administration of the Xloud Object Storage service.
Administrators design and manage storage policies that govern data placement and
replication, maintain the consistent hash rings mapping objects to physical drives,
monitor cluster health, enforce quotas, and apply security hardening across the proxy
and storage tiers.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

<p>Topics in This Guide</p>

<CardGroup>
  <Card title="Architecture" icon="network" href="/services/object-storage/architecture">
    Cluster topology — proxy tier, storage tier, and consistent hash ring mechanics
  </Card>

  <Card title="Storage Policies" icon="layers" href="/services/object-storage/storage-policies">
    Design and manage data placement tiers using named storage policies
  </Card>

  <Card title="Ring Management" icon="refresh-cw" href="/services/object-storage/ring-management">
    Add drives, adjust weights, rebalance rings, and distribute ring files to all nodes
  </Card>

  <Card title="Replication" icon="copy" href="/services/object-storage/replication">
    Monitor replication health, check cluster consistency, and manage quarantined objects
  </Card>

  <Card title="Quotas" icon="gauge" href="/services/object-storage/quotas">
    Enforce per-account and per-container storage limits
  </Card>

  <Card title="Security" icon="shield" href="/services/object-storage/security">
    TLS enforcement, temp URL key rotation, audit logging, and cross-project ACL governance
  </Card>

  <Card title="Monitoring" icon="activity" href="/services/object-storage/monitoring">
    Track cluster capacity, proxy metrics, replication latency, and quarantine counts
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/object-storage/admin-troubleshooting">
    Diagnose 507 storage errors, proxy latency, and ring inconsistencies
  </Card>
</CardGroup>

***

<p>Prerequisites</p>

<Note>
  **Required before proceeding**

  * Administrator credentials sourced via `openrc.sh`
  * SSH access to storage nodes for ring management operations
  * Access to XDeploy for service configuration changes
  * Familiarity with distributed storage concepts (consistent hashing, replication zones)
</Note>

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Object Storage User Guide" icon="book-open" href="/services/object-storage/user-guide">
    Step-by-step instructions for managing containers and objects
  </Card>

  <Card title="Key Manager Admin Guide" icon="lock" href="/services/key-manager/admin-guide">
    Configure encryption keys for server-side object storage encryption
  </Card>

  <Card title="Load Balancer Admin Guide" icon="combine" href="/services/load-balancer/admin-guide">
    Place a load balancer in front of object storage proxy nodes
  </Card>

  <Card title="Identity" icon="fingerprint" href="/cli-setup">
    Manage service accounts and RBAC policies for object storage access
  </Card>
</CardGroup>


# Object Storage Admin Troubleshooting
Source: https://docs.xloud.tech/services/object-storage/admin-troubleshooting

Diagnose and resolve platform-level Xloud Object Storage issues — 507 storage errors, proxy latency, ring inconsistencies, and slow replication.

## Overview

This guide covers platform-level object storage issues that require administrator access.
For user-facing issues such as 403 access errors or upload timeouts, see the
[Object Storage Troubleshooting](/services/object-storage/troubleshooting) guide.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Diagnostic Checklist

```bash title="Overall cluster health" theme={null}
xavs-storage-recon --all
```

```bash title="Disk usage across all nodes" theme={null}
xavs-storage-recon --diskusage
```

```bash title="Replication status" theme={null}
xavs-storage-recon --replication
```

```bash title="Ring file consistency" theme={null}
xavs-storage-recon --md5
```

***

## Platform Issues

<AccordionGroup>
  <Accordion title="507 Insufficient Storage on object PUT" icon="hard-drive">
    **Cause**: One or more storage nodes targeted by the ring have insufficient free space
    to accept the write.

    **Diagnosis**:

    ```bash title="Check disk usage per node" theme={null}
    xavs-storage-recon --diskusage
    ```

    **Resolution**:

    * Identify nodes or drives above 85% utilization
    * Expand storage capacity by adding new drives (see [Ring Management](/services/object-storage/ring-management))
    * Alternatively, rebalance the ring to shift weight toward nodes with available space:
      ```bash title="Adjust weight on high-capacity node" theme={null}
      xavs-ring-builder object.builder set_weight <device-id> <reduced-weight>
      xavs-ring-builder object.builder rebalance
      xavs-ring-builder object.builder write_ring
      ```
  </Accordion>

  <Accordion title="Slow reads or high proxy latency" icon="clock">
    **Cause**: Replication traffic competing with foreground I/O, or a storage node
    with degraded drives experiencing high read latency.

    **Diagnosis**:

    ```bash title="Check replication load" theme={null}
    xavs-storage-recon --replication --verbose
    ```

    If a specific node shows high `replication_time`, inspect that node's disk I/O:

    ```bash title="Check disk I/O on storage node (SSH required)" theme={null}
    iostat -x 1 5
    ```

    **Resolution**:

    * Consider throttling the replicator with `--concurrency 1` during peak hours
    * If a specific drive is degraded, reduce its ring weight to shift load away
    * Replace drives showing high latency or recurring I/O errors in `dmesg`
  </Accordion>

  <Accordion title="Ring inconsistency between nodes" icon="git-branch">
    **Cause**: The updated ring file was not distributed to all nodes after a rebalance.

    **Diagnosis**:

    ```bash title="Check MD5 of ring files on all nodes" theme={null}
    xavs-storage-recon --md5
    ```

    **Resolution**: Nodes with mismatched MD5 hashes have stale ring files. Redistribute
    the ring to affected nodes:

    ```bash title="Copy ring files to an affected node" theme={null}
    scp /etc/xavs-object-storage/*.ring.gz <node-ip>:/etc/xavs-object-storage/
    ```

    Restart the object-server and replicator on the affected node after distribution.
  </Accordion>

  <Accordion title="High quarantine count on a specific node" icon="shield-alert">
    **Cause**: Drive failure, bit-rot, or network errors during replication causing
    data corruption detected by the auditor.

    **Diagnosis**:

    ```bash title="Check quarantine counts" theme={null}
    xavs-storage-recon --quarantined --verbose
    ```

    ```bash title="Check drive health (SSH to node)" theme={null}
    smartctl -a /dev/<device>
    dmesg | grep -i "error\|fail\|ata"
    ```

    **Resolution**:

    1. If the drive is failing, set its ring weight to 0 and rebalance to drain data
    2. Replace the physical drive
    3. Add the replacement drive to the ring and rebalance
    4. The replicator will restore the quarantined objects from healthy replicas

    <Warning>
      Do not simply delete quarantined objects — they may be the only remaining copy
      if other replicas are also corrupted. Always verify healthy replicas exist before
      any quarantine cleanup.
    </Warning>
  </Accordion>

  <Accordion title="Proxy service returns 503 Service Unavailable" icon="server">
    **Cause**: The proxy-server cannot reach a quorum of storage nodes for an operation.

    **Diagnosis**:

    ```bash title="Check proxy container status" theme={null}
    docker ps --filter name=swift-proxy
    docker logs swift-proxy --tail 50
    ```

    ```bash title="Verify storage nodes are reachable from proxy" theme={null}
    xavs-storage-recon --all | grep -i "error\|fail"
    ```

    **Resolution**:

    * Verify storage node containers are running: `docker ps --filter name=swift`
    * Check network connectivity from proxy hosts to storage nodes on ports 6200, 6201, 6202
    * If nodes are degraded, the proxy will still serve reads from available replicas but
      writes require the configured replica quorum
  </Accordion>
</AccordionGroup>

***

## Log Locations

| Component        | Log Command                           |
| ---------------- | ------------------------------------- |
| Proxy server     | `docker logs swift-proxy`             |
| Object server    | `docker logs swift-object`            |
| Container server | `docker logs swift-container`         |
| Account server   | `docker logs swift-account`           |
| Replicator       | `docker logs swift-object-replicator` |

***

## Next Steps

<CardGroup>
  <Card title="Object Storage Troubleshooting (User)" href="/services/object-storage/troubleshooting">
    User-facing issues — 403 errors, upload timeouts, versioning failures
  </Card>

  <Card title="Ring Management" href="/services/object-storage/ring-management">
    Add capacity and redistribute data after failures
  </Card>

  <Card title="Replication" href="/services/object-storage/replication">
    Monitor and restore data durability
  </Card>

  <Card title="Monitoring" href="/services/object-storage/monitoring">
    Proactively catch issues before they become outages
  </Card>
</CardGroup>


# Object Storage Architecture
Source: https://docs.xloud.tech/services/object-storage/architecture

Understand the Xloud Object Storage cluster topology — proxy tier, storage tier, consistent hash ring mechanics, S3 API middleware, and data flow for all.

## Overview

Xloud Object Storage uses a three-tier, fully distributed architecture. Proxy nodes handle all API requests and authentication. Storage nodes persist object data on local drives. The consistent hash ring maps every object to its target storage locations without a central metadata server. This design eliminates single points of failure and enables horizontal scaling of each tier independently.

<Note>
  **Prerequisites**

  * Familiarity with Xloud Object Storage [storage policies](/services/object-storage/storage-policies)
  * Admin access to review cluster topology
</Note>

***

## Cluster Topology

```mermaid theme={null}
graph TD
    Client["API Client\n(Swift / S3)"] --> ProxyLB["Load Balancer :443"]
    ProxyLB --> P1["Proxy Node 1\nproxy-server"]
    ProxyLB --> P2["Proxy Node 2\nproxy-server"]

    P1 --> AuthM["Keystone Auth Middleware"]
    P1 --> S3MW["S3 API Middleware"]
    P1 --> AR["Account Ring"]
    P1 --> CR["Container Ring"]
    P1 --> OR["Object Ring(s)"]

    AR --> SN1["Storage Node 1\naccount-server\ncontainer-server\nobject-server"]
    CR --> SN1
    OR --> SN1
    OR --> SN2["Storage Node 2\nobject-server"]
    OR --> SN3["Storage Node 3\nobject-server"]

    SN1 <-->|Replication| SN2
    SN2 <-->|Replication| SN3
    SN3 <-->|Replication| SN1

    subgraph "Proxy Tier (Stateless)"
        P1
        P2
        AuthM
        S3MW
    end

    subgraph "Ring Layer"
        AR
        CR
        OR
    end

    subgraph "Storage Tier"
        SN1
        SN2
        SN3
    end
```

<Info>
  Proxy nodes are fully stateless — they hold only ring files (updated via ring distribution). All persistent state lives on storage nodes. Adding proxy nodes scales API throughput without touching the storage tier.
</Info>

***

## Component Descriptions

<AccordionGroup>
  <Accordion title="Proxy Server" icon="server">
    The proxy server is the single entry point for all client requests (Swift and S3 API). It performs:

    * Token validation via Keystone auth middleware
    * Ring lookups to identify target storage nodes for each request
    * Parallel writes to all replica nodes for PUT operations
    * Read fan-out and quorum resolution for GET operations
    * Transparent S3 API translation via `s3api` middleware

    Proxy nodes never store object data. They are horizontally scalable and stateless.
  </Accordion>

  <Accordion title="Account Server" icon="user">
    The account server manages project-level metadata:

    * Tracks all containers belonging to a project
    * Stores account-level statistics (bytes used, object count, container count)
    * Enforces quota limits in conjunction with the proxy
    * Served from the account ring — one partition per account
  </Accordion>

  <Accordion title="Container Server" icon="folder">
    The container server manages container-level metadata:

    * Lists all objects within a container (object listings)
    * Stores container-level statistics and custom metadata headers
    * Container records are replicated across the container ring
    * Object listings are eventually consistent — updates propagate asynchronously via the updater
  </Accordion>

  <Accordion title="Object Server" icon="file">
    The object server handles the actual object data:

    * Stores objects on local XFS or ext4 filesystems
    * Each object stored as a file at a path derived from its MD5 hash
    * Handles PUT, GET, DELETE, HEAD, and COPY operations
    * Writes metadata (content-type, custom headers) as extended file attributes
    * Generates a unique transaction ID for every operation
  </Accordion>

  <Accordion title="Replication Engine" icon="refresh-cw">
    The replication engine runs continuously on every storage node to maintain the configured replica count:

    * **Object replicator**: Compares local partition hashes with remote nodes; pushes missing objects via rsync or direct HTTP
    * **Container replicator**: Synchronizes container database records across ring replicas
    * **Account replicator**: Synchronizes account database records
    * Replication is partition-based — the ring divides the hash space into partitions, and each partition's primary and handoff nodes are replicated to
  </Accordion>

  <Accordion title="Background Services" icon="settings">
    Additional background services maintain cluster health:

    | Service           | Function                                                                                                      |
    | ----------------- | ------------------------------------------------------------------------------------------------------------- |
    | **Auditor**       | Reads every stored object and verifies checksum integrity. Quarantines corrupted objects.                     |
    | **Updater**       | Processes failed container and account update queues asynchronously. Resolves eventually-consistent listings. |
    | **Expirer**       | Deletes objects that have reached their `X-Delete-At` or `X-Delete-After` expiry timestamp.                   |
    | **Reconstructor** | EC-specific: reconstructs missing or corrupted EC fragments from surviving shards.                            |
  </Accordion>

  <Accordion title="S3 API Middleware" icon="code">
    The `s3api` middleware translates S3-format requests into Swift internal requests transparently:

    * Mounted in the proxy pipeline before the auth middleware
    * Translates S3 bucket operations to Swift container operations
    * Translates S3 object operations to Swift object operations
    * Handles S3 authentication (HMAC-SHA256 signature v4)
    * Translates S3 ACLs to Swift ACL headers
    * Supports multipart upload via Swift dynamic large objects
    * Supports object versioning via Swift versioning middleware

    <Info>
      S3 and Swift APIs share the same underlying storage. An object uploaded via S3 API is immediately accessible via the Swift API using the same account/container/object path structure, and vice versa.
    </Info>
  </Accordion>
</AccordionGroup>

***

## Consistent Hash Ring

The consistent hash ring is the core distribution mechanism. It determines which storage nodes hold each object without any central directory server.

```mermaid theme={null}
graph LR
    Object["Object Name\n(MD5 hash)"] --> Part["Partition\n(high bits of hash)"]
    Part --> Node1["Primary Node"]
    Part --> Node2["Replica Node 1"]
    Part --> Node3["Replica Node 2"]

    Ring["Ring File\n(.ring.gz)"] -->|"lookup(partition)"| Node1
    Ring -->|"lookup(partition)"| Node2
    Ring -->|"lookup(partition)"| Node3
```

Ring mechanics:

| Concept             | Description                                                             |
| ------------------- | ----------------------------------------------------------------------- |
| **Partition power** | `2^partition_power` partitions in the ring (typically 2^18 = 262,144)   |
| **Partition**       | A slice of the hash space. Every object maps to exactly one partition.  |
| **Device**          | A physical drive with assigned weight (capacity proportion)             |
| **Weight**          | Determines what fraction of partitions a device receives                |
| **Replica count**   | How many distinct devices hold each partition's data                    |
| **Zone**            | Fault domain grouping — ring enforces replicas land in distinct zones   |
| **Region**          | Geographic grouping — for geo-redundant deployments across data centers |

<Tip>
  Higher partition power means more partitions and finer-grained data distribution, but larger ring files. Use `2^18` for clusters up to \~200 storage nodes. Use `2^20` for very large clusters.
</Tip>

***

## Object Request Flow

<Tabs>
  <Tab title="Write (PUT)" icon="upload">
    ```mermaid theme={null}
    sequenceDiagram
        participant Client
        participant LB as Load Balancer
        participant Proxy as Proxy Server
        participant Ring as Object Ring
        participant S1 as Storage Node 1
        participant S2 as Storage Node 2
        participant S3 as Storage Node 3

        Client->>LB: PUT /v1/AUTH_proj/container/object
        LB->>Proxy: Forward request
        Proxy->>Proxy: Keystone token validation
        Proxy->>Ring: lookup(MD5(object path))
        Ring-->>Proxy: Primary: S1, Replicas: S2, S3
        Proxy->>S1: PUT object (replica 1) — parallel
        Proxy->>S2: PUT object (replica 2) — parallel
        Proxy->>S3: PUT object (replica 3) — parallel
        S1-->>Proxy: 201 Created
        S2-->>Proxy: 201 Created
        S3-->>Proxy: 201 Created
        Proxy->>Proxy: Quorum check (2 of 3 required)
        Proxy-->>Client: 201 Created
    ```

    The proxy writes to all replicas in parallel. It returns success to the client once a write quorum (default: `(replicas // 2) + 1`) confirms the write.
  </Tab>

  <Tab title="Read (GET)" icon="download">
    ```mermaid theme={null}
    sequenceDiagram
        participant Client
        participant Proxy as Proxy Server
        participant Ring as Object Ring
        participant S1 as Storage Node 1
        participant S2 as Storage Node 2

        Client->>Proxy: GET /v1/AUTH_proj/container/object
        Proxy->>Proxy: Keystone token validation
        Proxy->>Ring: lookup(MD5(object path))
        Ring-->>Proxy: Primary: S1, Replicas: S2, S3
        Proxy->>S1: GET object (primary)
        S1-->>Proxy: 200 OK + data stream
        Proxy-->>Client: 200 OK + data stream
        Note over Proxy,S2: S2/S3 not contacted on successful primary read
    ```

    Reads go to the primary node first. If the primary is unreachable, the proxy automatically falls back to replica nodes — transparent to the client.
  </Tab>

  <Tab title="S3 API" icon="code">
    ```mermaid theme={null}
    sequenceDiagram
        participant Client as S3 Client
        participant Proxy as Proxy + s3api middleware
        participant Keystone
        participant Storage as Storage Nodes

        Client->>Proxy: PUT /bucket/object (S3 signature v4)
        Proxy->>Proxy: s3api: validate HMAC signature
        Proxy->>Keystone: Exchange S3 creds for Swift token
        Keystone-->>Proxy: Swift auth token
        Proxy->>Proxy: Translate S3 → Swift request
        Proxy->>Storage: Swift PUT /v1/AUTH_proj/bucket/object
        Storage-->>Proxy: 201 Created
        Proxy->>Proxy: Translate Swift → S3 response
        Proxy-->>Client: 200 OK (S3 format)
    ```

    The S3 middleware translates the entire request and response lifecycle. The underlying storage operation is identical to a native Swift write.
  </Tab>
</Tabs>

***

## Replication Zones and Fault Domains

Storage nodes are grouped into zones for fault domain separation. The ring builder enforces replica placement across distinct zones.

| Zone   | Typical Mapping           | Failure Isolated        |
| ------ | ------------------------- | ----------------------- |
| Zone 1 | Rack 1 / PDU A / Switch A | Any single rack failure |
| Zone 2 | Rack 2 / PDU B / Switch B | Any single rack failure |
| Zone 3 | Rack 3 / PDU C / Switch C | Any single rack failure |

A 3-replica policy distributes one replica per zone. A full zone failure (rack down, switch failure) results in zero data loss and read/write operations continue using the two surviving zones.

<Warning>
  For geo-redundant deployments, configure regions in addition to zones. Each region hosts a complete replica set. Cross-region replication introduces higher write latency — design policies accordingly.
</Warning>

***

## Capacity Planning

<CardGroup>
  <Card title="Replication Overhead" icon="layers">
    3-replica policy: usable capacity = raw capacity ÷ 3.
    For 60 TB raw storage across 10 nodes, usable capacity = 20 TB.
  </Card>

  <Card title="EC Overhead" icon="shield">
    8+4 EC policy: usable capacity = raw capacity × (8 ÷ 12) = 66.7%.
    For 60 TB raw storage, usable capacity = 40 TB — 2× more efficient than 3-replica.
  </Card>

  <Card title="Minimum Nodes per Policy" icon="server">
    Replication (3×): minimum 3 nodes in 3 distinct zones.
    EC 8+4: minimum 12 nodes. EC 4+2: minimum 6 nodes.
  </Card>

  <Card title="Ring Rebalancing" icon="refresh-cw">
    Adding nodes triggers a ring rebalance. Set `min_part_hours` (minimum 1 hour) to limit
    partition moves per cycle and prevent rebalance storms during rapid node additions.
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="Storage Policies" href="/services/object-storage/storage-policies">
    Configure replication, EC, and multi-tier storage policies
  </Card>

  <Card title="Ring Management" href="/services/object-storage/ring-management">
    Add drives, adjust weights, and distribute updated rings
  </Card>

  <Card title="Replication" href="/services/object-storage/replication">
    Monitor replication health and manage quarantined objects
  </Card>

  <Card title="Monitoring" href="/services/object-storage/monitoring">
    Track cluster capacity and proxy request metrics
  </Card>
</CardGroup>


# Object Storage CLI Reference
Source: https://docs.xloud.tech/services/object-storage/cli-reference

Complete openstack container and object CLI commands for managing Xloud Object Storage — containers, objects, access control, and large objects.

## Overview

The `openstack container` and `openstack object` command groups manage Swift-compatible object storage — containers, objects, metadata, and access policies.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python swiftclient installed: `pip install python-swiftclient`
</Note>

***

## Containers

<CodeGroup>
  ```bash title="List containers" theme={null}
  openstack container list
  ```

  ```bash title="Create container" theme={null}
  openstack container create my-bucket
  ```

  ```bash title="Show container details" theme={null}
  openstack container show my-bucket
  ```

  ```bash title="Set container metadata" theme={null}
  openstack container set --property owner=team-a my-bucket
  ```

  ```bash title="Make container public" theme={null}
  openstack container set \
    --property "X-Container-Read=.r:*,.rlistings" \
    my-bucket
  ```

  ```bash title="Delete container" theme={null}
  openstack container delete my-bucket
  ```
</CodeGroup>

***

## Objects

<CodeGroup>
  ```bash title="List objects in container" theme={null}
  openstack object list my-bucket
  openstack object list --prefix logs/ my-bucket
  ```

  ```bash title="Upload object" theme={null}
  openstack object create my-bucket ./report.pdf
  ```

  ```bash title="Upload with custom name" theme={null}
  openstack object create --name backups/2026-01-01.tar.gz my-bucket ./backup.tar.gz
  ```

  ```bash title="Upload entire directory" theme={null}
  swift upload my-bucket ./my-directory/
  ```

  ```bash title="Show object details" theme={null}
  openstack object show my-bucket report.pdf
  ```

  ```bash title="Download object" theme={null}
  openstack object save my-bucket report.pdf
  ```

  ```bash title="Download to specific path" theme={null}
  openstack object save --file /tmp/report.pdf my-bucket report.pdf
  ```

  ```bash title="Delete object" theme={null}
  openstack object delete my-bucket report.pdf
  ```
</CodeGroup>

***

## Large Objects (Segmented)

<CodeGroup>
  ```bash title="Upload large file in segments (swift CLI)" theme={null}
  swift upload --segment-size 1G --use-slo my-bucket large-file.iso
  ```

  ```bash title="List segments" theme={null}
  openstack object list my-bucket_segments
  ```
</CodeGroup>

***

## Temporary URLs

<CodeGroup>
  ```bash title="Generate temp URL (swift CLI)" theme={null}
  swift tempurl GET 3600 \
    /v1/AUTH_<project-id>/my-bucket/report.pdf \
    <temp-url-key>
  ```

  ```bash title="Set temp URL key on account" theme={null}
  swift post -m "Temp-Url-Key:<your-secret-key>"
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Create Container Guide" href="/services/object-storage/create-container">
    Step-by-step guide to creating containers and uploading objects
  </Card>

  <Card title="Access Control Guide" href="/services/object-storage/access-control">
    Configure container and object access policies
  </Card>
</CardGroup>


# Create an Object Storage Container
Source: https://docs.xloud.tech/services/object-storage/create-container

Provision object storage containers in Xloud Object Storage using the Dashboard or CLI. Configure container name, access policy, and storage policy.

## Overview

Containers (buckets) are the top-level namespaces within your object storage account.
All objects must reside in a container. Container names must be unique within your
account, and the storage policy assigned at creation time cannot be changed afterward.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Create a Container

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Object Storage">
        Navigate to
        **Storage > Object Storage**.

        Click **Create Container** to open the creation dialog.
      </Step>

      <Step title="Configure container settings">
        | Field                | Description                                                     |
        | -------------------- | --------------------------------------------------------------- |
        | **Container Name**   | Unique name within your account (e.g., `app-backups`)           |
        | **Container Access** | `Private` (default) or `Public` — public enables anonymous read |
        | **Storage Policy**   | Select the replication policy appropriate for the data tier     |

        <Warning>
          Setting container access to **Public** makes all objects in the container
          accessible to anyone with the storage endpoint URL — including external parties.
          Use public access only for intentionally public assets such as static web content.
        </Warning>
      </Step>

      <Step title="Create the container">
        Click **Confirm**. The container appears in the list immediately.

        <Check>Container appears in the Containers list and is ready for object uploads.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Create the container">
        ```bash title="Create a private container" theme={null}
        openstack container create app-backups
        ```

        ```bash title="Create with a specific storage policy" theme={null}
        openstack container create \
          --storage-policy <policy-name> \
          app-backups
        ```
      </Step>

      <Step title="Verify creation">
        ```bash title="Show container metadata" theme={null}
        openstack container show app-backups
        ```

        <Check>Container shows `object_count: 0` and no error in metadata.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Container Naming Rules

| Rule           | Detail                                                                    |
| -------------- | ------------------------------------------------------------------------- |
| **Uniqueness** | Must be unique within your account (project)                              |
| **Length**     | 1–256 characters                                                          |
| **Characters** | Any UTF-8 character except `/`                                            |
| **Case**       | Case-sensitive — `app-backups` and `App-Backups` are different containers |

***

## List and Manage Containers

<CodeGroup>
  ```bash title="List containers in your account" theme={null}
  openstack container list
  ```

  ```bash title="Show container metadata" theme={null}
  openstack container show app-backups
  ```

  ```bash title="Delete an empty container" theme={null}
  openstack container delete app-backups
  ```
</CodeGroup>

<Danger>
  Containers with objects cannot be deleted until all objects are removed. Use
  `openstack object list app-backups` to view contents before deletion.
</Danger>

***

## Next Steps

<CardGroup>
  <Card title="Upload Objects" href="/services/object-storage/upload-objects">
    Upload files and manage objects within your container
  </Card>

  <Card title="Access Control" href="/services/object-storage/access-control">
    Configure read and write ACLs for container access
  </Card>

  <Card title="Versioning" href="/services/object-storage/versioning">
    Enable version retention on your container
  </Card>

  <Card title="Storage Policies" href="/services/object-storage/storage-policies">
    Learn about storage policy options available on your platform
  </Card>
</CardGroup>


# Large Object Uploads
Source: https://docs.xloud.tech/services/object-storage/large-objects

Upload files larger than 5 GB in Xloud Object Storage using Static Large Objects (SLO) or Dynamic Large Objects (DLO) for multi-part segment management.

## Overview

Individual objects in Xloud Object Storage are limited to 5 GB per upload request.
Files larger than 5 GB must be split into segments and uploaded as a Large Object —
either a Static Large Object (SLO) or Dynamic Large Object (DLO). Both produce a
manifest object that transparently concatenates segments when downloaded.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## SLO vs DLO Comparison

| Method                         | Manifest                                    | Best For                                | Segment Discovery                       |
| ------------------------------ | ------------------------------------------- | --------------------------------------- | --------------------------------------- |
| **SLO** (Static Large Object)  | Explicit JSON manifest listing each segment | Large files with known segments         | Explicit — you define each segment path |
| **DLO** (Dynamic Large Object) | Object with `X-Object-Manifest` header      | Streaming uploads of unknown total size | Automatic — by name prefix convention   |

***

## Static Large Object (SLO)

Use SLO when the complete file is available before upload begins.

<Steps>
  <Step title="Split the file into segments">
    ```bash title="Split file into 1 GB segments" theme={null}
    split -b 1G large-backup.tar.gz large-backup.tar.gz.part-
    ```
  </Step>

  <Step title="Create a segments container">
    ```bash title="Create dedicated segments container" theme={null}
    openstack container create app-backups-segments
    ```
  </Step>

  <Step title="Upload each segment">
    ```bash title="Upload all segments" theme={null}
    for f in large-backup.tar.gz.part-*; do
      openstack object create app-backups-segments "$f"
    done
    ```
  </Step>

  <Step title="Build and upload the SLO manifest">
    Collect the ETag of each segment (shown in the upload output) and create a manifest:

    ```json title="manifest.json" theme={null}
    [
      {
        "path": "app-backups-segments/large-backup.tar.gz.part-aa",
        "etag": "<etag-from-upload>",
        "size_bytes": 1073741824
      },
      {
        "path": "app-backups-segments/large-backup.tar.gz.part-ab",
        "etag": "<etag-from-upload>",
        "size_bytes": 524288000
      }
    ]
    ```

    Upload the manifest with the multipart-manifest query parameter:

    ```bash title="Create SLO manifest" theme={null}
    curl -X PUT \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      --data-binary @manifest.json \
      "https://object.<your-domain>/v1/<account>/app-backups/large-backup.tar.gz?multipart-manifest=put"
    ```

    <Check>The large object is now accessible at `app-backups/large-backup.tar.gz` and
    downloads transparently concatenate all segments.</Check>
  </Step>
</Steps>

***

## Dynamic Large Object (DLO)

Use DLO for streaming uploads where the total object size is not known in advance.

<Steps>
  <Step title="Upload segments with a common prefix">
    Upload each segment with a naming convention that groups them by prefix:

    ```bash title="Upload DLO segments" theme={null}
    openstack object create app-backups large-backup/segment-0001
    openstack object create app-backups large-backup/segment-0002
    openstack object create app-backups large-backup/segment-0003
    ```
  </Step>

  <Step title="Create the DLO manifest object">
    Create a zero-byte manifest object pointing to the segment prefix:

    ```bash title="Create DLO manifest" theme={null}
    curl -X PUT \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "X-Object-Manifest: app-backups/large-backup/" \
      -H "Content-Length: 0" \
      "https://object.<your-domain>/v1/<account>/app-backups/large-backup"
    ```

    <Check>Accessing `app-backups/large-backup` transparently returns all segments concatenated in alphabetical order.</Check>
  </Step>
</Steps>

***

## Download a Large Object

Downloading a large object (SLO or DLO) is transparent — use the same commands as a
normal object:

```bash title="Download large object" theme={null}
openstack object save app-backups large-backup.tar.gz
```

The proxy server automatically retrieves and concatenates all segments.

***

## Next Steps

<CardGroup>
  <Card title="Upload Objects" href="/services/object-storage/upload-objects">
    Standard object upload for files under 5 GB
  </Card>

  <Card title="Versioning" href="/services/object-storage/versioning">
    Enable versioning on containers that hold large objects
  </Card>

  <Card title="Troubleshooting" href="/services/object-storage/troubleshooting">
    Resolve large object upload timeouts and manifest errors
  </Card>

  <Card title="Storage Policies" href="/services/object-storage/storage-policies">
    Choose the right storage policy for large object containers
  </Card>
</CardGroup>


# Object Storage Monitoring
Source: https://docs.xloud.tech/services/object-storage/monitoring

Monitor Xloud Object Storage cluster health — track capacity utilization, proxy request metrics, replication latency, and quarantined object counts.

## Overview

Effective monitoring of the object storage cluster ensures early detection of capacity
constraints, performance degradation, and data integrity issues. This guide covers the
key metrics and commands for ongoing operational visibility.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Cluster Capacity

<Tabs>
  <Tab title="Capacity overview" icon="database">
    ```bash title="Storage capacity across all nodes" theme={null}
    xavs-storage-recon --diskusage --verbose
    ```

    Capacity thresholds:

    | Metric                  | Warning | Critical | Action                       |
    | ----------------------- | ------- | -------- | ---------------------------- |
    | Node capacity used      | 70%     | 85%      | Plan capacity expansion      |
    | Single drive capacity   | 80%     | 90%      | Add drives or rebalance ring |
    | Cluster-wide free space | \< 20%  | \< 10%   | Immediate expansion required |

    <Warning>
      When any storage node exceeds 85% capacity, the ring rebalancer may be unable to
      place new replicas, causing `507 Insufficient Storage` errors for writes.
      Plan capacity expansion before reaching 70% utilization.
    </Warning>
  </Tab>

  <Tab title="Per-node usage" icon="hard-drive">
    ```bash title="Disk usage summary per node" theme={null}
    xavs-storage-recon --diskusage
    ```

    The output shows each node's total capacity, used bytes, and percentage utilized.
    Identify outliers — nodes significantly above the cluster average indicate uneven
    data distribution, which may require ring weight adjustments.
  </Tab>
</Tabs>

***

## Proxy Metrics

<Tabs>
  <Tab title="Recon endpoint" icon="gauge">
    The proxy-server exposes metrics on the recon middleware endpoint:

    ```bash title="Check proxy load" theme={null}
    curl -s http://<proxy-node-ip>:6000/recon/load
    ```

    ```bash title="Check proxy memory" theme={null}
    curl -s http://<proxy-node-ip>:6000/recon/mem
    ```

    ```bash title="Check proxy async pending updates" theme={null}
    curl -s http://<proxy-node-ip>:6000/recon/async
    ```
  </Tab>

  <Tab title="Key proxy metrics" icon="activity">
    Monitor these proxy-level metrics:

    | Metric            | Description                         | Alert Threshold                  |
    | ----------------- | ----------------------------------- | -------------------------------- |
    | **Request rate**  | Requests per second per proxy node  | Baseline + 3× standard deviation |
    | **Error rate**    | 4xx and 5xx responses as % of total | > 5% 5xx errors                  |
    | **GET latency**   | p95 response time for object reads  | > 500ms p95                      |
    | **PUT latency**   | p95 response time for object writes | > 1000ms p95                     |
    | **Async pending** | Container/account updates queued    | > 1000 pending                   |
  </Tab>
</Tabs>

***

## Replication Health

```bash title="Replication status across all nodes" theme={null}
xavs-storage-recon --replication
```

```bash title="Check for quarantined (corrupted) objects" theme={null}
xavs-storage-recon --quarantined
```

```bash title="Verify ring file consistency across nodes" theme={null}
xavs-storage-recon --md5
```

Replication health alerts:

| Condition                   | Severity | Response                                  |
| --------------------------- | -------- | ----------------------------------------- |
| `replication_time` > 300s   | Warning  | Investigate slow nodes                    |
| `replication_last` > 600s   | Critical | Check replicator daemon status            |
| Quarantine count increasing | Critical | Check drive health, replace failed drives |
| MD5 mismatch                | Critical | Redistribute ring files immediately       |

***

## Integration with XIMP

For continuous monitoring, connect the object storage recon endpoint to XIMP
(Xloud Infrastructure Monitoring Platform):

```yaml title="Prometheus scrape config for object storage" theme={null}
scrape_configs:
  - job_name: 'xavs-object-storage-recon'
    static_configs:
      - targets: ['<proxy-node-1>:6000', '<proxy-node-2>:6000']
    metrics_path: '/recon/metrics'
```

<Tip>
  Configure alerting rules in XIMP for the critical thresholds above. Set notification
  channels for the on-call team to respond to 507 storage errors and quarantine
  count spikes promptly.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Replication" href="/services/object-storage/replication">
    Deep-dive into replication health and quarantine management
  </Card>

  <Card title="Ring Management" href="/services/object-storage/ring-management">
    Expand capacity by adding drives and rebalancing rings
  </Card>

  <Card title="Admin Troubleshooting" href="/services/object-storage/admin-troubleshooting">
    Respond to monitoring alerts and diagnose failures
  </Card>

  <Card title="Quotas" href="/services/object-storage/quotas">
    Set limits to prevent individual projects from consuming all capacity
  </Card>
</CardGroup>


# Object Storage Quotas
Source: https://docs.xloud.tech/services/object-storage/quotas

Enforce per-account and per-container storage quotas in Xloud Object Storage — set byte limits, object count limits, and monitor usage.

## Overview

Quotas prevent individual projects from consuming excessive object storage capacity.
Quotas are enforced at the account level (project-wide) or container level. When a
quota is exceeded, the storage service returns `413 Request Entity Too Large`.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Account-Level Quotas

Account quotas limit the total storage consumed by a project across all containers.

<Tabs>
  <Tab title="Set account quota" icon="settings">
    ```bash title="Set account-level quota (100 GB)" theme={null}
    openstack object store account set \
      --property X-Account-Meta-Quota-Bytes=107374182400
    ```

    ```bash title="View current account usage vs quota" theme={null}
    openstack object store account show \
      -c x-account-bytes-used \
      -c x-account-meta-quota-bytes
    ```
  </Tab>

  <Tab title="Remove account quota" icon="trash-2">
    ```bash title="Remove account quota" theme={null}
    openstack object store account set \
      --property X-Account-Meta-Quota-Bytes=
    ```
  </Tab>
</Tabs>

***

## Container-Level Quotas

Container quotas limit storage within a specific container.

<Tabs>
  <Tab title="Set byte quota" icon="database">
    ```bash title="Set container byte quota (10 GB)" theme={null}
    openstack container set \
      --property X-Container-Meta-Quota-Bytes=10737418240 \
      app-uploads
    ```
  </Tab>

  <Tab title="Set object count quota" icon="list">
    ```bash title="Set container object count quota" theme={null}
    openstack container set \
      --property X-Container-Meta-Quota-Count=10000 \
      app-uploads
    ```

    <Tip>
      Set both byte and count quotas on shared containers — count quotas prevent
      namespace abuse from applications that create excessive small objects.
    </Tip>
  </Tab>

  <Tab title="View container quota" icon="eye">
    ```bash title="View container quota and usage" theme={null}
    openstack container show app-uploads | grep -i "quota\|object"
    ```
  </Tab>
</Tabs>

***

## Quota Reference

| Quota Type             | Header                         | Scope                     | Enforcement           |
| ---------------------- | ------------------------------ | ------------------------- | --------------------- |
| Account byte limit     | `X-Account-Meta-Quota-Bytes`   | All containers in account | Returns 413 on exceed |
| Container byte limit   | `X-Container-Meta-Quota-Bytes` | Single container          | Returns 413 on exceed |
| Container object count | `X-Container-Meta-Quota-Count` | Single container          | Returns 413 on exceed |

***

## Common Quota Sizes

| Size   | Bytes           |
| ------ | --------------- |
| 10 GB  | `10737418240`   |
| 50 GB  | `53687091200`   |
| 100 GB | `107374182400`  |
| 500 GB | `536870912000`  |
| 1 TB   | `1099511627776` |

***

## Next Steps

<CardGroup>
  <Card title="Security" href="/services/object-storage/security">
    Apply access controls and TLS hardening alongside quotas
  </Card>

  <Card title="Monitoring" href="/services/object-storage/monitoring">
    Track capacity usage across nodes and accounts
  </Card>

  <Card title="Storage Policies" href="/services/object-storage/storage-policies">
    Design storage tiers to optimize cost and capacity
  </Card>

  <Card title="Admin Troubleshooting" href="/services/object-storage/admin-troubleshooting">
    Diagnose quota enforcement and 413 errors
  </Card>
</CardGroup>


# Object Storage Replication
Source: https://docs.xloud.tech/services/object-storage/replication

Monitor replication health in Xloud Object Storage — check replica consistency, manage quarantined objects, and verify data durability across storage nodes.

## Overview

The replicator daemon continuously ensures each object has the configured number of
replicas across different zones, detecting and repairing divergence. Monitoring
replication health is essential for maintaining data durability guarantees.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Monitor Replication

<Tabs>
  <Tab title="Cluster-wide status" icon="activity">
    ```bash title="Check replication status across all nodes" theme={null}
    xavs-storage-recon --replication
    ```

    ```bash title="Verbose replication check" theme={null}
    xavs-storage-recon --replication --verbose
    ```

    Key replication metrics:

    | Metric             | Healthy Value              | Concern Threshold      |
    | ------------------ | -------------------------- | ---------------------- |
    | `replication_time` | \< 60 seconds              | > 300 seconds          |
    | `replication_last` | Recent timestamp           | Older than 600 seconds |
    | `object_count`     | Consistent across replicas | Divergence > 1%        |
  </Tab>

  <Tab title="Single node check" icon="server">
    ```bash title="Check replication on a specific storage node" theme={null}
    xavs-storage-recon --replication -v <storage-node-ip>
    ```
  </Tab>

  <Tab title="Overall cluster health" icon="heart-pulse">
    ```bash title="Comprehensive cluster health check" theme={null}
    xavs-storage-recon --all
    ```

    ```bash title="Disk usage across all nodes" theme={null}
    xavs-storage-recon --diskusage
    ```

    ```bash title="Check for quarantined objects" theme={null}
    xavs-storage-recon --quarantined
    ```
  </Tab>
</Tabs>

***

## Quarantined Objects

The auditor daemon detects data corruption (bit-rot, write errors) through checksum
verification. Corrupted objects are moved to a quarantine directory and excluded from
reads until a healthy replica is served instead.

<Warning>
  A high quarantine count indicates data corruption — potentially caused by drive
  failures, bit rot, or network errors during replication. Investigate and replace
  affected drives promptly. Quarantined objects are excluded from reads until a
  healthy replica is found.
</Warning>

```bash title="Check quarantine counts by node" theme={null}
xavs-storage-recon --quarantined --verbose
```

```bash title="View quarantined objects on a node (SSH to node)" theme={null}
ls /var/lib/xavs-object-storage/quarantined/
```

If quarantine counts are high on a specific node:

1. Check drive health with `smartctl` or the hardware vendor tool
2. Replace failing drives and add replacement devices to the ring
3. Remove the degraded device from the ring to allow data to drain

***

## Replication Configuration

Key replication parameters configurable through XDeploy:

| Parameter      | Description                                       | Default |
| -------------- | ------------------------------------------------- | ------- |
| `concurrency`  | Number of parallel replication threads per daemon | 1       |
| `interval`     | Seconds between replication passes                | 30      |
| `node_timeout` | Seconds before marking a replica push as failed   | 10      |

Adjust `concurrency` during off-peak hours to accelerate replication after large ring
changes:

```bash title="Temporarily increase replication concurrency (XDeploy config)" theme={null}
# Edit object storage configuration → replicator section
# Set concurrency = 4, then deploy
xavs-ansible deploy -t swift
```

***

## Next Steps

<CardGroup>
  <Card title="Ring Management" href="/services/object-storage/ring-management">
    Add or remove drives that affect replication targets
  </Card>

  <Card title="Monitoring" href="/services/object-storage/monitoring">
    Set up capacity and replication health monitoring
  </Card>

  <Card title="Admin Troubleshooting" href="/services/object-storage/admin-troubleshooting">
    Diagnose replication failures and high-latency nodes
  </Card>

  <Card title="Storage Policies" href="/services/object-storage/storage-policies">
    Review replication factors for each storage policy
  </Card>
</CardGroup>


# Ring Management
Source: https://docs.xloud.tech/services/object-storage/ring-management

Manage Xloud Object Storage consistent hash rings — add and remove devices, adjust weights, rebalance rings, and distribute updated ring files to all nodes.

## Overview

The consistent hash rings determine where every object, container, and account lives
in the cluster. Ring changes require building new ring files and distributing them
to all nodes. Object rebalancing occurs gradually as the replicator daemon synchronizes
data to its new target locations.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Ring Structure

Each ring file (e.g., `object.ring.gz`) contains:

| Component      | Description                                                           |
| -------------- | --------------------------------------------------------------------- |
| **Devices**    | All registered storage drives with zone, region, IP, port, and weight |
| **Partitions** | Virtual partition slots distributed across devices based on weight    |
| **Replicas**   | Number of copies maintained for each partition                        |

***

## Add a Storage Device

Adding new drives expands cluster capacity. New drives start with weight 0 and are
gradually weighted up to prevent sudden data movement storms.

<Steps>
  <Step title="Add the device to the ring builder">
    ```bash title="Add device to object ring" theme={null}
    xavs-ring-builder object.builder add \
      --region 1 \
      --zone 1 \
      --ip <storage-node-ip> \
      --port 6200 \
      --device <device-name> \
      --weight 100
    ```
  </Step>

  <Step title="Rebalance the ring">
    ```bash title="Rebalance ring" theme={null}
    xavs-ring-builder object.builder rebalance
    ```

    <Warning>
      Rebalancing moves data between nodes. In large clusters, a full rebalance can
      take hours and generate significant network traffic. Use `--max-balance` to
      limit the percentage of partitions moved per rebalance cycle.
    </Warning>
  </Step>

  <Step title="Write and distribute the ring file">
    ```bash title="Write the updated ring file" theme={null}
    xavs-ring-builder object.builder write_ring
    ```

    Distribute `object.ring.gz` to `/etc/xavs-object-storage/` on all proxy and
    storage nodes.
  </Step>
</Steps>

***

## Remove a Storage Device

Graceful device removal involves reducing weight to 0 and rebalancing before
physically removing the device.

<Steps>
  <Step title="Set device weight to 0">
    ```bash title="Drain device weight" theme={null}
    xavs-ring-builder object.builder set_weight <device-id> 0
    ```
  </Step>

  <Step title="Rebalance to drain the device">
    ```bash title="Rebalance to migrate data off device" theme={null}
    xavs-ring-builder object.builder rebalance
    ```
  </Step>

  <Step title="Wait for replication to complete">
    Monitor with `xavs-storage-recon --replication` until the device shows zero
    pending objects and no replication errors.
  </Step>

  <Step title="Remove the device from the ring">
    ```bash title="Remove device and final rebalance" theme={null}
    xavs-ring-builder object.builder remove <device-id>
    xavs-ring-builder object.builder rebalance
    xavs-ring-builder object.builder write_ring
    ```

    Redistribute the updated ring file to all nodes.
  </Step>
</Steps>

***

## Verify Ring Consistency

After distributing updated ring files, verify all nodes are using the same ring:

```bash title="Check MD5 hash of ring files on all nodes" theme={null}
xavs-storage-recon --md5
```

All nodes should report identical MD5 hashes for each ring file. Mismatched hashes
indicate stale ring files on specific nodes.

***

## Ring Builder Operations Reference

<CodeGroup>
  ```bash title="Show ring builder contents" theme={null}
  xavs-ring-builder object.builder show
  ```

  ```bash title="List devices in the ring" theme={null}
  xavs-ring-builder object.builder list
  ```

  ```bash title="Show ring balance report" theme={null}
  xavs-ring-builder object.builder rebalance --dry-run
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Replication" href="/services/object-storage/replication">
    Monitor data migration after ring changes
  </Card>

  <Card title="Storage Policies" href="/services/object-storage/storage-policies">
    Understand how storage policies map to ring files
  </Card>

  <Card title="Monitoring" href="/services/object-storage/monitoring">
    Track rebalancing progress and disk usage
  </Card>

  <Card title="Admin Troubleshooting" href="/services/object-storage/admin-troubleshooting">
    Diagnose ring inconsistencies and rebalancing failures
  </Card>
</CardGroup>


# Object Storage Security
Source: https://docs.xloud.tech/services/object-storage/security

Harden Xloud Object Storage — enforce TLS on proxy nodes, rotate temp URL keys, configure audit logging, and govern cross-project container access.

## Overview

Object Storage security covers the proxy-layer communication security, temporary URL
key management, audit logging, and governance of cross-project container sharing.
This guide covers the key hardening areas for platform administrators.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Hardening Guidelines

<AccordionGroup>
  <Accordion title="TLS enforcement on proxy nodes" icon="lock">
    All proxy-server endpoints must use TLS:

    * Configure the proxy-server with SSL certificates issued through Xloud Key Manager
    * Disable plain HTTP access at the load balancer frontend — redirect all port 80
      traffic to 443
    * Set HSTS headers in the proxy pipeline:
      ```
      strict-transport-security: max-age=31536000
      ```
    * Rotate TLS certificates 30 days before expiration using an automated renewal workflow
    * Verify TLS configuration after every certificate rotation:
      ```bash title="Verify TLS certificate on proxy endpoint" theme={null}
      openssl s_client -connect <proxy-ip>:443 -servername <proxy-hostname> < /dev/null
      ```
  </Accordion>

  <Accordion title="Temp URL key management" icon="link">
    Temporary URLs are signed with an account-level key. Compromise of this key allows
    generation of arbitrary temporary URLs for all objects in the account:

    * Use randomly generated keys of at least 32 bytes
    * Rotate the `Temp-URL-Key` quarterly:
      ```bash title="Rotate temp URL signing key" theme={null}
      openstack object store account set \
        --property Temp-URL-Key=$(openssl rand -hex 32)
      ```
    * Set `Temp-URL-Key-2` as a secondary key during rotation to avoid invalidating
      existing in-flight URLs immediately:
      ```bash title="Set secondary temp URL key for rotation overlap" theme={null}
      openstack object store account set \
        --property Temp-URL-Key-2=$(openssl rand -hex 32)
      ```
  </Accordion>

  <Accordion title="Object access audit logging" icon="file-text">
    The proxy-server logs every API request including the authenticated user, container,
    object path, HTTP method, and response code. Configure log forwarding with:

    * 90-day minimum retention for compliance frameworks
    * Alerting on unusual patterns:
      * Mass object deletions
      * Access from unexpected source IPs
      * Spike in 4xx errors indicating credential scanning
    * Storage of logs in a separate protected container — restrict write access to the
      proxy log-shipping service account only
  </Accordion>

  <Accordion title="Cross-project container ACL governance" icon="users">
    Cross-project container sharing via ACLs requires careful governance:

    * Audit all containers with non-empty read or write ACLs quarterly:
      ```bash title="Find containers with ACLs (admin)" theme={null}
      openstack container list --all-projects --long
      ```
    * Require documented business justification for any cross-project ACL
    * Immediately revoke ACLs for decommissioned projects
    * Never grant write ACLs to external user IDs — use dedicated service accounts
      traceable to a specific application
  </Accordion>
</AccordionGroup>

***

## Security Checklist

| Control                    | Frequency             | Command                                   |
| -------------------------- | --------------------- | ----------------------------------------- |
| TLS certificate valid      | Ongoing               | `openssl s_client -connect <proxy>:443`   |
| TLS certificate rotation   | 30 days before expiry | Key Manager renewal workflow              |
| Temp URL key rotation      | Quarterly             | `account set --property Temp-URL-Key=...` |
| Cross-project ACL audit    | Quarterly             | `container list --all-projects --long`    |
| Audit log retention review | Annually              | Verify 90-day minimum                     |

***

## Next Steps

<CardGroup>
  <Card title="Quotas" href="/services/object-storage/quotas">
    Enforce per-account and per-container storage limits
  </Card>

  <Card title="Monitoring" href="/services/object-storage/monitoring">
    Set up cluster health and access pattern monitoring
  </Card>

  <Card title="Key Manager Admin Guide" href="/services/key-manager/admin-guide">
    Manage TLS certificates used by proxy nodes
  </Card>

  <Card title="Admin Troubleshooting" href="/services/object-storage/admin-troubleshooting">
    Diagnose security-related access errors
  </Card>
</CardGroup>


# Object Storage Policies
Source: https://docs.xloud.tech/services/object-storage/storage-policies

Configure storage policies for replication, erasure coding, and tiered object placement.

## Overview

Storage policies define how object data is placed, replicated, and protected across storage nodes. Each container is assigned to exactly one policy at creation time — the policy cannot be changed after the container is created. Multiple policies enable tiered storage (standard replication, erasure coding, SSD-backed performance tiers, and archival tiers).

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * Object Storage service deployed and healthy
  * Storage rings built for each policy (see [Ring Management](/services/object-storage/ring-management))
</Note>

***

## Policy Types

Xloud Object Storage supports three storage policy types. Each maps to a distinct object ring file and enforces different data placement and durability behavior.

<Tabs>
  <Tab title="Replication" icon="copy">
    **Replication policies** store N identical copies of every object across distinct storage zones. Each replica is fully readable independently — no reconstruction is needed.

    | Parameter           | Typical Value | Description                               |
    | ------------------- | ------------- | ----------------------------------------- |
    | `replica_count`     | `3`           | Number of full copies stored per object   |
    | Min drives writable | `2 of 3`      | Write quorum for consistency              |
    | Read quorum         | `1 of 3`      | Any single replica can serve a GET        |
    | Storage overhead    | `3×`          | Total space = object size × replica count |
    | Recovery time       | Fast          | Replication copies entire objects         |

    <Tip>
      3-replica replication is the recommended default for general-purpose workloads. It tolerates simultaneous loss of any one zone with no data loss and no performance penalty during reads.
    </Tip>
  </Tab>

  <Tab title="Erasure Coding" icon="shield">
    **Erasure coding (EC) policies** encode objects into data fragments and parity fragments. The object can be reconstructed from any sufficient subset of fragments. EC significantly reduces storage overhead while maintaining high durability.

    | Parameter                 | Example (8+4)            | Description                |
    | ------------------------- | ------------------------ | -------------------------- |
    | `ec_type`                 | `liberasurecode_rs_vand` | EC algorithm               |
    | `ec_num_data_fragments`   | `8`                      | Data shards per object     |
    | `ec_num_parity_fragments` | `4`                      | Parity shards per object   |
    | Storage overhead          | `1.5×`                   | vs. `3×` for replication   |
    | Fault tolerance           | Any 4 nodes              | Can lose 4 of 12 fragments |
    | Read cost                 | Higher CPU               | Requires fragment assembly |

    <Warning>
      Erasure coding requires at least as many storage nodes as `ec_num_data_fragments + ec_num_parity_fragments`. An 8+4 EC policy requires at minimum 12 distinct storage targets. EC read latency is higher than replication — avoid EC for latency-sensitive workloads.
    </Warning>

    Supported EC algorithms:

    * `liberasurecode_rs_vand` — Vandermonde Reed-Solomon (recommended)
    * `liberasurecode_rs_cauchy` — Cauchy Reed-Solomon
    * `flat_xor_hd` — XOR-based (limited parity)
    * `isa_l_rs_vand` — Intel ISA-L accelerated (requires ISA-L library)
  </Tab>

  <Tab title="Multi-Tier" icon="database">
    **Multi-tier policies** use separate rings targeting specific device classes (SSD vs. HDD). The ring builder restricts device placement to nodes tagged with the required device class, ensuring data lands on the correct hardware tier.

    | Policy     | Target Hardware  | Ring File          | Use Case                               |
    | ---------- | ---------------- | ------------------ | -------------------------------------- |
    | `gold`     | SSD / NVMe       | `object-1.ring.gz` | Frequently accessed, latency-sensitive |
    | `standard` | Mixed or HDD     | `object.ring.gz`   | General workloads                      |
    | `archive`  | High-density HDD | `object-2.ring.gz` | Infrequent access, cost-optimized      |

    <Tip>
      Label storage nodes with device class metadata during ring building. The ring builder enforces placement constraints — a ring built for SSD nodes will never place replicas on HDD-only nodes.
    </Tip>
  </Tab>
</Tabs>

***

## Policy Configuration Reference

Storage policies are defined in the Object Storage configuration managed by XDeploy. The configuration is applied during deployment via `xavs-ansible deploy -t swift`.

```ini title="swift.conf — storage policy definition" theme={null}
[storage-policy:0]
name = standard
default = yes
aliases = Policy-0, default

[storage-policy:1]
name = gold
aliases = ssd, performance

[storage-policy:2]
name = archive
aliases = ec, cold
policy_type = erasure_coding
ec_type = liberasurecode_rs_vand
ec_num_data_fragments = 8
ec_num_parity_fragments = 4
ec_object_segment_size = 1048576
```

| Field                     | Description                                                          |
| ------------------------- | -------------------------------------------------------------------- |
| `[storage-policy:N]`      | Section index — must match the ring file suffix (`object-N.ring.gz`) |
| `name`                    | Human-readable identifier used in container creation                 |
| `default`                 | `yes` for the policy applied without explicit selection              |
| `aliases`                 | Comma-separated alternate names                                      |
| `policy_type`             | `replication` (default) or `erasure_coding`                          |
| `ec_type`                 | EC algorithm — required when `policy_type = erasure_coding`          |
| `ec_num_data_fragments`   | Data shards per EC stripe                                            |
| `ec_num_parity_fragments` | Parity shards per EC stripe                                          |
| `deprecated`              | `yes` blocks new containers from using this policy                   |

***

## View and Manage Policies

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Object Storage" icon="compass">
        Navigate to
        **Storage > Object Storage**. The storage policy is shown as a column
        in the container list and is selectable during container creation.
      </Step>

      <Step title="Create a container with a specific policy" icon="plus">
        Click **Create Container**. In the **Storage Policy** dropdown, select the desired
        policy (e.g., `gold`, `archive`). Leaving the field blank assigns the default policy.

        <Warning>
          The storage policy field is only visible if multiple policies are configured in
          your cluster. If only one policy exists, containers are automatically assigned to it.
        </Warning>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="List available policies" icon="list">
        ```bash title="List storage policies" theme={null}
        openstack object store policy list
        ```

        ```bash title="Show policy detail" theme={null}
        openstack object store policy show gold
        ```

        Policy attributes returned:

        | Field        | Description                                     |
        | ------------ | ----------------------------------------------- |
        | `name`       | Human-readable identifier                       |
        | `aliases`    | Alternate names                                 |
        | `default`    | `True` if this is the default policy            |
        | `deprecated` | `True` if new containers cannot use this policy |
        | `index`      | Numeric index matching the ring file suffix     |
      </Step>

      <Step title="Create a container on a specific policy" icon="folder-plus">
        ```bash title="Create container on gold (SSD) policy" theme={null}
        openstack container create --storage-policy gold my-fast-container
        ```

        ```bash title="Create container on archive (EC) policy" theme={null}
        openstack container create --storage-policy archive my-archive-container
        ```
      </Step>

      <Step title="Verify the assigned policy" icon="circle-check">
        ```bash title="Show container details" theme={null}
        openstack container show my-fast-container
        ```

        The `storage_policy` field in the output confirms the assigned policy.

        <Check>The container is created and the correct storage policy is displayed.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## S3 API Compatibility

Xloud Object Storage exposes an S3-compatible API endpoint alongside the native Swift API. All storage policies are accessible via both APIs.

<Tabs>
  <Tab title="S3 API Access" icon="code">
    The S3-compatible endpoint uses the same underlying storage policies. Buckets created via S3 API map to Swift containers and inherit policy assignment behavior.

    ```bash title="Create bucket via S3 API (default policy)" theme={null}
    aws s3 mb s3://my-bucket \
      --endpoint-url https://object.<your-domain>

    ```

    ```bash title="Upload object via S3 API" theme={null}
    aws s3 cp myfile.tar.gz s3://my-bucket/ \
      --endpoint-url https://object.<your-domain>
    ```

    <Info>
      S3 bucket-to-policy mapping is configured by your administrator. Contact your Xloud
      administrator to assign a specific storage policy to S3 buckets, or use the Swift
      API to create containers with explicit policy selection.
    </Info>

    Supported S3 API operations:

    * Bucket operations: `CreateBucket`, `ListBuckets`, `DeleteBucket`, `GetBucketLocation`
    * Object operations: `PutObject`, `GetObject`, `DeleteObject`, `HeadObject`, `ListObjectsV2`
    * Multipart: `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`
    * ACLs: `GetBucketAcl`, `PutBucketAcl`, `GetObjectAcl`
    * Versioning: `GetBucketVersioning`, `PutBucketVersioning`, `ListObjectVersions`
  </Tab>

  <Tab title="Swift API Access" icon="terminal">
    The native Swift API provides the most control, including explicit storage policy selection.

    ```bash title="Authenticate and create container on archive policy" theme={null}
    swift \
      --os-auth-url https://identity.<your-domain>/v3 \
      --os-project-name myproject \
      --os-username myuser \
      --os-password mypassword \
      --os-user-domain-name Default \
      --os-project-domain-name Default \
      post -H "X-Storage-Policy: archive" my-archive-bucket
    ```

    ```bash title="Upload object via Swift CLI" theme={null}
    swift upload my-archive-bucket large-dataset.tar.gz
    ```

    ```bash title="Verify container policy" theme={null}
    swift stat my-archive-bucket
    ```

    The `X-Storage-Policy` header in the stat output confirms the assigned policy.
  </Tab>
</Tabs>

***

## Multi-Cloud Access Patterns

Xloud Object Storage integrates with external cloud object storage providers. Tenant virtual machines can access multiple object storage systems using standard CLI tools and SDKs.

<AccordionGroup>
  <Accordion title="Rclone — universal multi-cloud sync" icon="refresh-cw">
    Rclone provides a unified interface for Xloud Object Storage, AWS S3, Google Cloud Storage, and Azure Blob Storage.

    ```bash title="Configure Rclone for Xloud Object Storage" theme={null}
    rclone config create xloud-swift swift \
      auth https://identity.<your-domain>/v3 \
      user myuser \
      key mypassword \
      tenant myproject \
      auth_version 3
    ```

    ```bash title="Sync from AWS S3 to Xloud Object Storage" theme={null}
    rclone sync s3:my-aws-bucket xloud-swift:my-local-container
    ```

    ```bash title="Mount Xloud container as local filesystem" theme={null}
    rclone mount xloud-swift:my-container /mnt/xloud-storage \
      --vfs-cache-mode writes &
    ```
  </Accordion>

  <Accordion title="AWS CLI — S3-compatible access" icon="cloud">
    The AWS CLI connects to the Xloud S3-compatible endpoint using standard credentials.

    ```bash title="Configure AWS CLI for Xloud S3 endpoint" theme={null}
    aws configure set aws_access_key_id YOUR_ACCESS_KEY
    aws configure set aws_secret_access_key YOUR_SECRET_KEY
    aws configure set default.region us-east-1
    ```

    ```bash title="List buckets" theme={null}
    aws s3 ls --endpoint-url https://object.<your-domain>
    ```

    ```bash title="Sync local directory to Xloud" theme={null}
    aws s3 sync ./backup/ s3://my-backup-bucket \
      --endpoint-url https://object.<your-domain>
    ```
  </Accordion>

  <Accordion title="Python boto3 — programmatic access" icon="code">
    ```python title="boto3 client for Xloud S3-compatible endpoint" theme={null}
    import boto3

    client = boto3.client(
        's3',
        endpoint_url='https://object.<your-domain>',
        aws_access_key_id='YOUR_ACCESS_KEY',
        aws_secret_access_key='YOUR_SECRET_KEY',
        region_name='default'
    )

    # List buckets
    response = client.list_buckets()
    for bucket in response['Buckets']:
        print(bucket['Name'])

    # Upload object
    client.upload_file('myfile.tar.gz', 'my-bucket', 'backups/myfile.tar.gz')
    ```
  </Accordion>

  <Accordion title="Glance integration — image backend" icon="disc">
    Xloud Image Service can store VM images and snapshots directly in Object Storage, eliminating local disk requirements on the image service nodes.

    ```ini title="glance-api.conf — Swift backend" theme={null}
    [glance_store]
    stores = swift
    default_store = swift
    swift_store_auth_version = 3
    swift_store_auth_address = https://identity.<your-domain>/v3
    swift_store_container = glance-images
    swift_store_create_container_on_put = True
    ```

    ```ini title="glance-api.conf — S3-compatible backend (Ceph RGW)" theme={null}
    [glance_store]
    stores = s3
    default_store = s3
    s3_store_host = https://object.<your-domain>
    s3_store_access_key = YOUR_ACCESS_KEY
    s3_store_secret_key = YOUR_SECRET_KEY
    s3_store_bucket = glance-images
    s3_store_create_bucket_on_put = True
    ```

    <Tip>
      Using Object Storage as the Glance backend allows image sharing across all cluster nodes without NFS or shared filesystem dependencies.
    </Tip>
  </Accordion>
</AccordionGroup>

***

## Deprecated Policies

Mark a policy as deprecated to prevent new containers from using it while maintaining full access for existing containers.

```bash title="Deprecate a policy via XDeploy configuration" theme={null}
# In the XDeploy Object Storage configuration:
# Set deprecated = yes on the target policy section
# Redeploy: xavs-ansible deploy -t swift
```

<Info>
  Deprecated policies remain fully functional for existing containers. Only new container creation using the deprecated policy is blocked. Migrate data before removing the policy configuration entirely.
</Info>

***

## Best Practices

<CardGroup>
  <Card title="Plan Policies Before Deployment" icon="map">
    Policy indexes and names are permanent once containers are created. Design your tier
    structure (standard, gold, archive) before the first container is provisioned.
  </Card>

  <Card title="Use EC for Archival Data" icon="shield">
    Erasure coding at 8+4 reduces storage overhead from 3× to 1.5× with equivalent
    or better durability. Apply EC to infrequently accessed data that tolerates
    higher read latency.
  </Card>

  <Card title="Tag Rings by Device Class" icon="tag">
    Use ring builder device class labels (SSD, HDD) to enforce hardware affinity.
    Never mix device classes in a single ring — it defeats the purpose of tiering.
  </Card>

  <Card title="Set Sensible Defaults" icon="settings">
    Configure the most common data tier as the default policy. Operators who need
    high-performance or archival storage explicitly specify the policy at container
    creation time.
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="Ring Management" href="/services/object-storage/ring-management">
    Build and distribute ring files for each storage policy
  </Card>

  <Card title="Architecture" href="/services/object-storage/architecture">
    Understand how rings, proxy servers, and storage nodes interact
  </Card>

  <Card title="Replication" href="/services/object-storage/replication">
    Monitor replication health and manage quarantined objects
  </Card>

  <Card title="Quotas" href="/services/object-storage/quotas">
    Set per-account and per-container storage limits
  </Card>
</CardGroup>


# Temporary URLs
Source: https://docs.xloud.tech/services/object-storage/temp-urls

Generate time-limited, cryptographically signed URLs for sharing Xloud Object Storage objects without granting permanent access.

## Overview

Temporary URLs (TempURLs) let you share individual objects from Xloud Object Storage with anyone — no Xloud account required — for a limited time window. The URL is cryptographically signed with an HMAC-SHA256 digest, so it cannot be guessed or extended. When the expiry time passes, the URL stops working automatically.

Common use cases: pre-signed download links for customers, time-limited file sharing with external partners, secure upload URLs for untrusted clients, and CI/CD artifact distribution.

<Note>
  **Prerequisites**

  * Object Storage access with at least project member role
  * The `python-swiftclient` package installed (`pip install python-swiftclient`) or the `openstack` CLI
  * A Temporary URL key set on your account
</Note>

***

## Set a Temporary URL Key

Before generating TempURLs, set a secret key on your account. This key signs all TempURLs — keep it confidential.

<Tabs>
  <Tab title="CLI (swift)" icon="terminal">
    ```bash title="Set account-level TempURL key" theme={null}
    swift post -m "Temp-URL-Key: $(openssl rand -hex 32)"
    ```

    ```bash title="Verify the key is set" theme={null}
    swift stat | grep Temp-URL-Key
    ```

    <Tip>
      You can set a second key (`Temp-URL-Key-2`) to enable key rotation without invalidating existing URLs.
    </Tip>
  </Tab>

  <Tab title="CLI (openstack)" icon="terminal">
    ```bash title="Set TempURL key via object store account" theme={null}
    openstack object store account set \
      --property "Temp-URL-Key=$(openssl rand -hex 32)"
    ```

    ```bash title="Verify" theme={null}
    openstack object store account show | grep Temp-URL
    ```
  </Tab>
</Tabs>

***

## Generate a Temporary URL

<Tabs>
  <Tab title="CLI (swift-temp-url)" icon="terminal">
    The `swift-temp-url` command generates signed URLs directly:

    ```bash title="Generate a 24-hour download URL" theme={null}
    swift-temp-url GET 86400 \
      /v1/AUTH_<project-id>/my-container/my-file.zip \
      <your-tempurl-key>
    ```

    This outputs a path like:

    ```
    /v1/AUTH_abc123/my-container/my-file.zip?temp_url_sig=abc...&temp_url_expires=1742000000
    ```

    Prepend your Swift endpoint to get the full URL:

    ```bash title="Full shareable URL" theme={null}
    echo "https://object.<your-domain>$(swift-temp-url GET 86400 /v1/AUTH_<project-id>/my-container/my-file.zip <key>)"
    ```
  </Tab>

  <Tab title="Python" icon="code">
    ```python title="Generate TempURL in Python" theme={null}
    import hmac
    import hashlib
    import time

    def generate_temp_url(method, expires_in, path, key):
        expires = int(time.time()) + expires_in
        hmac_body = f"{method}\n{expires}\n{path}"
        sig = hmac.new(
            key.encode('utf-8'),
            hmac_body.encode('utf-8'),
            hashlib.sha256
        ).hexdigest()
        return f"{path}?temp_url_sig={sig}&temp_url_expires={expires}"

    key = "your-tempurl-key"
    path = "/v1/AUTH_<project-id>/my-container/report.pdf"

    url = generate_temp_url("GET", 3600, path, key)
    print(f"https://object.<your-domain>{url}")
    ```
  </Tab>

  <Tab title="Bash" icon="terminal">
    ```bash title="Pure bash TempURL generator" theme={null}
    #!/bin/bash
    METHOD="GET"
    EXPIRES=$(($(date +%s) + 86400))   # 24 hours
    PATH_STR="/v1/AUTH_<project-id>/my-container/my-file.zip"
    KEY="your-tempurl-key"
    ENDPOINT="https://object.<your-domain>"

    HMAC_BODY="${METHOD}\n${EXPIRES}\n${PATH_STR}"
    SIG=$(printf "${HMAC_BODY}" | openssl dgst -sha256 -hmac "${KEY}" -hex | awk '{print $2}')

    echo "${ENDPOINT}${PATH_STR}?temp_url_sig=${SIG}&temp_url_expires=${EXPIRES}"
    ```
  </Tab>
</Tabs>

***

## Upload-Only Temporary URLs

Generate a PUT TempURL to allow a client to upload a file to a specific object path without any read access:

```bash title="Generate a 1-hour upload URL" theme={null}
swift-temp-url PUT 3600 \
  /v1/AUTH_<project-id>/uploads/submission.zip \
  <your-tempurl-key>
```

The client uploads with:

```bash title="Client-side upload with TempURL" theme={null}
curl -X PUT \
  "https://object.<your-domain>/v1/AUTH_.../uploads/submission.zip?temp_url_sig=...&temp_url_expires=..." \
  --upload-file /local/path/submission.zip
```

<Warning>
  PUT TempURLs allow anyone with the URL to overwrite the target object. Scope them to a unique object path and keep expiry windows short (minutes, not hours) for uploads.
</Warning>

***

## URL Parameters Reference

| Parameter           | Description                                                                    |
| ------------------- | ------------------------------------------------------------------------------ |
| `temp_url_sig`      | HMAC-SHA256 signature over method, expiry, and path                            |
| `temp_url_expires`  | Unix timestamp after which the URL is invalid                                  |
| `temp_url_prefix`   | (Optional) Restrict the URL to a path prefix instead of a single object        |
| `temp_url_ip_range` | (Optional) Restrict URL use to a specific IP or CIDR range                     |
| `filename`          | (Optional) Override the `Content-Disposition` filename in the browser download |

```bash title="Force a browser download filename" theme={null}
# Append &filename=report-q1.pdf to the TempURL
"https://object.<your-domain>/...?temp_url_sig=...&temp_url_expires=...&filename=report-q1.pdf"
```

***

## Key Rotation

Rotate TempURL keys without immediately breaking existing URLs by using both key slots:

<Steps>
  <Step title="Set new key in slot 2" icon="key">
    ```bash title="Add new key to slot 2" theme={null}
    swift post -m "Temp-URL-Key-2: $(openssl rand -hex 32)"
    ```

    Existing URLs signed with key 1 remain valid.
  </Step>

  <Step title="Migrate to new key" icon="rotate-cw">
    Update all URL generation processes to use the new key value from slot 2.
  </Step>

  <Step title="Replace old key" icon="check">
    Once all old URLs have expired, move the new key to slot 1 and clear slot 2:

    ```bash title="Promote new key to slot 1" theme={null}
    NEW_KEY="your-new-key"
    swift post -m "Temp-URL-Key: ${NEW_KEY}"
    swift post -m "Temp-URL-Key-2:"
    ```
  </Step>
</Steps>

***

## Security Considerations

<AccordionGroup>
  <Accordion title="Keep expiry windows short" icon="clock">
    Set the minimum expiry needed for the use case. Downloads that should complete in minutes should not have 24-hour URLs. An attacker who intercepts a URL has access until expiry.
  </Accordion>

  <Accordion title="Use IP range restrictions for sensitive data" icon="shield">
    The `temp_url_ip_range` parameter restricts URL use to a specific source IP or CIDR:

    ```bash title="Restrict to a single IP" theme={null}
    # Add to the signed path parameters before generating
    swift-temp-url GET 3600 \
      "/v1/AUTH_.../sensitive.zip?temp_url_ip_range=203.0.113.5" \
      <key>
    ```
  </Accordion>

  <Accordion title="Never expose your TempURL key" icon="lock">
    The TempURL key signs all URLs for your account. Treat it like a password. Do not embed it in client-side code, public repositories, or logs. Rotate it if exposure is suspected.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Access Control" href="/services/object-storage/access-control">
    Container ACLs and account-level access policies for permanent access grants
  </Card>

  <Card title="Object Versioning" href="/services/object-storage/versioning">
    Retain previous versions of objects to recover from accidental overwrites
  </Card>

  <Card title="Large Objects" href="/services/object-storage/large-objects">
    Upload objects larger than 5 GB using multi-part Static or Dynamic Large Objects
  </Card>

  <Card title="Object Storage Security" href="/services/object-storage/security">
    Server-side encryption, TLS, and hardening guidance for object storage
  </Card>
</CardGroup>


# Object Storage Troubleshooting
Source: https://docs.xloud.tech/services/object-storage/troubleshooting

Resolve common Xloud Object Storage issues — 403 access errors, upload timeouts, versioning failures, and temporary URL authorization problems.

## Overview

This guide covers user-facing object storage issues. For platform-level issues such as
storage node failures, ring inconsistencies, or proxy outages, see the
[Admin Troubleshooting](/services/object-storage/admin-troubleshooting) guide.

***

## Common Issues

<AccordionGroup>
  <Accordion title="403 Forbidden when accessing a container" icon="lock">
    **Cause**: Your authentication token belongs to a project that does not have read
    or write access to the container.

    **Diagnosis**:

    ```bash title="Check container ACL" theme={null}
    openstack container show <container-name> | grep -i "read\|write"
    ```

    **Resolution**:

    * If the read ACL does not include your project or user ID, request the container
      owner to add your identity to the ACL
    * If the container is owned by your project and you still get 403, verify your role
      assignment includes the `object-store:read` capability
    * For public containers that return 403, verify the ACL contains `.r:*`
  </Accordion>

  <Accordion title="Object upload times out for large files" icon="clock">
    **Cause**: The proxy server timeout is shorter than the time needed to upload the
    file. Files over 5 GB must use the SLO or DLO large object mechanism.

    **Resolution**: Use the large object upload procedure described in the
    [Large Objects](/services/object-storage/large-objects) guide. Split files into
    1 GB or smaller segments.

    For files under 5 GB that still time out, check network bandwidth between your
    client and the proxy endpoint.
  </Accordion>

  <Accordion title="Versioned object not recoverable" icon="history">
    **Cause**: The archive container was deleted, or object versioning was disabled before
    the overwrite occurred.

    **Diagnosis**:

    ```bash title="List archived versions" theme={null}
    openstack object list <archive-container>
    ```

    **Resolution**:

    * If the archive container exists, list its contents and locate the version by
      timestamp prefix
    * If the archive container is missing, versions stored there are unrecoverable
    * Versioning must be enabled before overwrites occur — it cannot be applied retroactively
  </Accordion>

  <Accordion title="Temporary URL returns 401 Unauthorized" icon="link">
    **Cause**: The temporary URL has expired, or the signing key used to generate it
    has been changed on the account.

    **Diagnosis**:

    ```bash title="Check temp URL key" theme={null}
    openstack object store account show | grep -i temp
    ```

    **Resolution**:

    * If no key is configured, set one first:
      ```bash title="Set temp URL key" theme={null}
      openstack object store account set \
        --property Temp-URL-Key=<your-secret-key>
      ```
    * If the key was recently changed, regenerate the temporary URL using the new key
    * Set `Temp-URL-Key-2` as a secondary key during rotation to avoid invalidating
      in-flight URLs immediately
  </Accordion>

  <Accordion title="Object list returns empty when objects exist" icon="search">
    **Cause**: Object listing uses a prefix filter that doesn't match, or the container
    ACL does not include `.rlistings` for public access.

    **Diagnosis**:

    ```bash title="List without prefix filter" theme={null}
    openstack object list <container>
    ```

    **Resolution**:

    * Remove any prefix filter to see all objects
    * For public containers, add `.rlistings` to the read ACL if directory listing should
      be accessible:
      ```bash theme={null}
      openstack container set \
        --property "X-Container-Read=.r:*,.rlistings" \
        <container>
      ```
  </Accordion>
</AccordionGroup>

***

## Diagnostic Commands

<CodeGroup>
  ```bash title="Check account status and usage" theme={null}
  openstack object store account show
  ```

  ```bash title="Show container metadata and ACLs" theme={null}
  openstack container show <container>
  ```

  ```bash title="List objects in a container" theme={null}
  openstack object list <container>
  ```

  ```bash title="Show object metadata" theme={null}
  openstack object show <container> <object>
  ```

  ```bash title="Check temp URL key configuration" theme={null}
  openstack object store account show | grep -i temp
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Admin Troubleshooting" href="/services/object-storage/admin-troubleshooting">
    Platform-level issues — 507 errors, ring inconsistencies, proxy latency
  </Card>

  <Card title="Access Control" href="/services/object-storage/access-control">
    Review and update container ACL configuration
  </Card>

  <Card title="Versioning" href="/services/object-storage/versioning">
    Enable versioning to prevent future data loss from overwrites
  </Card>

  <Card title="Large Objects" href="/services/object-storage/large-objects">
    Resolve timeouts with proper large object upload procedures
  </Card>
</CardGroup>


# Upload and Manage Objects
Source: https://docs.xloud.tech/services/object-storage/upload-objects

Upload, download, list, update, and delete objects in Xloud Object Storage containers using the Dashboard and CLI. Includes metadata and bulk operations.

## Overview

Objects are the files stored within your containers. Each object has a name, payload,
content type, and optional custom metadata. Objects can be any size — files over 5 GB
require the Large Object mechanism (see [Large Objects](/services/object-storage/large-objects)).

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  A container must exist before objects can be uploaded. See
  [Create a Container](/services/object-storage/create-container) to provision one.
</Note>

***

## Upload Objects

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the container">
        Navigate to **Storage > Object Storage** and click the container name.
      </Step>

      <Step title="Upload files">
        Click **Upload File** and select the file from your local filesystem. For multiple
        files, use the **Upload Folder** option to upload directory contents recursively.

        <Tip>
          Set custom metadata during upload by expanding the **Custom Metadata** section.
          Add key-value pairs for tagging, workflow state, or application-specific properties.
        </Tip>
      </Step>

      <Step title="Confirm upload">
        Uploaded objects appear in the container file browser with their size and last
        modified timestamp.

        <Check>Object is listed in the container with the correct file size.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="Upload a single file" theme={null}
      openstack object create app-backups backup-2025-03-18.tar.gz
      ```

      ```bash title="Upload with custom metadata" theme={null}
      openstack object create \
        --property backup-type=daily \
        --property retention-days=90 \
        app-backups backup-2025-03-18.tar.gz
      ```

      ```bash title="Upload all files from a directory" theme={null}
      for f in /backups/*.gz; do
        openstack object create app-backups "$f"
      done
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Download Objects

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to the container, click the object name, and click **Download**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Download an object" theme={null}
    openstack object save app-backups backup-2025-03-18.tar.gz
    ```

    ```bash title="Download to a specific path" theme={null}
    openstack object save \
      --file /tmp/backup-restore.tar.gz \
      app-backups backup-2025-03-18.tar.gz
    ```
  </Tab>
</Tabs>

***

## List and Inspect Objects

<CodeGroup>
  ```bash title="List objects in a container" theme={null}
  openstack object list app-backups
  ```

  ```bash title="Show object metadata" theme={null}
  openstack object show app-backups backup-2025-03-18.tar.gz
  ```

  ```bash title="List with long format (size, date)" theme={null}
  openstack object list app-backups --long
  ```

  ```bash title="Filter by prefix" theme={null}
  openstack object list app-backups --prefix backup-2025
  ```
</CodeGroup>

***

## Generate Temporary URLs

Temporary URLs provide time-limited access to objects without requiring authentication
tokens — useful for sharing objects with external parties.

<Tabs>
  <Tab title="Setup temp URL key" icon="key">
    First, configure a temporary URL signing key on the account:

    ```bash title="Set temporary URL key" theme={null}
    openstack object store account set \
      --property Temp-URL-Key=$(openssl rand -hex 32)
    ```

    Store this key securely — it is required to generate all temporary URLs.
  </Tab>

  <Tab title="Generate a temp URL" icon="link">
    ```bash title="Generate a 1-hour temporary download URL" theme={null}
    openstack object create app-backups backup.tar.gz  # ensure object exists

    # Using the swift CLI for temp URL generation
    swift tempurl GET 3600 \
      /v1/<account>/app-backups/backup.tar.gz \
      <temp-url-key>
    ```

    <Tip>
      Set `Temp-URL-Key-2` as a secondary key to allow key rotation without invalidating
      in-flight temporary URLs.
    </Tip>
  </Tab>
</Tabs>

***

## Delete Objects

<CodeGroup>
  ```bash title="Delete a single object" theme={null}
  openstack object delete app-backups backup-2025-03-18.tar.gz
  ```

  ```bash title="Delete multiple objects" theme={null}
  openstack object delete app-backups obj1 obj2 obj3
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Access Control" href="/services/object-storage/access-control">
    Configure ACLs to control who can access the container and its objects
  </Card>

  <Card title="Large Objects" href="/services/object-storage/large-objects">
    Upload files larger than 5 GB using SLO or DLO
  </Card>

  <Card title="Versioning" href="/services/object-storage/versioning">
    Protect objects from accidental overwrites with version retention
  </Card>

  <Card title="Troubleshooting" href="/services/object-storage/troubleshooting">
    Resolve upload failures and access errors
  </Card>
</CardGroup>


# Object Storage User Guide
Source: https://docs.xloud.tech/services/object-storage/user-guide

Create containers, upload objects, configure access control, enable versioning, and work with large objects in Xloud Object Storage.

<p>Overview</p>

Xloud Object Storage provides a scalable, durable store for unstructured data. Data is
organized into containers (buckets) within your project account. Each container holds
objects of any size along with user-defined metadata. Access is controlled through ACLs,
enabling secure sharing within and across projects.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Note>
  Object Storage is accessed via the Object Storage API endpoint. Ensure your credentials
  file sets `OS_AUTH_URL` and that your project has the `object-store` service available.
  Verify access with `openstack object store account show`.
</Note>

***

<p>Topics in This Guide</p>

<CardGroup>
  <Card title="Create a Container" icon="folder-plus" href="/services/object-storage/create-container">
    Provision object storage containers with storage policy selection and access settings
  </Card>

  <Card title="Upload Objects" icon="upload" href="/services/object-storage/upload-objects">
    Upload, download, list, and delete objects using the Dashboard and CLI
  </Card>

  <Card title="Access Control" icon="list-checks" href="/services/object-storage/access-control">
    Configure read and write ACLs on containers for cross-project sharing
  </Card>

  <Card title="Versioning" icon="history" href="/services/object-storage/versioning">
    Enable object version retention to protect against accidental overwrites
  </Card>

  <Card title="Large Objects" icon="file-archive" href="/services/object-storage/large-objects">
    Upload files over 5 GB using Static and Dynamic Large Object mechanisms
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/object-storage/troubleshooting">
    Resolve 403 errors, upload timeouts, versioning failures, and temp URL issues
  </Card>
</CardGroup>

***

<p>Storage Hierarchy</p>

| Concept            | Description                                                              |
| ------------------ | ------------------------------------------------------------------------ |
| **Account**        | Your project's top-level object storage namespace — holds all containers |
| **Container**      | A logical bucket within the account — all objects reside in a container  |
| **Object**         | A stored data item — any file, document, image, or binary blob           |
| **Metadata**       | Custom key-value pairs on accounts, containers, or objects               |
| **Storage Policy** | Named configuration defining how container data is replicated            |
| **Temporary URL**  | A time-limited signed URL for sharing object access without tokens       |

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Object Storage Admin Guide" icon="shield-check" href="/services/object-storage/admin-guide">
    Configure storage policies, rings, replication, and quotas
  </Card>

  <Card title="Key Manager" icon="lock" href="/services/key-manager/user-guide">
    Manage encryption keys for server-side object storage encryption
  </Card>

  <Card title="Identity" icon="fingerprint" href="/cli-setup">
    Configure authentication tokens for programmatic object storage access
  </Card>

  <Card title="Load Balancer" icon="combine" href="/services/load-balancer/user-guide">
    Distribute object storage proxy requests behind a load balancer
  </Card>
</CardGroup>


# Object Versioning
Source: https://docs.xloud.tech/services/object-storage/versioning

Enable object version retention on Xloud Object Storage containers to preserve previous versions when objects are overwritten or deleted.

## Overview

Enable versioning on a container to retain all previous versions of objects. Each time
an object is overwritten, the previous version is preserved in a linked archive container.
This protects against accidental overwrites and provides a manual recovery mechanism.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

<Warning>
  Versioning must be enabled before overwrites occur — it cannot be applied retroactively.
  Objects overwritten before versioning was enabled have no archived versions.
</Warning>

***

## How Versioning Works

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant Primary as Primary Container (app-backups)
    participant Archive as Archive Container (app-backups-versions)

    Client->>Primary: PUT object.tar.gz (v2 content)
    Primary->>Archive: Move current version (v1) to archive
    Primary->>Primary: Store new version (v2)
    Note over Archive: app-backups-versions/object.tar.gz/001

    Client->>Primary: PUT object.tar.gz (v3 content)
    Primary->>Archive: Move v2 to archive
    Primary->>Primary: Store v3
    Note over Archive: app-backups-versions/object.tar.gz/002
```

***

## Enable Versioning

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create the archive container">
        Create a separate container to store previous versions. Navigate to
        **Storage > Object Storage** and click **Create Container**.
        Name it `app-backups-versions` (or similar).

        <Info>
          Create the archive container before enabling versioning on the primary container.
          The archive container must already exist.
        </Info>
      </Step>

      <Step title="Enable versioning on the primary container">
        Navigate to the primary container settings, click **Edit**, and enable
        **Object Versioning**. Specify the archive container name.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create the archive container">
        ```bash title="Create versioning archive" theme={null}
        openstack container create app-backups-versions
        ```
      </Step>

      <Step title="Enable versioning on the primary container">
        ```bash title="Enable versioning" theme={null}
        openstack container set \
          --property "X-Versions-Location=app-backups-versions" \
          app-backups
        ```
      </Step>

      <Step title="Verify versioning is active">
        ```bash title="Check versioning header" theme={null}
        openstack container show app-backups | grep -i version
        ```

        <Check>Container metadata shows `X-Versions-Location` set to the archive container name.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Recover a Previous Version

When an object is overwritten, the previous version is stored in the archive container
with a name that includes the original object name and a timestamp prefix.

```bash title="List archived versions" theme={null}
openstack object list app-backups-versions | grep object.tar.gz
```

```bash title="Download a specific archived version" theme={null}
openstack object save app-backups-versions \
  "app-backups/object.tar.gz/001t<timestamp>"
```

To restore a version, copy it back to the primary container:

```bash title="Restore an archived version" theme={null}
# Download the archived version
openstack object save app-backups-versions \
  "app-backups/object.tar.gz/001t<timestamp>" \
  --file /tmp/restored.tar.gz

# Re-upload to the primary container
openstack object create app-backups /tmp/restored.tar.gz \
  --name object.tar.gz
```

***

## Disable Versioning

```bash title="Disable versioning" theme={null}
openstack container set \
  --property "X-Versions-Location=" \
  app-backups
```

<Warning>
  Disabling versioning stops archiving new overwrites but does not remove existing
  archived versions. The archive container and its contents remain and accrue storage costs.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Upload Objects" href="/services/object-storage/upload-objects">
    Upload and manage objects within versioned containers
  </Card>

  <Card title="Access Control" href="/services/object-storage/access-control">
    Configure ACLs on both primary and archive containers
  </Card>

  <Card title="Large Objects" href="/services/object-storage/large-objects">
    Handle files over 5 GB with SLO and DLO mechanisms
  </Card>

  <Card title="Troubleshooting" href="/services/object-storage/troubleshooting">
    Resolve versioning configuration and recovery issues
  </Card>
</CardGroup>


# Action Plan Policies
Source: https://docs.xloud.tech/services/optimization/admin-guide/action-policies

Configure Xloud Optimization action plan execution policies — manual approval workflows, automatic execution for trusted strategies, and plan expiry controls.

## Overview

Action plan policies control whether migration plans require human approval before
execution or execute automatically after audit completion. The default policy requires
explicit approval — ensuring operators review every migration before it occurs. Automatic
execution is available for trusted, validated strategies in controlled environments.

***

## Policy Options

<CardGroup>
  <Card title="Manual Approval (Default)" icon="eye">
    All action plans require an explicit `execute` call. Operators review each
    migration before approving. No workloads move without human oversight.
  </Card>

  <Card title="Automatic Execution" icon="play">
    Plans execute immediately after audit completion without operator review.
    Suitable for fully tested strategies in non-production environments.
  </Card>
</CardGroup>

***

## Manual Approval Policy

The default policy. All action plans are created in `RECOMMENDED` state and require
explicit execution via the Dashboard or CLI.

This policy is appropriate for:

* Production environments with change management requirements
* Strategies involving mission-critical instances
* Clusters where maintenance windows must be respected

No configuration change is needed — this is the platform default.

***

## Automatic Execution Policy

Enable automatic execution by setting `auto_trigger` at audit creation:

```bash title="Create auto-executing audit" theme={null}
watcher audit create \
  --goal server_consolidation \
  --auto-trigger True
```

For scheduled audits, include `--auto-trigger True` in the audit creation command. All
resulting action plans execute automatically within 5–10 minutes of audit completion.

<Warning>
  Enable automatic execution only for non-production environments or after thorough
  testing of the strategy on your cluster topology. Automatic execution can trigger
  live migrations at any time — including during business hours if the audit schedule
  is misconfigured.
</Warning>

***

## Action Plan Expiry

Plans become stale when the cluster state changes significantly after audit completion.
Configure a maximum plan age to prevent outdated migrations from executing.

<Tabs>
  <Tab title="XDeploy" icon="browser">
    <Steps>
      <Step title="Open Advanced Configuration" icon="settings">
        Open **XDeploy** and navigate to **Advanced Configuration**. In the **Service Tree**
        (left panel), select **watcher**.
      </Step>

      <Step title="Edit the Expiry Setting" icon="file-pen">
        Click **New File** or select an existing `watcher.conf` from the **File Browser**
        (right panel). Add the following in the **Code Editor** (center panel):

        ```ini title="/etc/xavs/config/watcher/watcher.conf" theme={null}
        [DEFAULT]
        action_plan_expiry = 24
        ```

        Plans older than `action_plan_expiry` hours are automatically set to `CANCELLED` state.
        A new audit must run to generate a current plan.
      </Step>

      <Step title="Save and Apply" icon="circle-check">
        Click **Save Current File**. Return to **Operations** and run **reconfigure** to
        apply the expiry policy to the API and Decision Engine.

        <Check>Action plan expiry policy configured and applied via XDeploy.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly:

    ```ini title="/etc/xavs/watcher/watcher.conf" theme={null}
    [DEFAULT]
    action_plan_expiry = 24
    ```

    Plans older than `action_plan_expiry` hours are automatically set to `CANCELLED` state.
    A new audit must run to generate a current plan.

    ```bash title="Restart API and Decision Engine after config change" theme={null}
    docker restart watcher_api watcher_decision_engine
    ```
  </Tab>
</Tabs>

***

## Role-Based Execution Control

By default, the `admin` role can create audits, approve plans, and execute them.
The `member` role has read-only access to audits and plans.

To restrict execution to cloud administrators only, verify the default policy:

```bash title="Check RBAC policy" theme={null}
docker exec watcher_api \
  oslopolicy-list-redundant \
  --config-file /etc/watcher/watcher.conf
```

The default policies ensure:

* Plan **execution** requires the `admin` role
* Plan **viewing** is available to `member` and `reader` roles
* Plan **creation** (via audit) requires the `admin` role

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Create a test audit and verify the resulting action plan shows `RECOMMENDED`
    state (not auto-executed) when manual approval is configured.

    <Check>Action plan requires explicit approval before execution can begin.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create test audit without auto-trigger" theme={null}
    watcher audit create \
      --goal server_consolidation \
      --name policy-test
    ```

    ```bash title="Verify plan requires approval" theme={null}
    watcher actionplan list \
      -f value -c state
    ```

    Expected: `RECOMMENDED` (not `PENDING` or `ONGOING`)

    <Check>Plan state is `RECOMMENDED` confirming manual approval is required.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Scheduling" href="/services/optimization/admin-guide/scheduling">
    Set up recurring audits and configure their execution policy.
  </Card>

  <Card title="Security" href="/services/optimization/admin-guide/security">
    Configure RBAC to restrict execution to the admin role.
  </Card>

  <Card title="Compute Integration" href="/services/optimization/admin-guide/compute-integration">
    Ensure compute hosts support live migration for action execution.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/admin-guide/troubleshooting">
    Diagnose auto-trigger failures and execution policy issues.
  </Card>
</CardGroup>


# Optimization Architecture
Source: https://docs.xloud.tech/services/optimization/admin-guide/architecture

Understand the Xloud Optimization architecture — Decision Engine, strategy plugins, data collectors, Planner, and Applier component roles and.

## Overview

The Xloud Optimization is composed of three collaborating services: the API and
Decision Engine (analysis), the Planner (action plan generation), and the Applier (execution).
Understanding the architecture helps administrators tune the system, diagnose failures,
and integrate custom data sources or strategies.

<Warning>
  This guide requires administrator privileges. Changes to strategy configuration and
  data source settings affect all ongoing and future audits platform-wide.
</Warning>

***

## Component Diagram

```mermaid theme={null}
graph TD
    subgraph Input
        DS1[Telemetry / Prometheus<br/>CPU, RAM, Temperature]
        DS2[Compute API<br/>Host inventory, VM placement]
    end

    subgraph Decision Engine
        DE[Decision Engine]
        STR[Strategy Plugins]
        SCORER[Efficiency Scorer]
    end

    subgraph Planning
        PL[Planner]
        AP[(Action Plan DB)]
    end

    subgraph Execution
        APP[Applier]
        WF[Workflow Engine<br/>Taskflow]
        NOVA[Xloud Compute API<br/>Live Migration]
    end

    DS1 -->|Metrics| DE
    DS2 -->|Cluster state| DE
    DE -->|Select strategy| STR
    STR -->|Evaluate placements| SCORER
    SCORER -->|Recommended actions| PL
    PL --> AP
    AP -->|Started by operator| APP
    APP -->|Start via| WF
    WF -->|Migrate instances| NOVA
```

***

## Components

<AccordionGroup>
  <Accordion title="Decision Engine" icon="cpu">
    The analytical core. On each audit, the Decision Engine:

    1. Collects the cluster data model from the Compute API (host inventory, VM placement)
    2. Fetches time-series metrics from configured data sources (Prometheus, Telemetry)
    3. Selects the strategy plugin for the requested goal
    4. Runs the strategy algorithm to identify suboptimal placements
    5. Passes the resulting recommended actions to the Planner

    Deployed as: `watcher_decision_engine` container on controller nodes.
  </Accordion>

  <Accordion title="Planner" icon="clipboard-list">
    Receives the recommended action set from the Decision Engine and applies dependency
    ordering — ensuring migrations that depend on free capacity from an earlier migration
    execute in the correct sequence.

    The Planner is embedded in the Decision Engine container and does not run as a
    separate service.
  </Accordion>

  <Accordion title="Applier" icon="play">
    Executes approved action plans through a Taskflow workflow engine. The Applier:

    * Processes actions in priority order
    * Handles retries for transient failures
    * Updates action state in the database in real time
    * Halts execution on non-retriable failures

    Deployed as: `watcher_applier` container on controller nodes.
  </Accordion>

  <Accordion title="API" icon="code">
    Exposes the REST API used by the Dashboard and CLI to create audits, view action
    plans, and execute or cancel plans.

    Deployed as: `watcher_api` container on controller nodes.
  </Accordion>
</AccordionGroup>

***

## Deployment Topology

<Tree>
  <Tree.Folder name="Controller Nodes">
    <Tree.File name="watcher_api — REST API" />

    <Tree.File name="watcher_decision_engine — Analysis + Planner" />

    <Tree.File name="watcher_applier — Plan execution" />
  </Tree.Folder>
</Tree>

***

## Data Flow: Audit to Execution

```mermaid theme={null}
sequenceDiagram
    participant OP as Operator
    participant API as Watcher API
    participant DE as Decision Engine
    participant DB as Watcher DB
    participant APP as Applier
    participant NOVA as Xloud Compute

    OP->>API: Create audit (goal: consolidation)
    API->>DB: Store audit (state: PENDING)
    API->>DE: Dispatch audit task
    DE->>DE: Collect cluster model + metrics
    DE->>DE: Apply strategy algorithm
    DE->>DB: Store action plan (state: RECOMMENDED)
    DB->>OP: Notify — action plan ready
    OP->>API: Start action plan
    API->>APP: Dispatch execution task
    APP->>NOVA: POST /servers/{id}/action (live-migration)
    NOVA-->>APP: Migration complete
    APP->>DB: Update action (state: SUCCEEDED)
    APP->>DB: Update plan (state: SUCCEEDED)
```

***

## Service Configuration File

```bash title="Configuration file location" theme={null}
cat /etc/xavs/watcher/watcher.conf
```

Key sections:

| Section                                   | Purpose                                             |
| ----------------------------------------- | --------------------------------------------------- |
| `[DEFAULT]`                               | Service identity, RPC transport, action plan expiry |
| `[watcher_cluster_data_model_collectors]` | Data source configuration                           |
| `[watcher_strategies.*]`                  | Per-strategy tuning parameters                      |
| `[keystone_authtoken]`                    | Xloud Identity authentication                       |
| `[database]`                              | Database connection                                 |

***

## Next Steps

<CardGroup>
  <Card title="Strategy Configuration" href="/services/optimization/admin-guide/strategy-config">
    Configure and tune optimization strategy plugins.
  </Card>

  <Card title="Data Sources" href="/services/optimization/admin-guide/data-sources">
    Connect Prometheus and Telemetry data sources to the Decision Engine.
  </Card>

  <Card title="Scheduling" href="/services/optimization/admin-guide/scheduling">
    Automate recurring audits with audit templates and schedules.
  </Card>

  <Card title="Security" href="/services/optimization/admin-guide/security">
    Configure RBAC and service account credentials for the Optimization.
  </Card>
</CardGroup>


# Compute Integration
Source: https://docs.xloud.tech/services/optimization/admin-guide/compute-integration

Configure Xloud Compute for Optimization compatibility — verify shared storage, live migration capability, and host state for optimization action execution.

## Overview

The Optimization relies on Xloud Compute for both cluster state collection and
migration execution. All optimization actions that produce live migrations require
compute hosts to support live migration — which in turn requires shared storage for
instance disks. This page covers the compute requirements, verification steps, and
configuration adjustments needed before enabling optimization in production.

<Note>
  **Prerequisites**

  * Administrator privileges
  * Xloud Compute deployed and operational
  * Xloud Distributed Storage or equivalent shared storage for instance volumes
</Note>

***

## Shared Storage Requirement

<Warning>
  Live migration requires that instance disks are backed by shared storage (Xloud
  Distributed Storage or equivalent). Instances with local ephemeral disks cannot
  be live-migrated. Such instances will cause actions to fail during plan execution.
</Warning>

Verify that the instances targeted by optimization are using shared storage:

```bash title="Check instance storage backend" theme={null}
openstack server show <instance-id> \
  -f value -c "os-extended-volumes:volumes_attached"
```

Instances with no attached volumes (or with ephemeral disk only) cannot be live-migrated.
Work with project owners to migrate critical ephemeral-disk instances to volume-backed
equivalents before enabling optimization strategies that include those instances.

***

## Verify Compute Host Availability

Optimization actions only target compute hosts that are up, enabled, and not in
maintenance state.

```bash title="Check all compute services" theme={null}
openstack compute service list --service nova-compute
```

| Column     | Required Value |
| ---------- | -------------- |
| **State**  | `up`           |
| **Status** | `enabled`      |

Hosts with `state: down` or `status: disabled` are automatically excluded from
optimization targets. Disabled hosts do not receive live-migrated instances.

***

## Verify Live Migration Capability

```bash title="Check migration capability per host" theme={null}
openstack hypervisor show <hostname> \
  -f value -c hypervisor_hostname -c state -c status
```

Perform a test live migration between two hosts in the optimization segment:

```bash title="Test live migration between hosts" theme={null}
openstack server migrate <test-instance-id> \
  --live-migration \
  --host <destination-host>

openstack server show <test-instance-id> \
  -f value -c status
```

Expected: instance returns to `ACTIVE` status on the destination host.

***

## CPU Compatibility for Live Migration

Live migration between hosts with different CPU architectures or feature sets can fail.
Configure a common CPU baseline in the Compute service to ensure compatibility across
all hosts in the optimization segment.

<Tabs>
  <Tab title="XDeploy" icon="browser">
    <Steps>
      <Step title="Open Advanced Configuration" icon="settings">
        Open **XDeploy** and navigate to **Advanced Configuration**. In the **Service Tree**
        (left panel), select **nova**.
      </Step>

      <Step title="Edit the Compute Configuration" icon="file-pen">
        Click **New File** or select an existing `nova.conf` from the **File Browser**
        (right panel). Add the following in the **Code Editor** (center panel):

        ```ini title="/etc/xavs/config/nova/nova.conf" theme={null}
        [libvirt]
        cpu_mode = custom
        cpu_model = Cascadelake-Server-noTSX
        ```
      </Step>

      <Step title="Save and Apply" icon="circle-check">
        Click **Save Current File**. Return to **Operations** and run **reconfigure** to
        apply the CPU compatibility settings across all compute hosts.

        <Check>CPU compatibility baseline configured and applied via XDeploy.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly:

    ```ini title="nova.conf -- CPU compatibility" theme={null}
    [libvirt]
    cpu_mode = custom
    cpu_model = Cascadelake-Server-noTSX
    ```
  </Tab>
</Tabs>

<Tip>
  Use `cpu_mode = custom` with a CPU model that represents the lowest common denominator
  across all compute hosts in the optimization segment. This is especially important in
  clusters with heterogeneous hardware generations.
</Tip>

***

## Host Aggregates for Optimization Scope

Use host aggregates to define which hosts are eligible for optimization. Assign all
hosts intended for consolidation to a single aggregate, then scope your audits to
that aggregate:

```bash title="Create optimization aggregate" theme={null}
openstack aggregate create optimization-zone
openstack aggregate add host optimization-zone compute-01
openstack aggregate add host optimization-zone compute-02
openstack aggregate add host optimization-zone compute-03
```

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Hypervisors** (admin view). Verify:

    * All compute hosts show `State: Up` and `Status: Enabled`
    * No hosts are in maintenance or down state

    <Check>All compute hosts are up and enabled, ready to serve as migration targets.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Verify all hosts are available" theme={null}
    openstack compute service list \
      --service nova-compute \
      -f table -c Host -c State -c Status
    ```

    ```bash title="Check per-host utilization" theme={null}
    openstack hypervisor list --long \
      -f table -c Hostname -c "Running VMs" -c "vCPUs Used" -c "Memory MB Used"
    ```

    <Check>All hosts show `State: up`, `Status: enabled`, and capacity is visible for optimization planning.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Action Policies" href="/services/optimization/admin-guide/action-policies">
    Configure manual approval vs automatic execution for optimization plans.
  </Card>

  <Card title="Security" href="/services/optimization/admin-guide/security">
    Secure the service account used by the Applier to call the Compute API.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/admin-guide/troubleshooting">
    Diagnose live migration failures during plan execution.
  </Card>

  <Card title="Compute Admin Guide" href="/services/compute/admin-guide">
    Configure compute hosts, shared storage, and live migration settings.
  </Card>
</CardGroup>


# Custom Strategies
Source: https://docs.xloud.tech/services/optimization/admin-guide/custom-strategies

Implement and deploy custom Xloud Optimization strategy plugins — extend the Decision Engine with proprietary optimization algorithms for specialized.

## Overview

The Optimization supports custom optimization strategies through a Python plugin
interface. A custom strategy implements the `BaseStrategy` class, defines its own
goal association, and is registered via Python entry points. Custom strategies enable
specialized optimization logic for non-standard cluster topologies, proprietary metric
sources, or regulatory compliance placement requirements.

<Note>
  Custom strategies are deployed as Python packages on the controller node running the
  Decision Engine. A Decision Engine restart is required after installing a new strategy.
</Note>

***

## Strategy Interface

Custom strategies inherit from `watcher.decision_engine.strategy.strategies.base.BaseStrategy`
and must implement three methods:

```python title="Custom strategy skeleton" theme={null}
from watcher.decision_engine.strategy.strategies import base

class MyCustomStrategy(base.BaseStrategy):
    """Placement strategy for specialized workloads."""

    NAME = "my_custom_strategy"
    DISPLAY_NAME = "My Custom Strategy"
    GOAL_NAME = "server_consolidation"  # Associate with an existing goal

    def pre_execute(self):
        """Called before the main analysis — validate data sources."""
        pass

    def do_execute(self, audit):
        """Main analysis loop — build the action plan."""
        cluster_model = self.compute_model
        for host in cluster_model.compute_nodes.values():
            if self._is_underutilized(host):
                self._migrate_instances_from(host)

    def post_execute(self):
        """Called after analysis — cleanup."""
        pass

    def _is_underutilized(self, host):
        used_vcpu = host.vcpus - host.free_disk_gb  # Example metric
        return used_vcpu / host.vcpus < 0.15

    def _migrate_instances_from(self, source_host):
        for instance in self.compute_model.mapping.get_node_instances(source_host):
            self.solution.add_action(
                action_type="migrate",
                input_parameters={
                    "source_node": source_host.hostname,
                    "destination_node": self._find_target_host(instance),
                    "migration_type": "live"
                }
            )
```

***

## Register the Strategy

Add the strategy to the Python package entry points:

```ini title="setup.cfg" theme={null}
[entry_points]
watcher_strategies =
    my_custom_strategy = mypackage.strategies:MyCustomStrategy
```

Build and install the package:

```bash title="Install custom strategy package" theme={null}
pip install -e /path/to/mypackage
```

***

## Deploy to the Decision Engine Container

```bash title="Copy strategy package into container" theme={null}
docker cp /path/to/mypackage/ watcher_decision_engine:/opt/mypackage/
docker exec watcher_decision_engine pip install /opt/mypackage/
```

```bash title="Restart Decision Engine" theme={null}
docker restart watcher_decision_engine
```

```bash title="Verify strategy is registered" theme={null}
watcher strategy list
```

The custom strategy should appear in the list with its `NAME` value.

***

## Associate with a Goal

Custom strategies must be associated with an existing goal or a new goal created for
the purpose:

```bash title="List available goals" theme={null}
watcher goal list
```

If a new goal is needed, register it alongside the strategy entry point:

```ini title="setup.cfg" theme={null}
[entry_points]
watcher_goals =
    my_custom_goal = mypackage.goals:MyCustomGoal

watcher_strategies =
    my_custom_strategy = mypackage.strategies:MyCustomStrategy
```

***

## Test the Custom Strategy

```bash title="Run an audit with the custom strategy" theme={null}
watcher audit create \
  --goal server_consolidation \
  --strategy my_custom_strategy \
  --name custom-strategy-test
```

```bash title="Monitor audit progress" theme={null}
watcher audit show custom-strategy-test
```

```bash title="Review generated actions" theme={null}
watcher actionplan list \
  --audit <audit-uuid>
```

***

## Next Steps

<CardGroup>
  <Card title="Strategy Configuration" href="/services/optimization/admin-guide/strategy-config">
    Configure tuning parameters for built-in and custom strategies.
  </Card>

  <Card title="Data Sources" href="/services/optimization/admin-guide/data-sources">
    Connect the metric sources your custom strategy uses.
  </Card>

  <Card title="Architecture" href="/services/optimization/admin-guide/architecture">
    Review the Decision Engine plugin loading mechanism.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/admin-guide/troubleshooting">
    Diagnose custom strategy import and registration errors.
  </Card>
</CardGroup>


# Data Sources
Source: https://docs.xloud.tech/services/optimization/admin-guide/data-sources

Configure Xloud Optimization data sources — connect Prometheus, Telemetry, and the Compute API to enable metric-backed optimization strategies.

## Overview

The Decision Engine relies on data sources to build its cluster model and collect
performance metrics. The available data source determines which optimization strategies
can be used. The Compute API data source is always active. Prometheus and Telemetry
are optional and unlock additional strategies.

***

## Data Source Overview

| Data Source                | Provides                                              | Required For                                |
| -------------------------- | ----------------------------------------------------- | ------------------------------------------- |
| **Compute API**            | Host inventory, vCPU/memory usage, instance placement | All strategies — always active              |
| **Prometheus**             | Infrastructure metrics (node CPU, temperature)        | `outlet_temperature`, `saving_energy`       |
| **Telemetry (Ceilometer)** | Historical per-instance CPU, memory metrics           | `workload_stabilization`, `noisy_neighbor`  |
| **IPMI**                   | Server power state, inlet temperature                 | `outlet_temperature` (physical temperature) |

***

## Compute API Data Source

The Compute API data source is enabled by default and requires no additional configuration.
It provides:

* Real-time host inventory and hypervisor utilization
* Current instance placement (which instance runs on which host)
* vCPU and memory allocation per host

```bash title="Verify Compute API data source is working" theme={null}
docker exec -it watcher_decision_engine python3 -c "
from watcher.decision_engine.model.collector.nova import NovaClusterDataModelCollector
print('Compute collector loaded successfully')
"
```

***

## Prometheus Data Source

<Tabs>
  <Tab title="XDeploy" icon="browser">
    <Steps>
      <Step title="Enable Prometheus Monitoring" icon="toggle-left">
        Open **XDeploy** and navigate to **Configuration**. Select the **Monitoring** tab
        and toggle **Enable Prometheus** to **Yes**.
      </Step>

      <Step title="Configure Custom Data Source Settings (Optional)" icon="settings">
        For advanced Prometheus configuration, navigate to **Advanced Configuration**.
        In the **Service Tree** (left panel), select **watcher**. Click **New File** or
        select an existing `watcher.conf` from the **File Browser** (right panel).

        Add the following in the **Code Editor** (center panel):

        ```ini title="/etc/xavs/config/watcher/watcher.conf" theme={null}
        [watcher_cluster_data_model_collectors.prometheus]
        enabled = True
        host = 10.0.1.71
        port = 9291

        [prometheus_client]
        host = 10.0.1.71
        port = 9291
        ```
      </Step>

      <Step title="Save and Apply" icon="circle-check">
        Click **Save Current File**. Return to **Operations** and run **reconfigure** to
        apply the changes to the Decision Engine.

        <Check>Prometheus data source configured and applied via XDeploy.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly:

    ```ini title="/etc/xavs/watcher/watcher.conf" theme={null}
    [watcher_cluster_data_model_collectors.prometheus]
    enabled = True
    host = 10.0.1.71
    port = 9291

    [prometheus_client]
    host = 10.0.1.71
    port = 9291
    ```

    Restart the Decision Engine after changes:

    ```bash title="Restart Decision Engine" theme={null}
    docker restart watcher_decision_engine
    ```
  </Tab>
</Tabs>

<AccordionGroup>
  <Accordion title="Validation" icon="circle-check">
    ```bash title="Test Prometheus connectivity" theme={null}
    curl -s "http://10.0.1.71:9291/api/v1/query?query=up" \
      | jq '.status'
    ```

    Expected: `"success"`

    ```bash title="Verify temperature metrics are available" theme={null}
    curl -s "http://10.0.1.71:9291/api/v1/query?query=node_hwmon_temp_celsius" \
      | jq '.data.result | length'
    ```

    Expected: a non-zero count of temperature sensor results.
  </Accordion>
</AccordionGroup>

***

## Telemetry Data Source

Telemetry integration requires the Xloud Telemetry service (Ceilometer) to be deployed
and collecting per-instance metrics.

<AccordionGroup>
  <Accordion title="Configuration" icon="settings">
    ```ini title="/etc/xavs/watcher/watcher.conf" theme={null}
    [collector]
    collector_plugins = compute, ceilometer

    [ceilometer_client]
    endpoint_type = internalURL
    ```
  </Accordion>

  <Accordion title="Validation" icon="circle-check">
    ```bash title="Check available metrics from Telemetry" theme={null}
    openstack metric metric list --limit 20
    ```

    Verify that per-instance CPU metrics are present:

    ```bash title="Check cpu_util metrics" theme={null}
    openstack metric resource list \
      --type instance \
      | head -5
    ```

    For `workload_stabilization`, at least 2–4 hours of metric history is required
    before the strategy produces meaningful recommendations.
  </Accordion>
</AccordionGroup>

***

## Data Source and Strategy Matrix

| Goal                   | Compute API |    Prometheus   | Telemetry |
| ---------------------- | :---------: | :-------------: | :-------: |
| Server Consolidation   |   Required  |        -        |     -     |
| Energy Savings         |   Required  |     Optional    |     -     |
| Zone Rebalancing       |   Required  |        -        |     -     |
| Thermal Optimization   |   Required  | Required (temp) |     -     |
| Workload Stabilization |   Required  |        -        |  Required |
| Noisy Neighbor         |   Required  |        -        |  Required |

***

## Next Steps

<CardGroup>
  <Card title="Strategy Configuration" href="/services/optimization/admin-guide/strategy-config">
    Tune strategy parameters for each configured data source.
  </Card>

  <Card title="Custom Strategies" href="/services/optimization/admin-guide/custom-strategies">
    Build strategies using data from these configured sources.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/admin-guide/troubleshooting">
    Diagnose data source connectivity failures.
  </Card>

  <Card title="Architecture" href="/services/optimization/admin-guide/architecture">
    Review how data sources feed into the Decision Engine pipeline.
  </Card>
</CardGroup>


# Scheduling Audits
Source: https://docs.xloud.tech/services/optimization/admin-guide/scheduling

Automate Xloud Optimization audits with audit templates and schedules — configure recurring optimization runs and manage the audit template lifecycle.

## Overview

Scheduled audits run automatically on a recurring basis without manual initiation.
The scheduling workflow has two steps: create an audit template that defines the goal
and scope, then create a scheduled audit from that template with a cron expression or
interval. Scheduled audits are ideal for nightly consolidation runs, peak-hour workload
stabilization checks, and post-recovery zone rebalancing.

<Note>
  **Prerequisites**

  * Administrator privileges
  * At least one data source configured for the selected goal
</Note>

***

## Audit Templates

Audit templates are reusable configurations that serve as the basis for both manual
one-off audits and scheduled recurring audits.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Audit Templates" icon="compass">
        Navigate to
        **Optimization → Audit Templates**.
      </Step>

      <Step title="Create a template" icon="plus">
        Click **Create Audit Template** and fill in:

        | Field           | Description                 | Example                        |
        | --------------- | --------------------------- | ------------------------------ |
        | **Name**        | Template identifier         | `daily-consolidation`          |
        | **Goal**        | Optimization objective      | `server_consolidation`         |
        | **Scope**       | Target scope                | `CLUSTER`                      |
        | **Description** | Optional documentation note | `Nightly server consolidation` |
      </Step>

      <Step title="Save" icon="circle-check">
        Click **Create**. The template is now available for manual audits and scheduling.

        <Check>Audit template created and visible in the template list.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create an audit template" theme={null}
    watcher audittemplate create daily-consolidation \
      --goal server_consolidation \
      --description "Nightly server consolidation"
    ```

    ```bash title="List audit templates" theme={null}
    watcher audittemplate list
    ```

    ```bash title="Show template details" theme={null}
    watcher audittemplate show daily-consolidation
    ```
  </Tab>
</Tabs>

***

## Schedule a Recurring Audit

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Audit Templates** and click **Actions → Create Audit** on your
    template. In the audit creation form, enable **Auto Trigger** and set a cron
    expression in the scheduling field.

    | Cron Example  | Schedule                     |
    | ------------- | ---------------------------- |
    | `0 2 * * *`   | Daily at 02:00 AM            |
    | `0 */6 * * *` | Every 6 hours                |
    | `0 2 * * 0`   | Weekly on Sunday at 02:00 AM |

    <Tip>
      Schedule consolidation audits during off-peak hours (02:00–06:00 AM) to minimize
      the impact of live migrations on active workloads.
    </Tip>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create a scheduled audit" theme={null}
    watcher audit create \
      --audit-template daily-consolidation \
      --auto-trigger True \
      --goal server_consolidation
    ```

    ```bash title="List all audits including scheduled" theme={null}
    watcher audit list
    ```

    <Note>
      The `--auto-trigger True` flag instructs the Decision Engine to execute the
      resulting action plan automatically after audit completion, without manual approval.
      Use this only for non-production environments or after thoroughly validating the
      strategy on your topology.
    </Note>
  </Tab>
</Tabs>

***

## Auto-Trigger vs Manual Approval

| Mode                      | Behaviour                             | Recommended For                        |
| ------------------------- | ------------------------------------- | -------------------------------------- |
| Manual approval (default) | Plan requires explicit `execute` call | Production environments                |
| Auto-trigger              | Plan executes immediately after audit | Dev/test, or validated strategies only |

<Warning>
  Enable `auto_trigger` only after thoroughly testing the strategy behaviour on your
  cluster topology. Auto-trigger can initiate live migrations during business hours
  if the schedule is misconfigured.
</Warning>

***

## Manage Scheduled Audits

```bash title="List all audits" theme={null}
watcher audit list
```

```bash title="Delete a scheduled audit" theme={null}
watcher audit delete <audit-uuid>
```

```bash title="Delete an audit template" theme={null}
watcher audittemplate delete daily-consolidation
```

***

## Action Plan Expiry

Action plans become stale when the cluster state changes significantly after audit
completion. Configure expiry to prevent old plans from being executed.

<Tabs>
  <Tab title="XDeploy" icon="browser">
    <Steps>
      <Step title="Open Advanced Configuration" icon="settings">
        Open **XDeploy** and navigate to **Advanced Configuration**. In the **Service Tree**
        (left panel), select **watcher**.
      </Step>

      <Step title="Edit the Expiry Setting" icon="file-pen">
        Click **New File** or select an existing `watcher.conf` from the **File Browser**
        (right panel). Add the following in the **Code Editor** (center panel):

        ```ini title="/etc/xavs/config/watcher/watcher.conf" theme={null}
        [DEFAULT]
        action_plan_expiry = 24
        ```

        Plans older than `action_plan_expiry` hours are automatically invalidated. A new audit
        must be run to generate a fresh plan.
      </Step>

      <Step title="Save and Apply" icon="circle-check">
        Click **Save Current File**. Return to **Operations** and run **reconfigure** to
        apply the expiry setting.

        <Check>Action plan expiry configured and applied via XDeploy.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly:

    ```ini title="/etc/xavs/watcher/watcher.conf" theme={null}
    [DEFAULT]
    action_plan_expiry = 24
    ```

    Plans older than `action_plan_expiry` hours are automatically invalidated. A new audit
    must be run to generate a fresh plan.
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Action Plan Policies" href="/services/optimization/admin-guide/action-policies">
    Configure manual vs automatic action plan execution policies.
  </Card>

  <Card title="Strategy Configuration" href="/services/optimization/admin-guide/strategy-config">
    Tune strategy parameters for scheduled audits.
  </Card>

  <Card title="Audit History (User)" href="/services/optimization/user-guide/audit-history">
    Review scheduled audit results and trends over time.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/admin-guide/troubleshooting">
    Diagnose scheduled audit failures and data source issues.
  </Card>
</CardGroup>


# Security
Source: https://docs.xloud.tech/services/optimization/admin-guide/security

Secure the Xloud Optimization — configure the Applier service account, restrict RBAC execution policies, enable API audit logging, and protect the.

## Overview

The Optimization requires a dedicated service account with sufficient privileges to
call the Compute API for live migration. Hardening this account — and restricting who
can approve and execute action plans — prevents unauthorized workload movement and ensures
a complete audit trail for all optimization activity.

<Note>
  **Prerequisites**

  * Administrator privileges on both the Optimization and Xloud Identity
  * Xloud Compute deployed and operational
  * Optimization services running (API, Decision Engine, Applier)
</Note>

***

## Service Account Hardening

The Applier uses a dedicated service account to authenticate against the Compute API.
This account must have sufficient permissions to perform live migrations but should be
scoped to the minimum necessary privileges.

### Create a Dedicated Service Account

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create the user" icon="user-plus">
        Navigate to **Identity > Users** (admin view) and click **Create User**.

        | Field        | Value                                 |
        | ------------ | ------------------------------------- |
        | **Username** | `watcher-service`                     |
        | **Email**    | `watcher@internal.xloud.tech`         |
        | **Project**  | `service`                             |
        | **Role**     | `admin` (required for live migration) |

        Set a strong password and save it to your secrets manager.
      </Step>

      <Step title="Restrict to service project" icon="folder-lock">
        Ensure the `watcher-service` user has the `admin` role **only** in the `service`
        project — not in tenant projects. This limits blast radius if the account is compromised.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Create the Applier service account" theme={null}
    openstack user create watcher-service \
      --project service \
      --password "$(openssl rand -base64 32)"

    openstack role add admin \
      --user watcher-service \
      --project service
    ```

    ```bash title="Verify account permissions" theme={null}
    openstack role assignment list \
      --user watcher-service \
      --names
    ```
  </Tab>
</Tabs>

***

## Protect watcher.conf Credentials

The `watcher.conf` file contains the Applier's service account credentials. Restrict
access to this file on controller nodes and verify the authentication configuration.

<Tabs>
  <Tab title="XDeploy" icon="browser">
    <Steps>
      <Step title="Automatic Credential Management" icon="shield-check">
        Service account credentials and authentication settings are **automatically managed**
        by XDeploy during deployment. XDeploy generates the `[keystone_authtoken]` section
        with the correct service account, endpoint URLs, and permissions.

        No manual configuration is required for standard deployments.
      </Step>

      <Step title="Custom Authentication Settings (Optional)" icon="settings">
        For advanced troubleshooting or custom authentication configuration, navigate to
        **Advanced Configuration**. In the **Service Tree** (left panel), select **watcher**.
        Click **New File** or select an existing `watcher.conf` from the **File Browser**
        (right panel).

        Modify the `[keystone_authtoken]` section in the **Code Editor** (center panel):

        ```ini title="/etc/xavs/config/watcher/watcher.conf" theme={null}
        [keystone_authtoken]
        www_authenticate_uri = http://10.0.1.71:5000
        auth_url = http://10.0.1.71:5000/v3
        memcached_servers = 10.0.1.71:11211
        auth_type = password
        project_domain_name = Default
        user_domain_name = Default
        project_name = service
        username = watcher-service
        password = <service-account-password>
        ```
      </Step>

      <Step title="Save and Apply" icon="circle-check">
        Click **Save Current File**. Return to **Operations** and run **reconfigure** to
        apply the authentication changes.

        <Check>Service credentials configured and applied via XDeploy.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Restrict access to the configuration file on controller nodes:

    ```bash title="Restrict watcher.conf permissions" theme={null}
    chmod 640 /etc/xavs/watcher/watcher.conf
    chown root:watcher /etc/xavs/watcher/watcher.conf
    ```

    Verify the `[keystone_authtoken]` section uses the dedicated account:

    ```ini title="/etc/xavs/watcher/watcher.conf -- keystone_authtoken section" theme={null}
    [keystone_authtoken]
    www_authenticate_uri = http://10.0.1.71:5000
    auth_url = http://10.0.1.71:5000/v3
    memcached_servers = 10.0.1.71:11211
    auth_type = password
    project_domain_name = Default
    user_domain_name = Default
    project_name = service
    username = watcher-service
    password = <service-account-password>
    ```
  </Tab>
</Tabs>

<Warning>
  Never store credentials in plain text outside `/etc/xavs/`. Do not commit `watcher.conf`
  to version control. If the password must be rotated, update `watcher.conf` and restart
  the Applier container.
</Warning>

***

## RBAC Execution Policies

The Optimization enforces role-based access control on all API operations.
The default policy grants:

| Operation          | Required Role      |
| ------------------ | ------------------ |
| Create audit       | `admin`            |
| View audit results | `member`, `reader` |
| Start action plan  | `admin`            |
| Start action plan  | `admin`            |
| Cancel action plan | `admin`            |
| View action plans  | `member`, `reader` |

### Verify Default Policies

```bash title="List active RBAC policies" theme={null}
docker exec watcher_api \
  oslopolicy-list-redundant \
  --config-file /etc/watcher/watcher.conf
```

### Restrict Execution to Named Administrators

To restrict action plan execution to a dedicated ops team without granting full `admin`,
create a custom project-scoped role and override the policy:

```bash title="Create an optimizer-operator role" theme={null}
openstack role create optimizer-operator
openstack role add optimizer-operator \
  --user ops-user \
  --project service
```

```yaml title="/etc/xavs/watcher/policy.yaml — execution override" theme={null}
"action_plan:execute": "role:optimizer-operator or role:admin"
```

```bash title="Restart API after policy change" theme={null}
docker restart watcher_api
```

***

## API Audit Logging

Enable verbose API request logging to record who approved and executed each action plan:

```ini title="/etc/xavs/watcher/watcher.conf — audit logging" theme={null}
[DEFAULT]
use_journal = True

[oslo_middleware]
enable_proxy_headers_parsing = True
```

Logs are written to the container's journal and to the XAVS log volume:

```bash title="Follow real-time API logs" theme={null}
docker logs -f watcher_api
```

```bash title="Search for execution events" theme={null}
docker logs watcher_api 2>&1 \
  | grep "action_plan.*execute"
```

***

## TLS for API Communications

The Optimization API should be served behind the HAProxy endpoint which handles
TLS termination. Verify that all client traffic reaches the API via HTTPS:

```bash title="Verify API endpoint (should be HTTPS in production)" theme={null}
openstack endpoint list \
  --service optimization \
  -f table -c Interface -c URL
```

For internal controller communication, TLS on the RPC transport (RabbitMQ) is managed
by the platform-wide `[oslo_messaging_rabbit]` configuration.

***

## Rotation: Service Account Password

When rotating the Applier service account password:

<Steps>
  <Step title="Update the password in Xloud Identity" icon="key">
    ```bash title="Set a new password" theme={null}
    openstack user set watcher-service \
      --password "<new-strong-password>"
    ```
  </Step>

  <Step title="Update watcher.conf" icon="file-pen">
    Edit `/etc/xavs/watcher/watcher.conf` and update the `password` field under
    `[keystone_authtoken]`.
  </Step>

  <Step title="Restart all Optimizer containers" icon="rotate">
    ```bash title="Restart all Optimization services" theme={null}
    docker restart watcher_api watcher_decision_engine watcher_applier
    ```
  </Step>

  <Step title="Verify authentication" icon="circle-check">
    ```bash title="Confirm API is reachable after rotation" theme={null}
    watcher audit list
    ```

    <Check>Audit list returns without authentication errors — password rotation complete.</Check>
  </Step>
</Steps>

***

## Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Optimization → Audits**. Attempt to create an audit
    as a non-admin user — the action should be blocked with an authorization error.

    <Check>Non-admin users cannot create audits or execute action plans.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Test with a non-admin token" theme={null}
    openstack --os-auth-type password \
      --os-username member-user \
      --os-password member-pass \
      --os-project-name member-project \
      optimize audit create --goal server_consolidation
    ```

    Expected: `HTTP 403 Forbidden — Policy does not allow this request to be performed.`

    ```bash title="Verify watcher.conf permissions" theme={null}
    stat /etc/xavs/watcher/watcher.conf
    ```

    Expected: `Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: ( XXXX/watcher)`

    <Check>File permissions and RBAC policy both restrict access to authorized roles.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Action Policies" href="/services/optimization/admin-guide/action-policies">
    Configure manual approval vs automatic execution policies for action plans.
  </Card>

  <Card title="Compute Integration" href="/services/optimization/admin-guide/compute-integration">
    Verify the service account has compute permissions for live migration.
  </Card>

  <Card title="Architecture" href="/services/optimization/admin-guide/architecture">
    Review which containers use the service account credentials.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/admin-guide/troubleshooting">
    Diagnose authentication failures and permission errors.
  </Card>
</CardGroup>


# Strategy Configuration
Source: https://docs.xloud.tech/services/optimization/admin-guide/strategy-config

Configure Xloud Optimization strategy plugins — tune thresholds, periods, and metrics for each optimization strategy to match your cluster topology.

## Overview

Strategies are the algorithm plugins used by the Decision Engine to analyze the cluster
and generate migration recommendations. Each strategy is tied to one or more optimization
goals and accepts tuning parameters that control sensitivity thresholds, look-back periods,
and metric selection. This page documents all available strategies and their configuration
parameters.

***

## Available Strategies

| Strategy                 | Goal                   | Algorithm                                                       | Data Source Required     |
| ------------------------ | ---------------------- | --------------------------------------------------------------- | ------------------------ |
| `server_consolidation`   | Server Consolidation   | Bin-packing — fill hosts before activating new ones             | Compute API              |
| `outlet_temperature`     | Thermal Optimization   | Heatmap — migrate from hot racks based on inlet temperature     | Prometheus (temperature) |
| `workload_stabilization` | Workload Stabilization | Statistical variance analysis per instance                      | Telemetry time-series    |
| `saving_energy`          | Energy Savings         | Consolidate + flag empty hosts for power-off                    | Compute API              |
| `zone_migration`         | Zone Rebalancing       | Even distribution across availability zones                     | Compute API              |
| `noisy_neighbor`         | Noisy Neighbor         | CPU steal contention detection between co-located instances     | Telemetry                |
| `basic_consolidation`    | Consolidation (basic)  | Threshold-based: migrate from hosts below utilization threshold | Compute API only         |

***

## Configure Strategy Parameters

### Per-Audit Parameters

Override defaults at audit creation time without changing the service configuration:

```bash title="Create audit with custom parameters" theme={null}
openstack watcher audit create \
  --goal server_consolidation \
  --parameter threshold=0.15 \
  --parameter period=7200 \
  --name custom-threshold-audit
```

### Service-Wide Defaults

Set platform defaults that apply to all audits which do not override specific parameters.

<Tabs>
  <Tab title="XDeploy" icon="browser">
    <Steps>
      <Step title="Enable Dynamic Cluster Optimization" icon="toggle-left">
        Open **XDeploy** and navigate to **Configuration**. Select the **Advance Features** tab
        and toggle **Enable Dynamic Cluster Optimization** to **Yes**. Click **Save Configuration**.
      </Step>

      <Step title="Configure Strategy Defaults" icon="settings">
        Navigate to **Advanced Configuration**. In the **Service Tree** (left panel), select
        **watcher**. Click **New File** or select an existing `watcher.conf` from the
        **File Browser** (right panel).

        Add the following in the **Code Editor** (center panel):

        ```ini title="/etc/xavs/config/watcher/watcher.conf" theme={null}
        [watcher_strategies.server_consolidation]
        threshold = 0.2
        period = 3600

        [watcher_strategies.workload_stabilization]
        metric = cpu_util
        granularity = 300
        period = 7200

        [watcher_strategies.outlet_temperature]
        threshold = 35.0
        period = 3600
        ```
      </Step>

      <Step title="Save and Apply" icon="circle-check">
        Click **Save Current File**. Return to **Operations** and run **reconfigure** to
        apply the strategy defaults to the Decision Engine.

        <Check>Strategy defaults configured and applied via XDeploy.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Edit the configuration file directly:

    ```ini title="/etc/xavs/watcher/watcher.conf" theme={null}
    [watcher_strategies.server_consolidation]
    threshold = 0.2
    period = 3600

    [watcher_strategies.workload_stabilization]
    metric = cpu_util
    granularity = 300
    period = 7200

    [watcher_strategies.outlet_temperature]
    threshold = 35.0
    period = 3600
    ```

    Restart the Decision Engine after configuration changes:

    ```bash title="Restart Decision Engine" theme={null}
    docker restart watcher_decision_engine
    ```
  </Tab>
</Tabs>

***

## Parameter Reference

| Parameter     | Strategy                                      | Default    | Description                                          |
| ------------- | --------------------------------------------- | ---------- | ---------------------------------------------------- |
| `threshold`   | `server_consolidation`, `basic_consolidation` | `0.2`      | Fraction below which a host is underutilized         |
| `period`      | All                                           | `3600`     | Look-back window in seconds for metric aggregation   |
| `granularity` | Telemetry-backed                              | `300`      | Metric sample granularity in seconds                 |
| `metric`      | `workload_stabilization`                      | `cpu_util` | Metric used for stability scoring                    |
| `threshold`   | `outlet_temperature`                          | `35.0`     | Inlet temperature (°C) above which to evacuate racks |

***

## Verify Strategy Loading

```bash title="Verify strategy plugins are loaded" theme={null}
docker exec watcher_decision_engine python3 -c "
from stevedore import driver
drv = driver.DriverManager(
    namespace='watcher_strategies',
    name='server_consolidation',
    invoke_on_load=False
)
print('Strategy loaded:', drv.driver)
"
```

```bash title="List all registered strategies" theme={null}
openstack watcher strategy list
```

***

## Xloud Production Enhancements

<Info>**Xloud-Developed** — These enhancements ship with XAVS and are active on all Optimization strategies.</Info>

### Server Group Awareness

All 14 strategies automatically respect server group affinity and anti-affinity constraints during automated migrations. Hard policies (affinity, anti-affinity) block invalid migrations. Soft policies adjust destination scoring weights.

See [Server Groups](/services/compute/server-groups) for configuration.

### DRS Safety Checks

The workload balancing strategy includes production safety guardrails:

| Check                          | Description                                                                       |
| ------------------------------ | --------------------------------------------------------------------------------- |
| **Max iterations**             | Configurable limit on migrations per audit cycle                                  |
| **Capacity validation**        | Pre-migration check ensures destination has sufficient resources                  |
| **Concurrent migration limit** | Prevents overloading the cluster with simultaneous migrations                     |
| **Server group weight**        | Destination scoring adjusts based on soft-affinity/anti-affinity group membership |

### Automated Storage Tiering

The storage tier balance strategy automatically moves volumes between NVMe, SSD, and HDD pools based on IOPS activity and volume age. Configurable promotion and demotion thresholds. See [Storage Tiers](/services/storage/storage-tiers).

***

## Next Steps

<CardGroup>
  <Card title="Data Sources" href="/services/optimization/admin-guide/data-sources">
    Connect the data sources required by each strategy.
  </Card>

  <Card title="Custom Strategies" href="/services/optimization/admin-guide/custom-strategies">
    Implement and deploy custom optimization strategy plugins.
  </Card>

  <Card title="Scheduling" href="/services/optimization/admin-guide/scheduling">
    Schedule recurring audits using the configured strategies.
  </Card>

  <Card title="Architecture" href="/services/optimization/admin-guide/architecture">
    Review how strategies fit into the Decision Engine pipeline.
  </Card>
</CardGroup>


# Troubleshooting
Source: https://docs.xloud.tech/services/optimization/admin-guide/troubleshooting

Diagnose and resolve Xloud Optimization failures — audit errors, action plan execution failures, live migration issues, data source connectivity, and.

## Overview

This guide covers the most common failure modes in the Optimization: audits that
fail to complete, action plans that stall during execution, live migration errors from
the Compute API, and data source connectivity issues. Each section includes log locations,
diagnostic commands, and remediation steps.

***

## Quick Diagnostic Reference

<AccordionGroup>
  <Accordion title="Check all Optimization container health" icon="heart-pulse">
    ```bash title="Check container status" theme={null}
    docker ps --filter name=watcher \
      --format "table {{.Names}}\t{{.Status}}"
    ```

    All three containers must show `(healthy)`:

    * `watcher_api`
    * `watcher_decision_engine`
    * `watcher_applier`

    ```bash title="Check for recent errors in all containers" theme={null}
    for c in watcher_api watcher_decision_engine watcher_applier; do
      echo "=== $c ==="; docker logs --tail 20 $c 2>&1 | grep -E "ERROR|CRITICAL"
    done
    ```
  </Accordion>

  <Accordion title="Audit state glossary" icon="book">
    | State       | Meaning                                   |
    | ----------- | ----------------------------------------- |
    | `PENDING`   | Queued, waiting for Decision Engine       |
    | `ONGOING`   | Decision Engine is running the strategy   |
    | `SUCCEEDED` | Audit complete — action plan generated    |
    | `FAILED`    | Audit failed — check Decision Engine logs |
    | `CANCELLED` | Manually cancelled by an operator         |
  </Accordion>

  <Accordion title="Action plan state glossary" icon="book">
    | State         | Meaning                            |
    | ------------- | ---------------------------------- |
    | `RECOMMENDED` | Awaiting operator approval         |
    | `PENDING`     | Started, waiting for Applier       |
    | `ONGOING`     | Applier is executing actions       |
    | `SUCCEEDED`   | All actions completed successfully |
    | `FAILED`      | One or more actions failed         |
    | `CANCELLED`   | Expired or manually cancelled      |
  </Accordion>
</AccordionGroup>

***

## Audit Failures

### Audit Stuck in PENDING

The audit is queued but the Decision Engine has not picked it up.

```bash title="Check Decision Engine is running" theme={null}
docker ps --filter name=watcher_decision_engine
docker logs watcher_decision_engine --tail 50
```

**Common causes:**

* Decision Engine container is stopped or unhealthy
* RabbitMQ messaging connection is broken
* All Decision Engine workers are busy with another audit

```bash title="Restart Decision Engine" theme={null}
docker restart watcher_decision_engine
```

***

### Audit Fails with Strategy Error

```bash title="Show audit details" theme={null}
watcher audit show <audit-uuid> \
  -f value -c state -c scope
```

```bash title="Check Decision Engine logs for strategy errors" theme={null}
docker logs watcher_decision_engine 2>&1 \
  | grep -A 5 "ERROR.*strategy\|exception in.*strategy"
```

**Common causes and fixes:**

| Symptom            | Cause                                         | Fix                                                                                                           |
| ------------------ | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `NoDataFound`      | Data source not configured or unreachable     | Configure Prometheus or Telemetry — see [Data Sources](/services/optimization/admin-guide/data-sources)       |
| `InsufficientData` | Not enough metric history                     | Wait 2–4 hours for Telemetry to accumulate history                                                            |
| `StrategyNotFound` | Custom strategy not registered                | Reinstall the strategy package and restart Decision Engine                                                    |
| `NoCandidateFound` | All hosts are above the utilization threshold | Adjust strategy parameters — see [Strategy Configuration](/services/optimization/admin-guide/strategy-config) |

***

### Audit Succeeds but Generates Empty Action Plan

The audit completed successfully but no migrations were recommended.

This is expected behaviour when:

* All hosts are within the target utilization range (no consolidation needed)
* All instances are already on their optimal host
* The cluster is fully balanced for the selected goal

```bash title="Check current utilization" theme={null}
openstack hypervisor list --long \
  -f table -c Hostname -c "vCPUs Used" -c "Memory MB Used"
```

If the cluster appears underutilized but no actions were generated, lower the strategy
threshold parameters — see [Strategy Configuration](/services/optimization/admin-guide/strategy-config).

***

## Action Plan Execution Failures

### Action Plan Stuck in PENDING

The plan was approved but the Applier has not started execution.

```bash title="Check Applier logs" theme={null}
docker logs watcher_applier --tail 50
```

```bash title="Check action plan details" theme={null}
watcher actionplan show <plan-uuid>
```

**Common causes:**

* Applier container is stopped
* Plan has expired (exceeded `action_plan_expiry`)
* Taskflow workflow database is locked

```bash title="Restart Applier" theme={null}
docker restart watcher_applier
```

If the plan is expired, create a new audit to generate a fresh plan.

***

### Live Migration Action Fails

The Applier attempted a migration but Xloud Compute rejected it.

```bash title="Check action-level failure details" theme={null}
watcher action list \
  --action-plan <plan-uuid> \
  -f table -c uuid -c action_type -c state -c description
```

```bash title="Check Applier logs for migration errors" theme={null}
docker logs watcher_applier 2>&1 \
  | grep -A 10 "ERROR.*migrate\|MigrationError"
```

**Common migration errors:**

<AccordionGroup>
  <Accordion title="Instance has no shared storage" icon="hard-drive">
    **Error**: `LiveMigrationWithOldNovaNotSupported` or migration times out.

    The instance disk is backed by local ephemeral storage and cannot be live-migrated.

    **Fix**: Verify the instance is volume-backed before running optimization:

    ```bash title="Check instance storage" theme={null}
    openstack server show <instance-id> \
      -f value -c "os-extended-volumes:volumes_attached"
    ```

    Instances with no attached volumes must be excluded from optimization scope or
    migrated to volume-backed equivalents by the project owner.
  </Accordion>

  <Accordion title="CPU feature mismatch" icon="cpu">
    **Error**: `MigrationPreCheckError: Guest requires CPU feature not present on destination`.

    Compute hosts have different CPU feature sets and no common baseline is configured.

    **Fix**: Set a common CPU model in `nova.conf` on all compute hosts:

    ```ini title="nova.conf — CPU compatibility" theme={null}
    [libvirt]
    cpu_mode = custom
    cpu_model = Cascadelake-Server-noTSX
    ```

    See [Compute Integration](/services/optimization/admin-guide/compute-integration) for details.
  </Accordion>

  <Accordion title="Destination host has insufficient capacity" icon="server">
    **Error**: `NoValidHost: No valid host was found`.

    The destination host identified during the audit no longer has sufficient vCPU or
    memory available (cluster state changed between audit and execution).

    **Fix**: Run a new audit to generate a fresh plan reflecting current cluster state.
    Lower `action_plan_expiry` to prevent stale plans from executing:

    ```ini title="watcher.conf" theme={null}
    [DEFAULT]
    action_plan_expiry = 6
    ```
  </Accordion>

  <Accordion title="Source or destination host is disabled" icon="circle-xmark">
    **Error**: `HTTPBadRequest: Cannot live migrate to disabled host`.

    A host was disabled between audit completion and plan execution.

    **Fix**: Re-enable the host or run a new audit with current host availability.

    ```bash title="Re-enable a host" theme={null}
    openstack compute service set --enable <hostname> nova-compute
    ```
  </Accordion>
</AccordionGroup>

***

## Data Source Issues

### Prometheus Not Reachable

Strategies that require Prometheus (`outlet_temperature`, `saving_energy`) fail with
`NoDataFound`.

```bash title="Test Prometheus connectivity from Decision Engine" theme={null}
docker exec watcher_decision_engine \
  curl -s "http://10.0.1.71:9291/api/v1/query?query=up" | python3 -m json.tool
```

Expected: `"status": "success"` with results.

```bash title="Check Prometheus section in watcher.conf" theme={null}
grep -A 5 "\[prometheus_client\]" /etc/xavs/watcher/watcher.conf
```

***

### Telemetry Metrics Missing

Strategies that require Telemetry (`workload_stabilization`, `noisy_neighbor`) fail
with `InsufficientData` or generate no recommendations.

```bash title="Check Telemetry collector is configured" theme={null}
grep -A 5 "\[ceilometer_client\]" /etc/xavs/watcher/watcher.conf
```

```bash title="Verify metrics exist in Telemetry" theme={null}
openstack metric resource list --type instance | head -5
```

If no instance resources are listed, the Telemetry service is not collecting metrics.
Verify Xloud Telemetry is deployed and the `ceilometer` compute agent is enabled.

***

## Authentication Failures

### 401 Unauthorized in Applier Logs

The Applier service account credentials are invalid or expired.

```bash title="Check Applier authentication errors" theme={null}
docker logs watcher_applier 2>&1 | grep "401\|Unauthorized\|keystoneauth"
```

```bash title="Test the service account token" theme={null}
openstack --os-username watcher-service \
  --os-password "<service-account-password>" \
  --os-project-name service \
  token issue
```

If token issue fails, the service account credentials in `watcher.conf` are incorrect.
Update the `[keystone_authtoken]` section and restart all Optimizer containers:

```bash title="Restart after credential update" theme={null}
docker restart watcher_api watcher_decision_engine watcher_applier
```

***

## Log Locations

| Component       | Log Location                                   |
| --------------- | ---------------------------------------------- |
| API             | `docker logs watcher_api`                      |
| Decision Engine | `docker logs watcher_decision_engine`          |
| Applier         | `docker logs watcher_applier`                  |
| Full log files  | `/var/log/kolla/watcher/` (on controller host) |

```bash title="Search all Optimizer logs for errors" theme={null}
grep -r "ERROR\|CRITICAL" /var/log/kolla/watcher/ \
  | tail -50
```

***

## Next Steps

<CardGroup>
  <Card title="Strategy Configuration" href="/services/optimization/admin-guide/strategy-config">
    Adjust thresholds and parameters when audits generate no recommendations.
  </Card>

  <Card title="Compute Integration" href="/services/optimization/admin-guide/compute-integration">
    Verify shared storage and CPU compatibility for live migration.
  </Card>

  <Card title="Data Sources" href="/services/optimization/admin-guide/data-sources">
    Diagnose Prometheus and Telemetry connectivity failures.
  </Card>

  <Card title="Security" href="/services/optimization/admin-guide/security">
    Resolve service account authentication failures.
  </Card>
</CardGroup>


# Optimization CLI Reference
Source: https://docs.xloud.tech/services/optimization/cli-reference

Complete watcher CLI commands for running audits, reviewing action plans, and executing optimization actions on Xloud Compute clusters.

## Overview

The `watcher` CLI manages resource optimization — goals, audits, audit templates, action plans, actions, and strategies for workload consolidation and efficiency.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Watcher client installed: `pip install python-watcherclient`
</Note>

***

## Goals

<CodeGroup>
  ```bash title="List optimization goals" theme={null}
  watcher goal list
  ```

  ```bash title="Show goal details" theme={null}
  watcher goal show server_consolidation
  ```
</CodeGroup>

***

## Audits

<CodeGroup>
  ```bash title="List audits" theme={null}
  watcher audit list
  ```

  ```bash title="Create one-shot audit" theme={null}
  watcher audit create \
    --goal server_consolidation \
    --strategy basic
  ```

  ```bash title="Create audit (auto-selects strategy)" theme={null}
  watcher audit create \
    --goal vm_workload_consolidation
  ```

  ```bash title="Create continuous audit" theme={null}
  watcher audit create \
    --goal server_consolidation \
    --audit-type CONTINUOUS \
    --interval 3600
  ```

  ```bash title="Show audit" theme={null}
  watcher audit show <audit-uuid>
  ```

  ```bash title="Delete audit" theme={null}
  watcher audit delete <audit-uuid>
  ```
</CodeGroup>

***

## Audit Templates

<CodeGroup>
  ```bash title="List audit templates" theme={null}
  watcher audittemplate list
  ```

  ```bash title="Create audit template" theme={null}
  watcher audittemplate create \
    --goal server_consolidation \
    --strategy basic \
    weekly-consolidation
  ```

  ```bash title="Show audit template" theme={null}
  watcher audittemplate show weekly-consolidation
  ```

  ```bash title="Delete audit template" theme={null}
  watcher audittemplate delete weekly-consolidation
  ```
</CodeGroup>

***

## Action Plans

<CodeGroup>
  ```bash title="List action plans" theme={null}
  watcher actionplan list
  ```

  ```bash title="Show action plan" theme={null}
  watcher actionplan show <plan-uuid>
  ```

  ```bash title="Start action plan" theme={null}
  watcher actionplan start <plan-uuid>
  ```

  ```bash title="Delete action plan" theme={null}
  watcher actionplan delete <plan-uuid>
  ```
</CodeGroup>

***

## Actions

<CodeGroup>
  ```bash title="List actions in a plan" theme={null}
  watcher action list --action-plan <plan-uuid>
  ```

  ```bash title="Show action details" theme={null}
  watcher action show <action-uuid>
  ```
</CodeGroup>

***

## Strategies

<CodeGroup>
  ```bash title="List strategies" theme={null}
  watcher strategy list
  ```

  ```bash title="Show strategy details" theme={null}
  watcher strategy show basic
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Run Audit Guide" href="/services/optimization/user-guide/run-audit">
    Step-by-step guide to running your first optimization audit
  </Card>

  <Card title="Action Plans Guide" href="/services/optimization/user-guide/action-plans">
    Review and execute optimization action plans
  </Card>
</CardGroup>


# Cluster Optimization
Source: https://docs.xloud.tech/services/optimization/index

Automated resource optimization for Xloud — workload placement, consolidation, and dynamic scaling.

Maximize infrastructure efficiency with automated workload analysis and intelligent
placement. Xloud Cluster Optimization continuously audits your compute cluster, generates
actionable improvement plans, and executes workload rebalancing — reducing resource
waste, lowering energy consumption, and improving workload performance.

<Card title="XAVS — Advanced Virtualization Solutions" icon="external-link" href="https://xloud.tech/xavs">
  Product details on xloud.tech
</Card>

***

<p>Cluster Optimization</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/optimization/user-guide">
    Run optimization audits, select goals and strategies, review action plans, and
    execute workload rebalancing from the Dashboard or CLI.
  </Card>

  <Card title="Admin Guide" icon="shield-check" href="/services/optimization/admin-guide">
    Configure optimization strategies, data sources, scheduled audits, and integration
    with compute metrics for platform-wide resource management.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/optimization/cli-reference">
    Complete command reference for managing goals, audits, action plans, and strategies
    using the openstack CLI.
  </Card>

  <Card title="Compute Service" icon="server" href="/services/compute">
    Xloud Compute provides the workload and host metrics that drive optimization
    decisions and executes the resulting migration actions.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Intelligent Workload Consolidation" icon="layers" href="/services/optimization/user-guide/run-audit">
    Identify underutilized hosts and consolidate workloads to reduce the active host
    footprint — enabling idle hosts to enter low-power states or be decommissioned.
  </Card>

  <Card title="Thermal Optimization" icon="thermometer" href="/services/optimization/user-guide/optimization-goals">
    Monitor inlet temperatures and redistribute workloads away from overheating compute
    racks, protecting hardware and extending its operational lifespan.
  </Card>

  <Card title="Workload Stability" icon="activity" href="/services/optimization/user-guide/optimization-goals">
    Detect noisy-neighbor interference and migrate workloads to hosts where they
    can achieve stable, consistent performance metrics.
  </Card>

  <Card title="Energy Efficiency" icon="zap" href="/services/optimization/user-guide/optimization-goals">
    Optimize placement to minimize the number of active compute hosts, reducing
    power consumption during off-peak periods.
  </Card>

  <Card title="Scheduled Audits" icon="calendar" href="/services/optimization/admin-guide/scheduling">
    Run optimization analyses on a schedule — daily, weekly, or triggered by capacity
    events — without manual initiation.
  </Card>

  <Card title="Review Before Execute" icon="eye" href="/services/optimization/user-guide/action-plans">
    Every audit produces a human-readable action plan. Review and approve actions
    before execution, or configure fully automated application.
  </Card>
</CardGroup>

***

<p>How It Works</p>

```mermaid theme={null}
graph TD
    A[Cluster Metrics<br/>vCPU, RAM, Temp] -->|Collected| B[Decision Engine]
    B -->|Apply strategy| C[Optimization Planner]
    C -->|Generate| D[Action Plan]
    D -->|Reviewed by operator| E{Start?}
    E -->|Yes| F[Action Applier]
    E -->|No| G[Discard / Modify]
    F -->|Start migrations| H[Xloud Compute API]
    H -->|Live migrate workloads| I[Optimized Cluster]
```

***

<p>Optimization Goals</p>

| Goal                      | Description                                           | Common Strategy          |
| ------------------------- | ----------------------------------------------------- | ------------------------ |
| Server Consolidation      | Reduce the number of active compute hosts             | `server_consolidation`   |
| Thermal Optimization      | Reduce inlet temperatures by redistributing heat load | `outlet_temperature`     |
| Workload Stabilization    | Improve instance performance consistency              | `workload_stabilization` |
| Energy Savings            | Minimize active host count to reduce power draw       | `saving_energy`          |
| Zone Rebalancing          | Distribute workloads evenly across availability zones | `zone_migration`         |
| Noisy Neighbor Mitigation | Isolate CPU/memory contention between instances       | `noisy_neighbor`         |

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Workload placement and live migration executed by the Optimization
  </Card>

  <Card title="Instance HA" icon="heart-pulse" href="/services/instance-ha/index">
    Recovery events that trigger rebalancing workflows
  </Card>

  <Card title="Xloud Identity" icon="fingerprint" href="/services/identity/index">
    RBAC policies governing who can approve and execute optimization actions
  </Card>
</CardGroup>


# Action Plans
Source: https://docs.xloud.tech/services/optimization/user-guide/action-plans

Review Xloud Optimization action plans — understand recommended migrations, inspect individual actions, and approve or reject plans before execution.

## Overview

Every successful audit generates an action plan — an ordered list of workload migrations
that achieve the selected optimization goal. Action plans must be reviewed and explicitly
approved before execution begins. This approval step ensures operators have full visibility
into what will be moved, and when, before any workload is disrupted.

<Note>
  **Prerequisites**

  * A completed audit with state `SUCCEEDED`
  * Project access with the `member` role or above
</Note>

***

## Action Plan Structure

An action plan contains:

| Component      | Description                                                                                 |
| -------------- | ------------------------------------------------------------------------------------------- |
| **Plan UUID**  | Unique identifier for the plan                                                              |
| **State**      | Current plan state: `RECOMMENDED`, `PENDING`, `ONGOING`, `SUCCEEDED`, `FAILED`, `CANCELLED` |
| **Actions**    | Ordered list of individual workload operations                                              |
| **Efficacy**   | Estimated improvement percentage for the selected goal                                      |
| **Audit UUID** | Reference to the audit that generated this plan                                             |

Each action within the plan describes:

* The operation type (e.g., `migrate`)
* Source instance and source host
* Destination host (pre-selected by the strategy)
* Execution priority (lower numbers run first)

***

## View Action Plans

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Action Plans" icon="compass">
        Navigate to
        **Optimization → Action Plans**.
      </Step>

      <Step title="Select a plan" icon="clipboard-list">
        Click the action plan name or UUID to open the detail view.
      </Step>

      <Step title="Review individual actions" icon="eye">
        The detail view shows the full list of recommended actions:

        | Column               | Description                               |
        | -------------------- | ----------------------------------------- |
        | **Action**           | Operation type: `migrate`                 |
        | **Input Parameters** | Source instance ID and destination host   |
        | **State**            | `RECOMMENDED` — awaiting approval         |
        | **Priority**         | Execution order — lower numbers run first |
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all action plans" theme={null}
    watcher actionplan list
    ```

    ```bash title="Show action plan details" theme={null}
    watcher actionplan show <action-plan-uuid>
    ```

    ```bash title="List individual actions in a plan" theme={null}
    watcher action list \
      --action-plan <action-plan-uuid>
    ```

    Each action shows: `action_type`, `input_parameters` (source + destination),
    `state`, and `parents` (dependencies between actions).
  </Tab>
</Tabs>

***

## Start an Action Plan

<Warning>
  Approving an action plan initiates live migrations immediately on execution.
  Verify that target hosts have sufficient capacity before approving. Review the
  full action list to ensure no mission-critical instances are scheduled for migration
  during active business hours.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    On the action plan detail page, click **Start**. The plan state changes from
    `RECOMMENDED` to `PENDING`, indicating it is ready for execution.

    <Tip>
      Review the **Efficacy** metric on the plan detail page. A high efficacy score
      (above 70%) indicates a meaningful optimization improvement.
    </Tip>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Action plan approval transitions the state to `PENDING`. In the CLI workflow,
    you execute the plan directly — approval and execution are combined:

    ```bash title="Execute (which implicitly approves) the plan" theme={null}
    watcher actionplan start <action-plan-uuid>
    ```

    To review before executing:

    ```bash title="Review actions" theme={null}
    watcher action list \
      --action-plan <action-plan-uuid> \
      -f table -c uuid -c action_type -c input_parameters -c state
    ```
  </Tab>
</Tabs>

***

## Reject an Action Plan

If the action plan is not appropriate — e.g., it targets instances that should
not be migrated during the current window — reject it and run a new audit later.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    On the action plan detail page, click **Delete**. The plan state changes to
    `CANCELLED` and no migrations are executed.
  </Tab>

  <Tab title="CLI" icon="terminal">
    Action plan cancellation is not directly supported via the CLI. The plan expires
    automatically after the `action_plan_expiry` period configured by your administrator
    (default: 24 hours). Run a new audit to generate a fresh plan.
  </Tab>
</Tabs>

***

## Action Plan States

| State         | Meaning                            |
| ------------- | ---------------------------------- |
| `RECOMMENDED` | Plan generated, awaiting approval  |
| `PENDING`     | Started, queued for execution      |
| `ONGOING`     | Execution in progress              |
| `SUCCEEDED`   | All actions completed successfully |
| `FAILED`      | One or more actions failed         |
| `CANCELLED`   | Plan was rejected or expired       |

***

## Next Steps

<CardGroup>
  <Card title="Execute Actions" href="/services/optimization/user-guide/execute-actions">
    Run an approved action plan and monitor migration progress.
  </Card>

  <Card title="Audit History" href="/services/optimization/user-guide/audit-history">
    Review past audits and their associated action plans.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/user-guide/troubleshooting">
    Resolve failed actions and cancelled plans.
  </Card>

  <Card title="Run an Audit" href="/services/optimization/user-guide/run-audit">
    Generate a new action plan by running a fresh audit.
  </Card>
</CardGroup>


# Audit History
Source: https://docs.xloud.tech/services/optimization/user-guide/audit-history

Review Xloud Optimization audit history — track optimization trends, view past action plan outcomes, and validate ongoing cluster efficiency improvements.

## Overview

Every optimization audit is persisted with its full timeline — goal, strategy, start
time, completion time, and the associated action plan outcome. Reviewing audit history
helps you track optimization trends, identify recurring imbalance patterns, and validate
that executed plans have produced lasting improvements.

<Note>
  **Prerequisites**

  * At least one completed audit in your environment
  * Project access with the `member` role or above
</Note>

***

## View Audit History

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Audits" icon="compass">
        Navigate to
        **Optimization → Audits**.
      </Step>

      <Step title="Review audit list" icon="eye">
        The audit list shows all historical audits sorted by creation time (newest first).

        | Column         | Description                           |
        | -------------- | ------------------------------------- |
        | **Name**       | Audit display name                    |
        | **Goal**       | Optimization objective                |
        | **State**      | `SUCCEEDED`, `FAILED`, or `CANCELLED` |
        | **Created At** | Audit creation timestamp              |
        | **Interval**   | Scope of the audit (CLUSTER or ZONE)  |
      </Step>

      <Step title="Open an audit" icon="zoom-in">
        Click any audit to view its full detail, including:

        * The strategy that was applied
        * Any parameters that overrode the defaults
        * A link to the associated action plan

        Click the **Action Plan UUID** to view the migration plan and its execution outcome.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all audits" theme={null}
    watcher audit list
    ```

    ```bash title="Filter by goal" theme={null}
    watcher audit list \
      --goal server_consolidation
    ```

    ```bash title="Show audit summary" theme={null}
    watcher audit show <audit-uuid> -f json
    ```

    ```bash title="List action plan for an audit" theme={null}
    watcher actionplan list \
      --audit <audit-uuid>
    ```
  </Tab>
</Tabs>

***

## Audit State Reference

| State       | Meaning                                                       |
| ----------- | ------------------------------------------------------------- |
| `PENDING`   | Queued, not yet started                                       |
| `ONGOING`   | Decision Engine is running the analysis                       |
| `SUCCEEDED` | Audit complete and action plan generated                      |
| `FAILED`    | Audit could not complete due to a data source or engine error |
| `CANCELLED` | Manually cancelled before completion                          |

***

## Analyze Optimization Trends

Use audit history to identify patterns that indicate structural cluster issues:

<AccordionGroup>
  <Accordion title="Recurring empty action plans" icon="clipboard">
    If `server_consolidation` audits consistently produce empty action plans (no
    recommended migrations), the cluster is already well-balanced — or the utilization
    threshold is set too high and no hosts fall below it.

    Check current host utilization:

    ```bash title="Check host utilization" theme={null}
    openstack hypervisor list --long
    ```

    If hosts are genuinely well-distributed, no action is needed. If hosts appear
    underutilized but no plan is generated, ask your administrator to lower the
    `threshold` parameter in the strategy configuration.
  </Accordion>

  <Accordion title="Frequent noisy-neighbor audits" icon="volume-x">
    If `noisy_neighbor` audits consistently find high-contention pairs, the cluster
    may have insufficient capacity to separate contending workloads. Consider adding
    compute capacity or reviewing instance sizing.
  </Accordion>

  <Accordion title="Plans generated but never executed" icon="pause-circle">
    Review the action plan list for plans in `PENDING` (approved but not executed)
    or `RECOMMENDED` (not yet approved) state. Stale plans are outdated within 24 hours
    by default — run a new audit to generate a current plan.
  </Accordion>
</AccordionGroup>

***

## Export Audit History

```bash title="Export audit list to JSON" theme={null}
watcher audit list -f json > audit-history.json
```

```bash title="Export with full details for each audit" theme={null}
for audit_id in $(watcher audit list -f value -c uuid); do
  watcher audit show $audit_id -f json
done > audit-details.json
```

***

## Next Steps

<CardGroup>
  <Card title="Run an Audit" href="/services/optimization/user-guide/run-audit">
    Start a new optimization audit to continue improving cluster efficiency.
  </Card>

  <Card title="Action Plans" href="/services/optimization/user-guide/action-plans">
    Review and approve pending action plans from past audits.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/user-guide/troubleshooting">
    Diagnose failed audits and strategies reporting insufficient data.
  </Card>

  <Card title="Admin Guide — Scheduling" href="/services/optimization/admin-guide/scheduling">
    Automate recurring audits on a schedule for continuous optimization.
  </Card>
</CardGroup>


# Execute Actions
Source: https://docs.xloud.tech/services/optimization/user-guide/execute-actions

Execute approved Xloud Optimization action plans — trigger live migrations, monitor execution progress, and verify optimization outcomes.

## Overview

After reviewing and approving an action plan, you execute it to trigger the live
migrations. Actions execute sequentially in priority order — each migration must complete
before the next begins. Live migrations are non-disruptive for running instances: the
workload continues serving requests while its memory state is transferred to the
destination host. Plan execution can be cancelled at any time — in-progress migrations
complete, pending actions are halted.

<Note>
  **Prerequisites**

  * An action plan in `RECOMMENDED` or `PENDING` state
  * All compute hosts with sufficient capacity to accept migrating instances
  * Shared storage (Xloud Distributed Storage) for all instances in the plan
</Note>

***

## Execute an Action Plan

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the action plan" icon="compass">
        Navigate to **Optimization → Action Plans** and click the plan name.
      </Step>

      <Step title="Start the action plan" icon="circle-check">
        If the plan state is `RECOMMENDED`, click **Start** first.
      </Step>

      <Step title="Execute" icon="play">
        Click **Start**. The plan state transitions to `ONGOING`.

        Action statuses update in real time:

        | Status      | Meaning                                    |
        | ----------- | ------------------------------------------ |
        | `PENDING`   | Action queued                              |
        | `ONGOING`   | Migration in progress                      |
        | `SUCCEEDED` | Action completed                           |
        | `FAILED`    | Action failed — execution halts            |
        | `CANCELLED` | Execution cancelled before this action ran |
      </Step>

      <Step title="Monitor completion" icon="activity">
        Watch the action list update as each migration completes. The plan state changes
        to `SUCCEEDED` when all actions finish.

        <Check>Plan state is `SUCCEEDED` and all actions show `SUCCEEDED`.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Execute the plan" icon="play">
        ```bash title="Start action plan" theme={null}
        watcher actionplan start <action-plan-uuid>
        ```
      </Step>

      <Step title="Monitor plan progress" icon="clock">
        ```bash title="Check plan state" theme={null}
        watcher actionplan show <action-plan-uuid> \
          -f value -c state
        ```
      </Step>

      <Step title="Watch individual actions" icon="eye">
        ```bash title="List action states" theme={null}
        watcher action list \
          --action-plan <action-plan-uuid> \
          -f table -c uuid -c action_type -c state
        ```
      </Step>

      <Step title="Verify completion" icon="circle-check">
        ```bash title="Confirm all actions succeeded" theme={null}
        watcher actionplan show <action-plan-uuid> \
          -f value -c state
        ```

        Expected: `SUCCEEDED`

        <Check>Plan state is `SUCCEEDED` and migrations are complete.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Verify Post-Execution Placement

After execution, verify that instances have been redistributed as expected.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Compute > Hypervisors** (admin view) to view the updated host utilization.
    Compare the host instance counts and resource utilization before and after execution.

    <Check>Previously underutilized hosts now show reduced instance counts.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Check hypervisor utilization" theme={null}
    openstack hypervisor list --long
    ```

    ```bash title="Verify instance placement" theme={null}
    openstack server list --all \
      -f table -c ID -c Name -c Status -c "OS-EXT-SRV-ATTR:host"
    ```

    Compare `running_vms` and `memory_mb_used` against pre-execution values.

    <Check>Host utilization is more balanced and the active host footprint is reduced.</Check>
  </Tab>
</Tabs>

***

## Cancel Execution Mid-Plan

If a plan needs to be stopped after execution has started:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    On the action plan detail page, click **Cancel**. The in-progress migration
    completes; all pending actions are marked `CANCELLED`.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Cancel a running action plan" theme={null}
    watcher actionplan cancel <action-plan-uuid>
    ```

    ```bash title="Verify cancellation" theme={null}
    watcher actionplan show <action-plan-uuid> \
      -f value -c state
    ```

    Expected: `CANCELLED`
  </Tab>
</Tabs>

<Tip>
  After cancellation, run a new audit to generate an updated action plan that reflects
  the current (partially optimized) cluster state.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Audit History" href="/services/optimization/user-guide/audit-history">
    Review the completed audit and its execution results over time.
  </Card>

  <Card title="Action Plans" href="/services/optimization/user-guide/action-plans">
    Review the action plan detail and individual action outcomes.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/user-guide/troubleshooting">
    Resolve failed actions and migration errors.
  </Card>

  <Card title="Run an Audit" href="/services/optimization/user-guide/run-audit">
    Run another audit to continue optimization after the current plan completes.
  </Card>
</CardGroup>


# Optimization Goals
Source: https://docs.xloud.tech/services/optimization/user-guide/optimization-goals

Understand Xloud Optimization goals — server consolidation, thermal optimization, workload stabilization, energy savings, and noisy neighbor mitigation.

## Overview

An optimization goal defines the objective of an audit. When you create an audit, you
select a goal and the Optimization automatically chooses the most appropriate
strategy algorithm to achieve it. Goals translate high-level operational intent — "reduce
my host footprint" or "fix noisy-neighbor complaints" — into concrete migration plans.

<Note>
  **Prerequisites**

  * Optimization enabled on your platform
  * Project access with the `member` role or above
  * Metrics collected for the look-back period (typically 1 hour minimum)
</Note>

***

## Available Goals

<CardGroup>
  <Card title="Server Consolidation" icon="layers">
    Migrate workloads from underutilized hosts to consolidate the active footprint.
    Idle hosts can enter low-power states after workloads are moved off.
  </Card>

  <Card title="Thermal Optimization" icon="thermometer">
    Redistribute workloads away from compute racks with high inlet temperatures,
    protecting hardware and extending operational lifespan.
  </Card>

  <Card title="Workload Stabilization" icon="activity">
    Identify instances with erratic CPU or memory consumption and migrate them to
    hosts where they achieve consistent, stable performance.
  </Card>

  <Card title="Energy Savings" icon="zap">
    Consolidate workloads to the minimum number of hosts during off-peak periods,
    enabling idle nodes to reduce power draw.
  </Card>

  <Card title="Zone Rebalancing" icon="network">
    Evenly redistribute workloads across availability zones after recovery events
    or uneven initial placement.
  </Card>

  <Card title="Noisy Neighbor Mitigation" icon="volume-x">
    Detect instances causing CPU or memory contention and isolate them from
    co-located workloads to restore performance for affected instances.
  </Card>
</CardGroup>

***

## Goal Reference

| Goal                   | CLI Name                 | Strategy                       | Data Required            | Trigger Scenario                          |
| ---------------------- | ------------------------ | ------------------------------ | ------------------------ | ----------------------------------------- |
| Server Consolidation   | `server_consolidation`   | Bin-packing                    | Compute API              | High host count, low average utilization  |
| Thermal Optimization   | `thermal_optimization`   | Outlet temperature heatmap     | Prometheus (temperature) | High temperature alerts from monitoring   |
| Workload Stabilization | `workload_stabilization` | Statistical variance analysis  | Telemetry time-series    | Sporadic guest performance complaints     |
| Energy Savings         | `saving_energy`          | Consolidate + mark idle hosts  | Compute API              | Off-peak scheduled run                    |
| Zone Rebalancing       | `zone_migration`         | Even zone distribution         | Compute API              | Post-recovery or uneven initial placement |
| Noisy Neighbor         | `noisy_neighbor`         | CPU steal contention detection | Telemetry per-instance   | Guest CPU steal complaints                |

***

## Selecting a Goal

Choose the goal that matches the problem you are solving:

<AccordionGroup>
  <Accordion title="My cluster has too many active hosts" icon="layers">
    Use **Server Consolidation** (`server_consolidation`). The strategy identifies hosts
    running below the utilization threshold (default: 20%) and generates migrations that
    pack those workloads onto fewer hosts.
  </Accordion>

  <Accordion title="Some instances are performing erratically" icon="activity">
    Use **Workload Stabilization** (`workload_stabilization`). The strategy analyzes
    per-instance CPU and memory time-series data to identify high-variance instances
    and moves them to less-contended hosts.

    Requires Telemetry metrics to be collected for the look-back period (minimum 2 hours
    of data recommended).
  </Accordion>

  <Accordion title="A compute rack is running hot" icon="thermometer">
    Use **Thermal Optimization** (`thermal_optimization`). The strategy reads inlet
    temperature data from Prometheus and generates a plan to move workloads away from
    the hottest racks.

    Requires Prometheus with temperature sensor metrics configured as a data source.
  </Accordion>

  <Accordion title="After a failover, zones are imbalanced" icon="network">
    Use **Zone Rebalancing** (`zone_migration`). The strategy redistributes instances
    evenly across availability zones without requiring external metric data.
  </Accordion>

  <Accordion title="Guests are complaining about noisy neighbors" icon="volume-x">
    Use **Noisy Neighbor** (`noisy_neighbor`). The strategy detects instance pairs with
    high CPU contention co-located on the same host and separates them.

    Requires per-instance CPU steal metrics from Telemetry.
  </Accordion>
</AccordionGroup>

***

## Goal Availability

Not all goals may be available on your deployment. Available goals depend on the data
sources and strategies enabled by your administrator.

```bash title="List available goals" theme={null}
watcher goal list
```

If a goal is not listed, contact your administrator to enable the required data source. Your administrator can configure this through [XDeploy](/deployment).

***

## Next Steps

<CardGroup>
  <Card title="Run an Audit" href="/services/optimization/user-guide/run-audit">
    Create and run an audit using one of the available optimization goals.
  </Card>

  <Card title="Action Plans" href="/services/optimization/user-guide/action-plans">
    Review the migration plan generated by a completed audit.
  </Card>

  <Card title="Audit History" href="/services/optimization/user-guide/audit-history">
    Review past audits and track optimization trends over time.
  </Card>

  <Card title="Optimization Admin Guide" href="/services/optimization/admin-guide">
    Administrator reference for configuring strategies and data sources.
  </Card>
</CardGroup>


# Run an Optimization Audit
Source: https://docs.xloud.tech/services/optimization/user-guide/run-audit

Create and execute Optimization audits in Xloud — select a goal, configure scope, submit the audit, and monitor progress to action plan generation.

## Overview

An audit is a point-in-time analysis of the cluster. When you create an audit, the
Optimization collects current cluster metrics, applies the selected strategy
algorithm, and generates a prioritized action plan. Audits are fast — typically 30
seconds to 2 minutes depending on cluster size and data source responsiveness.

<Note>
  **Prerequisites**

  * Optimization enabled on your platform
  * At least 1 hour of metric data collected (for telemetry-backed strategies)
  * Project access with the `member` role or above
</Note>

***

## Create and Run an Audit

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Audits" icon="compass">
        Navigate to
        **Optimization → Audits**.
      </Step>

      <Step title="Create a new audit" icon="plus">
        Click **Create Audit** and fill in the parameters:

        | Field            | Description                | Example                       |
        | ---------------- | -------------------------- | ----------------------------- |
        | **Name**         | Display name for the audit | `daily-consolidation-2026-03` |
        | **Goal**         | Optimization objective     | `server_consolidation`        |
        | **Scope**        | Target scope               | `CLUSTER`                     |
        | **Strategy**     | Optional override          | Leave blank for default       |
        | **Auto Trigger** | Run on a schedule          | Optional                      |

        <Tip>
          Leave **Strategy** blank to use the default strategy for the selected goal.
          Override only if your administrator has configured custom strategies.
        </Tip>
      </Step>

      <Step title="Confirm the audit" icon="play">
        Click **Create**. The audit status progresses:

        `PENDING` → `ONGOING` → `SUCCEEDED`

        <Check>Audit reaches `SUCCEEDED` status and an action plan is generated.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="List available goals" icon="list">
        ```bash title="List optimization goals" theme={null}
        watcher goal list
        ```
      </Step>

      <Step title="Create and run the audit" icon="play">
        ```bash title="Create audit" theme={null}
        watcher audit create \
          --goal server_consolidation \
          --name daily-consolidation-2026-03
        ```
      </Step>

      <Step title="Monitor status" icon="clock">
        ```bash title="Check audit status" theme={null}
        watcher audit show daily-consolidation-2026-03 \
          -f value -c state
        ```

        Poll until state is `SUCCEEDED`.
      </Step>

      <Step title="Verify completion" icon="circle-check">
        ```bash title="Confirm audit succeeded" theme={null}
        watcher audit show daily-consolidation-2026-03
        ```

        <Check>State is `SUCCEEDED` and an action plan UUID is listed.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Audit States

| State       | Meaning                                                             |
| ----------- | ------------------------------------------------------------------- |
| `PENDING`   | Audit queued, waiting for Decision Engine capacity                  |
| `ONGOING`   | Decision Engine is collecting metrics and computing recommendations |
| `SUCCEEDED` | Audit complete — action plan generated                              |
| `FAILED`    | Audit could not complete — check error detail                       |
| `CANCELLED` | Manually cancelled before completion                                |

***

## Audit Scope Options

| Scope     | Description                                            | When to Use                 |
| --------- | ------------------------------------------------------ | --------------------------- |
| `CLUSTER` | Analyze all hosts and instances in the cluster         | Full cluster rebalancing    |
| `ZONE`    | Analyze only instances in a specific availability zone | Zone-specific consolidation |

```bash title="Run zone-scoped audit" theme={null}
watcher audit create \
  --goal zone_migration \
  --scope ZONE \
  --name zone-a-rebalance
```

***

## Strategy Parameters

Override default strategy thresholds when creating an audit:

```bash title="Audit with custom consolidation threshold" theme={null}
watcher audit create \
  --goal server_consolidation \
  --parameter threshold=0.15 \
  --parameter period=7200 \
  --name low-threshold-consolidation
```

| Parameter     | Strategy               | Default | Description                                           |
| ------------- | ---------------------- | ------- | ----------------------------------------------------- |
| `threshold`   | `server_consolidation` | `0.2`   | Utilization ratio below which a host is underutilized |
| `period`      | All                    | `3600`  | Look-back window in seconds                           |
| `granularity` | Telemetry-backed       | `300`   | Metric sample granularity in seconds                  |

***

## Next Steps

<CardGroup>
  <Card title="Action Plans" href="/services/optimization/user-guide/action-plans">
    Review and approve the migration plan generated by your audit.
  </Card>

  <Card title="Execute Actions" href="/services/optimization/user-guide/execute-actions">
    Start a recommended action plan to rebalance workloads.
  </Card>

  <Card title="Optimization Goals" href="/services/optimization/user-guide/optimization-goals">
    Understand which goal to select for different operational scenarios.
  </Card>

  <Card title="Troubleshooting" href="/services/optimization/user-guide/troubleshooting">
    Resolve audit failures and empty action plan scenarios.
  </Card>
</CardGroup>


# Optimization Troubleshooting — User Guide
Source: https://docs.xloud.tech/services/optimization/user-guide/troubleshooting

Resolve common Xloud Optimization issues — empty action plans, stuck audits, failed migrations, and cancelled action plans.

## Overview

This page covers common Optimization issues encountered by operators — audits that
produce empty plans, audits stuck in `ONGOING`, migrations that fail during execution, and
plans that revert after completion. For platform-level issues such as Decision Engine
failures or data source connectivity, see the
[Admin Troubleshooting](/services/optimization/admin-guide/troubleshooting) guide.

***

## Common Issues

<AccordionGroup>
  <Accordion title="Audit completes but produces an empty action plan" icon="clipboard">
    **Cause**: The cluster is already optimally placed for the selected goal — the strategy
    found no hosts below the utilization threshold and no migrations are recommended.

    **Resolution**:

    Check current host utilization to confirm whether consolidation is genuinely needed:

    ```bash title="Check per-host utilization" theme={null}
    openstack hypervisor list --long
    ```

    If all hosts show healthy, even utilization — this is expected behaviour. No action
    is needed.

    If hosts appear imbalanced but no plan was generated, the strategy threshold may
    be too conservative:

    ```bash title="Create audit with lower threshold" theme={null}
    watcher audit create \
      --goal server_consolidation \
      --parameter threshold=0.1 \
      --name lower-threshold-audit
    ```

    <Tip>
      The default consolidation threshold is 0.2 (20%). Lowering it to 0.1 (10%)
      means more hosts qualify as underutilized and are included in the migration plan.
    </Tip>
  </Accordion>

  <Accordion title="Audit stuck in ONGOING state" icon="clock">
    **Cause**: The Decision Engine is waiting for metric data from a slow or unavailable
    data source (Prometheus or Telemetry).

    **Resolution**:

    ```bash title="Check audit status and duration" theme={null}
    watcher audit show <audit-uuid> \
      -f value -c state -c created_at
    ```

    If the audit has been `ONGOING` for more than 5 minutes, contact your administrator
    to check Decision Engine and data source connectivity. Your administrator can configure this through [XDeploy](/deployment).

    For non-telemetry goals (e.g., `server_consolidation`, `zone_migration`), audits
    should complete within 30–90 seconds. Longer durations indicate a data collection
    issue.
  </Accordion>

  <Accordion title="Action fails with migration error" icon="circle-x">
    **Cause**: A live migration failed — commonly due to insufficient memory on the
    target host, a CPU model incompatibility between source and destination hosts, or
    a storage connectivity issue.

    **Resolution**:

    ```bash title="Show failed action details" theme={null}
    watcher action show <action-uuid> -f json
    ```

    Review the `fault` field for the specific migration error. Common errors:

    | Error                 | Cause                                         | Fix                                         |
    | --------------------- | --------------------------------------------- | ------------------------------------------- |
    | `No valid host found` | Target host has insufficient capacity         | Add compute capacity or adjust plan         |
    | `CPU compatibility`   | CPU model mismatch between hosts              | Configure `cpu_mode=custom` on all hosts    |
    | `Disk not found`      | Instance uses local disk (not shared storage) | Verify instance uses shared storage backend |

    After resolving the root cause, create a new audit to generate a fresh plan.
  </Accordion>

  <Accordion title="Action plan shows CANCELLED state" icon="x-circle">
    **Cause**: A previous action in the plan failed, causing the Applier to halt and
    cancel all remaining actions automatically.

    **Resolution**: Review the failed action to identify the root cause:

    ```bash title="List actions and find the failed one" theme={null}
    watcher action list \
      --action-plan <action-plan-uuid> \
      -f table -c uuid -c action_type -c state
    ```

    Fix the root cause (capacity, CPU compatibility, storage), then run a new audit
    to generate an updated plan reflecting the current cluster state.
  </Accordion>

  <Accordion title="Optimizations revert after execution" icon="rotate">
    **Cause**: Another process — the compute scheduler placing new instances, auto-scaling,
    or manual migrations — is placing instances back on hosts that were just emptied by
    the optimization.

    **Resolution**: Coordinate with team members performing manual migrations during
    optimization windows. Consider applying compute host aggregates or availability zone
    constraints to prevent the scheduler from re-populating hosts that were intentionally
    consolidated.
  </Accordion>
</AccordionGroup>

***

## Diagnostic Commands

```bash title="List all audits with states" theme={null}
watcher audit list \
  -f table -c uuid -c name -c state -c created_at
```

```bash title="Show full audit detail" theme={null}
watcher audit show <audit-uuid> -f json
```

```bash title="List action plans with states" theme={null}
watcher actionplan list \
  -f table -c uuid -c state -c audit_uuid
```

```bash title="Show individual action failures" theme={null}
watcher action show <action-uuid> -f json
```

***

## Next Steps

<CardGroup>
  <Card title="Run an Audit" href="/services/optimization/user-guide/run-audit">
    Create a new audit after resolving the issue.
  </Card>

  <Card title="Audit History" href="/services/optimization/user-guide/audit-history">
    Review past audits to identify recurring patterns.
  </Card>

  <Card title="Admin Troubleshooting" href="/services/optimization/admin-guide/troubleshooting">
    Platform-level diagnostics for Decision Engine and data source failures.
  </Card>

  <Card title="Compute Admin Guide" href="/services/compute/admin-guide">
    Verify shared storage and live migration capability for optimization actions.
  </Card>
</CardGroup>


# Orchestration Administration
Source: https://docs.xloud.tech/services/orchestration/admin-guide

Administer the Xloud Orchestration service — configure the engine, manage the stack domain, tune performance, and secure template-based deployments.

Configure, secure, and operate the Xloud Orchestration service for production environments.

***

<CardGroup>
  <Card title="Architecture" icon="server" href="/services/orchestration/architecture">
    Service components, ports, and request flow through the Orchestration engine
  </Card>

  <Card title="Configuration" icon="settings" href="/services/orchestration/configuration">
    Key configuration options, stack domain setup, quotas, and XDeploy integration
  </Card>

  <Card title="Scaling the Service" icon="trending-up" href="/services/orchestration/scaling">
    Deploy multiple engine workers and tune performance for large deployments
  </Card>

  <Card title="Security" icon="shield-check" href="/services/orchestration/security">
    Stack domain users, trust-based authorization, and policy configuration
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/orchestration/admin-troubleshooting">
    Diagnose engine failures, API errors, stack domain issues, and plugin faults
  </Card>
</CardGroup>


# Orchestration Admin Troubleshooting
Source: https://docs.xloud.tech/services/orchestration/admin-troubleshooting

Diagnose and resolve Xloud Orchestration service-level issues — engine failures, API errors, stack domain problems, and resource plugin faults.

## Overview

Admin-level Orchestration issues differ from user-facing stack failures. They typically
involve the service itself — engine workers not starting, the API becoming unreachable,
trust or stack domain misconfiguration, or a resource plugin failing to load. Use the
service log files and `openstack orchestration service list` as primary diagnostic tools.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Diagnostic Reference

<AccordionGroup>
  <Accordion title="Engine workers not processing stacks" icon="circle-x">
    **Symptoms**: Stacks remain in `CREATE_IN_PROGRESS` indefinitely. No events appear
    in `openstack stack event list`. `openstack orchestration service list` shows engine
    workers as `down`.

    **Diagnosis**:

    ```bash title="Check service status" theme={null}
    openstack orchestration service list
    ```

    ```bash title="Check engine container logs (XDeploy/XAVS deployment)" theme={null}
    docker logs heat_engine --tail=100
    ```

    ```bash title="Check message queue connectivity" theme={null}
    docker exec heat_engine python3 -c "
    import kombu
    conn = kombu.Connection('amqp://user:pass@rabbitmq/')
    conn.ensure_connection()
    print('RabbitMQ connection OK')
    "
    ```

    **Common causes and resolutions**:

    | Cause                              | Resolution                                                                                                           |
    | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
    | Engine container exited on startup | Check `docker logs heat_engine` for the error. Common causes: database connection failure, misconfigured `heat.conf` |
    | RabbitMQ connection refused        | Verify RabbitMQ is running: `docker ps \| grep rabbit`. Check `transport_url` in engine configuration                |
    | Database migration not applied     | Run `docker exec heat_engine heat-manage db_sync` to apply pending migrations                                        |
    | Stack domain not configured        | Check `stack_domain_admin` and `stack_domain_admin_password` in `heat.conf`                                          |

    **Restart the engine**:

    ```bash title="Restart engine container" theme={null}
    docker restart heat_engine
    ```
  </Accordion>

  <Accordion title="API returning 500 or refusing connections" icon="server-crash">
    **Symptoms**: Dashboard shows Orchestration as unavailable. CLI commands return
    `503 Service Unavailable` or connection refused on port 8004.

    **Diagnosis**:

    ```bash title="Check API container status" theme={null}
    docker ps --filter name=heat_api
    docker logs heat_api --tail=50
    ```

    ```bash title="Test API endpoint directly" theme={null}
    curl -s http://localhost:8004/
    ```

    ```bash title="Check HAProxy backend health" theme={null}
    echo "show stat" | socat stdio /var/run/haproxy/admin.sock | grep heat
    ```

    **Common causes and resolutions**:

    | Cause                                    | Resolution                                                                        |
    | ---------------------------------------- | --------------------------------------------------------------------------------- |
    | API container not running                | `docker start heat_api`                                                           |
    | Keystone endpoint not registered         | Verify: `openstack endpoint list \| grep orchestration`                           |
    | SSL certificate expired (if TLS enabled) | Renew certificate and restart API container                                       |
    | HAProxy backend marked DOWN              | Check network connectivity between HAProxy and the API container; restart the API |
  </Accordion>

  <Accordion title="Stack domain user creation fails" icon="user-x">
    **Symptoms**: Stacks containing `WaitCondition` or auto-scaling resources fail
    with errors mentioning `StackDomainUser` or `TrustActionMismatch`. Users cannot
    create stacks that require credentials delegation.

    **Diagnosis**:

    ```bash title="Verify stack domain exists" theme={null}
    openstack domain list | grep heat
    ```

    ```bash title="Verify stack domain admin user" theme={null}
    openstack user list --domain heat
    ```

    ```bash title="Test stack domain admin credentials" theme={null}
    openstack --os-username heat_domain_admin \
              --os-user-domain-name heat \
              --os-password <password> \
              token issue
    ```

    **Common causes and resolutions**:

    | Cause                                                 | Resolution                                                            |
    | ----------------------------------------------------- | --------------------------------------------------------------------- |
    | `heat` domain does not exist                          | Re-run `xavs-ansible deploy -t heat` to recreate the domain           |
    | Stack domain admin password incorrect                 | Update `heat_domain_admin_password` in `passwords.yml` and redeploy   |
    | `stack_domain_admin` setting missing from `heat.conf` | Verify XDeploy configuration and redeploy                             |
    | Xloud Identity service unreachable from engine        | Check network connectivity between the engine container and port 5000 |
  </Accordion>

  <Accordion title="Resource plugin fails to load or raises errors" icon="puzzle-x">
    **Symptoms**: Specific resource types consistently fail with `InvalidTemplateVersion`
    or `ResourceTypeUnavailable`. The engine log shows import errors.

    **Diagnosis**:

    ```bash title="List available resource types" theme={null}
    openstack orchestration resource type list
    ```

    ```bash title="Show resource type schema" theme={null}
    openstack orchestration resource type show Xloud::Compute::Server
    ```

    ```bash title="Check engine log for plugin errors" theme={null}
    docker logs heat_engine 2>&1 | grep -i "plugin\|resource_type\|ImportError"
    ```

    **Common causes and resolutions**:

    | Cause                                 | Resolution                                                                                                                        |
    | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
    | Dependent service not enabled         | Some resource types require specific services. `Xloud::Networking::FloatingIP` requires networking; verify the service is enabled |
    | Plugin version mismatch after upgrade | Restart the engine after upgrades: `docker restart heat_engine`                                                                   |
    | Custom plugin missing                 | If using custom resource plugins, verify the plugin file exists in the engine's plugin directory and has correct permissions      |
  </Accordion>

  <Accordion title="Large stacks time out or fail under load" icon="clock">
    **Symptoms**: Stacks with many resources (100+) frequently time out or take much
    longer than expected. Engine workers appear idle despite stacks being queued.

    **Diagnosis**:

    ```bash title="Check engine worker count" theme={null}
    openstack orchestration service list | grep heat-eng | wc -l
    ```

    ```bash title="Check message queue depth" theme={null}
    docker exec rabbitmq rabbitmqctl list_queues name messages
    ```

    **Resolutions**:

    | Action                        | Setting                              |
    | ----------------------------- | ------------------------------------ |
    | Increase engine workers       | `heat_engine_workers: 8` (or higher) |
    | Increase RPC timeout          | `heat_rpc_response_timeout: 300`     |
    | Increase database pool        | `heat_db_max_pool_size: 20`          |
    | Verify convergence mode is on | `heat_convergence_engine: true`      |

    Apply changes by updating globals and redeploying:

    ```bash title="Redeploy with new settings" theme={null}
    xavs-ansible deploy -t heat
    ```
  </Accordion>
</AccordionGroup>

***

## Log Locations

| Service              | Log Path                                                             |
| -------------------- | -------------------------------------------------------------------- |
| Orchestration Engine | `docker logs heat_engine` or `/var/log/kolla/heat/heat-engine.log`   |
| Orchestration API    | `docker logs heat_api` or `/var/log/kolla/heat/heat-api.log`         |
| CloudWatch API       | `docker logs heat_api_cfn` or `/var/log/kolla/heat/heat-api-cfn.log` |

***

## Next Steps

<CardGroup>
  <Card title="Configuration" href="/services/orchestration/configuration">
    Review and update service configuration through XDeploy
  </Card>

  <Card title="Scaling the Service" href="/services/orchestration/scaling">
    Add engine workers to resolve throughput and timeout issues
  </Card>

  <Card title="Security" href="/services/orchestration/security">
    Diagnose stack domain and trust authorization problems
  </Card>

  <Card title="User Troubleshooting" href="/services/orchestration/troubleshooting">
    Stack-level diagnostics for CREATE\_FAILED and template errors
  </Card>
</CardGroup>


# Orchestration Service Architecture
Source: https://docs.xloud.tech/services/orchestration/architecture

Understand the Xloud Orchestration service components — API, Engine, and resource plugins — and how they process templates to provision cloud infrastructure.

## Overview

Xloud Orchestration is built on a three-tier architecture: an API tier that accepts
template submissions, an engine tier that resolves dependencies and drives resource
provisioning, and a plugin layer that delegates individual resource operations to the
appropriate cloud service API. The engine is stateless — all stack state is persisted
in a relational database, enabling horizontal engine scaling.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Service Topology

```mermaid theme={null}
graph TD
    U([User / Dashboard]) -->|Submit template| API[Orchestration API<br/>:8004]
    API -->|Parse template| ENG[Orchestration Engine]
    ENG -->|Store state| DB[(Database)]
    ENG -->|Resolve graph| GRAPH[Dependency Resolver]
    GRAPH -->|Provision resource| COMPUTE[Xloud Compute API<br/>:8774]
    GRAPH -->|Provision resource| NET[Xloud Networking API<br/>:9696]
    GRAPH -->|Provision resource| STORE[Xloud Block Storage API<br/>:8776]
    GRAPH -->|Provision resource| ID[Xloud Identity API<br/>:5000]
    COMPUTE & NET & STORE & ID -->|Resource status| ENG
    ENG -->|Stack status + outputs| API
    API -->|Response| U
    style API fill:#197560,color:#fff
    style ENG fill:#197560,color:#fff
    style GRAPH fill:#3F8F7E,color:#fff
    style DB fill:#3F8F7E,color:#fff
```

***

## Service Components

| Component                     | Port     | Runs On          | Description                                                                                                                |
| ----------------------------- | -------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------- |
| **Orchestration API**         | 8004     | Controller nodes | REST API endpoint — accepts template submissions, parameter updates, and stack lifecycle commands                          |
| **Orchestration Engine**      | Internal | Controller nodes | Core processing service — parses templates, builds dependency graphs, drives resource provisioning, and tracks stack state |
| **CloudWatch-Compatible API** | 8000     | Controller nodes | Alarm and metric compatibility endpoint used by `WaitCondition` signal handling and scaling policy webhooks                |
| **Resource Plugins**          | N/A      | Engine process   | In-process plugins that translate resource type operations into calls to the underlying service APIs                       |

<Note>
  The Orchestration API and CloudWatch-compatible API are separate processes. Both must
  be running for auto-scaling and `WaitCondition` resources to function correctly.
  Port 8000 must be accessible from instances that use `WaitCondition` signal URLs.
</Note>

***

## How the Engine Processes a Template

```mermaid theme={null}
sequenceDiagram
    participant User
    participant API as Orchestration API
    participant Engine as Orchestration Engine
    participant DB as Database
    participant Plugin as Resource Plugin
    participant Cloud as Cloud Service API

    User->>API: POST /v1/stacks (template + params)
    API->>Engine: Dispatch create request
    Engine->>DB: Store stack record (CREATE_IN_PROGRESS)
    Engine->>Engine: Parse template, resolve parameters
    Engine->>Engine: Build dependency graph
    loop For each resource (in dependency order)
        Engine->>Plugin: create(resource_type, properties)
        Plugin->>Cloud: API call (e.g., POST /servers)
        Cloud-->>Plugin: Resource ID + initial state
        Plugin-->>Engine: Resource created
        Engine->>DB: Update resource status
    end
    Engine->>DB: Update stack status (CREATE_COMPLETE)
    Engine-->>API: Stack ready
    API-->>User: 201 Created + stack ID
```

***

## Request Flow Details

### Template Parsing

The engine parses the YAML template and evaluates all `parameters` sections, applying
default values where the caller did not provide overrides. Intrinsic functions are
not evaluated at parse time — they are resolved lazily when a resource is being
provisioned and its dependencies are known.

### Dependency Graph Resolution

The engine constructs a directed acyclic graph (DAG) from:

* Explicit `depends_on` declarations in resource definitions
* Implicit dependencies detected from `get_resource` and `get_attr` references in
  resource properties

Resources without dependencies are provisioned in parallel. Resources with dependencies
are queued until all prerequisite resources reach `CREATE_COMPLETE`.

### Resource Plugin Architecture

Each resource type (`Xloud::Compute::Server`, `Xloud::Networking::Net`, etc.) is
handled by a dedicated plugin that implements a standard interface:

| Plugin Method      | Trigger             | Description                                                   |
| ------------------ | ------------------- | ------------------------------------------------------------- |
| `create()`         | Stack create        | Calls the service API to provision the resource               |
| `update()`         | Stack update        | Applies property changes — either in-place or via replacement |
| `delete()`         | Stack delete        | Removes the resource from the cloud service                   |
| `get_reference()`  | `get_resource` call | Returns the resource's primary ID                             |
| `get_live_state()` | Stack drift check   | Polls the service API for the current resource state          |

***

## Database Schema Overview

The Orchestration service persists all state in a dedicated database schema:

| Table             | Contents                                                         |
| ----------------- | ---------------------------------------------------------------- |
| `stack`           | Stack metadata, status, template, and owner                      |
| `resource`        | Individual resource records with type, properties, and status    |
| `resource_data`   | Per-resource key-value metadata (e.g., resource IDs, attributes) |
| `event`           | Ordered log of all stack and resource state transitions          |
| `software_config` | Configuration objects for software deployment resources          |
| `stack_tag`       | Tag associations for stacks                                      |

***

## Next Steps

<CardGroup>
  <Card title="Configuration" href="/services/orchestration/configuration">
    Configure the Orchestration service, quotas, and stack domain
  </Card>

  <Card title="Scaling the Service" href="/services/orchestration/scaling">
    Deploy multiple engine workers for high-throughput deployments
  </Card>

  <Card title="Security" href="/services/orchestration/security">
    Stack domain users, trust authorization, and policy configuration
  </Card>

  <Card title="Admin Troubleshooting" href="/services/orchestration/admin-troubleshooting">
    Diagnose engine failures and service-level issues
  </Card>
</CardGroup>


# Auto-Scaling with Xloud Orchestration
Source: https://docs.xloud.tech/services/orchestration/autoscaling

Configure auto-scaling groups with Heat templates. Scale instances based on Prometheus metrics and webhooks.

## Overview

Xloud Orchestration provides native auto-scaling through three coordinated resource types: a
<Tooltip>scaling group</Tooltip>,
a scaling policy, and an alarm trigger. When a metric threshold is breached — such as CPU
utilization exceeding 80% — the alarm fires a webhook that activates the scaling policy, which
adds or removes instances from the group.

Prometheus is the monitoring backend for alarm-driven scaling. It replaces legacy telemetry
stacks (Ceilometer/Aodh) and provides a more reliable, standards-based metrics pipeline with
support for multi-dimensional labels, alerting rules, and long-retention storage.

<Note>
  **Prerequisites**

  * Xloud Orchestration enabled in your project
  * A compatible image and flavor for scaled instances
  * Prometheus deployed and scraping compute metrics ([see Prometheus integration](/integrations/prometheus))
  * Basic familiarity with [orchestration templates](/services/orchestration/template-guide)
</Note>

***

## Architecture

```mermaid theme={null}
graph LR
    PROM[Prometheus<br/>Metrics] -->|Evaluate alert rules| ALERT[Alertmanager]
    ALERT -->|POST to signal URL| POLICY[Scaling Policy]
    POLICY -->|Scale out / in| GROUP[Auto-Scaling Group]
    GROUP -->|Launch instance| NOVA[Compute]
    GROUP -->|Terminate instance| NOVA
    style POLICY fill:#197560,color:#fff
    style GROUP fill:#197560,color:#fff
    style PROM fill:#145C4C,color:#fff
```

### Scaling Components

| Component              | Resource Type                | Purpose                                                              |
| ---------------------- | ---------------------------- | -------------------------------------------------------------------- |
| **Auto-Scaling Group** | `OS::Heat::AutoScalingGroup` | Manages a pool of identical instances with min/max size constraints  |
| **Scaling Policy**     | `OS::Heat::ScalingPolicy`    | Defines how to adjust the group — add N, remove N, or set exact size |
| **Signal URL**         | Produced by `ScalingPolicy`  | Webhook endpoint activated by Prometheus Alertmanager or manual POST |
| **Prometheus**         | External                     | Scrapes instance metrics, evaluates alert rules, fires webhooks      |

***

## Use Cases

| Use Case                           | Description                                                              | Trigger              |
| ---------------------------------- | ------------------------------------------------------------------------ | -------------------- |
| **Elastic web / application tier** | Scale web server instances based on HTTP request rate or CPU utilization | Prometheus alert     |
| **CI/CD build farm**               | Add worker nodes during active builds, shrink on idle                    | Schedule or webhook  |
| **Batch processing cluster**       | Provision compute nodes for heavy batch jobs, release when complete      | Manual or scheduled  |
| **Dev/test resource pools**        | Automatically scale out environments for short-lived test runs           | On-demand webhook    |
| **Disaster recovery warm pool**    | Maintain standby instances that scale out during failover events         | Alertmanager webhook |

***

## Orchestration Templates

### Static Cluster Template

Use this template when you need a fixed number of instances deployed as a named group. Each
instance is declared as a discrete resource — suitable for small, stable clusters.

```yaml title="static-cluster.yaml" theme={null}
heat_template_version: 2016-10-14

description: Static 3-node compute cluster

parameters:
  image:
    type: string
    default: Ubuntu-22.04
  flavor:
    type: string
    default: m1.small
  network:
    type: string
    default: private
  key_name:
    type: string

resources:
  vm1:
    type: OS::Nova::Server
    properties:
      name: cluster-node-1
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - network: { get_param: network }

  vm2:
    type: OS::Nova::Server
    properties:
      name: cluster-node-2
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - network: { get_param: network }

  vm3:
    type: OS::Nova::Server
    properties:
      name: cluster-node-3
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - network: { get_param: network }

outputs:
  vm1_ip:
    value: { get_attr: [vm1, first_address] }
  vm2_ip:
    value: { get_attr: [vm2, first_address] }
  vm3_ip:
    value: { get_attr: [vm3, first_address] }
```

### Auto-Scaling Template

This template creates a web tier that scales between 1 and 10 instances. The scaling policy
signal URLs are exposed as stack outputs and can be wired into Prometheus Alertmanager webhook
receivers.

```yaml title="autoscaling-stack.yaml" theme={null}
heat_template_version: 2016-10-14

description: >
  Auto-scaling web tier with scale-out and scale-in policies.
  Signal URLs are consumed by Prometheus Alertmanager webhook receivers.

parameters:
  image:
    type: string
    label: Instance Image

  flavor:
    type: string
    label: Instance Flavor
    default: m1.small

  key_name:
    type: string
    label: Key Pair

  network:
    type: string
    label: Network

  min_size:
    type: number
    default: 1
    constraints:
      - range: { min: 1, max: 10 }

  max_size:
    type: number
    default: 10
    constraints:
      - range: { min: 2, max: 20 }

resources:

  # Auto-scaling group — manages the instance pool
  web_asg:
    type: OS::Heat::AutoScalingGroup
    properties:
      min_size: { get_param: min_size }
      max_size: { get_param: max_size }
      desired_capacity: { get_param: min_size }
      resource:
        type: OS::Nova::Server
        properties:
          image: { get_param: image }
          flavor: { get_param: flavor }
          key_name: { get_param: key_name }
          networks:
            - network: { get_param: network }
          user_data: |
            #!/bin/bash
            apt-get update -y
            apt-get install -y nginx
            systemctl enable --now nginx

  # Scale-out policy — add 1 instance per trigger
  scale_out_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      auto_scaling_group_id: { get_resource: web_asg }
      adjustment_type: change_in_capacity
      scaling_adjustment: 1
      cooldown: 60

  # Scale-in policy — remove 1 instance per trigger
  scale_in_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      auto_scaling_group_id: { get_resource: web_asg }
      adjustment_type: change_in_capacity
      scaling_adjustment: -1
      cooldown: 120

outputs:
  scale_out_url:
    description: Webhook URL to trigger scale-out (wire into Alertmanager)
    value: { get_attr: [scale_out_policy, signal_url] }

  scale_in_url:
    description: Webhook URL to trigger scale-in (wire into Alertmanager)
    value: { get_attr: [scale_in_policy, signal_url] }

  current_size:
    description: Current instance count in the scaling group
    value: { get_attr: [web_asg, current_size] }
```

***

## Adjustment Types

| `adjustment_type`            | Behavior                                            | Example                                          |
| ---------------------------- | --------------------------------------------------- | ------------------------------------------------ |
| `change_in_capacity`         | Add or remove N instances relative to current count | `scaling_adjustment: 2` adds 2 instances         |
| `exact_capacity`             | Set the group to exactly N instances                | `scaling_adjustment: 5` sets group size to 5     |
| `percent_change_in_capacity` | Change capacity by a percentage of current size     | `scaling_adjustment: 25` adds 25% more instances |

***

## Prometheus Integration

Prometheus Alertmanager delivers scaling signals by sending an HTTP POST to the policy signal
URL. Configure a webhook receiver in your Alertmanager configuration:

```yaml title="alertmanager.yml" theme={null}
route:
  receiver: "default"
  routes:
    - match:
        alertname: "HighCpuUsage"
      receiver: "scale-out"
    - match:
        alertname: "LowCpuUsage"
      receiver: "scale-in"

receivers:
  - name: "scale-out"
    webhook_configs:
      - url: "<scale_out_url from stack output>"
        send_resolved: false

  - name: "scale-in"
    webhook_configs:
      - url: "<scale_in_url from stack output>"
        send_resolved: false
```

A matching Prometheus alert rule that fires when average CPU exceeds 80% for 2 minutes:

```yaml title="alert-rules.yml" theme={null}
groups:
  - name: autoscaling
    rules:
      - alert: HighCpuUsage
        expr: avg(rate(node_cpu_seconds_total{mode!="idle"}[2m])) by (job) > 0.80
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "CPU usage above 80% — triggering scale-out"

      - alert: LowCpuUsage
        expr: avg(rate(node_cpu_seconds_total{mode!="idle"}[10m])) by (job) < 0.20
        for: 10m
        labels:
          severity: info
        annotations:
          summary: "CPU usage below 20% — triggering scale-in"
```

<Tip>
  Use longer evaluation windows (5–10 minutes) for scale-in rules to avoid prematurely
  terminating instances during short idle periods. Scale-out rules can use shorter windows
  (1–2 minutes) to respond faster to load spikes.
</Tip>

***

## Deploy and Trigger Scaling

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Deploy the auto-scaling stack" icon="rocket">
        Navigate to **Orchestration > Stacks** and click **Create Stack**.

        In the **Prepare Template** step, upload `autoscaling-stack.yaml`. In the
        **Orchestration Information** step, fill in the parameters:

        | Parameter  | Example Value  | Description                     |
        | ---------- | -------------- | ------------------------------- |
        | `image`    | `Ubuntu-22.04` | Base image for scaled instances |
        | `flavor`   | `m1.small`     | Instance size                   |
        | `key_name` | `my-keypair`   | SSH key pair for access         |
        | `network`  | `private`      | Network for instances           |
        | `min_size` | `1`            | Minimum instance count          |
        | `max_size` | `10`           | Maximum instance count          |

        Click **Confirm**.

        <Check>Stack reaches **Create Complete**. The scaling group shows the initial instance count.</Check>
      </Step>

      <Step title="Retrieve webhook URLs" icon="link">
        Open the stack detail page and select the **Detail** tab (Outputs card). Copy the values for
        `scale_out_url` and `scale_in_url` — these are used as Alertmanager webhook
        receiver URLs.
      </Step>

      <Step title="Manually trigger scale-out" icon="trending-up">
        To test scaling without waiting for an alert, send an HTTP POST to the signal URL:

        ```bash title="Trigger scale-out via webhook" theme={null}
        curl -X POST "<scale_out_url>"
        ```

        <Check>The scaling group adds one instance. Check the **Stack Resources** tab to confirm the new member.</Check>
      </Step>

      <Step title="Monitor the group" icon="activity">
        Return to **Orchestration > Stacks** and open the stack. The **Stack Resources**
        tab shows the current group resources. The **Stack Events** tab shows scaling
        events in real time as Prometheus alerts fire and Alertmanager posts to the signal URLs.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Deploy the stack" icon="rocket">
        ```bash title="Create the auto-scaling stack" theme={null}
        openstack stack create \
          --template autoscaling-stack.yaml \
          --parameter image=Ubuntu-22.04 \
          --parameter key_name=my-keypair \
          --parameter network=private \
          --parameter min_size=1 \
          --parameter max_size=10 \
          --wait \
          web-asg-stack
        ```
      </Step>

      <Step title="Retrieve webhook URLs" icon="link">
        ```bash title="Get scaling webhook URLs" theme={null}
        openstack stack output show web-asg-stack scale_out_url -c output_value -f value
        openstack stack output show web-asg-stack scale_in_url -c output_value -f value
        ```

        Store these URLs in your Alertmanager webhook receiver configuration.
      </Step>

      <Step title="Trigger scale-out manually" icon="trending-up">
        ```bash title="Signal scale-out" theme={null}
        SCALE_OUT_URL=$(openstack stack output show web-asg-stack scale_out_url \
          -c output_value -f value)
        curl -X POST "$SCALE_OUT_URL"
        ```

        ```bash title="Verify group size increased" theme={null}
        openstack stack resource list web-asg-stack
        ```

        <Check>The auto-scaling group shows one additional member instance.</Check>
      </Step>

      <Step title="Check current group size" icon="bar-chart">
        ```bash title="Show current instance count" theme={null}
        openstack stack output show web-asg-stack current_size -c output_value -f value
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Cooldown Periods

Cooldown prevents rapid successive scaling events from destabilizing your workload. The
`cooldown` value is specified in seconds per scaling policy.

| Scenario                                      | Recommended Scale-Out Cooldown | Recommended Scale-In Cooldown |
| --------------------------------------------- | ------------------------------ | ----------------------------- |
| Fast-booting instances (cloud image, no init) | 30–60 s                        | 60–90 s                       |
| Instances with cloud-init provisioning        | 90–120 s                       | 120–180 s                     |
| Instances requiring application warm-up       | 120–180 s                      | 180–300 s                     |

<Warning>
  Setting cooldown too low on scale-in can cause thrashing — where instances are terminated
  before the remaining group has stabilized under the new load distribution. Use a scale-in
  cooldown at least twice the scale-out cooldown.
</Warning>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Stack fails to create with CREATE_FAILED" icon="circle-x">
    **Cause**: Insufficient quota, unavailable flavor, or invalid image name.

    **Resolution**:

    ```bash title="Check stack events for the error message" theme={null}
    openstack stack event list web-asg-stack --nested-depth 5
    ```

    Review the `resource_status_reason` field. Common causes:

    * Compute quota exceeded — check with `openstack quota show`
    * Image not found — verify with `openstack image list`
    * Flavor not available in the target availability zone
  </Accordion>

  <Accordion title="Scale-out webhook returns 401 or 403" icon="lock">
    **Cause**: The signal URL contains a temporary token that has expired, or the URL was
    copied incorrectly.

    **Resolution**: Retrieve a fresh signal URL from the stack output:

    ```bash title="Refresh signal URL" theme={null}
    openstack stack output show web-asg-stack scale_out_url -c output_value -f value
    ```

    Signal URLs are valid as long as the stack exists. Update your Alertmanager config with
    the current URL after any stack update.
  </Accordion>

  <Accordion title="Auto-scaling group does not grow beyond min_size" icon="trending-up">
    **Cause**: Alertmanager is not reaching the signal URL, or the Prometheus alert is not
    firing.

    **Resolution**:

    1. Verify Alertmanager is running: `curl http://<alertmanager-host>:9093/-/healthy`
    2. Check alert state in Prometheus UI under **Alerts**
    3. Confirm the webhook receiver URL in Alertmanager config matches the stack output
    4. Test manually: `curl -X POST "<scale_out_url>"` — if this works, the stack is healthy
  </Accordion>

  <Accordion title="Instances in the group show BUILD or ERROR" icon="clock">
    **Cause**: Compute capacity exhausted on available hosts, or image boot failure.

    **Resolution**:

    ```bash title="List instances in the scaling group" theme={null}
    openstack stack resource list web-asg-stack --nested-depth 2
    ```

    Identify failed instances and check their events:

    ```bash title="Check instance events" theme={null}
    openstack server event list <instance-id>
    ```
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Template Guide" href="/services/orchestration/template-guide">
    Learn intrinsic functions and conditions used in scaling templates
  </Card>

  <Card title="Manage Stacks" href="/services/orchestration/stacks">
    Update, suspend, and manage the auto-scaling stack lifecycle
  </Card>

  <Card title="Prometheus Integration" href="/integrations/prometheus">
    Configure Prometheus scrape targets and alert rules for scaling triggers
  </Card>

  <Card title="Xloud Load Balancer" href="/services/load-balancer">
    Front auto-scaling groups with a load balancer for traffic distribution
  </Card>
</CardGroup>


# Orchestration CLI Reference
Source: https://docs.xloud.tech/services/orchestration/cli-reference

Complete openstack stack CLI commands for managing Xloud Orchestration — create, update, delete, and inspect stacks, resources, events, and outputs.

## Overview

The `openstack stack` command group manages infrastructure stacks defined by orchestration templates — create, update, suspend, resume, and delete stacks, inspect resources and events, and retrieve stack outputs.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python heatclient installed: `pip install python-heatclient`
</Note>

***

## Stacks

### Create and Update

<CodeGroup>
  ```bash title="Create stack from template" theme={null}
  openstack stack create \
    --template stack.yaml \
    my-stack
  ```

  ```bash title="Create with parameters" theme={null}
  openstack stack create \
    --template stack.yaml \
    --parameter image=Ubuntu-22.04 \
    --parameter flavor=m1.small \
    --parameter network=private \
    --wait \
    my-stack
  ```

  ```bash title="Create from environment file" theme={null}
  openstack stack create \
    --template stack.yaml \
    --environment env.yaml \
    --wait \
    my-stack
  ```

  ```bash title="Update stack" theme={null}
  openstack stack update \
    --template stack.yaml \
    --parameter flavor=m1.medium \
    --wait \
    my-stack
  ```

  ```bash title="Preview stack changes (dry run)" theme={null}
  openstack stack update \
    --template stack.yaml \
    --dry-run \
    my-stack
  ```
</CodeGroup>

### List and Inspect

<CodeGroup>
  ```bash title="List stacks" theme={null}
  openstack stack list
  openstack stack list --nested
  ```

  ```bash title="Show stack details" theme={null}
  openstack stack show my-stack
  ```

  ```bash title="Show stack in JSON" theme={null}
  openstack stack show my-stack --format json
  ```
</CodeGroup>

### Lifecycle

<CodeGroup>
  ```bash title="Suspend stack" theme={null}
  openstack stack suspend my-stack
  ```

  ```bash title="Resume stack" theme={null}
  openstack stack resume my-stack
  ```

  ```bash title="Abandon stack (keep resources)" theme={null}
  openstack stack abandon my-stack
  ```

  ```bash title="Delete stack" theme={null}
  openstack stack delete my-stack
  openstack stack delete --wait my-stack
  ```
</CodeGroup>

***

## Resources

<CodeGroup>
  ```bash title="List resources in a stack" theme={null}
  openstack stack resource list my-stack
  openstack stack resource list --nested-depth 3 my-stack
  ```

  ```bash title="Show resource details" theme={null}
  openstack stack resource show my-stack <resource-name>
  ```

  ```bash title="List resource types" theme={null}
  openstack orchestration resource type list
  ```

  ```bash title="Show resource type schema" theme={null}
  openstack orchestration resource type show OS::Nova::Server
  ```
</CodeGroup>

***

## Events

<CodeGroup>
  ```bash title="List stack events" theme={null}
  openstack stack event list my-stack
  ```

  ```bash title="List with nested resources" theme={null}
  openstack stack event list --nested-depth 5 my-stack
  ```

  ```bash title="Show event details" theme={null}
  openstack stack event show my-stack <resource-name> <event-id>
  ```
</CodeGroup>

***

## Outputs

<CodeGroup>
  ```bash title="List stack outputs" theme={null}
  openstack stack output list my-stack
  ```

  ```bash title="Show a specific output" theme={null}
  openstack stack output show my-stack <output-key>
  ```

  ```bash title="Extract output value" theme={null}
  openstack stack output show my-stack scale_out_url \
    -c output_value -f value
  ```
</CodeGroup>

***

## Template Validation

<CodeGroup>
  ```bash title="Validate template" theme={null}
  openstack orchestration template validate \
    --template stack.yaml
  ```

  ```bash title="Validate with parameters" theme={null}
  openstack orchestration template validate \
    --template stack.yaml \
    --parameter flavor=m1.small
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="Template Guide" href="/services/orchestration/template-guide">
    Write and validate orchestration templates
  </Card>

  <Card title="Auto-Scaling" href="/services/orchestration/autoscaling">
    Configure auto-scaling groups and alarm triggers
  </Card>
</CardGroup>


# Orchestration Configuration
Source: https://docs.xloud.tech/services/orchestration/configuration

Configure the Xloud Orchestration service — key settings, stack domain setup, default quotas, and XDeploy integration for production deployments.

## Overview

The Orchestration service is configured through XDeploy global variables and
service-specific configuration files. Key configuration areas include the stack domain
(used for trust delegation), engine worker settings, quota defaults, and integration
with the CloudWatch-compatible alarm endpoint.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Key Configuration Options

The following settings control core Orchestration service behavior. All settings are
managed through XDeploy.

| Setting                                    | Default      | Description                                                |
| ------------------------------------------ | ------------ | ---------------------------------------------------------- |
| `enable_heat`                              | `"no"`       | Enable the Orchestration service                           |
| `heat_engine_workers`                      | `4`          | Number of engine worker processes per controller node      |
| `heat_api_workers`                         | `4`          | Number of API worker processes per controller node         |
| `heat_max_stacks_per_tenant`               | `100`        | Maximum stacks per project                                 |
| `heat_max_resources_per_stack`             | `1000`       | Maximum resources in a single stack                        |
| `heat_max_nested_stack_depth`              | `5`          | Maximum depth for nested stack hierarchies                 |
| `heat_convergence_engine`                  | `true`       | Enable convergence mode for parallel resource provisioning |
| `heat_default_deployment_signal_transport` | `CFN_SIGNAL` | Default signal transport for `WaitCondition` resources     |

<Note>
  Enable the Orchestration service by setting `enable_heat: "yes"` in
  `/etc/xavs/globals.d/_50_orchestration.yml` and running `xavs-ansible deploy -t heat`.
</Note>

***

## Enable the Service

<Tabs>
  <Tab title="XDeploy" icon="gauge">
    <Steps>
      <Step title="Navigate to Configuration" icon="settings">
        In XDeploy, navigate to **Configuration → Services → Orchestration**.
      </Step>

      <Step title="Enable the service" icon="toggle-right">
        Set **Enable Orchestration** to **Yes** and configure the engine worker count
        appropriate for your controller node capacity (typically 2–4 workers per CPU core).
      </Step>

      <Step title="Apply configuration" icon="play">
        Click **Save** and then **Deploy → Orchestration** to apply the configuration.

        <Check>
          The Orchestration API is accessible at `http://<controller-ip>:8004/v1`.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create configuration override" icon="file-code">
        ```bash title="Create orchestration globals" theme={null}
        cat > /etc/xavs/globals.d/_50_orchestration.yml << 'EOF'
        enable_heat: "yes"
        heat_engine_workers: 4
        heat_api_workers: 4
        heat_max_stacks_per_tenant: 100
        EOF
        ```
      </Step>

      <Step title="Deploy the service" icon="play">
        ```bash title="Deploy Orchestration" theme={null}
        xavs-ansible deploy -t heat
        ```
      </Step>

      <Step title="Verify the service" icon="circle-check">
        ```bash title="Check Orchestration endpoint" theme={null}
        openstack orchestration service list
        ```

        <Check>All engine and API services show status `up`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Stack Domain Setup

The stack domain is a dedicated Xloud Identity domain used for trust delegation.
When a template creates resources that require credentials (e.g., `WaitCondition`
signals, auto-scaling webhooks), the engine uses a stack domain user — scoped to
the stack's project — rather than the submitting user's credentials.

<Warning>
  The stack domain must be configured before deploying stacks that use
  `WaitCondition` or scaling policy resources. Stacks using only basic compute
  and network resources do not require the stack domain.
</Warning>

Stack domain configuration is handled automatically by XDeploy during the
Orchestration deployment. The following variables control the domain:

| Setting                      | Description                                                     |
| ---------------------------- | --------------------------------------------------------------- |
| `heat_domain_name`           | Name of the stack domain in Xloud Identity (default: `heat`)    |
| `heat_domain_admin`          | Admin user for the stack domain                                 |
| `heat_domain_admin_password` | Password for the stack domain admin (stored in `passwords.yml`) |

***

## Default Quotas

Orchestration quotas limit per-project resource consumption. Defaults are set
cluster-wide; you can override them per-project.

| Quota       | Default | Description                                     |
| ----------- | ------- | ----------------------------------------------- |
| `stacks`    | 100     | Maximum stacks per project                      |
| `resources` | 1000    | Maximum resources across all stacks per project |

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to the admin quota settings to adjust global
    quota defaults. For per-project overrides, navigate to **Identity > Projects** (admin view),
    select a project, and click **Manage Quota**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show orchestration quotas for a project" theme={null}
    openstack quota show --project <PROJECT_ID>
    ```

    ```bash title="Update stack quota for a project" theme={null}
    openstack quota set --stacks 200 <PROJECT_ID>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Scaling the Service" href="/services/orchestration/scaling">
    Configure multiple engine workers for high-throughput deployments
  </Card>

  <Card title="Security" href="/services/orchestration/security">
    Stack domain trust, policy configuration, and template injection prevention
  </Card>

  <Card title="Architecture" href="/services/orchestration/architecture">
    Understand service components and request processing flow
  </Card>

  <Card title="Admin Troubleshooting" href="/services/orchestration/admin-troubleshooting">
    Diagnose configuration errors and engine startup failures
  </Card>
</CardGroup>


# Create Your First Stack
Source: https://docs.xloud.tech/services/orchestration/getting-started

Learn how to define an orchestration template, deploy a stack on Xloud, and manage the stack lifecycle using the Dashboard and CLI.

## Overview

A <Tooltip>stack</Tooltip>
is the fundamental unit of Xloud Orchestration. You describe the desired infrastructure
in an orchestration template, submit it to the Orchestration API, and the engine provisions
every resource in dependency order, reporting the overall stack status when complete.

<Note>
  **Prerequisites**

  * A project with `member` or `admin` role
  * At least one image available in the Xloud Image Service
  * At least one flavor available for your project
  * A network available in your project
  * `xloud` CLI installed and authenticated (see [CLI Setup](/cli-setup))
</Note>

***

## What Is a Stack?

A stack groups related cloud resources into a single deployable unit:

| Concept       | Description                                                                   |
| ------------- | ----------------------------------------------------------------------------- |
| **Template**  | A YAML file declaring the resources, parameters, and outputs for a deployment |
| **Stack**     | The live instantiation of a template — the actual running resources           |
| **Parameter** | A runtime variable that customizes a template without editing it              |
| **Output**    | A value produced by the stack (e.g., an IP address) returned after creation   |
| **Resource**  | A single cloud object managed by the stack (instance, network, volume, etc.)  |

***

## Example Template

The following template launches a single compute instance with a configurable flavor.
Save it as `first-stack.yaml`:

```yaml title="first-stack.yaml" theme={null}
xloud_template_version: "2025-10-15"

description: >
  A minimal orchestration template that launches a single compute
  instance with a parameterized flavor and key pair.

parameters:
  instance_name:
    type: string
    label: Instance Name
    description: Display name for the compute instance
    default: my-first-instance

  flavor:
    type: string
    label: Flavor
    description: Compute flavor (vCPU and RAM profile)
    default: m1.small

  image:
    type: string
    label: Image
    description: OS image name or ID to boot from
    default: Ubuntu-22.04

  key_name:
    type: string
    label: Key Pair
    description: SSH key pair for instance access

  network:
    type: string
    label: Network
    description: Network to attach the instance to
    default: default

resources:
  my_instance:
    type: Xloud::Compute::Server
    properties:
      name: { get_param: instance_name }
      flavor: { get_param: flavor }
      image: { get_param: image }
      key_name: { get_param: key_name }
      networks:
        - network: { get_param: network }

outputs:
  instance_id:
    description: The ID of the created instance
    value: { get_resource: my_instance }

  instance_ip:
    description: The assigned IP address
    value: { get_attr: [my_instance, first_address] }
```

***

## Create a Stack

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Orchestration" icon="compass">
        Navigate to
        **Orchestration > Stacks**.
      </Step>

      <Step title="Open the Create Stack wizard">
        Click **Create Stack**. The 2-step wizard opens.
      </Step>

      <Step title="Step 1 — Prepare Template">
        | Field                    | Type                       | Required | Description                                                 |
        | ------------------------ | -------------------------- | -------- | ----------------------------------------------------------- |
        | **Template Content**     | Text area with file upload | Yes      | Paste YAML template or upload a `.yaml` file                |
        | **Environment Variable** | Text area with file upload | No       | Optional environment variables file for template parameters |

        The template is validated for YAML syntax. Click **Next** to proceed.

        <Tip>
          Environment variable files let you separate configuration from the template.
          Parameters in the environment file must match those defined in the template.
        </Tip>
      </Step>

      <Step title="Step 2 — Orchestration Information">
        | Field                          | Type   | Required | Default | Description                                                             |
        | ------------------------------ | ------ | -------- | ------- | ----------------------------------------------------------------------- |
        | **Stack Name**                 | Text   | Yes      | —       | Unique name for this stack                                              |
        | **Creation Timeout (Minutes)** | Number | Yes      | 60      | Time before creation is marked as failed                                |
        | **Fail Rollback**              | Radio  | Yes      | Enable  | Enable: delete resources on failure. Disable: keep resources on failure |

        Below these fields, **dynamic parameter fields** appear based on the template's
        `parameters` section. Each template parameter becomes a form field with:

        * Type mapped from template: `string` → text input, `number` → number input,
          `json` → JSON input, `boolean` → Yes/No radio
        * Default values pre-populated from template
        * Description shown as help text

        Click **Confirm** to create the stack.

        <Check>The stack appears in the list with status **Create In Progress**.</Check>
      </Step>

      <Step title="Monitor progress">
        Click the stack name to open the detail view. Four tabs are available:

        * **Detail** — Startup parameters (timeout, rollback), outputs, deployment parameters
        * **Stack Resources** — List of provisioned resources with links to their detail pages
        * **Stack Events** — Real-time provisioning events with timestamps and status
        * **YAML File** — Read-only view of the template YAML

        <Check>
          Status transitions to **Create Complete** when all resources are successfully
          provisioned.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Source credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Validate the template" icon="circle-check">
        ```bash title="Validate template syntax" theme={null}
        openstack orchestration template validate -t first-stack.yaml
        ```

        A valid template returns the parsed parameter and resource definitions.
        Fix any reported errors before proceeding.
      </Step>

      <Step title="Create the stack" icon="play">
        ```bash title="Create stack with parameters" theme={null}
        openstack stack create \
          --template first-stack.yaml \
          --parameter key_name=my-keypair \
          --parameter flavor=m1.small \
          --parameter network=default \
          --wait \
          my-first-stack
        ```

        The `--wait` flag blocks until the stack reaches a terminal state.

        | Flag          | Description                                                    |
        | ------------- | -------------------------------------------------------------- |
        | `--template`  | Path to local template file (or `--template-url` for remote)   |
        | `--parameter` | Override a template parameter (repeat for each parameter)      |
        | `--wait`      | Wait for stack creation to complete before returning           |
        | `--timeout`   | Maximum minutes to wait before marking as failed (default: 60) |
      </Step>

      <Step title="Verify stack status" icon="circle-check">
        ```bash title="Show stack details" theme={null}
        openstack stack show my-first-stack
        ```

        ```bash title="List stack resources" theme={null}
        openstack stack resource list my-first-stack
        ```

        ```bash title="Show stack outputs" theme={null}
        openstack stack output list my-first-stack
        openstack stack output show my-first-stack instance_ip
        ```

        <Check>
          `stack_status` shows `CREATE_COMPLETE`. Every resource in the list displays
          `CREATE_COMPLETE`.
        </Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## List and Inspect Stacks

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Orchestration > Stacks** to see all stacks in your project.
    Click any stack to view its detail, resources, events, and template.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all stacks" theme={null}
    openstack stack list
    ```

    ```bash title="Show stack detail" theme={null}
    openstack stack show my-first-stack
    ```

    ```bash title="List stack events" theme={null}
    openstack stack event list my-first-stack
    ```
  </Tab>
</Tabs>

***

## Delete a Stack

<Warning>
  Deleting a stack permanently deletes all resources it manages. This includes instances,
  volumes, networks, and any other resources created by the template. This operation
  cannot be undone.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    In the Stacks list, check the box next to your stack and click **Delete Stacks**.
    Confirm the deletion in the dialog.

    <Check>Stack status transitions to **Delete In Progress**, then disappears from the list.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a stack" theme={null}
    openstack stack delete --yes --wait my-first-stack
    ```

    <Check>`openstack stack list` no longer shows the deleted stack.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Template Guide" href="/services/orchestration/template-guide">
    Learn template structure, parameter types, and intrinsic functions
  </Card>

  <Card title="Resource Types" href="/services/orchestration/resources">
    Explore compute, network, storage, and identity resource definitions
  </Card>

  <Card title="Manage Stacks" href="/services/orchestration/stacks">
    Update, suspend, resume, and manage the stack lifecycle
  </Card>

  <Card title="Auto-Scaling" href="/services/orchestration/autoscaling">
    Scale instance groups automatically with alarm-driven policies
  </Card>
</CardGroup>


# Orchestration
Source: https://docs.xloud.tech/services/orchestration/index

Template-based infrastructure orchestration for Xloud Cloud Platform. Define, deploy, and manage cloud resources as code using declarative orchestration.

Define and deploy cloud infrastructure as code using declarative templates.
Xloud Orchestration enables repeatable, version-controlled provisioning of entire
application stacks — compute, networking, storage, and scaling policies — from a
single template file.

<Card title="XAVS — Advanced Virtualization Solutions" icon="external-link" href="https://xloud.tech/xavs">
  Product details on xloud.tech
</Card>

***

<p>Orchestration</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/orchestration/user-guide">
    Create stacks, manage resources, configure auto-scaling, and work with
    orchestration templates for your infrastructure.
  </Card>

  <Card title="Admin Guide" icon="shield-check" href="/services/orchestration/admin-guide">
    Configure the Orchestration service, manage the stack domain, tune
    performance, and secure template-based deployments.
  </Card>

  <Card title="Template Guide" icon="file-code" href="/services/orchestration/template-guide">
    Deep reference for template structure, parameter types, resource
    definitions, intrinsic functions, and conditions.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/orchestration/cli-reference">
    Command-line operations for stack create, update, list, show, delete,
    and template validation.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Template-Based Deployment" icon="file-code" href="/services/orchestration/template-guide">
    Describe your entire infrastructure in a single declarative template.
    Provision networks, instances, volumes, and policies with one API call.
  </Card>

  <Card title="Auto-Scaling" icon="trending-up" href="/services/orchestration/autoscaling">
    Define scaling groups and alarm-driven policies to automatically grow or
    shrink instance pools in response to real-time demand.
  </Card>

  <Card title="Stack Composition" icon="layers" href="/services/orchestration/stacks">
    Compose large deployments from smaller, reusable nested stacks. Share
    templates across projects and environments.
  </Card>

  <Card title="Resource Management" icon="boxes" href="/services/orchestration/resources">
    Manage the full lifecycle of every resource in a stack as a unit — create,
    update, rollback, suspend, resume, and delete together.
  </Card>

  <Card title="High Availability Support" icon="heart-pulse" href="/services/instance-ha/index">
    Model multi-zone, load-balanced, and auto-recovering application topologies
    directly in your orchestration templates.
  </Card>

  <Card title="Plugin Extensibility" icon="puzzle" href="/services/orchestration/configuration">
    Over 100 built-in resource types covering compute, networking, storage,
    identity, and orchestration primitives. Custom plugins supported.
  </Card>
</CardGroup>

***

<p>How It Works</p>

```mermaid theme={null}
graph TD
    U([User / Dashboard]) -->|Submit template| API[Orchestration API<br/>:8004]
    API -->|Parse & validate| ENG[Orchestration Engine]
    ENG -->|Resolve dependencies| DEPS[Dependency Graph]
    DEPS -->|Provision resources| COMPUTE[Xloud Compute]
    DEPS -->|Provision resources| NET[Xloud Networking]
    DEPS -->|Provision resources| STORE[Xloud Block Storage]
    DEPS -->|Provision resources| ID[Xloud Identity]
    COMPUTE & NET & STORE & ID -->|Report status| ENG
    ENG -->|Stack status| API
    API -->|Stack output| U
    style API fill:#197560,color:#fff
    style ENG fill:#197560,color:#fff
    style DEPS fill:#3F8F7E,color:#fff
```

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Virtual machine instances provisioned and managed by orchestration stacks
  </Card>

  <Card title="Xloud Block Storage" icon="hard-drive" href="/services/storage/index">
    Persistent volumes created and attached through orchestration templates
  </Card>

  <Card title="Xloud Networking" icon="network" href="/services/networking/index">
    Networks, subnets, routers, and floating IPs defined in stack templates
  </Card>

  <Card title="Xloud Load Balancer" icon="combine" href="/services/load-balancer">
    Load balancer resources for auto-scaling groups in orchestration stacks
  </Card>
</CardGroup>


# Resource Types
Source: https://docs.xloud.tech/services/orchestration/resources

Reference for all Xloud Orchestration resource types — compute, networking, storage, identity, and orchestration primitives with definition examples.

## Overview

Xloud Orchestration supports over 100 built-in resource types covering the full range
of cloud infrastructure. Each resource type maps to a specific cloud service API. The
Orchestration engine provisions resources in dependency order and tracks their lifecycle
as part of the parent stack.

<Note>
  Resource type names use the `Xloud::Service::ResourceType` convention. Available
  resource types depend on the services enabled in your cluster.
</Note>

***

## Resource Categories

| Category          | Service             | Common Resource Types                                                 |
| ----------------- | ------------------- | --------------------------------------------------------------------- |
| **Compute**       | Xloud Compute       | Server, KeyPair, ServerGroup                                          |
| **Networking**    | Xloud Networking    | Net, Subnet, Router, RouterInterface, FloatingIP, SecurityGroup, Port |
| **Storage**       | Xloud Block Storage | Volume, VolumeAttachment                                              |
| **Identity**      | Xloud Identity      | Project, User, Role                                                   |
| **Orchestration** | Xloud Orchestration | Stack, WaitCondition, AutoScalingGroup, ScalingPolicy                 |

***

## Resource Definition Structure

Every resource follows this structure:

```yaml title="resource-structure.yaml" theme={null}
resources:
  logical_resource_name:
    type: Xloud::Service::ResourceType    # Required
    depends_on:                           # Optional — explicit ordering
      - other_resource_name
    condition: my_condition               # Optional — conditional creation
    deletion_policy: Delete               # Optional — Retain | Snapshot | Delete
    properties:                           # Required — resource configuration
      property_name: value
      property_from_param: { get_param: param_name }
      property_from_resource: { get_resource: other_resource }
    metadata:                             # Optional — arbitrary key-value data
      custom_key: custom_value
```

### `depends_on` and Dependency Resolution

The engine automatically detects dependencies from intrinsic functions like
`get_resource` and `get_attr`. Use `depends_on` only for ordering requirements
that cannot be expressed through property references.

```yaml title="explicit-dependency.yaml" theme={null}
resources:
  network:
    type: Xloud::Networking::Net
    properties:
      name: app-network

  subnet:
    type: Xloud::Networking::Subnet
    depends_on: [network]               # Wait for network before creating subnet
    properties:
      network_id: { get_resource: network }
      cidr: "10.0.1.0/24"
```

***

## Resource Examples by Category

<AccordionGroup>
  <Accordion title="Compute Resources" icon="server">
    ### Xloud::Compute::Server

    ```yaml title="compute-server.yaml" theme={null}
    resources:
      web_server:
        type: Xloud::Compute::Server
        properties:
          name: web-server
          image: { get_param: image }
          flavor: { get_param: flavor }
          key_name: { get_param: key_name }
          security_groups:
            - { get_resource: web_sg }
          networks:
            - network: { get_param: network }
          user_data_format: RAW
          user_data: |
            #!/bin/bash
            apt-get install -y nginx
            systemctl enable --now nginx
    ```

    ### Xloud::Compute::KeyPair

    ```yaml title="keypair.yaml" theme={null}
    resources:
      deploy_key:
        type: Xloud::Compute::KeyPair
        properties:
          name: deploy-key
          save_private_key: true   # Private key returned as stack output

    outputs:
      private_key:
        value: { get_attr: [deploy_key, private_key] }
    ```

    ### Xloud::Compute::ServerGroup

    ```yaml title="server-group.yaml" theme={null}
    resources:
      anti_affinity_group:
        type: Xloud::Compute::ServerGroup
        properties:
          name: web-anti-affinity
          policies:
            - anti-affinity     # Spread instances across different hosts
    ```
  </Accordion>

  <Accordion title="Networking Resources" icon="network">
    ### Xloud::Networking::Net

    ```yaml title="network.yaml" theme={null}
    resources:
      app_network:
        type: Xloud::Networking::Net
        properties:
          name: app-network
          admin_state_up: true
    ```

    ### Xloud::Networking::Subnet

    ```yaml title="subnet.yaml" theme={null}
    resources:
      app_subnet:
        type: Xloud::Networking::Subnet
        properties:
          name: app-subnet
          network_id: { get_resource: app_network }
          cidr: "192.168.10.0/24"
          ip_version: 4
          dns_nameservers:
            - "8.8.8.8"
            - "1.1.1.1"
          enable_dhcp: true
    ```

    ### Xloud::Networking::Router and RouterInterface

    ```yaml title="router.yaml" theme={null}
    resources:
      router:
        type: Xloud::Networking::Router
        properties:
          name: app-router
          external_gateway_info:
            network: { get_param: external_network }

      router_interface:
        type: Xloud::Networking::RouterInterface
        properties:
          router_id: { get_resource: router }
          subnet_id: { get_resource: app_subnet }
    ```

    ### Xloud::Networking::FloatingIP

    ```yaml title="floating-ip.yaml" theme={null}
    resources:
      floating_ip:
        type: Xloud::Networking::FloatingIP
        properties:
          floating_network: { get_param: external_network }

      floating_ip_assoc:
        type: Xloud::Networking::FloatingIPAssociation
        properties:
          floatingip_id: { get_resource: floating_ip }
          port_id: { get_attr: [web_server, addresses, app-network, 0, port] }
    ```

    ### Xloud::Networking::SecurityGroup

    ```yaml title="security-group.yaml" theme={null}
    resources:
      web_sg:
        type: Xloud::Networking::SecurityGroup
        properties:
          name: web-security-group
          description: Allow HTTP, HTTPS, and SSH
          rules:
            - protocol: tcp
              port_range_min: 22
              port_range_max: 22
              remote_ip_prefix: "0.0.0.0/0"
            - protocol: tcp
              port_range_min: 80
              port_range_max: 80
              remote_ip_prefix: "0.0.0.0/0"
            - protocol: tcp
              port_range_min: 443
              port_range_max: 443
              remote_ip_prefix: "0.0.0.0/0"
    ```
  </Accordion>

  <Accordion title="Storage Resources" icon="hard-drive">
    ### Xloud::BlockStorage::Volume

    ```yaml title="volume.yaml" theme={null}
    resources:
      data_volume:
        type: Xloud::BlockStorage::Volume
        properties:
          name: app-data
          size: 100          # GB
          volume_type: ceph-ssd
          description: Application data volume
    ```

    ### Xloud::BlockStorage::VolumeAttachment

    ```yaml title="volume-attachment.yaml" theme={null}
    resources:
      data_volume_attach:
        type: Xloud::BlockStorage::VolumeAttachment
        properties:
          instance_uuid: { get_resource: web_server }
          volume_id: { get_resource: data_volume }
          mountpoint: /dev/vdb
    ```
  </Accordion>

  <Accordion title="Identity Resources" icon="fingerprint">
    ### Xloud::Identity::Project

    ```yaml title="project.yaml" theme={null}
    resources:
      app_project:
        type: Xloud::Identity::Project
        properties:
          name: my-app-project
          description: Project for the application team
          enabled: true
    ```

    ### Xloud::Identity::User

    ```yaml title="user.yaml" theme={null}
    resources:
      svc_user:
        type: Xloud::Identity::User
        properties:
          name: svc-deployer
          password: { get_param: svc_password }
          email: svc-deployer@example.com
          enabled: true
    ```
  </Accordion>

  <Accordion title="Orchestration Resources" icon="workflow">
    ### Xloud::Orchestration::Stack (Nested Stack)

    ```yaml title="nested-stack.yaml" theme={null}
    resources:
      database_tier:
        type: Xloud::Orchestration::Stack
        properties:
          template: { get_file: database-stack.yaml }
          parameters:
            flavor: { get_param: db_flavor }
            network: { get_resource: app_network }
    ```

    ### Xloud::Orchestration::WaitCondition

    Use `WaitCondition` to pause stack creation until an external signal is received
    (e.g., after cloud-init completes application installation):

    ```yaml title="wait-condition.yaml" theme={null}
    resources:
      wait_handle:
        type: Xloud::Orchestration::WaitConditionHandle

      app_ready:
        type: Xloud::Orchestration::WaitCondition
        depends_on: [app_server]
        properties:
          handle: { get_resource: wait_handle }
          timeout: 300          # Seconds to wait for signal
          count: 1

      app_server:
        type: Xloud::Compute::Server
        properties:
          user_data:
            str_replace:
              template: |
                #!/bin/bash
                apt-get install -y myapp
                # Signal completion
                curl -X POST "$WAIT_URL" -d '{"Status": "SUCCESS", "UniqueId": "app"}'
              params:
                $WAIT_URL: { get_resource: wait_handle }
    ```
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Template Guide" href="/services/orchestration/template-guide">
    Template structure, parameters, intrinsic functions, and conditions
  </Card>

  <Card title="Auto-Scaling" href="/services/orchestration/autoscaling">
    Auto-scaling group and scaling policy resource examples
  </Card>

  <Card title="Manage Stacks" href="/services/orchestration/stacks">
    Create, update, and manage the stack lifecycle
  </Card>

  <Card title="Troubleshooting" href="/services/orchestration/troubleshooting">
    Diagnose resource provisioning failures and dependency errors
  </Card>
</CardGroup>


# Scaling the Orchestration Service
Source: https://docs.xloud.tech/services/orchestration/scaling

Deploy multiple Xloud Orchestration engine workers, configure convergence mode, and tune performance settings for high-throughput template deployments.

## Overview

The Orchestration engine is a horizontally scalable service. Multiple engine worker
processes can run on a single controller node or across multiple controller nodes,
sharing work through a message queue. In convergence mode, the engine distributes
individual resource operations across all available workers, enabling parallel
provisioning of independent resources within a single stack.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Engine Worker Architecture

```mermaid theme={null}
graph TD
    API[Orchestration API<br/>:8004] -->|Publish task| MQ[Message Queue<br/>RabbitMQ]
    MQ -->|Consume task| W1[Engine Worker 1]
    MQ -->|Consume task| W2[Engine Worker 2]
    MQ -->|Consume task| W3[Engine Worker N]
    W1 & W2 & W3 -->|Read/write state| DB[(Database)]
    W1 -->|Provision resource| CLOUD[Cloud Service APIs]
    W2 -->|Provision resource| CLOUD
    W3 -->|Provision resource| CLOUD
    style API fill:#197560,color:#fff
    style MQ fill:#3F8F7E,color:#fff
```

***

## Worker Configuration

### Engine Workers Per Node

The `heat_engine_workers` setting controls how many engine worker processes run on
each controller node. Each worker is an independent process that picks up tasks from
the message queue.

<Tabs>
  <Tab title="XDeploy" icon="gauge">
    In XDeploy, navigate to **Configuration → Services → Orchestration** and adjust
    the **Engine Workers** slider. A common starting point is 2–4 workers per CPU core
    available on the controller node.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Set engine workers in globals" theme={null}
    # Edit /etc/xavs/globals.d/_50_orchestration.yml
    heat_engine_workers: 8      # Adjust based on controller CPU count
    heat_api_workers: 4
    ```

    ```bash title="Deploy to apply changes" theme={null}
    xavs-ansible deploy -t heat
    ```
  </Tab>
</Tabs>

### Worker Sizing Guidelines

| Controller vCPU Count | Recommended Engine Workers | Recommended API Workers |
| --------------------- | -------------------------- | ----------------------- |
| 4 vCPUs               | 2                          | 2                       |
| 8 vCPUs               | 4                          | 4                       |
| 16 vCPUs              | 8                          | 4                       |
| 32+ vCPUs             | 16                         | 8                       |

<Tip>
  Engine workers are I/O bound (waiting on cloud service API calls), not CPU bound.
  You can safely set `heat_engine_workers` higher than the physical CPU count.
  Monitor the message queue depth in XIMP to identify bottlenecks.
</Tip>

***

## Convergence Mode

Convergence mode enables the engine to process independent resources in a stack
concurrently across all available workers. This significantly reduces total stack
creation time for large templates.

| Mode                      | Behavior                                                | Best For                                                      |
| ------------------------- | ------------------------------------------------------- | ------------------------------------------------------------- |
| **Convergence** (default) | Resources are processed in parallel by multiple workers | Large stacks (50+ resources), independent resource graphs     |
| **Non-convergence**       | Resources processed sequentially by a single worker     | Simple stacks, environments where strict ordering is required |

```yaml title="Enable convergence in globals" theme={null}
heat_convergence_engine: true   # Default — recommended for production
```

<Note>
  Convergence mode requires the database to be accessible from all engine workers
  simultaneously. Ensure your database connection pool is sized appropriately:
  `heat_db_max_pool_size` should be at least `heat_engine_workers * 2`.
</Note>

***

## Performance Tuning

| Setting                        | Default | Description                                     |
| ------------------------------ | ------- | ----------------------------------------------- |
| `heat_engine_workers`          | `4`     | Engine worker processes per controller node     |
| `heat_api_workers`             | `4`     | API worker processes per controller node        |
| `heat_db_max_pool_size`        | `10`    | Max database connections per worker             |
| `heat_db_max_overflow`         | `20`    | Max overflow connections above the pool         |
| `heat_rpc_response_timeout`    | `120`   | Seconds before an RPC call is considered failed |
| `heat_max_stacks_per_tenant`   | `100`   | Per-project stack limit                         |
| `heat_max_resources_per_stack` | `1000`  | Per-stack resource limit                        |

***

## Monitoring Engine Health

```bash title="List active engine services" theme={null}
openstack orchestration service list
```

Expected output — all services show `status: up`:

```
+----------+-----------+------+--------+-------------------+-----+
| Hostname | Binary    | Port | Status | Updated At        | ... |
+----------+-----------+------+--------+-------------------+-----+
| ctrl-01  | heat-eng  | 0    | up     | 2026-03-18T10:00Z | ... |
| ctrl-01  | heat-api  | 8004 | up     | 2026-03-18T10:00Z | ... |
+----------+-----------+------+--------+-------------------+-----+
```

<Tip>
  Set up an alert in XIMP for `orchestration_engine_up == 0` to detect engine worker
  failures before they affect user deployments.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Configuration" href="/services/orchestration/configuration">
    Full configuration reference for the Orchestration service
  </Card>

  <Card title="Security" href="/services/orchestration/security">
    Trust-based authorization and policy configuration
  </Card>

  <Card title="Architecture" href="/services/orchestration/architecture">
    Engine internals and dependency resolution design
  </Card>

  <Card title="Admin Troubleshooting" href="/services/orchestration/admin-troubleshooting">
    Resolve engine worker failures and performance degradation
  </Card>
</CardGroup>


# Orchestration Security
Source: https://docs.xloud.tech/services/orchestration/security

Secure the Xloud Orchestration service — stack domain users, trust-based authorization, policy configuration, and template injection prevention.

## Overview

Xloud Orchestration introduces unique security considerations beyond standard service
policies. Templates can create users, assign roles, and invoke webhooks on behalf of
the submitting user — making trust delegation and template validation critical security
controls. This page covers the stack domain model, trust-based authorization,
policy configuration, and template injection prevention.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Stack Domain Users

### Why a Separate Domain Is Needed

When a stack contains resources that require long-running credentials — such as
`WaitCondition` signal URLs, auto-scaling webhooks, or software deployment agents —
the Orchestration engine cannot use the submitting user's token (which expires).
Instead, it creates a short-lived stack domain user scoped to the stack's project.

```mermaid theme={null}
graph TD
    U[User] -->|Submit stack| API[Orchestration API]
    API -->|Create trust| TRUST[Trust in Xloud Identity]
    API -->|Create domain user| DU[Stack Domain User<br/>heat-domain]
    STACK[Stack Resources] -->|Signal via webhook| API
    API -->|Validate using| DU
    DU -->|Scoped to project| PROJECT[User's Project]
    style TRUST fill:#197560,color:#fff
    style DU fill:#3F8F7E,color:#fff
```

### Domain User Lifecycle

| Event              | Action                                                                                |
| ------------------ | ------------------------------------------------------------------------------------- |
| Stack created      | Stack domain user created in the `heat` domain, scoped to the stack project           |
| Stack deleted      | Stack domain user is deleted automatically                                            |
| User token expires | Domain user credentials are refreshed automatically — the stack continues to function |

***

## Trust-Based Authorization

Xloud Orchestration uses Xloud Identity <Tooltip>trusts</Tooltip>
to delegate the submitting user's permissions to the engine for resource provisioning.

### How Trusts Work

1. When a stack is submitted, the engine requests a trust from Xloud Identity.
2. The trust grants the engine the submitting user's roles within the stack's project.
3. The engine uses the trust to authenticate when calling compute, networking, and
   storage APIs on behalf of the stack.
4. The trust is tied to the stack — deleting the stack revokes the trust.

### Reviewing Active Trusts

```bash title="List trusts for the current user" theme={null}
openstack trust list
```

```bash title="Show trust detail" theme={null}
openstack trust show <TRUST_ID>
```

<Warning>
  Users who are removed from a project while their stacks are still running will have
  their trusts invalidated. The Orchestration engine will fail to provision new resources
  for those stacks until the user is re-added or the stacks are re-created by a valid
  project member.
</Warning>

***

## Policy Configuration

Orchestration API access is governed by policies defined in `policy.yaml`. The default
policy restricts stack management to project members and administration to users with
the `admin` role.

### Default Policy Summary

| Operation                   | Default Policy                                  |
| --------------------------- | ----------------------------------------------- |
| Create stack                | `rule:project_member` — any project member      |
| Update stack                | `rule:project_member` — stack owner or admin    |
| Delete stack                | `rule:project_member` — stack owner or admin    |
| List stacks (all projects)  | `rule:admin_required` — admin only              |
| Show stack in other project | `rule:admin_required` — admin only              |
| Abandon stack               | `rule:admin_required` — admin only              |
| Validate template           | `rule:deny_stack_user` — not stack domain users |

### Overriding Policies

```yaml title="policy.yaml override example" theme={null}
# Allow project readers to list stacks (not just members)
stacks:list:
  rules:
    - project_reader

# Prevent non-admin users from creating nested stacks
stacks:create_with_nested:
  rules:
    - admin_required
```

Policies are applied via XDeploy configuration overrides in
`/etc/xavs/orchestration/policy.yaml`.

***

## Template Injection Prevention

<Danger>
  Orchestration templates are powerful — a malicious or misconfigured template can
  create users, assign roles, consume large quota, and trigger webhooks to external
  systems. Never execute untrusted templates from unknown sources.
</Danger>

### Security Controls for Template Execution

| Control                    | Description                                                                                                                  |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **Policy enforcement**     | Templates execute with the submitting user's roles. A project member cannot create resources outside their quota or project. |
| **Quota limits**           | `heat_max_stacks_per_tenant` and `heat_max_resources_per_stack` prevent runaway resource creation.                           |
| **Template validation**    | Run `openstack orchestration template validate` before deploying untrusted templates to inspect all declared resources.      |
| **Allowed resource types** | Configure `allowed_resources` in the engine configuration to restrict which resource types can be used.                      |
| **Restricted properties**  | `restricted_metadata_keys` prevents templates from setting arbitrary instance metadata.                                      |

### Review Template Before Execution

```bash title="Validate and inspect a template" theme={null}
openstack orchestration template validate \
  --show-nested \
  -t untrusted-template.yaml
```

The output lists every resource type the template will attempt to create. Verify that
all resource types are expected before deploying.

<Tip>
  For multi-tenant environments, configure `allowed_resources` in XDeploy to restrict
  the resource types available to non-admin users. This prevents project members from
  creating identity resources (users, roles) through templates.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Configuration" href="/services/orchestration/configuration">
    Configure the stack domain, quotas, and service settings
  </Card>

  <Card title="Architecture" href="/services/orchestration/architecture">
    Understand the trust delegation and engine processing flow
  </Card>

  <Card title="Admin Troubleshooting" href="/services/orchestration/admin-troubleshooting">
    Resolve stack domain and authorization failures
  </Card>

  <Card title="Xloud Identity" href="/services/identity/index">
    Manage domains, trusts, and role assignments in Xloud Identity
  </Card>
</CardGroup>


# Manage Stacks
Source: https://docs.xloud.tech/services/orchestration/stacks

Create, update, suspend, resume, and delete Xloud Orchestration stacks. Covers stack outputs, nested stacks, and update behavior through the Dashboard and CLI.

## Overview

Xloud Orchestration manages the complete lifecycle of infrastructure stacks. After initial
creation, a stack can be updated with a revised template or new parameter values, suspended
to release compute and network resources, resumed when needed, and eventually deleted to
clean up all associated resources in a single operation.

<Note>
  **Prerequisites**

  * A deployed stack (see [Create Your First Stack](/services/orchestration/getting-started))
  * `member` or `admin` role in your project
</Note>

***

## Stack Lifecycle

```mermaid theme={null}
stateDiagram-v2
    [*] --> CREATE_IN_PROGRESS
    CREATE_IN_PROGRESS --> CREATE_COMPLETE
    CREATE_IN_PROGRESS --> CREATE_FAILED

    CREATE_COMPLETE --> UPDATE_IN_PROGRESS
    UPDATE_IN_PROGRESS --> UPDATE_COMPLETE
    UPDATE_IN_PROGRESS --> UPDATE_FAILED

    CREATE_COMPLETE --> SUSPEND_IN_PROGRESS
    UPDATE_COMPLETE --> SUSPEND_IN_PROGRESS
    SUSPEND_IN_PROGRESS --> SUSPEND_COMPLETE

    SUSPEND_COMPLETE --> RESUME_IN_PROGRESS
    RESUME_IN_PROGRESS --> RESUME_COMPLETE

    CREATE_COMPLETE --> DELETE_IN_PROGRESS
    UPDATE_COMPLETE --> DELETE_IN_PROGRESS
    SUSPEND_COMPLETE --> DELETE_IN_PROGRESS
    DELETE_IN_PROGRESS --> DELETE_COMPLETE
    DELETE_IN_PROGRESS --> DELETE_FAILED
    DELETE_COMPLETE --> [*]
```

***

## Update a Stack

Stack updates apply changes to an existing stack using a revised template or updated
parameter values. The engine computes a diff between the current and desired state,
then creates, updates, or deletes resources as needed.

<Warning>
  Stack updates can cause resource replacement. Resources that do not support in-place
  updates are deleted and recreated, which may cause downtime. Review the planned
  changes using `--dry-run` before applying to production stacks.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the stack" icon="layers">
        Navigate to **Orchestration > Stacks** and click the stack name to
        open its detail view.
      </Step>

      <Step title="Update the template">
        Click the **More** dropdown on the stack row and select **Update Template**.
        The same 2-step wizard opens, pre-populated with the current template and
        parameter values. The stack name is read-only.
      </Step>

      <Step title="Apply changes">
        Modify the template or parameter values, then click **Confirm**.

        <Check>
          Stack status transitions to **Update In Progress**, then **Update Complete**
          when all changes are applied.
        </Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Update stack with revised template" theme={null}
    openstack stack update \
      --template updated-template.yaml \
      --parameter flavor=m1.large \
      --wait \
      my-stack
    ```

    ```bash title="Preview update changes (dry run)" theme={null}
    openstack stack update \
      --template updated-template.yaml \
      --dry-run \
      my-stack
    ```

    ```bash title="Update only a parameter value (no template change)" theme={null}
    openstack stack update \
      --existing \
      --parameter flavor=m1.large \
      --wait \
      my-stack
    ```

    | Flag         | Description                                                  |
    | ------------ | ------------------------------------------------------------ |
    | `--existing` | Reuse the current template; only update specified parameters |
    | `--dry-run`  | Preview planned resource changes without executing them      |
    | `--rollback` | Automatically roll back if the update fails                  |
    | `--wait`     | Block until the update reaches a terminal state              |
  </Tab>
</Tabs>

***

## Suspend and Resume a Stack

Suspending a stack stops all running instances and releases compute resources while
preserving the stack definition for later resumption.

<Note>
  Suspend and Resume are **CLI-only** operations. They are not available in the
  Dashboard.
</Note>

<CodeGroup>
  ```bash title="Suspend a stack" theme={null}
  openstack stack suspend --wait my-stack
  ```

  ```bash title="Resume a suspended stack" theme={null}
  openstack stack resume --wait my-stack
  ```
</CodeGroup>

***

## Stack Outputs

Outputs expose values produced by the stack — IP addresses, resource IDs, URLs, and
other runtime data.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Open the stack detail page and select the **Detail** tab (Outputs card). All defined outputs
    and their current values are listed.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all outputs" theme={null}
    openstack stack output list my-stack
    ```

    ```bash title="Show a specific output" theme={null}
    openstack stack output show my-stack instance_ip
    ```

    ```bash title="Show output in JSON" theme={null}
    openstack stack output show my-stack instance_ip -f json
    ```
  </Tab>
</Tabs>

***

## Nested Stacks

A nested stack is a stack created as a resource within a parent stack using the
`Xloud::Orchestration::Stack` resource type. Nesting enables modular template design
— common patterns such as networking tiers, database clusters, or load balancer
configurations can be extracted into reusable child templates.

```yaml title="parent-stack-with-nested.yaml" theme={null}
resources:

  network_tier:
    type: Xloud::Orchestration::Stack
    properties:
      template: { get_file: network-template.yaml }
      parameters:
        cidr: "10.0.0.0/24"
        external_network: { get_param: external_network }

  app_tier:
    type: Xloud::Orchestration::Stack
    depends_on: [network_tier]
    properties:
      template: { get_file: app-template.yaml }
      parameters:
        network: { get_attr: [network_tier, outputs, network_id] }
        flavor: { get_param: app_flavor }
```

<Note>
  Nested stacks appear as child stacks in the Dashboard under the parent stack's
  **Stack Resources** tab. Each child stack has its own resource list, events, and outputs.
</Note>

***

## Delete a Stack

<Danger>
  Deleting a stack permanently destroys all resources it manages — including instances,
  volumes marked for deletion, networks, and floating IPs. Resources with
  `deletion_policy: Retain` in the template are preserved.
</Danger>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    In the Stacks list, click **Delete** (the first row action) on the stack row.
    Confirm the operation in the dialog. You can also select multiple stacks using
    checkboxes and click **Delete** in the batch actions bar.

    The **More** dropdown also offers **Abandon Stack** — this removes the stack
    record but preserves the deployed resources (unlike Delete which removes
    everything).

    <Check>Stack transitions to **Delete In Progress** and then disappears from the list.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a stack" theme={null}
    openstack stack delete --yes --wait my-stack
    ```

    ```bash title="Delete multiple stacks" theme={null}
    openstack stack delete --yes stack-1 stack-2 stack-3
    ```

    <Check>`openstack stack list` no longer shows the deleted stacks.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Auto-Scaling" href="/services/orchestration/autoscaling">
    Create auto-scaling groups and alarm-driven scaling policies
  </Card>

  <Card title="Template Guide" href="/services/orchestration/template-guide">
    Author templates with parameters, conditions, and intrinsic functions
  </Card>

  <Card title="Resource Types" href="/services/orchestration/resources">
    Full reference for all supported resource types and their properties
  </Card>

  <Card title="Troubleshooting" href="/services/orchestration/troubleshooting">
    Resolve stack update failures, rollback issues, and dependency errors
  </Card>
</CardGroup>


# Orchestration Template Guide
Source: https://docs.xloud.tech/services/orchestration/template-guide

Write Heat templates to define cloud infrastructure as code. Covers resource types, parameters, and outputs.

## Overview

An orchestration template is a YAML document that declaratively describes a set of
cloud resources and their relationships. The Orchestration engine reads the template,
resolves dependencies between resources, and provisions them in the correct order.
Templates are reusable — parameterized values allow the same template to deploy
different environments without modification.

<Note>
  **Prerequisites**

  * Familiarity with YAML syntax
  * Access to the Xloud Orchestration service in your project
  * `xloud` CLI installed for template validation
</Note>

***

## Template Structure

Every orchestration template has up to six top-level sections:

```yaml title="template-skeleton.yaml" theme={null}
xloud_template_version: "2025-10-15"   # Required — template format version

description: >                          # Optional — human-readable description
  Brief summary of what this template deploys.

parameter_groups:                       # Optional — group parameters for the UI
  - label: Instance Settings
    parameters: [instance_name, flavor]

parameters:                             # Optional — runtime variables
  instance_name:
    type: string
    default: my-instance

conditions:                             # Optional — conditional resource creation
  create_extra_volume:
    equals: [{ get_param: env }, production]

resources:                              # Required — cloud objects to create
  my_instance:
    type: Xloud::Compute::Server
    properties:
      name: { get_param: instance_name }

outputs:                                # Optional — values returned after creation
  server_ip:
    description: The instance IP address
    value: { get_attr: [my_instance, first_address] }
```

| Section                  | Required | Purpose                                                             |
| ------------------------ | -------- | ------------------------------------------------------------------- |
| `xloud_template_version` | Yes      | Specifies the template format version                               |
| `description`            | No       | Human-readable summary shown in the Dashboard                       |
| `parameter_groups`       | No       | Groups parameters into labeled sections in the Dashboard UI         |
| `parameters`             | No       | Defines input variables that customize the template at runtime      |
| `conditions`             | No       | Named boolean expressions controlling conditional resource creation |
| `resources`              | Yes      | The cloud resources to create, update, or delete                    |
| `outputs`                | No       | Values extracted from created resources and returned to the caller  |

***

## Parameters

Parameters make templates reusable across different environments and projects.

### Parameter Types

| Type                   | Description                      | Example Value      |
| ---------------------- | -------------------------------- | ------------------ |
| `string`               | Arbitrary text value             | `"m1.large"`       |
| `number`               | Integer or float                 | `3`                |
| `boolean`              | `true` or `false`                | `true`             |
| `json`                 | Arbitrary JSON object or array   | `{"key": "value"}` |
| `comma_delimited_list` | CSV string interpreted as a list | `"az1,az2,az3"`    |

### Parameter Definition Reference

```yaml title="parameters-reference.yaml" theme={null}
parameters:

  flavor:
    type: string
    label: Instance Flavor
    description: The compute flavor (vCPU and RAM profile) for the instance
    default: m1.small
    constraints:
      - allowed_values: [m1.small, m1.medium, m1.large]
        description: Must be a supported flavor

  instance_count:
    type: number
    label: Instance Count
    description: Number of instances to launch in the scaling group
    default: 2
    constraints:
      - range: { min: 1, max: 20 }
        description: Between 1 and 20 instances

  enable_monitoring:
    type: boolean
    label: Enable Monitoring
    description: Whether to install the monitoring agent on boot
    default: false

  tags:
    type: json
    label: Instance Tags
    description: Key-value metadata to attach to each instance
    default: { "env": "dev", "team": "platform" }
```

<Tip>
  Validate parameters before deploying: run `openstack orchestration template validate -t your-template.yaml`
  to catch type mismatches and constraint violations before the stack is submitted.
</Tip>

***

## Resources

Resources are the core of every template. Each resource has a logical name, a type,
and a set of properties.

```yaml title="resource-definition.yaml" theme={null}
resources:

  web_server:
    type: Xloud::Compute::Server
    depends_on: [web_network_port]        # Explicit dependency
    deletion_policy: Retain               # Retain | Snapshot | Delete (default)
    update_policy:
      rolling_update:
        max_batch_size: 1
        pause_time: PT30S
    properties:
      name: web-server-01
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - port: { get_resource: web_network_port }
      user_data: |
        #!/bin/bash
        apt-get install -y nginx
```

***

## Intrinsic Functions

Intrinsic functions allow template sections to reference other parts of the template
dynamically at runtime.

| Function       | Syntax                                                | Description                                     |
| -------------- | ----------------------------------------------------- | ----------------------------------------------- |
| `get_param`    | `{ get_param: param_name }`                           | Returns the value of a template parameter       |
| `get_resource` | `{ get_resource: resource_name }`                     | Returns the ID of another resource in the stack |
| `get_attr`     | `{ get_attr: [resource_name, attribute] }`            | Returns a specific attribute of a resource      |
| `str_replace`  | `{ str_replace: { template: "...", params: {...} } }` | Performs string substitution                    |
| `list_join`    | `{ list_join: [",", [value1, value2]] }`              | Joins a list of values with a delimiter         |
| `if`           | `{ if: [condition_name, true_value, false_value] }`   | Returns one of two values based on a condition  |
| `equals`       | `{ equals: [value_a, value_b] }`                      | Returns `true` if both values are equal         |
| `not`          | `{ not: condition }`                                  | Negates a condition                             |
| `and`          | `{ and: [condition_a, condition_b] }`                 | Returns `true` if all conditions are true       |
| `or`           | `{ or: [condition_a, condition_b] }`                  | Returns `true` if any condition is true         |

### Function Examples

```yaml title="intrinsic-function-examples.yaml" theme={null}
resources:

  # get_param — retrieve a parameter value
  my_instance:
    type: Xloud::Compute::Server
    properties:
      flavor: { get_param: flavor }

  # get_resource — reference another resource's ID
  volume_attachment:
    type: Xloud::BlockStorage::VolumeAttachment
    properties:
      instance_uuid: { get_resource: my_instance }
      volume_id: { get_resource: my_volume }

  # get_attr — retrieve a resource attribute
  floating_ip_assoc:
    type: Xloud::Networking::FloatingIPAssociation
    properties:
      floatingip_id: { get_resource: my_floating_ip }
      port_id: { get_attr: [my_instance, addresses, default, 0, port] }

  # str_replace — build a user_data script from parameters
  bootstrap_instance:
    type: Xloud::Compute::Server
    properties:
      user_data:
        str_replace:
          template: |
            #!/bin/bash
            echo "Deploying to $ENV_NAME" > /etc/motd
          params:
            $ENV_NAME: { get_param: environment }
```

***

## Conditions

Conditions enable selective resource creation based on parameter values. Define a
condition by name, then reference it in resource definitions with `condition:`.

```yaml title="conditions-example.yaml" theme={null}
parameters:
  environment:
    type: string
    default: dev
    constraints:
      - allowed_values: [dev, staging, production]

conditions:
  is_production:
    equals: [{ get_param: environment }, production]

resources:

  # Always created
  app_server:
    type: Xloud::Compute::Server
    properties:
      flavor: { if: [is_production, m1.large, m1.small] }
      image: { get_param: image }

  # Only created in production
  monitoring_agent:
    type: Xloud::Compute::Server
    condition: is_production
    properties:
      flavor: m1.small
      image: { get_param: image }
```

***

## Outputs

Outputs expose values from created resources back to the caller. They are visible in
the Dashboard stack detail view and retrievable via CLI.

```yaml title="outputs-example.yaml" theme={null}
outputs:

  instance_id:
    description: UUID of the compute instance
    value: { get_resource: my_instance }

  instance_ip:
    description: Primary fixed IP address
    value: { get_attr: [my_instance, first_address] }

  public_url:
    description: Publicly accessible application URL
    value:
      str_replace:
        template: "https://$IP/app"
        params:
          $IP: { get_attr: [my_floating_ip, floating_ip_address] }
```

***

## Next Steps

<CardGroup>
  <Card title="Resource Types" href="/services/orchestration/resources">
    Full reference for all supported resource types and their properties
  </Card>

  <Card title="Getting Started" href="/services/orchestration/getting-started">
    Deploy your first stack using a complete example template
  </Card>

  <Card title="Auto-Scaling" href="/services/orchestration/autoscaling">
    Use scaling groups and alarm policies in templates
  </Card>

  <Card title="Manage Stacks" href="/services/orchestration/stacks">
    Update and manage stacks through their full lifecycle
  </Card>
</CardGroup>


# Orchestration Troubleshooting
Source: https://docs.xloud.tech/services/orchestration/troubleshooting

Resolve common orchestration issues — stuck stacks, template errors, and resource creation failures.

## Overview

Most Orchestration issues fall into one of five categories: stack creation failures,
stacks stuck in progress, template validation errors, resource dependency failures, and
nested stack propagation issues. Use the event log as the primary diagnostic tool —
it records each resource transition with timestamps and error messages.

<Tip>
  Always check the stack event log first: `openstack stack event list <stack-name> --nested-depth 5`
  provides a chronological view of every resource action across all nested stacks.
</Tip>

***

## Troubleshooting Reference

<AccordionGroup>
  <Accordion title="Stack stuck in CREATE_FAILED" icon="circle-x">
    **Symptoms**: Stack creation stops and the status shows `CREATE_FAILED`. One or more
    resources show `CREATE_FAILED` in the resource list.

    **Diagnosis**:

    ```bash title="List stack events" theme={null}
    openstack stack event list my-stack --nested-depth 5
    ```

    ```bash title="Show the specific failed resource" theme={null}
    openstack stack resource show my-stack my_instance
    ```

    **Common causes and resolutions**:

    | Cause                                   | Resolution                                                            |
    | --------------------------------------- | --------------------------------------------------------------------- |
    | Invalid image name or ID                | Verify the image exists: `openstack image list --status active`       |
    | Flavor not found                        | Verify the flavor: `openstack flavor list`                            |
    | Network not found                       | Verify the network: `openstack network list`                          |
    | Quota exceeded                          | Check quota: `openstack quota show`                                   |
    | Security group rule conflict            | Review security group rules for duplicates or conflicting CIDR ranges |
    | No hosts available in availability zone | Check host capacity with your administrator                           |
    | Key pair not found                      | Verify: `openstack keypair list`                                      |

    **Recovery**:
    After fixing the root cause, re-deploy the stack. `CREATE_FAILED` stacks can be
    deleted and recreated, but cannot be resumed:

    ```bash title="Delete failed stack and redeploy" theme={null}
    openstack stack delete --yes my-stack
    openstack stack create --template template.yaml --wait my-stack
    ```
  </Accordion>

  <Accordion title="Stack stuck in CREATE_IN_PROGRESS or UPDATE_IN_PROGRESS" icon="clock">
    **Symptoms**: The stack status has not changed from `IN_PROGRESS` for an extended
    period (beyond the expected resource creation time).

    **Diagnosis**:

    ```bash title="Check current resource states" theme={null}
    openstack stack resource list my-stack
    ```

    ```bash title="Stream live events" theme={null}
    openstack stack event list my-stack --follow
    ```

    **Common causes and resolutions**:

    | Cause                                                   | Resolution                                                                                                                  |
    | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
    | `WaitCondition` waiting for a signal that never arrives | Check the instance console log: `openstack console log show <instance-id>`. Verify the signal URL in user\_data is correct. |
    | Instance in BUILD state for too long                    | Check compute host capacity and scheduler logs via XDeploy                                                                  |
    | Dependency cycle between resources                      | Review `depends_on` declarations and `get_resource` references for circular chains                                          |
    | External API timeout                                    | The Orchestration engine retries — wait for the configured timeout before intervening                                       |

    **Force-abort a stuck stack** (admin only):

    ```bash title="Abandon a stuck stack" theme={null}
    openstack stack abandon my-stack
    ```

    <Warning>
      `stack abandon` releases the stack from Orchestration management but does NOT
      delete the underlying resources. You must clean up instances, volumes, and
      networks manually.
    </Warning>
  </Accordion>

  <Accordion title="Template validation errors" icon="file-x">
    **Symptoms**: `openstack orchestration template validate` returns errors, or the
    Dashboard rejects the template at upload time.

    **Diagnosis**:

    ```bash title="Validate template" theme={null}
    openstack orchestration template validate -t my-template.yaml
    ```

    **Common error messages and resolutions**:

    | Error Message                           | Resolution                                                                                              |
    | --------------------------------------- | ------------------------------------------------------------------------------------------------------- |
    | `Unknown type: Xloud::Compute::Serverr` | Check for typos in resource type names — type names are case-sensitive                                  |
    | `Property X is not supported`           | Check the property name against the resource type documentation                                         |
    | `Parameter X not found`                 | A `get_param` references a parameter that is not defined in the `parameters` section                    |
    | `Circular dependency detected`          | A `depends_on` or `get_resource` chain creates a cycle — draw the dependency graph to identify the loop |
    | `Invalid YAML`                          | Use a YAML linter (e.g., `yamllint`) to find indentation or syntax errors                               |

    <Tip>
      Use two-space indentation consistently throughout your templates. Tabs are not
      valid in YAML and will cause parse failures.
    </Tip>
  </Accordion>

  <Accordion title="Resource dependency failures" icon="link-slash">
    **Symptoms**: A resource fails because a resource it depends on was not yet created,
    or an attribute reference returns an empty value.

    **Diagnosis**:

    ```bash title="Show resource detail and status" theme={null}
    openstack stack resource show my-stack resource_name
    ```

    ```bash title="List resource events" theme={null}
    openstack stack resource event list my-stack resource_name
    ```

    **Common causes and resolutions**:

    | Cause                                                       | Resolution                                                                                                  |
    | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
    | `get_attr` used on a resource still in `CREATE_IN_PROGRESS` | Add an explicit `depends_on` to ensure the source resource is complete before the dependent resource starts |
    | `get_attr` attribute path is incorrect                      | Check the resource type's attribute documentation for the correct path                                      |
    | Resource deleted out-of-band (manually)                     | The stack's view of the resource is stale — run `openstack stack update --existing` to reconcile            |
    | Volume attachment fails because instance is still building  | Add `depends_on: [my_instance]` to the `VolumeAttachment` resource                                          |
  </Accordion>

  <Accordion title="Nested stack errors" icon="layers">
    **Symptoms**: A parent stack fails or stalls because a child (nested) stack
    encounters an error.

    **Diagnosis**:

    ```bash title="List events across all nested stacks" theme={null}
    openstack stack event list my-parent-stack --nested-depth 5
    ```

    ```bash title="List nested stacks" theme={null}
    openstack stack resource list my-parent-stack
    # Find the Xloud::Orchestration::Stack resource, then:
    openstack stack show <child-stack-id>
    ```

    **Common causes and resolutions**:

    | Cause                                                   | Resolution                                                                                                               |
    | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
    | Child template file not accessible                      | Ensure `get_file:` paths are relative to the parent template directory, or use `template_url` with an accessible URL     |
    | Parameter mismatch between parent and child             | Verify that all parameters passed to the child template via `parameters:` match the parameter names defined in the child |
    | Child stack output referenced before creation completes | Add `depends_on` to resources that consume child stack outputs                                                           |
    | `get_attr` on nested stack refers to undefined output   | Verify the output name exists in the child template's `outputs` section                                                  |
  </Accordion>
</AccordionGroup>

***

## Diagnostic Command Reference

```bash title="Key troubleshooting commands" theme={null}
# List all stacks with status
openstack stack list

# Show stack detail and status reason
openstack stack show my-stack

# List resources with status
openstack stack resource list my-stack

# Show a specific resource
openstack stack resource show my-stack resource_name

# Stream events in real time
openstack stack event list my-stack --follow

# Events across all nested stacks
openstack stack event list my-stack --nested-depth 5

# Validate a template before deploying
openstack orchestration template validate -t my-template.yaml
```

***

## Next Steps

<CardGroup>
  <Card title="Getting Started" href="/services/orchestration/getting-started">
    Review stack creation fundamentals and template structure
  </Card>

  <Card title="Template Guide" href="/services/orchestration/template-guide">
    Correct template syntax, parameter types, and intrinsic functions
  </Card>

  <Card title="Admin Troubleshooting" href="/services/orchestration/admin-troubleshooting">
    Engine-level diagnostics and service configuration issues
  </Card>

  <Card title="Resource Types" href="/services/orchestration/resources">
    Verify correct resource type names and property definitions
  </Card>
</CardGroup>


# Orchestration User Guide
Source: https://docs.xloud.tech/services/orchestration/user-guide

Create and manage infrastructure stacks using Xloud Orchestration templates. Covers getting started, template authoring, resource types, auto-scaling, and.

Create, update, and manage cloud infrastructure stacks using declarative orchestration templates.

***

<CardGroup>
  <Card title="Getting Started" icon="rocket" href="/services/orchestration/getting-started">
    Create your first stack from a template and learn the stack lifecycle
  </Card>

  <Card title="Template Guide" icon="file-code" href="/services/orchestration/template-guide">
    Template structure, parameters, intrinsic functions, and conditions
  </Card>

  <Card title="Resource Types" icon="boxes" href="/services/orchestration/resources">
    Complete reference for compute, network, storage, and identity resources
  </Card>

  <Card title="Manage Stacks" icon="layers" href="/services/orchestration/stacks">
    Create, update, suspend, resume, and delete stacks through their lifecycle
  </Card>

  <Card title="Auto-Scaling" icon="trending-up" href="/services/orchestration/autoscaling">
    Scale instance groups automatically using scaling groups and alarm policies
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/orchestration/troubleshooting">
    Diagnose stack failures, stuck deployments, and template validation errors
  </Card>
</CardGroup>


# XSDS Architecture
Source: https://docs.xloud.tech/services/sds/admin-guide/architecture

Understand the distributed storage cluster components — MON, MGR, OSD, MDS, and RGW — and how they work together to deliver block, object, and file storage.

## Overview

The XSDS distributed storage cluster is composed of several service types that work
together to provide unified block, object, and file storage. Services are deployed
and managed by XDeploy.

<Tip>
  **XDeploy GUI** — The distributed storage cluster can be bootstrapped and managed through the [XSDS Storage](/deployment/xsds) interface. Storage tiers, CRUSH rules, and Ceph configuration are all accessible from the GUI. No manual file editing required.
</Tip>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Familiarity with distributed storage concepts — replication, erasure coding, and data placement
</Note>

***

## Architecture Diagram

```mermaid theme={null}
graph TD
    Client["Client\n(Block / Object / File)"] --> GW["Gateway Layer\n(RBD / S3 / NFS-SMB)"]
    GW --> MON["Monitor Nodes\n(Cluster State and Quorum)"]
    GW --> MGR["Manager Nodes\n(Stats and Orchestration)"]
    MON --> CRUSH["CRUSH Map\n(Placement Policy)"]
    CRUSH --> OSD1["OSD Node A\n(NVMe Tier)"]
    CRUSH --> OSD2["OSD Node B\n(SSD Tier)"]
    CRUSH --> OSD3["OSD Node C\n(HDD Tier)"]
    MGR --> DASH["Admin Dashboard\n(Monitoring and Alerts)"]
```

***

## Service Components

| Service           | Role                                                                                             | Minimum Count        |
| ----------------- | ------------------------------------------------------------------------------------------------ | -------------------- |
| **Monitor (MON)** | Maintains authoritative cluster state and quorum. Must have an odd number for majority voting.   | 3                    |
| **Manager (MGR)** | Provides metrics, orchestration API, and dashboard. Active-standby.                              | 2                    |
| **OSD**           | Object Storage Daemon — one per physical storage device. Handles I/O, replication, and recovery. | 3 per replica factor |
| **MDS**           | Metadata Server — required for shared file storage. Active-standby.                              | 2                    |
| **RGW**           | RADOS Gateway — provides the S3-compatible object storage API.                                   | 2 (HA pair)          |

***

## Component Deep Dive

<AccordionGroup>
  <Accordion title="Monitor (MON)" icon="eye">
    Monitors maintain the authoritative cluster state map, which includes:

    * **OSD map**: Which OSDs are up, down, in, or out
    * **CRUSH map**: Placement topology and rules
    * **PG map**: State of all placement groups
    * **MDS map**: Metadata server state (if using shared file storage)

    Monitors use Paxos consensus to agree on cluster state. A majority (quorum) of
    monitors must be reachable for the cluster to accept writes. With 3 monitors,
    the cluster survives 1 monitor failure. With 5 monitors, it survives 2.

    <Warning>
      Never run fewer than 3 monitors in production. A 2-monitor cluster loses quorum
      if either monitor fails, halting all write operations.
    </Warning>
  </Accordion>

  <Accordion title="Object Storage Daemon (OSD)" icon="hard-drive">
    Each OSD manages one physical storage device. OSDs are responsible for:

    * Serving client read/write requests
    * Replicating data to peer OSDs according to the CRUSH map
    * Running scrub operations to detect and repair data corruption
    * Reporting health status to monitors

    OSD state has two dimensions:

    * **up/down**: Whether the OSD process is running
    * **in/out**: Whether the OSD is participating in data distribution

    An OSD that is `down` but `in` triggers recovery. An OSD that is `out` has its
    data redistributed to remaining OSDs.
  </Accordion>

  <Accordion title="RADOS Gateway (RGW)" icon="globe">
    RGW provides the S3-compatible object storage API. It translates S3 API requests
    into RADOS operations against the underlying storage cluster.

    RGW is stateless — all state is stored in the cluster. Deploy at least 2 RGW
    instances behind a load balancer for high availability. XDeploy configures the
    HAProxy frontend automatically.

    RGW supports:

    * S3-compatible API (buckets, objects, ACLs, lifecycle policies)
    * Multi-site replication between XSDS clusters
    * Pre-signed URLs for time-limited object access
  </Accordion>

  <Accordion title="Metadata Server (MDS)" icon="folder-open">
    MDS manages the metadata for the distributed file system. Each client inode,
    directory, and file name is tracked by an active MDS instance.

    MDS instances are either active (serving metadata requests) or standby (ready
    to take over). On active MDS failure, a standby takes over within seconds.

    Multiple active MDS instances (multi-active MDS) can be configured for large
    deployments with high metadata operation rates. Contact your Xloud support team
    for multi-active MDS configuration guidance.
  </Accordion>
</AccordionGroup>

***

## Deployment Architecture

<Tabs>
  <Tab title="Single-Site" icon="server">
    A standard single-site XSDS cluster distributes OSDs across at least 3 hosts,
    with monitor and manager services co-located on the same hosts:

    ```mermaid theme={null}
    graph LR
        subgraph Node1["Storage Node 1"]
            MON1["MON"]
            MGR1["MGR"]
            OSD1a["OSD.0 (NVMe)"]
            OSD1b["OSD.1 (SSD)"]
        end
        subgraph Node2["Storage Node 2"]
            MON2["MON"]
            OSD2a["OSD.2 (NVMe)"]
            OSD2b["OSD.3 (SSD)"]
        end
        subgraph Node3["Storage Node 3"]
            MON3["MON"]
            MGR2["MGR (standby)"]
            OSD3a["OSD.4 (NVMe)"]
            OSD3b["OSD.5 (SSD)"]
        end
    ```
  </Tab>

  <Tab title="Multi-Site" icon="globe">
    For disaster recovery configurations, XSDS supports multi-site replication
    between two independent clusters. Each site runs its own full cluster with
    its own monitors, managers, and OSDs.

    Replication between sites is handled at the RGW layer (for object storage)
    or at the block device level via XDR (for block storage).

    See the [XDR Admin Guide](/services/disaster-recovery/admin-guide) for
    cross-site replication configuration.
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Cluster Management" href="/services/sds/admin-guide/cluster-management">
    Monitor cluster health, manage services, and perform operational procedures
  </Card>

  <Card title="Pool Management" href="/services/sds/admin-guide/pool-management">
    Create and configure storage pools with the appropriate protection scheme
  </Card>

  <Card title="CRUSH Maps" href="/services/sds/admin-guide/crush-maps">
    Define failure domains and device class rules for data placement
  </Card>

  <Card title="Capacity Planning" href="/services/sds/admin-guide/capacity-planning">
    Monitor utilization and plan cluster expansion before capacity is exhausted
  </Card>
</CardGroup>


# Capacity Planning
Source: https://docs.xloud.tech/services/sds/admin-guide/capacity-planning

Monitor XSDS cluster utilization, maintain safe capacity headroom, and plan storage expansion before capacity constraints impact performance or availability.

## Overview

Maintaining adequate free capacity in an XSDS cluster is critical for both performance
and data safety. At high utilization, the cluster cannot complete recovery operations
after OSD failures, and I/O performance degrades significantly. This page covers
monitoring, thresholds, and expansion procedures.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * SSH access to a cluster management node
  * Access to **XDeploy** (`https://connect.<your-domain>`) for node provisioning
</Note>

***

## Capacity Thresholds

| Utilization | Status    | Action Required                           |
| ----------- | --------- | ----------------------------------------- |
| \< 60%      | Healthy   | Monitor routinely                         |
| 60–70%      | Watch     | Begin planning expansion                  |
| 70–80%      | Warning   | Initiate expansion — order hardware       |
| 80–85%      | Critical  | Accelerate expansion — immediate action   |
| > 85%       | Emergency | Risk of degraded I/O and recovery failure |

<Warning>
  Above 85% utilization, the cluster may refuse writes and cannot complete data
  recovery after OSD failures. Maintain a minimum of 30% free capacity headroom.
</Warning>

***

## Monitoring Utilization

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **XDeploy → Storage → Capacity** for a graphical capacity overview
    showing per-pool and cluster-wide utilization with trend projections.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Cluster-wide capacity summary" theme={null}
    ceph df
    ```

    ```bash title="Per-pool capacity" theme={null}
    ceph df detail
    ```

    ```bash title="Per-OSD utilization" theme={null}
    ceph osd df tree
    ```

    ```bash title="PG autoscale status" theme={null}
    ceph osd pool autoscale-status
    ```

    Key metrics to monitor:

    * **Used %**: Alert at 70%, act at 80%
    * **PG distribution**: Imbalanced PGs cause some OSDs to bear disproportionate load
    * **Recovery I/O**: Active recovery competes with client I/O — schedule OSD additions
      during low-traffic windows where possible
  </Tab>
</Tabs>

***

## Capacity Calculations

<AccordionGroup>
  <Accordion title="Replicated pools" icon="copy">
    For a pool with replication factor `n`, usable capacity = raw capacity / `n`.

    | Raw Capacity | Replication Factor | Usable Capacity |
    | ------------ | ------------------ | --------------- |
    | 100 TB       | 3 (default)        | \~33 TB         |
    | 100 TB       | 2                  | \~50 TB         |

    Account for the 30% headroom recommendation:

    * 100 TB raw, factor 3 = \~33 TB usable
    * 30% headroom = \~10 TB reserved
    * Effective usable = \~23 TB
  </Accordion>

  <Accordion title="Erasure-coded pools" icon="code">
    For an erasure code profile `k+m`, usable capacity = raw capacity × `k/(k+m)`.

    | Profile | Overhead | Usable from 100 TB |
    | ------- | -------- | ------------------ |
    | 4+2     | 1.5×     | \~67 TB            |
    | 6+2     | 1.33×    | \~75 TB            |
    | 8+3     | 1.375×   | \~73 TB            |
  </Accordion>

  <Accordion title="Snapshot space" icon="camera">
    Snapshots consume incremental capacity proportional to the change rate after the
    snapshot is taken. A volume with 10% daily churn accumulates approximately 10% of
    its size in snapshot data per day per snapshot retained.

    Factor snapshot retention into capacity planning. For 7-day retention on a 10-TB
    pool with 10% daily churn: approximately 7 TB additional snapshot space required.
  </Accordion>
</AccordionGroup>

***

## Expanding the Cluster

<Steps>
  <Step title="Deploy new OSD node via XDeploy" icon="server">
    Navigate to **XDeploy → Infrastructure → Nodes → Add Node** and register the
    new storage node. XDeploy configures the OS, installs storage packages, and
    joins the node to the cluster.

    <Tip>
      Add at least 3 OSDs per expansion batch to ensure balanced data distribution
      across the cluster. Adding a single OSD may cause temporary imbalance.
    </Tip>
  </Step>

  <Step title="Verify OSD integration" icon="circle-check">
    ```bash title="Confirm new OSDs are up and in" theme={null}
    ceph osd tree
    ```

    New OSDs should show `up` and `in`. The cluster begins re-balancing data
    automatically once OSDs are registered.
  </Step>

  <Step title="Monitor rebalancing" icon="refresh-cw">
    ```bash title="Watch recovery progress" theme={null}
    watch ceph status
    ```

    Rebalancing completes when `ceph status` shows `HEALTH_OK` with no active
    recovery operations. Rebalancing speed depends on cluster size and network
    bandwidth.

    <Note>
      Recovery I/O competes with client I/O. If client performance is impacted during
      rebalancing, throttle recovery:

      ```bash title="Throttle recovery I/O" theme={null}
      ceph osd set-recovery-delay 5
      ```
    </Note>

    <Check>Cluster returns to `HEALTH_OK` with data distributed across all OSDs including new ones.</Check>
  </Step>
</Steps>

***

## Capacity Trend Monitoring

Configure XIMP alerts to proactively notify administrators before capacity reaches
critical thresholds:

| Alert             | Threshold       | XIMP Metric                   |
| ----------------- | --------------- | ----------------------------- |
| Capacity Warning  | Pool used > 70% | `xloud_storage_pool_used_pct` |
| Capacity Critical | Pool used > 80% | `xloud_storage_pool_used_pct` |
| OSD Near Full     | OSD used > 85%  | `xloud_storage_osd_used_pct`  |

Navigate to **Monitoring → Alerting → Alert Rules** in the XIMP portal and create
rules sourcing from the `xloud_storage` metric namespace.

***

## Next Steps

<CardGroup>
  <Card title="Cluster Management" href="/services/sds/admin-guide/cluster-management">
    Add OSDs and manage cluster health during expansion
  </Card>

  <Card title="Monitoring" href="/services/sds/admin-guide/monitoring">
    Configure XIMP alerts for capacity and health thresholds
  </Card>

  <Card title="Storage Tiers" href="/services/sds/admin-guide/storage-tiers">
    Add new tiers when expanding with different device classes
  </Card>

  <Card title="Troubleshooting" href="/services/sds/admin-guide/troubleshooting">
    Diagnose capacity-related HEALTH\_WARN states
  </Card>
</CardGroup>


# Cluster Management
Source: https://docs.xloud.tech/services/sds/admin-guide/cluster-management

Monitor XSDS cluster health, manage service placement, add and remove OSDs, and perform day-to-day operational procedures.

## Overview

Day-to-day cluster management involves monitoring health status, responding to warnings
and errors, managing OSD lifecycle, and performing maintenance operations. This page
covers the most common operational tasks for XSDS administrators.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * SSH access to a cluster node running the management CLI
  * Access to **XDeploy** (`https://connect.<your-domain>`)
</Note>

***

## Monitoring Cluster Health

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **XDeploy → Storage → Cluster Health** for a graphical overview of
    cluster status, OSD counts, capacity utilization, and active alerts.

    | Health State  | Meaning                                       | Action Required              |
    | ------------- | --------------------------------------------- | ---------------------------- |
    | `HEALTH_OK`   | All components healthy, data fully replicated | None                         |
    | `HEALTH_WARN` | Non-critical issue detected                   | Investigate and resolve      |
    | `HEALTH_ERR`  | Critical issue — data may be at risk          | Immediate attention required |
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="View cluster health summary" icon="heart-pulse">
        ```bash title="Cluster health summary" theme={null}
        ceph status
        ```
      </Step>

      <Step title="Investigate warnings" icon="search">
        ```bash title="Detailed health report" theme={null}
        ceph health detail
        ```

        Review each warning entry. Common warnings are documented in the
        [Troubleshooting](/services/sds/admin-guide/troubleshooting) guide.
      </Step>

      <Step title="View OSD status" icon="hard-drive">
        ```bash title="OSD tree — placement and status" theme={null}
        ceph osd tree
        ```

        ```bash title="OSD utilization" theme={null}
        ceph osd df tree
        ```

        <Check>All OSDs show `up` and `in` status. OSDs showing `down` or `out` require investigation.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Service Operations

Common cluster management operations performed through XDeploy or the CLI.

<Tabs>
  <Tab title="Service Management" icon="settings">
    ```bash title="List all cluster services" theme={null}
    ceph orch ls
    ```

    ```bash title="View service placement" theme={null}
    ceph orch ps
    ```

    ```bash title="Redeploy a specific service" theme={null}
    ceph orch redeploy <SERVICE_TYPE>.<SERVICE_ID>
    ```

    <Warning>
      Always verify cluster health is `HEALTH_OK` before adding or removing OSDs.
      Operations on an already-degraded cluster can cause data unavailability.
    </Warning>
  </Tab>

  <Tab title="OSD Operations" icon="hard-drive">
    ```bash title="Add an OSD on a new device" theme={null}
    ceph orch daemon add osd <HOST>:<DEVICE_PATH>
    ```

    ```bash title="Mark an OSD out (begin data evacuation)" theme={null}
    ceph osd out <OSD_ID>
    ```

    ```bash title="Remove an OSD after data evacuation" theme={null}
    ceph orch osd rm <OSD_ID>
    ```

    ```bash title="View OSD details" theme={null}
    ceph osd dump | grep "^osd\."
    ```
  </Tab>
</Tabs>

***

## OSD Lifecycle Management

<Steps>
  <Step title="Adding a new OSD" icon="plus">
    Deploy new OSDs through XDeploy:

    1. Navigate to **XDeploy → Storage → OSDs → Add OSD**
    2. Select the target host and available device
    3. XDeploy provisions and integrates the OSD into the cluster

    Alternatively via CLI:

    ```bash title="Add OSD on specific host and device" theme={null}
    ceph orch daemon add osd <HOSTNAME>:/dev/nvme1n1
    ```

    After adding, the cluster begins re-balancing data automatically.

    <Tip>
      Add OSDs in batches rather than one at a time. Adding multiple OSDs simultaneously
      reduces the number of rebalancing cycles and recovers faster than sequential additions.
    </Tip>
  </Step>

  <Step title="Removing a failed OSD" icon="trash">
    ```bash title="Mark OSD out to trigger data recovery" theme={null}
    ceph osd out <OSD_ID>
    ```

    Monitor recovery progress:

    ```bash title="Watch recovery progress" theme={null}
    watch ceph status
    ```

    Once recovery completes (`HEALTH_OK` with no active recovery), remove the OSD:

    ```bash title="Remove the OSD from the cluster" theme={null}
    ceph osd purge <OSD_ID> --yes-i-really-mean-it
    ```

    <Warning>
      Do not remove an OSD before recovery is complete. Removing an OSD during
      active recovery on a degraded cluster risks data loss.
    </Warning>
  </Step>

  <Step title="Replacing a failed disk" icon="refresh-cw">
    After physically replacing the failed disk, redeploy the OSD through XDeploy:

    1. Navigate to **XDeploy → Storage → OSDs → Replace OSD**
    2. Select the host and the new device
    3. XDeploy provisions the replacement OSD and the cluster begins rebalancing

    <Check>New OSD shows `up in` in `ceph osd tree` and cluster returns to `HEALTH_OK`.</Check>
  </Step>
</Steps>

***

## Maintenance Mode

Before performing maintenance on a storage node (firmware updates, hardware replacement,
OS maintenance), set the cluster to maintenance mode to prevent false recovery triggers:

```bash title="Enable maintenance mode for a node" theme={null}
ceph osd set noout
ceph osd set norebalance
```

Perform your maintenance, then restore normal operation:

```bash title="Disable maintenance mode" theme={null}
ceph osd unset noout
ceph osd unset norebalance
```

<Warning>
  Do not leave `noout` and `norebalance` flags set for extended periods. If an OSD
  failure occurs while `noout` is set, the cluster will not re-replicate data to
  compensate, increasing the risk of data loss.
</Warning>

***

## Next Steps

<CardGroup>
  <Card title="Pool Management" href="/services/sds/admin-guide/pool-management">
    Create and configure storage pools for different workload types
  </Card>

  <Card title="CRUSH Maps" href="/services/sds/admin-guide/crush-maps">
    Manage failure domains and device class routing
  </Card>

  <Card title="Capacity Planning" href="/services/sds/admin-guide/capacity-planning">
    Monitor utilization and plan OSD additions before capacity is exhausted
  </Card>

  <Card title="Troubleshooting" href="/services/sds/admin-guide/troubleshooting">
    Diagnose HEALTH\_WARN states, OSD failures, and slow request issues
  </Card>
</CardGroup>


# CRUSH Maps
Source: https://docs.xloud.tech/services/sds/admin-guide/crush-maps

Configure the XSDS CRUSH map to define failure domains, device class rules, and data placement topology across nodes, racks, and data center rooms.

## Overview

The CRUSH map defines the hierarchical topology of the storage cluster — how nodes,
racks, and rooms are structured, and how data is distributed across failure domains.
Correct CRUSH configuration ensures data is spread across independent failure domains
for maximum availability.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * SSH access to a cluster management node
  * Understanding of your physical infrastructure topology (host, rack, room layout)
</Note>

***

## CRUSH Hierarchy

The CRUSH hierarchy places storage devices into a tree of buckets. Data placement
rules traverse this tree to select OSDs from distinct failure domains.

```mermaid theme={null}
graph TD
    DC["Data Center"] --> ROOM1["Room A"] & ROOM2["Room B"]
    ROOM1 --> RACK1["Rack 01"] & RACK2["Rack 02"]
    ROOM2 --> RACK3["Rack 03"]
    RACK1 --> HOST1["host-1"] & HOST2["host-2"]
    RACK2 --> HOST3["host-3"]
    RACK3 --> HOST4["host-4"] & HOST5["host-5"]
    HOST1 --> OSD1["osd.0 (nvme)"] & OSD2["osd.1 (ssd)"]
    HOST2 --> OSD3["osd.2 (nvme)"] & OSD4["osd.3 (ssd)"]
```

***

## Viewing the CRUSH Map

<Tabs>
  <Tab title="OSD Tree" icon="list">
    ```bash title="View OSD tree with device classes" theme={null}
    ceph osd tree
    ```

    This shows the current hierarchy and the device class assigned to each OSD.
  </Tab>

  <Tab title="Full CRUSH Map" icon="map">
    ```bash title="Export and decompile the CRUSH map" theme={null}
    ceph osd getcrushmap -o /tmp/crushmap.bin
    crushtool -d /tmp/crushmap.bin -o /tmp/crushmap.txt
    cat /tmp/crushmap.txt
    ```

    The compiled CRUSH map defines:

    * **Device class assignments** — which OSDs belong to NVMe, SSD, or HDD classes
    * **Bucket hierarchy** — OSDs → hosts → racks → data center rooms
    * **Replication rules** — how many copies to place and across which failure domains
  </Tab>
</Tabs>

***

## Device Class Rules

Device class CRUSH rules route data to a specific storage tier. Create one rule
per device class to enable multi-tier storage.

<Steps>
  <Step title="Create a device class rule" icon="plus">
    ```bash title="Create SSD device class rule" theme={null}
    ceph osd crush rule create-replicated \
      replicated_rule_ssd default host ssd
    ```

    ```bash title="Create NVMe device class rule" theme={null}
    ceph osd crush rule create-replicated \
      replicated_rule_nvme default host nvme
    ```

    ```bash title="Create HDD device class rule" theme={null}
    ceph osd crush rule create-replicated \
      replicated_rule_hdd default host hdd
    ```

    Valid device classes: `nvme`, `ssd`, `hdd`.
  </Step>

  <Step title="Assign rule to a pool" icon="database">
    ```bash title="Assign CRUSH rule to pool" theme={null}
    ceph osd pool set <POOL_NAME> crush_rule replicated_rule_ssd
    ```

    <Note>
      In a single-device-class cluster (all SSD), changing a pool's CRUSH rule from
      `replicated_rule` to `replicated_rule_ssd` involves zero data movement — data
      is already on the correct devices.
    </Note>
  </Step>

  <Step title="Verify rule assignment" icon="circle-check">
    ```bash title="List rules and their pools" theme={null}
    ceph osd crush rule dump --format json | python3 -m json.tool
    ```

    ```bash title="Show pool CRUSH rule" theme={null}
    ceph osd pool get <POOL_NAME> crush_rule
    ```

    <Check>Pool reports the new CRUSH rule name and OSD tree shows data on the correct device class.</Check>
  </Step>
</Steps>

***

## Managing OSD Device Classes

OSDs are automatically classified by device type at deployment. Verify or override
classifications as needed.

<Tabs>
  <Tab title="View Classifications" icon="list">
    ```bash title="List all OSD device classes" theme={null}
    ceph osd crush class ls
    ```

    ```bash title="List OSDs in a specific class" theme={null}
    ceph osd crush class ls-osd ssd
    ```
  </Tab>

  <Tab title="Reclassify an OSD" icon="settings">
    ```bash title="Remove current classification" theme={null}
    ceph osd crush rm-device-class <OSD_ID>
    ```

    ```bash title="Set new device class" theme={null}
    ceph osd crush set-device-class ssd <OSD_ID>
    ```

    <Warning>
      In a multi-class cluster with existing data, reclassifying an OSD to a different
      class can trigger data migration if pools have class-specific CRUSH rules. Verify
      cluster health is `HEALTH_OK` before reclassifying.
    </Warning>
  </Tab>
</Tabs>

***

## Failure Domain Configuration

Configure CRUSH rules to spread replicas across independent failure domains. The
failure domain level determines how many simultaneous failures the cluster can
survive without data loss.

| Failure Domain | Survives                                      | Recommended For                     |
| -------------- | --------------------------------------------- | ----------------------------------- |
| `host`         | Any number of OSD failures on different hosts | Small clusters (\< 10 hosts)        |
| `rack`         | Entire rack failures (power, networking)      | Medium clusters with physical racks |
| `room`         | Room-level failures (fire, flood)             | Large data center deployments       |

```bash title="Create a rack-level failure domain rule" theme={null}
ceph osd crush rule create-replicated \
  replicated_rule_ssd_rack default rack ssd
```

<Tip>
  For most production deployments, `host`-level failure domains provide the right
  balance of protection and OSD count requirements. Use `rack`-level failure domains
  only when you have at least 3 physical racks in your deployment.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Pool Management" href="/services/sds/admin-guide/pool-management">
    Create pools and assign the CRUSH rules you've just configured
  </Card>

  <Card title="Storage Tiers" href="/services/sds/admin-guide/storage-tiers">
    Wire device class rules to Cinder volume types for multi-tier storage
  </Card>

  <Card title="Cluster Management" href="/services/sds/admin-guide/cluster-management">
    Monitor cluster health and manage OSDs in your configured topology
  </Card>

  <Card title="Troubleshooting" href="/services/sds/admin-guide/troubleshooting">
    Diagnose CRUSH-related issues including imbalanced data distribution
  </Card>
</CardGroup>


# Monitoring
Source: https://docs.xloud.tech/services/sds/admin-guide/monitoring

Monitor XSDS cluster health, OSD status, and I/O performance through XIMP integration — key metrics, alert thresholds, and observability configuration.

## Overview

XSDS cluster health and performance data is exported to XIMP for centralized monitoring
and alerting. This page covers the key metrics to monitor, recommended alert thresholds,
and how to configure the integration.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * XIMP deployed and accessible (see [XIMP Admin Guide](/services/monitoring/admin-guide))
  * Metric scrape target configured for the XSDS cluster metrics endpoint
</Note>

***

## Key Metrics and Alert Thresholds

| Metric                  | Namespace                            | Alert Threshold      | Action                              |
| ----------------------- | ------------------------------------ | -------------------- | ----------------------------------- |
| Cluster health          | `xloud_storage_health`               | `HEALTH_WARN`        | Investigate immediately             |
| OSD `down` count        | `xloud_storage_osd_down`             | > 0                  | Replace or recover failed OSD       |
| Pool used %             | `xloud_storage_pool_used_pct`        | > 70%                | Plan capacity expansion             |
| Recovery I/O rate       | `xloud_storage_recovery_bytes_sec`   | Sustained > 200 MB/s | Consider I/O throttling             |
| PG `inconsistent` count | `xloud_storage_pg_inconsistent`      | > 0                  | Run `ceph health detail` and repair |
| Replication lag (RGW)   | `xloud_storage_rgw_sync_lag_sec`     | Sustained > 30s      | Check network bandwidth to RGW      |
| OSD apply latency       | `xloud_storage_osd_apply_latency_ms` | > 20 ms              | Investigate OSD or disk health      |

***

## Configuring XIMP Integration

<Steps>
  <Step title="Verify metrics endpoint" icon="activity">
    The XSDS cluster exposes metrics on the management node. Verify the endpoint
    is reachable:

    ```bash title="Check metrics endpoint" theme={null}
    curl http://<MGMT_NODE_IP>:9283/metrics | head -20
    ```

    Port `9283` is the default metrics exporter port deployed by XDeploy.
  </Step>

  <Step title="Add scrape target in XIMP" icon="plus">
    Navigate to **Monitoring → Administration → Scrape Targets → Add Target**:

    | Field               | Value                                                    |
    | ------------------- | -------------------------------------------------------- |
    | **URL**             | `http://<MGMT_NODE_IP>:9283/metrics`                     |
    | **Scrape Interval** | `60s` (storage metrics don't need sub-minute resolution) |
    | **Labels**          | `service=xsds`, `cluster=<CLUSTER_NAME>`                 |

    Or via CLI:

    ```bash title="Add XSDS scrape target" theme={null}
    ximp target add \
      --url http://<MGMT_NODE_IP>:9283/metrics \
      --interval 60s \
      --label service=xsds \
      --label cluster=prod-storage
    ```
  </Step>

  <Step title="Create alert rules" icon="bell">
    Navigate to **Monitoring → Alerting → Alert Rules** and create rules for each
    threshold in the table above.

    Example alert rule for pool utilization:

    ```yaml title="alert-storage-capacity.yaml" theme={null}
    name: xsds-pool-capacity-warning
    metric: xloud_storage_pool_used_pct
    condition: ">"
    threshold: 70
    evaluation_period: 10m
    severity: warning
    notification_channels:
      - ops-email
    ```

    <Check>Alert rules appear in the Active Rules list and evaluate against live storage metrics.</Check>
  </Step>
</Steps>

***

## Built-In Dashboards

The XIMP portal includes pre-built XSDS dashboards. Navigate to
**Monitoring → Dashboards** and search for "XSDS" or "Storage":

| Dashboard                 | Shows                                                 |
| ------------------------- | ----------------------------------------------------- |
| **XSDS Cluster Overview** | Health state, OSD counts, capacity, recovery activity |
| **XSDS Pool Utilization** | Per-pool used %, available bytes, PG counts           |
| **XSDS OSD Performance**  | Per-OSD latency, IOPS, throughput                     |
| **XSDS Recovery**         | Active recovery operations, estimated completion time |

<Tip>
  Pin the "XSDS Cluster Overview" dashboard to your XIMP home screen for constant
  visibility during on-call rotations.
</Tip>

***

## Cluster CLI Health Check

For quick health checks without opening the XIMP portal, use the management CLI
directly from a cluster node:

```bash title="Quick health overview" theme={null}
ceph status
```

```bash title="OSD performance snapshot" theme={null}
ceph osd perf
```

```bash title="Pool I/O statistics (5-second window)" theme={null}
ceph osd pool stats
```

```bash title="Active slow requests" theme={null}
ceph health detail | grep -i slow
```

***

## Next Steps

<CardGroup>
  <Card title="XIMP Admin Guide" href="/services/monitoring/admin-guide">
    Configure the monitoring platform that collects and displays XSDS metrics
  </Card>

  <Card title="Capacity Planning" href="/services/sds/admin-guide/capacity-planning">
    Use utilization metrics to plan cluster expansion before thresholds are reached
  </Card>

  <Card title="Troubleshooting" href="/services/sds/admin-guide/troubleshooting">
    Diagnose the issues surfaced by monitoring alerts
  </Card>

  <Card title="XIMP Alert Rules" href="/services/monitoring/admin-guide/alert-channels">
    Configure notification channels for storage health alerts
  </Card>
</CardGroup>


# Pool Management
Source: https://docs.xloud.tech/services/sds/admin-guide/pool-management

Create and configure replicated and erasure-coded storage pools, set application types, and manage pool settings in XSDS.

## Overview

Pools are the logical containers for stored data. Each pool has a defined data protection
policy (replication or erasure coding), a device class mapping, and a PG count. Creating
pools correctly at the outset avoids disruptive reconfiguration later.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * SSH access to a cluster management node
  * CRUSH map configured with appropriate device class rules (see [CRUSH Maps](/services/sds/admin-guide/crush-maps))
</Note>

***

## Creating Pools

<Tabs>
  <Tab title="Replicated Pool" icon="copy">
    <Steps>
      <Step title="Create the pool" icon="plus">
        ```bash title="Create a replicated pool" theme={null}
        ceph osd pool create <POOL_NAME> <PG_COUNT> <PG_COUNT> replicated
        ```

        For most pools, set PG count to `128` initially — the PG autoscaler will
        adjust automatically as data volume grows.
      </Step>

      <Step title="Set replication factor" icon="copy">
        ```bash title="Set replication size (factor)" theme={null}
        ceph osd pool set <POOL_NAME> size 3
        ceph osd pool set <POOL_NAME> min_size 2
        ```

        `size` is the total number of copies. `min_size` is the minimum needed to
        serve I/O (allows degraded operation with 2 copies during OSD failure recovery).
      </Step>

      <Step title="Assign a CRUSH rule" icon="map">
        Associate the pool with a CRUSH rule that targets the correct device class:

        ```bash title="Set CRUSH rule on pool" theme={null}
        ceph osd pool set <POOL_NAME> crush_rule replicated_rule_ssd
        ```

        Use `ceph osd crush rule ls` to list available rules.
      </Step>

      <Step title="Enable pool application" icon="check">
        Tag the pool with its application type so the cluster knows how to manage it:

        ```bash title="Enable RBD (block storage) on pool" theme={null}
        ceph osd pool application enable <POOL_NAME> rbd
        ```

        Valid application types: `rbd` (block), `rgw` (object), `cephfs` (file).

        <Check>Pool appears in `ceph osd pool ls detail` with correct application and replication settings.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Erasure-Coded Pool" icon="code">
    <Steps>
      <Step title="Create an erasure code profile" icon="settings">
        ```bash title="Create erasure code profile (4+2 on HDD)" theme={null}
        ceph osd erasure-code-profile set ec-4-2 \
          k=4 m=2 crush-device-class=hdd
        ```

        ```bash title="Create erasure code profile (8+3 for large archives)" theme={null}
        ceph osd erasure-code-profile set ec-8-3 \
          k=8 m=3 crush-device-class=hdd
        ```
      </Step>

      <Step title="Create the pool using the profile" icon="plus">
        ```bash title="Create erasure-coded pool" theme={null}
        ceph osd pool create <POOL_NAME> erasure ec-4-2
        ```
      </Step>

      <Step title="Enable pool application" icon="check">
        ```bash title="Enable RGW (object storage) on EC pool" theme={null}
        ceph osd pool application enable <POOL_NAME> rgw
        ```

        <Note>
          Erasure-coded pools cannot be used directly for block storage (RBD) without
          a replicated overlay pool. Use erasure coding primarily for object storage
          and large object archives.
        </Note>

        <Check>Pool appears in `ceph osd pool ls detail` with erasure code profile listed.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Managing Existing Pools

<Tabs>
  <Tab title="Inspect Pools" icon="search">
    ```bash title="List all pools with details" theme={null}
    ceph osd pool ls detail
    ```

    ```bash title="Show pool statistics" theme={null}
    ceph df detail
    ```

    ```bash title="Show specific pool configuration" theme={null}
    ceph osd pool get <POOL_NAME> all
    ```
  </Tab>

  <Tab title="Modify Pools" icon="settings">
    ```bash title="Enable PG autoscaling on a pool" theme={null}
    ceph osd pool set <POOL_NAME> pg_autoscale_mode on
    ```

    ```bash title="Set compression on a pool" theme={null}
    ceph osd pool set <POOL_NAME> compression_mode aggressive
    ceph osd pool set <POOL_NAME> compression_algorithm lz4
    ```

    ```bash title="Rename a pool" theme={null}
    ceph osd pool rename <OLD_NAME> <NEW_NAME>
    ```
  </Tab>

  <Tab title="Delete a Pool" icon="trash">
    <Danger>
      Pool deletion is irreversible and permanently destroys all data stored in the
      pool. Confirm with the requesting team that all data has been migrated or is
      no longer needed before proceeding.
    </Danger>

    ```bash title="Enable pool deletion (required safety flag)" theme={null}
    ceph config set mon mon_allow_pool_delete true
    ```

    ```bash title="Delete pool (requires double confirmation)" theme={null}
    ceph osd pool delete <POOL_NAME> <POOL_NAME> \
      --yes-i-really-really-mean-it
    ```

    ```bash title="Re-disable pool deletion after use" theme={null}
    ceph config set mon mon_allow_pool_delete false
    ```
  </Tab>
</Tabs>

***

## Pool Configuration Reference

| Parameter        | Command                                                | Notes                |
| ---------------- | ------------------------------------------------------ | -------------------- |
| Replication size | `ceph osd pool set <pool> size <n>`                    | Number of copies     |
| Minimum size     | `ceph osd pool set <pool> min_size <n>`                | Min copies for I/O   |
| CRUSH rule       | `ceph osd pool set <pool> crush_rule <rule>`           | Device class routing |
| PG autoscale     | `ceph osd pool set <pool> pg_autoscale_mode on`        | Automatic PG sizing  |
| Compression      | `ceph osd pool set <pool> compression_mode aggressive` | Inline compression   |
| Quotas           | `ceph osd pool set-quota <pool> max_bytes <bytes>`     | Capacity limit       |

***

## Next Steps

<CardGroup>
  <Card title="CRUSH Maps" href="/services/sds/admin-guide/crush-maps">
    Configure failure domains and device class rules for pool placement
  </Card>

  <Card title="Storage Tiers" href="/services/sds/admin-guide/storage-tiers">
    Map pools to Cinder volume types for multi-tier storage
  </Card>

  <Card title="Capacity Planning" href="/services/sds/admin-guide/capacity-planning">
    Monitor pool utilization and plan expansion
  </Card>

  <Card title="Troubleshooting" href="/services/sds/admin-guide/troubleshooting">
    Diagnose pool-related issues — PG warnings, capacity alerts
  </Card>
</CardGroup>


# Security
Source: https://docs.xloud.tech/services/sds/admin-guide/security

Secure XSDS deployments with encryption at rest, cluster authentication (cephx), key rotation, and network isolation for production environments.

## Overview

XSDS security encompasses encryption at rest for stored data, cluster authentication
between all services using cephx, network isolation to separate replication traffic
from client-facing traffic, and regular key rotation procedures.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Access to **XDeploy** (`https://connect.<your-domain>`) for OSD deployment settings
  * SSH access to cluster management node for cephx key operations
</Note>

***

## Encryption at Rest

XSDS supports OSD-level encryption at rest using dm-crypt. All data written to an
encrypted OSD is encrypted before it reaches the physical disk.

<Tabs>
  <Tab title="Enable Encryption" icon="lock">
    Encryption is configured at OSD deployment time through XDeploy.

    <Steps>
      <Step title="Configure encryption before deployment" icon="settings">
        Navigate to **XDeploy → Storage → OSD Deployment** and enable
        **Encrypt OSDs at deployment** before provisioning new OSDs.

        Encryption keys are managed by the Xloud Key Management service and rotated
        on a configurable schedule.
      </Step>

      <Step title="Verify encryption on deployed OSDs" icon="circle-check">
        ```bash title="Check if OSD device is encrypted" theme={null}
        ceph osd metadata <OSD_ID> | grep dmcrypt
        ```

        Encrypted OSDs report `"dmcrypt": true` in their metadata.

        <Check>OSD metadata confirms dm-crypt encryption is active.</Check>
      </Step>
    </Steps>

    <Warning>
      Encryption cannot be enabled on existing OSDs without redeploying them. Plan
      encryption requirements before initial OSD deployment. In-place encryption of
      existing OSDs requires data migration to new encrypted OSDs.
    </Warning>
  </Tab>

  <Tab title="Key Management" icon="key">
    Encryption keys for XSDS OSDs are managed by the Xloud Key Management (XKM) service.

    * Keys are generated per-OSD at deployment time
    * Keys are stored in the XKM service, not on the OSD nodes themselves
    * Key rotation schedules are configurable in **XDeploy → Security → Key Rotation**
    * Each OSD retrieves its key at startup via the XKM API

    <Note>
      If an OSD node is compromised, the attacker cannot decrypt disk contents without
      access to the corresponding key in XKM. The physical disks are unreadable outside
      the cluster.
    </Note>
  </Tab>
</Tabs>

***

## Cluster Authentication (cephx)

All cluster communication uses cephx, the XSDS cluster authentication framework.
Each service has its own key with minimum required capabilities.

<Tabs>
  <Tab title="View and Inspect Keys" icon="list">
    ```bash title="List all cephx keys" theme={null}
    ceph auth ls
    ```

    ```bash title="View capabilities for a specific key" theme={null}
    ceph auth get client.<NAME>
    ```

    Standard key names:

    * `client.admin` — full administrative access
    * `client.cinder` — used by the block storage service
    * `client.glance` — used by the image service
    * `client.nova` — used by the compute service
    * `client.rgw.<id>` — used by object storage gateways
  </Tab>

  <Tab title="Key Rotation" icon="refresh-cw">
    Rotate cephx keys periodically or immediately after a suspected compromise:

    ```bash title="Rotate a cephx key" theme={null}
    ceph auth rotate client.<NAME>
    ```

    <Note>
      Rotating a cephx key does not require redistributing the key file — updated
      capabilities take effect immediately in the MON database. The key itself (secret)
      remains the same after rotation unless you generate a new key entirely.
    </Note>

    To generate a completely new key for a service:

    ```bash title="Delete and recreate a key" theme={null}
    ceph auth del client.<NAME>
    ceph auth get-or-create client.<NAME> \
      mon 'profile rbd' \
      osd 'profile rbd pool=volumes'
    ```

    After generating a new key, update the corresponding keyring file on all service
    nodes and restart the affected services.
  </Tab>

  <Tab title="Least Privilege Caps" icon="shield">
    Grant each client key only the capabilities required for its role:

    | Service                | Recommended Capabilities                                                 |
    | ---------------------- | ------------------------------------------------------------------------ |
    | Cinder (block storage) | `mon 'profile rbd' osd 'profile rbd pool=volumes'`                       |
    | Glance (image service) | `mon 'profile rbd' osd 'profile rbd pool=images'`                        |
    | Nova (compute)         | `mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms'` |
    | RGW (object storage)   | `mon 'allow rw' osd 'allow rwx'`                                         |

    ```bash title="Update capabilities for a key" theme={null}
    ceph auth caps client.cinder \
      mon 'profile rbd' \
      osd 'profile rbd pool=volumes'
    ```
  </Tab>
</Tabs>

***

## Network Isolation

Configure a dedicated cluster network for OSD replication traffic to isolate
storage replication I/O from client-facing traffic.

| Network             | Purpose                                       | Traffic                                |
| ------------------- | --------------------------------------------- | -------------------------------------- |
| **Public network**  | Client-to-OSD I/O, MON communication, RGW API | Read/write requests from Compute nodes |
| **Cluster network** | OSD-to-OSD replication, recovery, scrubbing   | Internal replication traffic           |

<Tabs>
  <Tab title="Recommended Setup" icon="network">
    Separate physical interfaces or VLANs are recommended for high-throughput
    production environments:

    * **Public network**: 10 GbE or 25 GbE, shared with compute nodes
    * **Cluster network**: 25 GbE or faster, storage-only VLAN

    Network configuration is set in the cluster configuration and applied by XDeploy
    during initial deployment.

    <Tip>
      Isolating cluster (replication) traffic prevents recovery operations from
      impacting client I/O performance. During OSD failures, recovery traffic can
      easily saturate a shared 10 GbE link.
    </Tip>
  </Tab>

  <Tab title="Verify Network Config" icon="settings">
    ```bash title="Show cluster network configuration" theme={null}
    ceph config get osd cluster_network
    ceph config get osd public_network
    ```

    ```bash title="Check OSD binding" theme={null}
    ceph osd dump | grep "^osd\." | awk '{print $1, $14, $16}'
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Encryption at Rest" href="/services/storage/encryption">
    Block storage volume-level encryption managed through the Key Management service
  </Card>

  <Card title="Xloud Key Management" href="/services/key-manager/admin-guide">
    Manage and rotate the encryption keys used by XSDS and other services
  </Card>

  <Card title="Cluster Management" href="/services/sds/admin-guide/cluster-management">
    Ongoing operational management for a secured cluster
  </Card>

  <Card title="Troubleshooting" href="/services/sds/admin-guide/troubleshooting">
    Diagnose authentication and connectivity issues
  </Card>
</CardGroup>


# Storage Tiers
Source: https://docs.xloud.tech/services/sds/admin-guide/storage-tiers

Configure multi-tier storage in XSDS by mapping device class pools to Cinder volume types, enabling workload-appropriate media selection for NVMe, SSD, and HDD tiers.

## Overview

Multi-tier storage routes different workload classes to appropriate device media
through volume types in the block storage service and pool configurations in XSDS.
Tier configuration is managed through XDeploy's Storage Tiers panel, which
auto-generates the required pool, CRUSH rule, and block storage backend configuration.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Tip>
  **XDeploy GUI** — Storage tier detection, CRUSH rule creation, and pool configuration can be performed through the [XSDS Storage](/deployment/xsds) interface under the **Storage Tiers** tab. Click **Detect Tiers** to automatically discover device classes. No manual file editing required.
</Tip>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Device class CRUSH rules created for each tier (see [CRUSH Maps](/services/sds/admin-guide/crush-maps))
  * Access to **XDeploy** (`https://connect.<your-domain>`)
</Note>

***

## Tier Overview

| Tier | Device Class | Pool                | Volume Type | Use Case                          |
| ---- | ------------ | ------------------- | ----------- | --------------------------------- |
| NVMe | `nvme`       | `volumes-nvme`      | `ceph-nvme` | Databases, OLTP, latency-critical |
| SSD  | `ssd`        | `volumes` (default) | `ceph-ssd`  | General-purpose, web, application |
| HDD  | `hdd`        | `volumes-hdd`       | `ceph-hdd`  | Backup, archive, big data         |

***

## Configuring Tiers via XDeploy

<Steps>
  <Step title="Open Storage Tiers panel" icon="layers">
    Navigate to **XDeploy → Configuration → Storage Tiers**.

    The panel displays detected device classes in the cluster and lets you define
    which tiers to expose as volume types.
  </Step>

  <Step title="Add a new tier" icon="plus">
    Click **Add Tier** and configure:

    | Field            | Description                                                               |
    | ---------------- | ------------------------------------------------------------------------- |
    | **Tier Name**    | Display name (e.g., `NVMe Performance`)                                   |
    | **Device Class** | Physical media class: `nvme`, `ssd`, or `hdd`                             |
    | **Pool Name**    | Name for the storage pool (e.g., `volumes-nvme`)                          |
    | **Volume Type**  | Cinder volume type name exposed to users (e.g., `ceph-nvme`)              |
    | **Default**      | Whether this tier is the default for new volumes without an explicit type |

    <Tip>
      Set the fastest available tier as the default volume type. Users who don't
      specify a type explicitly receive the best performance tier.
    </Tip>
  </Step>

  <Step title="Apply configuration" icon="check">
    Click **Apply**. XDeploy automatically:

    1. Creates the pool with the correct CRUSH rule for the device class
    2. Registers the Cinder backend pointing to the new pool
    3. Creates the volume type with appropriate extra specs
    4. Updates the `_50_ceph_tiers.yml` configuration file

    <Check>New volume type appears in `openstack volume type list` and is available to tenants.</Check>
  </Step>
</Steps>

***

## Manual Tier Configuration

For environments where XDeploy is not managing tier configuration, configure tiers manually.

<Tabs>
  <Tab title="Create Pool and Rule" icon="database">
    ```bash title="Create CRUSH rule for NVMe" theme={null}
    ceph osd crush rule create-replicated \
      replicated_rule_nvme default host nvme
    ```

    ```bash title="Create NVMe-backed pool" theme={null}
    ceph osd pool create volumes-nvme 128 128 replicated
    ceph osd pool set volumes-nvme size 3
    ceph osd pool set volumes-nvme min_size 2
    ceph osd pool set volumes-nvme crush_rule replicated_rule_nvme
    ceph osd pool application enable volumes-nvme rbd
    ```
  </Tab>

  <Tab title="Register Cinder Backend" icon="settings">
    Add the new backend to your Cinder configuration (managed by XDeploy via
    `globals.d/_50_ceph_tiers.yml`):

    ```yaml title="/etc/xavs/globals.d/_50_ceph_tiers.yml" theme={null}
    cinder_ceph_backends:
      - name: ceph-ssd
        pool: volumes
        device_class: ssd
      - name: ceph-nvme
        pool: volumes-nvme
        device_class: nvme
      - name: rbd-1
        pool: volumes

    ceph_default_volume_type: ceph-nvme
    ```

    Then deploy the block storage configuration:

    ```bash title="Deploy Cinder configuration" theme={null}
    xavs-ansible deploy -t cinder
    ```
  </Tab>
</Tabs>

***

## Verifying Tier Configuration

<Tabs>
  <Tab title="Check Volume Types" icon="list">
    ```bash title="List all volume types" theme={null}
    openstack volume type list
    ```

    ```bash title="Show volume type extra specs" theme={null}
    openstack volume type show ceph-nvme -c extra_specs
    ```

    The `volume_backend_name` extra spec should match the Cinder backend name.
  </Tab>

  <Tab title="Test Volume Creation" icon="flask-conical">
    ```bash title="Create a test volume on the NVMe tier" theme={null}
    openstack volume create \
      --size 10 \
      --type ceph-nvme \
      test-nvme-volume
    ```

    ```bash title="Verify it reaches AVAILABLE status" theme={null}
    openstack volume show test-nvme-volume -c status
    ```

    ```bash title="Clean up test volume" theme={null}
    openstack volume delete test-nvme-volume
    ```

    <Check>Volume reaches `available` status — the tier is correctly configured end-to-end.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="CRUSH Maps" href="/services/sds/admin-guide/crush-maps">
    Configure device class rules that route each tier to the correct physical media
  </Card>

  <Card title="Pool Management" href="/services/sds/admin-guide/pool-management">
    Manage the pools backing each storage tier
  </Card>

  <Card title="Capacity Planning" href="/services/sds/admin-guide/capacity-planning">
    Monitor per-tier utilization and plan expansion
  </Card>

  <Card title="Block Storage — Volume Types" href="/services/storage/volume-types-admin">
    Advanced Cinder volume type configuration for QoS and backend selection
  </Card>
</CardGroup>


# Troubleshooting
Source: https://docs.xloud.tech/services/sds/admin-guide/troubleshooting

Diagnose and resolve XSDS cluster-level issues — HEALTH_WARN states, OSD failures, slow requests, PG inconsistencies, and capacity emergencies.

## Overview

This page covers cluster-level troubleshooting procedures for XSDS administrators.
For tenant-facing storage issues (stuck volumes, snapshot failures), see the
[XSDS User Guide Troubleshooting](/services/sds/user-guide/troubleshooting) page.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * SSH access to a cluster management node
</Note>

***

## Diagnostic Reference

Before investigating specific issues, collect the full health report:

```bash title="Full health report" theme={null}
ceph health detail
```

```bash title="Cluster status overview" theme={null}
ceph status
```

```bash title="Recent cluster log events" theme={null}
ceph log last 100
```

***

## Common Issues

<AccordionGroup>
  <Accordion title="HEALTH_WARN: too few PGs" icon="layers">
    **Cause**: A pool has fewer placement groups than recommended for its current
    data volume. Under-provisioned PGs cause I/O imbalance across OSDs.

    **Diagnosis**:

    ```bash title="Check PG autoscale status" theme={null}
    ceph osd pool autoscale-status
    ```

    **Resolution**:
    Enable PG auto-scaling to let the cluster manage PG counts automatically:

    ```bash title="Enable autoscaler on a pool" theme={null}
    ceph osd pool set <POOL_NAME> pg_autoscale_mode on
    ```

    <Tip>
      Enable `pg_autoscale_mode on` on all pools as a default configuration. The
      autoscaler prevents both under- and over-provisioned PG counts as pool data
      volumes change.
    </Tip>
  </Accordion>

  <Accordion title="OSD down after hardware failure" icon="hard-drive">
    **Cause**: A physical disk or node has failed, causing one or more OSDs to go down.

    **Diagnosis**:

    ```bash title="Identify failed OSDs" theme={null}
    ceph osd tree | grep down
    ```

    **Resolution**:

    1. Identify the failed OSD and its host from `ceph osd tree`
    2. Mark the OSD `out` to begin data recovery on remaining OSDs:
       ```bash title="Mark OSD out" theme={null}
       ceph osd out <OSD_ID>
       ```
    3. Monitor recovery: `watch ceph status`
    4. After recovery completes (status shows `HEALTH_OK`), replace the failed hardware
    5. Redeploy the OSD through XDeploy on the replacement device

    <Warning>
      Do not remove an OSD before the cluster has finished recovering data. Removing
      an OSD during recovery on an already-degraded cluster risks data loss.
    </Warning>
  </Accordion>

  <Accordion title="Slow requests / high latency" icon="clock">
    **Cause**: Blocked OSD operations, high CPU load on OSD nodes, or network congestion
    on the storage network.

    **Diagnosis**:

    ```bash title="Check for blocked operations" theme={null}
    ceph health detail | grep -i slow
    ```

    ```bash title="View OSD performance" theme={null}
    ceph osd perf
    ```

    **Resolution**:

    | Root Cause                            | Resolution                                                   |
    | ------------------------------------- | ------------------------------------------------------------ |
    | HDD fragmentation                     | Schedule `ceph osd scrub <OSD_ID>` during low-traffic period |
    | CPU throttling on OSD nodes           | Review CPU allocation in XDeploy                             |
    | Rebalancing competing with client I/O | Throttle recovery I/O (see below)                            |
    | Full disks on some OSDs               | Add capacity or rebalance via CRUSH weight                   |

    ```bash title="Throttle recovery I/O" theme={null}
    ceph osd set-recovery-delay 5
    ceph osd set-backfillfull-ratio 0.85
    ```
  </Accordion>

  <Accordion title="PG inconsistent" icon="circle-x">
    **Cause**: Data inconsistency detected between OSD replicas during scrubbing.
    This may indicate a hardware error (bad disk sector, bit flip).

    **Diagnosis**:

    ```bash title="Find inconsistent PGs" theme={null}
    ceph health detail | grep inconsistent
    ```

    ```bash title="Get PG details" theme={null}
    ceph pg <PG_ID> query
    ```

    **Resolution**:

    ```bash title="Repair inconsistent PG" theme={null}
    ceph pg repair <PG_ID>
    ```

    <Warning>
      PG repair selects the primary OSD's copy as authoritative and overwrites
      inconsistent replicas. If the primary copy is also corrupt, this may propagate
      corruption. Verify the application-level data integrity after repair.
    </Warning>
  </Accordion>

  <Accordion title="Pool near full / cluster full" icon="gauge">
    **Cause**: Pool or cluster capacity has reached a critical threshold, causing
    the cluster to throttle or reject writes.

    **Diagnosis**:

    ```bash title="Check cluster capacity" theme={null}
    ceph df
    ```

    ```bash title="Identify full OSDs" theme={null}
    ceph osd df | sort -k9 -rn | head -10
    ```

    **Resolution**:

    * **Immediate**: Raise the `full_ratio` temporarily to allow the cluster to accept
      writes while expansion is underway:
      ```bash title="Temporarily raise full ratio (emergency only)" theme={null}
      ceph osd set-full-ratio 0.97
      ```
    * **Short-term**: Delete unnecessary data, snapshots, or expired objects
    * **Long-term**: Add new OSD nodes through XDeploy (see [Capacity Planning](/services/sds/admin-guide/capacity-planning))

    <Danger>
      Operating above 90% utilization with the `full_ratio` raised is an emergency
      measure only. Data loss can occur if any additional OSD failures happen while
      the cluster is in this state.
    </Danger>
  </Accordion>

  <Accordion title="Object gateway (S3) returning errors" icon="globe">
    **Cause**: The RGW service is down, misconfigured, or the backing data pool is unavailable.

    **Diagnosis**:

    ```bash title="Check RGW service status" theme={null}
    ceph orch ps | grep rgw
    ```

    ```bash title="Check RGW logs" theme={null}
    ceph log last 50 | grep rgw
    ```

    **Resolution**:

    ```bash title="Restart RGW service" theme={null}
    ceph orch restart rgw.<SERVICE_ID>
    ```

    If RGW remains down after restart, check the data pool health:

    ```bash title="Check RGW data pool" theme={null}
    ceph osd pool ls detail | grep rgw
    ceph health detail | grep <RGW_POOL>
    ```
  </Accordion>
</AccordionGroup>

***

## When to Contact Support

Contact [support@xloud.tech](mailto:support@xloud.tech) if:

* `HEALTH_ERR` persists after initial investigation
* Data appears inaccessible or corrupt after OSD recovery
* PG repair does not resolve the inconsistency
* Cluster is full and expansion cannot be provisioned immediately

When opening a support ticket, include:

```bash title="Information to include with support ticket" theme={null}
ceph health detail
ceph status
ceph osd tree
ceph log last 200
```

***

## Next Steps

<CardGroup>
  <Card title="Cluster Management" href="/services/sds/admin-guide/cluster-management">
    Routine operational procedures — adding OSDs, maintenance mode
  </Card>

  <Card title="Monitoring" href="/services/sds/admin-guide/monitoring">
    Set up proactive alerts to catch issues before they become critical
  </Card>

  <Card title="Capacity Planning" href="/services/sds/admin-guide/capacity-planning">
    Prevent capacity emergencies with proactive expansion planning
  </Card>

  <Card title="User Guide Troubleshooting" href="/services/sds/user-guide/troubleshooting">
    Tenant-facing storage issues — stuck volumes, snapshot failures
  </Card>
</CardGroup>


# Software-Defined Storage CLI Reference
Source: https://docs.xloud.tech/services/sds/cli-reference

Complete ceph CLI commands for managing Xloud XSDS — pools, OSDs, health, CRUSH maps, RBD images, and object gateway.

## Overview

Xloud Software-Defined Storage (XSDS) is powered by Ceph. Use `ceph` and `rbd` CLI commands to manage the cluster, pools, block storage images, and object gateway.

<Note>
  **Prerequisites**

  * SSH access to a Ceph monitor or admin node
  * `ceph` and `rbd` CLI tools installed (`apt install ceph-common`)
  * Ceph keyring with admin permissions: `/etc/ceph/ceph.client.admin.keyring`
</Note>

***

## Cluster Health

<CodeGroup>
  ```bash title="Cluster health summary" theme={null}
  ceph health
  ceph health detail
  ```

  ```bash title="Cluster status" theme={null}
  ceph status
  ceph -s
  ```

  ```bash title="Monitor quorum" theme={null}
  ceph quorum_status --format json | jq .quorum_names
  ```

  ```bash title="OSD tree" theme={null}
  ceph osd tree
  ```

  ```bash title="Cluster usage" theme={null}
  ceph df
  ceph df detail
  ```
</CodeGroup>

***

## Pools

<CodeGroup>
  ```bash title="List pools" theme={null}
  ceph osd pool ls
  ceph osd pool ls detail
  ```

  ```bash title="Create replicated pool" theme={null}
  ceph osd pool create volumes 128 replicated
  ceph osd pool set volumes size 3
  ```

  ```bash title="Create erasure-coded pool" theme={null}
  ceph osd pool create ec-data 64 erasure default
  ```

  ```bash title="Show pool stats" theme={null}
  ceph osd pool stats volumes
  ```

  ```bash title="Get pool parameter" theme={null}
  ceph osd pool get volumes size
  ceph osd pool get volumes crush_rule
  ```

  ```bash title="Set pool parameter" theme={null}
  ceph osd pool set volumes size 3
  ```

  ```bash title="Delete pool" theme={null}
  ceph osd pool delete volumes volumes --yes-i-really-really-mean-it
  ```
</CodeGroup>

***

## OSDs

<CodeGroup>
  ```bash title="List OSDs" theme={null}
  ceph osd ls
  ```

  ```bash title="Show OSD stats" theme={null}
  ceph osd df
  ```

  ```bash title="Mark OSD out (before removal)" theme={null}
  ceph osd out osd.3
  ```

  ```bash title="Mark OSD down" theme={null}
  ceph osd down osd.3
  ```

  ```bash title="Remove OSD" theme={null}
  ceph osd purge osd.3 --yes-i-really-mean-it
  ```

  ```bash title="Reweight OSD" theme={null}
  ceph osd reweight osd.3 0.9
  ```
</CodeGroup>

***

## RBD (Block Storage Images)

<CodeGroup>
  ```bash title="List images in pool" theme={null}
  rbd ls volumes
  ```

  ```bash title="Create image" theme={null}
  rbd create --size 50G volumes/my-image
  ```

  ```bash title="Show image info" theme={null}
  rbd info volumes/my-image
  ```

  ```bash title="Resize image" theme={null}
  rbd resize --size 100G volumes/my-image
  ```

  ```bash title="Create snapshot" theme={null}
  rbd snap create volumes/my-image@snap-1
  ```

  ```bash title="List snapshots" theme={null}
  rbd snap ls volumes/my-image
  ```

  ```bash title="Delete snapshot" theme={null}
  rbd snap rm volumes/my-image@snap-1
  ```

  ```bash title="Delete image" theme={null}
  rbd rm volumes/my-image
  ```
</CodeGroup>

***

## Object Gateway (RGW)

<CodeGroup>
  ```bash title="List buckets (admin)" theme={null}
  radosgw-admin bucket list
  ```

  ```bash title="Show bucket stats" theme={null}
  radosgw-admin bucket stats --bucket my-bucket
  ```

  ```bash title="Create RGW user" theme={null}
  radosgw-admin user create \
    --uid my-user \
    --display-name "My User" \
    --email user@example.com
  ```

  ```bash title="Show user info and keys" theme={null}
  radosgw-admin user info --uid my-user
  ```
</CodeGroup>

***

## Next Steps

<CardGroup>
  <Card title="XSDS Architecture" href="/services/sds/admin-guide/architecture">
    Understand Ceph cluster components and data placement
  </Card>

  <Card title="Pool Management" href="/services/sds/admin-guide/pool-management">
    Create and configure Ceph storage pools
  </Card>
</CardGroup>


# Software-Defined Storage
Source: https://docs.xloud.tech/services/sds/index

Unified block, object, and file storage on commodity hardware — petabyte-scale, self-healing, and multi-protocol with XSDS.

Xloud Software-Defined Storage (XSDS) delivers unified block, object, and file storage
on commodity hardware — scaling from terabytes to petabytes with no single point of
failure. A self-healing architecture continuously monitors and redistributes data to
maintain durability, while multi-protocol access lets applications consume storage
through the interface that suits them best.

<Card title="XSDS — Software-Defined Storage" icon="external-link" href="https://xloud.tech/ceph">
  Product details, hardware compatibility matrix, performance benchmarks, and datasheet on xloud.tech
</Card>

***

<p>XSDS Documentation</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/sds/user-guide">
    Consume block volumes, object buckets, and shared file systems from your workloads.
    Covers access methods, data protection policies, and performance configuration.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/sds/admin-guide">
    Deploy and operate distributed storage clusters. Configure pools, CRUSH maps, storage
    tiers, encryption, and capacity planning for production environments.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/sds/cli-reference">
    Manage Ceph clusters, pools, OSDs, RBD images, and object storage buckets using
    ceph, rbd, and radosgw-admin CLI tools.
  </Card>

  <Card title="Block Storage" icon="hard-drive" href="/services/storage/user-guide">
    Persistent block volumes for Xloud Compute instances — powered by the XSDS backend.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Unified Storage" icon="layers" href="/services/sds/user-guide/storage-types">
    Single platform delivers block, object, and file storage — eliminating the need for
    separate storage appliances per protocol.
  </Card>

  <Card title="Self-Healing Architecture" icon="shield-check" href="/services/sds/admin-guide/architecture">
    Continuous health monitoring detects failed components and automatically re-replicates
    data to restore the configured redundancy level without operator intervention.
  </Card>

  <Card title="Petabyte Scalability" icon="trending-up" href="/services/sds/admin-guide/capacity-planning">
    Scale storage capacity by adding nodes to the cluster — no downtime, no data migration,
    and no architectural limits on growth.
  </Card>

  <Card title="Data Protection" icon="lock" href="/services/sds/user-guide/data-protection">
    Configurable replication and erasure coding protect data across nodes, racks, and
    data center rooms. Encryption at rest secures all stored data.
  </Card>

  <Card title="Performance Optimization" icon="zap" href="/services/sds/user-guide/performance">
    Inline deduplication, compression, and intelligent caching maximize throughput and
    minimize latency. Automated tiering places hot data on faster media.
  </Card>

  <Card title="Multi-Protocol Access" icon="network" href="/services/sds/user-guide/access-methods">
    Native object (S3-compatible), block (RBD), and file (CephFS) interfaces alongside
    NFS and SMB gateways for legacy application compatibility.
  </Card>
</CardGroup>

***

<p>Use Cases</p>

<CardGroup>
  <Card title="Scalable Cloud Storage" icon="cloud">
    Foundational storage layer for private cloud deployments — block volumes for instances,
    object storage for unstructured data, and file shares for collaborative workloads.
  </Card>

  <Card title="Big Data & Analytics" icon="chart-bar">
    High-throughput sequential I/O and massive capacity make XSDS ideal for data lake
    architectures, log aggregation, and analytics pipelines.
  </Card>

  <Card title="Content Delivery & Media" icon="film">
    S3-compatible object storage with horizontal scalability for large media libraries,
    software distribution, and static asset hosting.
  </Card>

  <Card title="Backup & Recovery" icon="rotate-ccw">
    Cost-efficient backup target using erasure coding for high durability at lower raw
    capacity overhead compared to full replication.
  </Card>
</CardGroup>

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Virtual machine instances that consume XSDS block volumes as persistent disks
  </Card>

  <Card title="Xloud Block Storage" icon="hard-drive" href="/services/storage/index">
    Persistent block volumes backed by the XSDS distributed storage backend
  </Card>

  <Card title="Disaster Recovery" icon="rotate-ccw" href="/services/disaster-recovery/index">
    Protect workloads and data with XDR replication and automated failover
  </Card>
</CardGroup>


# Access Methods
Source: https://docs.xloud.tech/services/sds/user-guide/access-methods

Connect to XSDS block, object, and file storage through the Dashboard, CLI, S3-compatible API, and native mount protocols.

## Overview

XSDS exposes storage through multiple access protocols. The appropriate method depends
on the consumer application and the storage type being accessed. This page covers all
supported access methods with configuration steps for each.

<Note>
  **Prerequisites**

  * An active Xloud account with project member access
  * For CLI access: `openstack` CLI installed ([CLI Setup](/cli-setup))
  * For S3 API access: S3 access keys generated from the Dashboard
</Note>

***

## Access Method Overview

| Access Method         | Storage Type  | Typical User                                        |
| --------------------- | ------------- | --------------------------------------------------- |
| **Xloud Dashboard**   | Block, Object | GUI-based management and exploration                |
| **`openstack` CLI**   | Block, Object | Scripted management and automation                  |
| **S3-compatible API** | Object        | Application integration, SDK access                 |
| **NFS / SMB mount**   | Shared File   | Legacy application and workstation access           |
| **iSCSI / RBD**       | Block         | Hypervisor-level attachment (managed automatically) |

***

## Dashboard Access

<Tabs>
  <Tab title="Block Storage" icon="hard-drive">
    <Steps>
      <Step title="Navigate to Volumes" icon="hard-drive">
        Log in to the **Xloud Dashboard** (`https://connect.<your-domain>`) and navigate to
        **Project → Volumes → Volumes**.
      </Step>

      <Step title="Manage volumes" icon="settings">
        From the volume list you can:

        * **Create** a new volume (click **Create Volume**)
        * **Attach/Detach** volumes to instances (via the Actions menu)
        * **Create Snapshot** for point-in-time backup
        * **Extend** volume size
        * **Transfer** volume to another project

        <Tip>
          Use the **Filter** bar to search volumes by name, status, or volume type across
          large projects.
        </Tip>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Object Storage" icon="box">
    <Steps>
      <Step title="Navigate to Object Store" icon="box">
        Navigate to **Project → Object Store → Containers**. Containers in the Dashboard
        correspond to S3 buckets.
      </Step>

      <Step title="Manage containers and objects" icon="folder">
        * **Create Container** to provision a new bucket
        * **Upload** files directly from the browser
        * **Set Access** to configure public or private access
        * **Generate Temp URL** for time-limited pre-signed links

        <Note>
          The Dashboard uses the Swift API naming convention ("containers") while the
          S3-compatible API uses "buckets". Both refer to the same underlying objects.
        </Note>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## CLI Access

<Tabs>
  <Tab title="Block Storage CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Common volume commands" icon="hard-drive">
        ```bash title="List volumes" theme={null}
        openstack volume list
        ```

        ```bash title="Create a volume" theme={null}
        openstack volume create \
          --size 50 \
          --type ceph-ssd \
          --description "Application data disk" \
          app-data-01
        ```

        ```bash title="Show volume details" theme={null}
        openstack volume show app-data-01
        ```

        ```bash title="Attach to a running instance" theme={null}
        openstack server add volume <INSTANCE_ID> app-data-01
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Object Storage CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Common object storage commands" icon="box">
        ```bash title="List containers" theme={null}
        openstack container list
        ```

        ```bash title="Create a container" theme={null}
        openstack container create my-bucket
        ```

        ```bash title="Upload an object" theme={null}
        openstack object create my-bucket /local/path/to/file.tar.gz
        ```

        ```bash title="Download an object" theme={null}
        openstack object save my-bucket file.tar.gz
        ```

        ```bash title="List objects in a container" theme={null}
        openstack object list my-bucket
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## S3-Compatible API

The XSDS object storage service exposes a fully S3-compatible API. Any application or
tool that supports the AWS S3 API can connect to XSDS without modification.

<Tabs>
  <Tab title="Generate Credentials" icon="key">
    <Steps>
      <Step title="Navigate to Access Keys" icon="key">
        In the Xloud Dashboard, navigate to **Project → Object Store → Access Keys**
        and click **Create Key**.
      </Step>

      <Step title="Save your credentials" icon="file-text">
        The Access Key ID and Secret Access Key are shown once. Store them securely
        — they cannot be retrieved after the dialog is closed.

        <Warning>
          Treat S3 access keys as sensitive credentials. Do not commit them to source
          control. Use environment variables or a secrets manager in application code.
        </Warning>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Python (boto3)" icon="code">
    ```python title="Connect with boto3" theme={null}
    import boto3

    s3 = boto3.client(
        's3',
        endpoint_url='https://object.<your-domain>',
        aws_access_key_id='YOUR_ACCESS_KEY',
        aws_secret_access_key='YOUR_SECRET_KEY',
        region_name='xloud-region-1'
    )

    # List buckets
    response = s3.list_buckets()
    for bucket in response['Buckets']:
        print(bucket['Name'])

    # Upload a file
    s3.upload_file('local_file.txt', 'my-bucket', 'remote_object.txt')

    # Download a file
    s3.download_file('my-bucket', 'remote_object.txt', 'downloaded_file.txt')
    ```
  </Tab>

  <Tab title="CLI (aws s3)" icon="terminal">
    ```bash title="Configure AWS CLI profile" theme={null}
    aws configure --profile xloud
    # AWS Access Key ID: YOUR_ACCESS_KEY
    # AWS Secret Access Key: YOUR_SECRET_KEY
    # Default region name: xloud-region-1
    # Default output format: json
    ```

    ```bash title="List buckets" theme={null}
    aws s3 ls --endpoint-url https://object.<your-domain> --profile xloud
    ```

    ```bash title="Upload a file" theme={null}
    aws s3 cp local_file.txt \
      s3://my-bucket/remote_object.txt \
      --endpoint-url https://object.<your-domain> \
      --profile xloud
    ```

    ```bash title="Sync a directory" theme={null}
    aws s3 sync ./local-dir/ \
      s3://my-bucket/remote-dir/ \
      --endpoint-url https://object.<your-domain> \
      --profile xloud
    ```
  </Tab>
</Tabs>

***

## NFS / SMB Mount (Shared File Storage)

<Tabs>
  <Tab title="NFS (Linux)" icon="terminal">
    <Steps>
      <Step title="Get the NFS export path" icon="folder-open">
        Navigate to **Project → Shared File Systems → Shares** and note the export
        path for your share (format: `<gateway-ip>:/<share-path>`).
      </Step>

      <Step title="Mount the share" icon="hard-drive">
        ```bash title="Create mount point" theme={null}
        mkdir -p /mnt/shared-data
        ```

        ```bash title="Mount via NFS" theme={null}
        mount -t nfs \
          -o vers=4,rw,hard,intr \
          <gateway-ip>:/<share-path> \
          /mnt/shared-data
        ```
      </Step>

      <Step title="Persist mount across reboots" icon="file-text">
        Add to `/etc/fstab`:

        ```bash title="/etc/fstab entry" theme={null}
        <gateway-ip>:/<share-path>  /mnt/shared-data  nfs  vers=4,rw,hard,intr  0  0
        ```

        <Check>Mount is accessible at `/mnt/shared-data` and persists after reboot.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="SMB (Windows)" icon="monitor">
    <Steps>
      <Step title="Get the SMB share path" icon="folder-open">
        Navigate to **Project → Shared File Systems → Shares** and note the SMB
        UNC path for your share (format: `\\<gateway-ip>\<share-name>`).
      </Step>

      <Step title="Map network drive" icon="hard-drive">
        In Windows Explorer, right-click **This PC** and select **Map network drive**.
        Enter the UNC path and provide your Xloud credentials when prompted.

        Or from PowerShell:

        ```powershell title="Map network drive (PowerShell)" theme={null}
        net use Z: \\<gateway-ip>\<share-name> /user:<USERNAME> <PASSWORD> /persistent:yes
        ```

        <Check>Drive Z: appears in Windows Explorer and is accessible to applications.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Data Protection" href="/services/sds/user-guide/data-protection">
    Configure replication and erasure coding to protect your data
  </Card>

  <Card title="Snapshots" href="/services/sds/user-guide/snapshots">
    Create and restore point-in-time snapshots for volumes and buckets
  </Card>

  <Card title="Performance" href="/services/sds/user-guide/performance">
    Storage tiering, deduplication, and caching to optimize I/O
  </Card>

  <Card title="Troubleshooting" href="/services/sds/user-guide/troubleshooting">
    Diagnose and resolve common access and connectivity issues
  </Card>
</CardGroup>


# Data Protection
Source: https://docs.xloud.tech/services/sds/user-guide/data-protection

Configure replication and erasure coding in XSDS to protect your data against hardware failures with the right balance of durability and storage efficiency.

## Overview

XSDS offers two data protection strategies: **replication** and **erasure coding**.
The appropriate choice depends on your durability requirements, acceptable storage
overhead, and the latency sensitivity of your workload.

<Note>
  **Prerequisites**

  * An active Xloud account with project member access
  * Contact your storage administrator to provision erasure-coded pools — they are
    configured at the cluster level by an administrator. Your administrator can configure this through [XDeploy](/deployment).
</Note>

***

## Protection Strategy Comparison

| Strategy                   | Storage Overhead    | Latency | Failure Tolerance           | Best For                                         |
| -------------------------- | ------------------- | ------- | --------------------------- | ------------------------------------------------ |
| **Replication (factor 3)** | 3× raw capacity     | Low     | 2 simultaneous OSD failures | Databases, boot volumes, latency-critical        |
| **Erasure Coding (4+2)**   | 1.5× raw capacity   | Higher  | 2 OSD failures              | Backups, archives, large object stores           |
| **Erasure Coding (8+3)**   | 1.375× raw capacity | Higher  | 3 OSD failures              | Very large datasets requiring maximum efficiency |

***

## Replication

Replication stores multiple complete copies of every object across separate OSDs.
The default replication factor is **3** — three full copies are maintained at all times.

<Tabs>
  <Tab title="How It Works" icon="copy">
    When data is written to a replicated pool, XSDS simultaneously writes it to the
    number of OSDs defined by the replication factor. Each copy is placed on a
    different host according to the CRUSH map, ensuring that host-level failures
    do not cause data loss.

    | Replication Factor | Copies   | Survives Simultaneous Failures | Storage Overhead |
    | ------------------ | -------- | ------------------------------ | ---------------- |
    | 2                  | 2 copies | 1 OSD failure                  | 2×               |
    | 3 (default)        | 3 copies | 2 OSD failures                 | 3×               |
    | 4                  | 4 copies | 3 OSD failures                 | 4×               |

    **Characteristics:**

    * Lower read/write latency compared to erasure coding
    * Higher raw storage overhead (3× capacity for factor-3)
    * Recovery is faster after an OSD failure
    * Best suited for latency-sensitive workloads (databases, boot volumes)
  </Tab>

  <Tab title="Selecting a Replicated Pool" icon="gauge">
    Volume types in the Xloud Dashboard map to specific storage pools. Replicated pools
    appear as volume types with names such as `ceph-ssd` or `ceph-nvme`.

    <Steps>
      <Step title="Check available volume types" icon="list">
        ```bash title="List available volume types" theme={null}
        openstack volume type list
        ```

        Contact your storage administrator to determine which volume types are backed
        by replicated pools versus erasure-coded pools. Your administrator can configure this through [XDeploy](/deployment).
      </Step>

      <Step title="Create a volume on a replicated pool" icon="plus">
        ```bash title="Create volume on SSD-replicated pool" theme={null}
        openstack volume create \
          --size 100 \
          --type ceph-ssd \
          prod-database-data
        ```

        <Check>Volume is created with the specified type and the replication factor
        defined by the pool configuration.</Check>
      </Step>
    </Steps>

    <Tip>
      Use replication factor 3 for all production block volumes and actively-accessed
      object data. Reserve factor 2 only for non-critical development environments.
    </Tip>
  </Tab>
</Tabs>

***

## Erasure Coding

Erasure coding divides data into data chunks and parity chunks, distributing them
across OSDs. The data can be reconstructed from any sufficient subset of chunks,
providing durability at significantly lower storage overhead than replication.

<Tabs>
  <Tab title="How It Works" icon="code">
    In an erasure-coded profile `k+m`, data is split into `k` data chunks and `m`
    parity chunks. The pool can tolerate the loss of any `m` OSDs and reconstruct
    the data from the remaining `k` chunks.

    | Profile | Data Chunks (k) | Parity Chunks (m) | Overhead | Failure Tolerance |
    | ------- | --------------- | ----------------- | -------- | ----------------- |
    | 4+2     | 4               | 2                 | 1.5×     | 2 OSD failures    |
    | 6+2     | 6               | 2                 | 1.33×    | 2 OSD failures    |
    | 8+3     | 8               | 3                 | 1.375×   | 3 OSD failures    |

    **Characteristics:**

    * Significantly lower storage overhead compared to replication
    * Higher CPU and network overhead for encoding/decoding
    * Best suited for large objects: backups, archives, cold data
    * Recovery after failure takes longer than replication
  </Tab>

  <Tab title="Using Erasure-Coded Pools" icon="gauge">
    Erasure-coded pools are configured by your storage administrator. Once provisioned,
    they appear as volume types or object storage containers using that pool.

    <Note>
      Erasure coding is enabled at the pool level during pool creation by an
      administrator. Contact your storage administrator to provision an erasure-coded
      pool for your project. Your administrator can configure this through [XDeploy](/deployment). See the [XSDS Admin Guide](/services/sds/admin-guide/pool-management)
      for pool creation procedures.
    </Note>

    To verify which pool a volume type uses:

    ```bash title="Show volume type details" theme={null}
    openstack volume type show <TYPE_NAME> -c extra_specs
    ```

    The `volume_backend_name` extra spec maps to the storage pool on the backend.
  </Tab>
</Tabs>

***

## Validation

Confirm your volume is using the expected protection policy:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Project → Volumes → Volumes** and click on the volume name.
    The **Volume Type** field shows which pool backs this volume. Contact your
    storage administrator to confirm the protection scheme for that pool type. Your administrator can configure this through [XDeploy](/deployment).
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show volume type" theme={null}
    openstack volume show <VOLUME_NAME> -c volume_type
    ```

    ```bash title="Show pool backing the volume type" theme={null}
    openstack volume type show <VOLUME_TYPE> -c extra_specs
    ```

    <Check>The `volume_backend_name` extra spec confirms which storage pool and
    protection scheme is active for your volume.</Check>
  </Tab>
</Tabs>

***

## Best Practices

<CardGroup>
  <Card title="Production Workloads" icon="server">
    Use replication factor 3 for all production databases, application volumes, and
    frequently-accessed data. The lower latency outweighs the higher storage cost for
    latency-sensitive workloads.
  </Card>

  <Card title="Backup & Archive" icon="archive">
    Use erasure-coded pools (4+2 or 6+2) for backup targets and archives. The storage
    efficiency savings are significant at scale, and higher latency is acceptable for
    infrequent access patterns.
  </Card>

  <Card title="Development Environments" icon="flask-conical">
    Replication factor 2 is sufficient for non-critical development and test volumes.
    Reduces storage consumption while maintaining single-failure protection.
  </Card>

  <Card title="Pre-Maintenance Snapshots" icon="copy">
    Combine data protection pools with regular snapshots for layered protection. Pools
    protect against hardware failures; snapshots protect against logical corruption and
    accidental deletion.
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="Snapshots" href="/services/sds/user-guide/snapshots">
    Create point-in-time snapshots to complement pool-level data protection
  </Card>

  <Card title="Storage Tiers" href="/services/sds/user-guide/performance">
    Route workloads to NVMe, SSD, or HDD tiers based on access patterns
  </Card>

  <Card title="XSDS Admin — Pool Management" href="/services/sds/admin-guide/pool-management">
    Create and configure replicated and erasure-coded pools (administrator)
  </Card>

  <Card title="XSDS Admin — CRUSH Maps" href="/services/sds/admin-guide/crush-maps">
    Configure failure domains and device class rules (administrator)
  </Card>
</CardGroup>


# Performance
Source: https://docs.xloud.tech/services/sds/user-guide/performance

Optimize XSDS storage performance using storage tiering, deduplication, compression, and read caching for your workload's I/O profile.

## Overview

XSDS provides several performance optimization features that can be configured
independently or combined to match your workload's I/O requirements. Understanding
how each feature works helps you select the right combination without unnecessary
overhead.

<Note>
  **Prerequisites**

  * An active Xloud account with project member access
  * Storage tiering, deduplication, and caching are configured at the pool level by
    an administrator — contact your storage administrator to enable these features. Your administrator can configure this through [XDeploy](/deployment).
</Note>

***

## Storage Tiering

XSDS supports multiple storage device classes within a single cluster. Administrators
configure volume types that map to specific device classes, allowing you to direct
each workload to the appropriate media tier.

| Tier     | Device Class    | Volume Type | Typical Use Case                                          |
| -------- | --------------- | ----------- | --------------------------------------------------------- |
| **NVMe** | NVMe SSD        | `ceph-nvme` | Databases, high-IOPS OLTP, latency-critical applications  |
| **SSD**  | SATA/SAS SSD    | `ceph-ssd`  | General-purpose workloads, web servers, application tiers |
| **HDD**  | Hard Disk Drive | `ceph-hdd`  | Backups, archives, cold data, large sequential workloads  |

<Tabs>
  <Tab title="Selecting a Tier" icon="gauge">
    <Steps>
      <Step title="View available volume types" icon="list">
        ```bash title="List volume types" theme={null}
        openstack volume type list
        ```

        Volume types prefixed with `ceph-` correspond to XSDS-backed tiers.
      </Step>

      <Step title="Create a volume on the target tier" icon="plus">
        ```bash title="Create an NVMe-tier volume" theme={null}
        openstack volume create \
          --size 200 \
          --type ceph-nvme \
          prod-db-nvme
        ```

        ```bash title="Create an HDD-tier archive volume" theme={null}
        openstack volume create \
          --size 10000 \
          --type ceph-hdd \
          archive-cold-data
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Automatic Tiering" icon="zap">
    Xloud can automate tiering based on access patterns — hot data migrates to faster
    media automatically while cold data moves to higher-capacity, lower-cost tiers.

    Automatic tiering is managed through the [Xloud Resource Optimizer](/services/optimization/user-guide).
    The optimizer monitors access patterns and issues migration recommendations that
    can be applied manually or executed automatically.

    <Tip>
      If your workload exhibits strong temporal locality (recent data is hot, older
      data is cold), automatic tiering can significantly reduce costs while maintaining
      performance for active data.
    </Tip>
  </Tab>
</Tabs>

***

## Deduplication and Compression

Inline deduplication and compression reduce the effective storage footprint of
compressible workloads.

<AccordionGroup>
  <Accordion title="Deduplication" icon="copy">
    Deduplication eliminates redundant data blocks across all objects in a pool.
    When two objects contain identical blocks, only one physical copy is stored.

    * Transparent to applications — no changes to client code required
    * Most effective for backup workloads (multiple similar backup sets)
    * Effectiveness varies: typical savings range from 1.5× to 4× for backup data
    * CPU-intensive — may reduce throughput on write-heavy workloads

    <Note>
      Deduplication is enabled at the pool level by an administrator. Check with your
      storage administrator whether deduplication is active on your assigned pools.
    </Note>
  </Accordion>

  <Accordion title="Compression" icon="minimize-2">
    Compression applies lossless compression to stored data before writing to disk.
    Common algorithms include LZ4 (fast, lower ratio) and ZSTD (slower, better ratio).

    * Transparent to applications — reads/writes use normal protocols
    * Most effective for text data, logs, and structured data formats (JSON, CSV)
    * Less effective for already-compressed formats (JPEG, MP4, ZIP, encrypted data)
    * Typical savings: 1.2× to 2× depending on data type

    To check whether compression is enabled on your pool:

    ```bash title="Check pool compression" theme={null}
    openstack volume type show <TYPE_NAME> -c extra_specs
    ```
  </Accordion>
</AccordionGroup>

***

## Read Caching

A tiered caching layer accelerates read-intensive workloads by promoting hot data
to a faster media tier (typically SSD or NVMe) while the bulk of data resides on
slower, higher-capacity devices.

<Tabs>
  <Tab title="How Caching Works" icon="zap">
    When the caching tier is active:

    1. Frequently-accessed data blocks are automatically promoted from the capacity
       tier to the cache tier
    2. Subsequent reads are served directly from the faster cache
    3. Cache eviction moves cold data back to the capacity tier without data loss

    * Effective for workloads with a working set significantly smaller than total dataset size
    * Cache promotion is automatic and policy-driven — no application changes required
    * Latency for cached reads approaches native NVMe/SSD latency

    <Tip>
      If your workload is primarily write-heavy or exhibits no temporal access locality,
      caching provides limited benefit. Use a native SSD or NVMe-backed pool instead.
    </Tip>
  </Tab>

  <Tab title="Enabling Caching" icon="settings">
    Read caching is configured at the pool level by an administrator. Contact your
    storage administrator to enable a cache tier for your storage pool. Your administrator can configure this through [XDeploy](/deployment).

    Once enabled, caching is transparent — your existing volumes automatically benefit
    from the cache without re-creating or migrating data.

    <Note>
      Cache pools are shared by all volumes in the backing pool. Heavy write workloads
      from one project can evict cache entries for others. Contact your administrator to
      discuss cache tier isolation options for production workloads. Your administrator can configure this through [XDeploy](/deployment).
    </Note>
  </Tab>
</Tabs>

***

## Performance Validation

Measure the effective I/O performance of your storage configuration:

<Tabs>
  <Tab title="Basic I/O Test" icon="terminal">
    From inside a Xloud Compute instance with a volume attached:

    ```bash title="Sequential write throughput (1 GB test)" theme={null}
    dd if=/dev/zero of=/dev/vdb bs=1M count=1024 oflag=direct
    ```

    ```bash title="Random read IOPS (4K blocks)" theme={null}
    fio --name=random-read \
      --filename=/dev/vdb \
      --rw=randread \
      --bs=4k \
      --numjobs=4 \
      --iodepth=32 \
      --runtime=60 \
      --group_reporting
    ```

    <Warning>
      Run I/O tests on a dedicated test volume, not on a volume containing production
      data. Direct device tests (`/dev/vdb`) will corrupt any file system on that device.
    </Warning>
  </Tab>

  <Tab title="Interpreting Results" icon="chart-bar">
    Expected performance ranges by tier (approximate, varies by cluster load):

    | Tier | Sequential Read   | Sequential Write | Random 4K IOPS |
    | ---- | ----------------- | ---------------- | -------------- |
    | NVMe | 3–5 GB/s          | 2–4 GB/s         | 200K–400K      |
    | SSD  | 500 MB/s–1.5 GB/s | 400 MB/s–1 GB/s  | 50K–150K       |
    | HDD  | 100–300 MB/s      | 80–200 MB/s      | 200–500        |

    If measured performance is significantly below these ranges, open a support ticket
    with the fio output — the storage team can investigate backend bottlenecks.
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Storage Types" href="/services/sds/user-guide/storage-types">
    Understand which storage interface best fits your workload's access pattern
  </Card>

  <Card title="Data Protection" href="/services/sds/user-guide/data-protection">
    Configure replication and erasure coding for durability
  </Card>

  <Card title="XSDS Admin — Storage Tiers" href="/services/sds/admin-guide/storage-tiers">
    Configure multi-tier storage pools and device class rules (administrator)
  </Card>

  <Card title="Resource Optimizer" href="/services/optimization/user-guide">
    Automate data placement across tiers based on access patterns
  </Card>
</CardGroup>


# Snapshots
Source: https://docs.xloud.tech/services/sds/user-guide/snapshots

Create, manage, and restore XSDS volume snapshots for point-in-time recovery, data cloning, and pre-maintenance protection.

## Overview

Snapshots capture the state of a volume at a specific point in time. They are
space-efficient — only changed blocks are stored after the initial snapshot is taken.
Snapshots can be used to restore a volume to a previous state or to create new volumes
pre-populated with the same data.

<Note>
  **Prerequisites**

  * An active Xloud account with project member access
  * Source volume must exist and be accessible
  * Sufficient snapshot quota in your project (check with your administrator)
</Note>

***

## Creating Snapshots

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Volumes" icon="hard-drive">
        Navigate to
        **Project → Volumes → Volumes**.
      </Step>

      <Step title="Create a snapshot" icon="copy">
        In the volume list, click the **Actions** dropdown next to the target volume and
        select **Create Snapshot**.

        | Field             | Description                                         |
        | ----------------- | --------------------------------------------------- |
        | **Snapshot Name** | Descriptive name including date or context          |
        | **Description**   | Optional — notes about the purpose of this snapshot |

        <Tip>
          Take snapshots before making significant configuration changes or before
          applying OS patches. Use a naming convention that includes the date:
          `myvolume-2026-03-18-pre-upgrade`.
        </Tip>
      </Step>

      <Step title="Verify the snapshot" icon="circle-check">
        Navigate to **Project → Volumes → Snapshots**. The snapshot displays with
        status **Available**.

        <Check>Snapshot is available and ready for restore or volume creation.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```
      </Step>

      <Step title="Create a snapshot" icon="copy">
        ```bash title="Create volume snapshot" theme={null}
        openstack volume snapshot create \
          --volume <VOLUME_NAME_OR_ID> \
          --description "Pre-maintenance snapshot $(date +%Y-%m-%d)" \
          snapshot-$(date +%Y%m%d)
        ```
      </Step>

      <Step title="Verify the snapshot" icon="circle-check">
        ```bash title="List snapshots" theme={null}
        openstack volume snapshot list
        ```

        ```bash title="Show snapshot details" theme={null}
        openstack volume snapshot show snapshot-$(date +%Y%m%d)
        ```

        Confirm `status` shows `available`.

        <Check>Status field shows `available` — snapshot is ready.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Restoring from Snapshots

<Tabs>
  <Tab title="Create Volume from Snapshot" icon="gauge">
    Create a new volume pre-populated with the snapshot data. This is the primary
    restore method — the original volume is preserved.

    <Steps>
      <Step title="Locate the snapshot" icon="search">
        Navigate to **Project → Volumes → Snapshots** and find the snapshot to restore.
      </Step>

      <Step title="Create a volume from the snapshot" icon="plus">
        Click **Actions → Create Volume** next to the snapshot.

        | Field           | Value                                             |
        | --------------- | ------------------------------------------------- |
        | **Volume Name** | Descriptive name for the restored volume          |
        | **Size**        | Must be equal to or larger than the snapshot size |
        | **Volume Type** | Select the appropriate storage tier               |

        <Tip>
          You can increase the volume size when restoring. This is useful when
          you need the restored data plus additional free space.
        </Tip>
      </Step>

      <Step title="Attach and mount the restored volume" icon="hard-drive">
        Attach the restored volume to the target instance:

        ```bash title="Attach restored volume" theme={null}
        openstack server add volume <INSTANCE_ID> <RESTORED_VOLUME_NAME>
        ```

        <Check>The restored volume appears as a new block device and contains
        the data from the snapshot point in time.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI Restore" icon="terminal">
    ```bash title="Create volume from snapshot" theme={null}
    openstack volume create \
      --snapshot <SNAPSHOT_NAME_OR_ID> \
      --size <SIZE_GB> \
      --type ceph-ssd \
      restored-volume-$(date +%Y%m%d)
    ```

    ```bash title="Verify the volume is available" theme={null}
    openstack volume show restored-volume-$(date +%Y%m%d) -c status
    ```

    ```bash title="Attach to instance" theme={null}
    openstack server add volume <INSTANCE_ID> restored-volume-$(date +%Y%m%d)
    ```
  </Tab>
</Tabs>

***

## Managing Snapshots

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Project → Volumes → Snapshots** to manage all project snapshots.

    Available actions:

    * **Create Volume** — create a new volume from the snapshot
    * **Edit Snapshot** — update the name or description
    * **Delete Snapshot** — permanently remove the snapshot

    <Warning>
      Snapshot deletion is permanent. Ensure the snapshot is no longer needed before
      deleting. A snapshot cannot be deleted if a volume was created from it and that
      volume still exists.
    </Warning>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List all snapshots" theme={null}
    openstack volume snapshot list --long
    ```

    ```bash title="Delete a snapshot" theme={null}
    openstack volume snapshot delete <SNAPSHOT_NAME_OR_ID>
    ```

    ```bash title="Check project snapshot quota" theme={null}
    openstack quota show --volume
    ```

    <Warning>
      Before deleting a snapshot, verify no volumes depend on it. Deleting a parent
      snapshot while dependent volumes exist is blocked — you must delete the child
      volumes first.
    </Warning>
  </Tab>
</Tabs>

***

## Consistency Considerations

<AccordionGroup>
  <Accordion title="Crash-consistent snapshots" icon="hard-drive">
    By default, XSDS snapshots are crash-consistent — they capture the on-disk state
    at the moment of the snapshot request. This is equivalent to pulling the power cord
    from the machine and is safe for most file systems (ext4, XFS) that use journaling
    for recovery.

    However, for databases and stateful applications, crash-consistent snapshots may
    capture in-flight transactions in an inconsistent state. The database will recover
    on next startup, but some in-flight transactions may be lost.
  </Accordion>

  <Accordion title="Application-consistent snapshots" icon="shield">
    For databases and transactional applications, coordinate the snapshot with
    application-level freeze/thaw procedures:

    ```bash title="PostgreSQL: flush and freeze" theme={null}
    psql -c "SELECT pg_start_backup('snapshot', true);"
    # Create the snapshot now
    openstack volume snapshot create --volume <VOL> snapshot-$(date +%Y%m%d)
    psql -c "SELECT pg_stop_backup();"
    ```

    ```bash title="MySQL: flush tables with read lock" theme={null}
    mysql -e "FLUSH TABLES WITH READ LOCK;"
    # Create the snapshot now
    openstack volume snapshot create --volume <VOL> snapshot-$(date +%Y%m%d)
    mysql -e "UNLOCK TABLES;"
    ```

    <Tip>
      For consistent multi-volume snapshots across a database and its transaction logs,
      freeze all volumes simultaneously before taking snapshots. This ensures the volumes
      are in a mutually consistent state.
    </Tip>
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Data Protection" href="/services/sds/user-guide/data-protection">
    Pool-level replication and erasure coding for hardware failure protection
  </Card>

  <Card title="Disaster Recovery" href="/services/disaster-recovery/user-guide">
    Replicate volumes to a DR site for site-level failure protection
  </Card>

  <Card title="Block Storage Snapshots" href="/services/storage/snapshots">
    Full lifecycle management for block volume snapshots
  </Card>

  <Card title="Troubleshooting" href="/services/sds/user-guide/troubleshooting">
    Diagnose snapshot creation failures and stuck snapshot states
  </Card>
</CardGroup>


# Storage Types
Source: https://docs.xloud.tech/services/sds/user-guide/storage-types

Understand block, object, and shared file storage in XSDS — capabilities, use cases, and how to choose the right interface for your workload.

## Overview

XSDS exposes three distinct storage interfaces from a single distributed platform.
Each interface suits a different application pattern — selecting the right type for
your workload is the first step to optimal performance and cost efficiency.

<Note>
  **Prerequisites**

  * An active Xloud account with project access
  * Familiarity with your application's I/O pattern (random vs sequential, latency vs throughput)
</Note>

***

## Storage Interface Comparison

| Interface               | Protocol           | Access Pattern                 | Typical Use Case                     |
| ----------------------- | ------------------ | ------------------------------ | ------------------------------------ |
| **Block Storage**       | RBD / iSCSI        | Random read/write, low latency | Instance boot disks, databases, OLTP |
| **Object Storage**      | S3-compatible HTTP | Large objects, high throughput | Backups, media files, data lakes     |
| **Shared File Storage** | NFS / SMB          | Concurrent multi-client        | Home directories, shared configs     |

***

## Block Storage

<Tabs>
  <Tab title="Overview" icon="hard-drive">
    Block storage volumes attach to Xloud Compute instances as persistent disks. Volumes
    exist independently of instance lifecycle — data survives instance deletion and can
    be re-attached to a different instance.

    **Capabilities:**

    | Capability    | Details                                                  |
    | ------------- | -------------------------------------------------------- |
    | Hot-attach    | Attach to running instances without rebooting            |
    | Online resize | Extend volume size without downtime (extend only)        |
    | Snapshots     | Point-in-time copies for recovery or cloning             |
    | Encryption    | At-rest encryption via the Xloud Key Management service  |
    | QoS           | IOPS and throughput limits configurable via volume types |

    <Tip>
      Select a volume type that matches your workload's I/O profile. NVMe-backed types
      suit latency-sensitive databases; HDD-backed types suit large sequential workloads
      like backups and archives.
    </Tip>
  </Tab>

  <Tab title="Best For" icon="circle-check">
    Block storage is ideal when your workload requires:

    * Exclusive read/write access from a single instance
    * Predictable low-latency I/O (sub-millisecond for NVMe-backed volumes)
    * POSIX-compliant file system on top of the volume (ext4, XFS, etc.)
    * Database workloads: PostgreSQL, MySQL, MongoDB, Redis

    See the [Xloud Block Storage guide](/services/storage/user-guide) for complete
    volume lifecycle management procedures.
  </Tab>
</Tabs>

***

## Object Storage

<Tabs>
  <Tab title="Overview" icon="box">
    Object storage provides an S3-compatible API for storing and retrieving unstructured
    data — images, backups, logs, media files, and application artifacts — at any scale.

    **Capabilities:**

    | Capability             | Details                                                              |
    | ---------------------- | -------------------------------------------------------------------- |
    | S3 API compatibility   | Works with standard S3 clients, SDKs, and tools                      |
    | Bucket ACLs            | Per-bucket and per-object access control                             |
    | Lifecycle policies     | Automate object expiration and tiering                               |
    | Object versioning      | Retain multiple versions of objects for accidental-delete protection |
    | Multi-part upload      | Efficient upload for large files (> 100 MB)                          |
    | Server-side encryption | Transparent encryption at write time                                 |
    | Static website hosting | Serve static content directly from buckets                           |

    <Note>
      Access credentials for object storage are separate from your Xloud Dashboard
      credentials. Generate S3-compatible access keys from the Dashboard under
      **Project → Object Store → Access Keys**.
    </Note>
  </Tab>

  <Tab title="Best For" icon="circle-check">
    Object storage is ideal when your workload requires:

    * Storing large volumes of unstructured data (images, videos, documents, logs)
    * Application integration via standard S3 SDK (boto3, AWS SDK, MinIO client)
    * Multi-region data distribution and static content delivery
    * Cost-efficient backup and archival with erasure-coded pools
  </Tab>
</Tabs>

***

## Shared File Storage

<Tabs>
  <Tab title="Overview" icon="folder-open">
    Shared file storage provides a POSIX-compliant distributed file system accessible
    from multiple instances simultaneously — suitable for home directories, shared
    configuration, and workloads that require concurrent read/write access.

    **Capabilities:**

    | Capability           | Details                                                     |
    | -------------------- | ----------------------------------------------------------- |
    | POSIX semantics      | Full file system compliance including permissions and links |
    | Multi-client access  | Concurrent read/write from many instances at once           |
    | NFS and SMB gateways | Compatible with Linux and Windows clients                   |
    | Per-directory quotas | Control storage consumption per share or directory          |
    | Directory snapshots  | Point-in-time recovery at the directory level               |

    <Warning>
      Shared file storage throughput is distributed across all connected clients. For
      high-IOPS workloads requiring exclusive access, use a dedicated block volume
      instead.
    </Warning>
  </Tab>

  <Tab title="Best For" icon="circle-check">
    Shared file storage is ideal when your workload requires:

    * Concurrent read/write access from multiple instances simultaneously
    * Legacy NFS-mounted application data directories
    * Shared configuration files across a cluster of application nodes
    * Collaborative workloads where multiple users write to the same directory tree
  </Tab>
</Tabs>

***

## Choosing the Right Type

<AccordionGroup>
  <Accordion title="My application is a relational database" icon="database">
    Use **Block Storage**. Databases require low-latency random I/O and exclusive
    access to their data files. Choose an NVMe or SSD-backed volume type for
    production OLTP databases.
  </Accordion>

  <Accordion title="I need to store uploaded user files or media" icon="image">
    Use **Object Storage**. Media files, documents, and user uploads benefit from
    the S3 API's simplicity, scalability, and built-in lifecycle management. Applications
    integrate via the S3 SDK — no mounting required.
  </Accordion>

  <Accordion title="Multiple instances need to read and write the same data" icon="users">
    Use **Shared File Storage**. If your application tier requires a common data
    directory accessible by all instances (e.g., a shared cache, log directory, or
    configuration tree), shared file storage provides concurrent access with POSIX
    semantics.
  </Accordion>

  <Accordion title="I need to store backups at low cost" icon="archive">
    Use **Object Storage** with an erasure-coded pool. Erasure coding reduces storage
    overhead to 1.33–1.5× compared to 3× for replicated storage, making it the most
    cost-efficient option for backup and archival workloads where access is infrequent.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Access Methods" href="/services/sds/user-guide/access-methods">
    Connect to each storage type — Dashboard, CLI, S3 API, and mount protocols
  </Card>

  <Card title="Data Protection" href="/services/sds/user-guide/data-protection">
    Configure replication and erasure coding for each storage type
  </Card>

  <Card title="Snapshots" href="/services/sds/user-guide/snapshots">
    Create and restore snapshots for block volumes and object buckets
  </Card>

  <Card title="Block Storage User Guide" href="/services/storage/user-guide">
    Full volume lifecycle management — create, attach, extend, and back up volumes
  </Card>
</CardGroup>


# Troubleshooting
Source: https://docs.xloud.tech/services/sds/user-guide/troubleshooting

Diagnose and resolve common XSDS user-facing issues — stuck volumes, snapshot failures, object storage performance, and access errors.

## Overview

This page covers common issues encountered when using XSDS block, object, and shared
file storage — including diagnosis steps and resolutions for each scenario.

<Note>
  **Prerequisites**

  * Access to the Xloud Dashboard and CLI (`openstack` CLI authenticated)
  * For advanced diagnostics, contact your storage administrator. Your administrator can configure this through [XDeploy](/deployment).
</Note>

***

## Common Issues

<AccordionGroup>
  <Accordion title="Volume stuck in 'creating' status" icon="clock">
    **Cause**: The storage scheduler could not place the volume on a suitable backend,
    or the backend is temporarily unavailable.

    **Diagnosis**:

    ```bash title="Check volume status and fault message" theme={null}
    openstack volume show <VOLUME_NAME> -c status -c fault
    ```

    Common causes and resolutions:

    | Cause                                         | Resolution                                                                                                                                                                 |
    | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | Insufficient capacity in the target pool      | Choose a different volume type or contact your administrator to expand capacity. Your administrator can configure this through [XDeploy](/deployment).                     |
    | Volume type references an unavailable backend | Try a different volume type; contact your administrator if the issue persists. Your administrator can configure this through [XDeploy](/deployment).                       |
    | Storage service temporarily unhealthy         | Wait 2–3 minutes and check status again; contact your administrator if it persists beyond 5 minutes. Your administrator can configure this through [XDeploy](/deployment). |
    | Quota exceeded                                | Check quota with `openstack quota show --volume` and request an increase from your administrator                                                                           |

    Contact your storage administrator if the volume remains in `creating` state for
    more than 5 minutes. Your administrator can configure this through [XDeploy](/deployment).
  </Accordion>

  <Accordion title="Volume stuck in 'deleting' status" icon="trash">
    **Cause**: A snapshot derived from this volume still exists, or there is an
    ongoing operation that holds a lock on the volume.

    **Diagnosis**:

    ```bash title="List snapshots from this volume" theme={null}
    openstack volume snapshot list --volume <VOLUME_NAME_OR_ID>
    ```

    **Resolution**:

    1. Delete all snapshots that were created from this volume:
       ```bash title="Delete child snapshot" theme={null}
       openstack volume snapshot delete <SNAPSHOT_ID>
       ```
    2. After all child snapshots are deleted, retry the volume deletion:
       ```bash title="Retry volume deletion" theme={null}
       openstack volume delete <VOLUME_NAME_OR_ID>
       ```
  </Accordion>

  <Accordion title="Snapshot creation fails" icon="x-circle">
    **Cause**: The source volume is attached and has in-flight I/O, or the snapshot
    quota for the project has been reached.

    **Diagnosis**:

    ```bash title="Check project snapshot quota" theme={null}
    openstack quota show --volume
    ```

    Look for `snapshots` in the output. If `used >= limit`, request a quota increase.

    **Resolution**:

    * For quota exceeded: contact your administrator to increase the snapshot quota. Your administrator can configure this through [XDeploy](/deployment).
    * For consistency issues: flush application writes before taking a snapshot of
      a database volume (see [Snapshots — Consistency](/services/sds/user-guide/snapshots))

    <Warning>
      Crash-consistent snapshots capture the on-disk state at the moment of the snapshot
      request. For databases and stateful applications, coordinate with application-level
      freeze/thaw procedures to ensure data integrity.
    </Warning>
  </Accordion>

  <Accordion title="Object storage — poor upload/download performance" icon="gauge">
    **Cause**: Large numbers of small objects, high latency between the client and
    the gateway, or network routing through the public internet for intra-cluster traffic.

    **Resolution**:

    * Use multi-part upload for objects larger than 100 MB:
      ```python title="boto3 multi-part upload" theme={null}
      s3.upload_file(
          'large_file.tar.gz', 'my-bucket', 'large_file.tar.gz',
          Config=boto3.s3.transfer.TransferConfig(
              multipart_threshold=1024*1024*100,  # 100 MB
              multipart_chunksize=1024*1024*50    # 50 MB chunks
          )
      )
      ```
    * For small-object workloads, batch objects into larger archives where the
      application permits
    * Verify network path to the storage endpoint — avoid routing through the public
      internet for intra-cluster traffic

    <Tip>
      Use the S3 API endpoint local to your region for lowest latency. Check
      **Project → Object Store → Endpoints** in the Dashboard for your regional endpoint URL.
    </Tip>
  </Accordion>

  <Accordion title="NFS mount fails or is inaccessible" icon="network">
    **Cause**: Firewall rules blocking NFS traffic, incorrect export path, or the
    NFS gateway service is unhealthy.

    **Diagnosis**:

    ```bash title="Test NFS gateway connectivity" theme={null}
    showmount -e <gateway-ip>
    ```

    ```bash title="Check mount connectivity" theme={null}
    rpcinfo -p <gateway-ip>
    ```

    **Resolution**:

    * Ensure security group rules on the client instance permit outbound traffic to
      the NFS gateway on ports 111 (portmapper) and 2049 (NFS)
    * Verify the export path matches exactly what was provided in the Dashboard
    * If `showmount` hangs, the NFS gateway may be temporarily unavailable — contact
      your storage administrator

    <Note>
      NFS port 2049 must be open in the security group applied to client instances.
      Navigate to **Project → Network → Security Groups** and verify the rule exists.
    </Note>
  </Accordion>

  <Accordion title="S3 authentication error (403 Forbidden)" icon="lock">
    **Cause**: The access key or secret key is incorrect, expired, or belongs to a
    different project.

    **Resolution**:

    1. Verify credentials in the Dashboard under **Project → Object Store → Access Keys**
    2. If the key was deleted or lost, generate a new key pair:
       * Navigate to **Project → Object Store → Access Keys → Create Key**
       * Update all applications and configuration files using the old key
    3. Confirm the endpoint URL matches your region:
       ```bash title="Verify S3 endpoint" theme={null}
       openstack catalog show object-store
       ```
  </Accordion>
</AccordionGroup>

***

## Diagnostics Reference

| Issue                    | First Diagnostic Command                         |
| ------------------------ | ------------------------------------------------ |
| Volume not creating      | `openstack volume show <VOL> -c status -c fault` |
| Quota check              | `openstack quota show --volume`                  |
| Snapshot list for volume | `openstack volume snapshot list --volume <VOL>`  |
| Object storage endpoint  | `openstack catalog show object-store`            |
| NFS gateway reachability | `showmount -e <gateway-ip>`                      |

***

## When to Contact Support

Contact [support@xloud.tech](mailto:support@xloud.tech) if:

* A volume has been stuck in `creating` or `deleting` state for more than 10 minutes
* The storage administrator cannot resolve the issue from the cluster admin CLI
* You observe data inconsistency after a snapshot restore
* Object storage bucket contents are missing unexpectedly

<Tip>
  When opening a support ticket, include the output of
  `openstack volume show <VOLUME_ID>` or `openstack volume snapshot show <SNAPSHOT_ID>` —
  the `fault` and `migration_status` fields are particularly useful for diagnosis.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="XSDS Admin Troubleshooting" href="/services/sds/admin-guide/troubleshooting">
    Cluster-level diagnostics for storage administrators — OSD failures, slow requests
  </Card>

  <Card title="Data Protection" href="/services/sds/user-guide/data-protection">
    Configure replication and erasure coding to reduce exposure to hardware failures
  </Card>

  <Card title="Snapshots" href="/services/sds/user-guide/snapshots">
    Best practices for creating consistent snapshots to minimize recovery risk
  </Card>

  <Card title="Support" href="mailto:support@xloud.tech">
    Contact Xloud support for issues that require cluster-level investigation
  </Card>
</CardGroup>


# Block Storage Administration
Source: https://docs.xloud.tech/services/storage/admin-guide

Configure and operate Xloud Block Storage — backends, volume types, storage tiers, quotas, encryption, and migration.

<p>Overview</p>

The Xloud Block Storage service delivers persistent block volumes to compute instances through a distributed architecture of API, scheduler, volume service, and backend driver components. As an administrator, you are responsible for configuring storage backends, defining volume types and QoS policies, managing storage tiers, enforcing project quotas, maintaining service health, and ensuring data security across your deployment.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Tabs>
  <Tab title="XDeploy" icon="server">
    Storage backends, backup configuration, and multi-backend setup are configured through XDeploy:

    <Steps>
      <Step title="Open Configuration" icon="settings">
        Navigate to **XDeploy → Configuration** and select the **Storage** tab.
      </Step>

      <Step title="Select storage backend" icon="database">
        Select the storage backend for your deployment: Ceph RBD, LVM, NFS, iSCSI,
        VMware VMDK, or Pure Storage. Configure backend-specific parameters such as
        pool names, volume groups, or NFS share paths.
      </Step>

      <Step title="Configure backups" icon="archive">
        Set the backup driver and target (Ceph, NFS, or object storage) in the backup
        configuration section.
      </Step>

      <Step title="Save and deploy" icon="rocket">
        Click **Save Configuration**, then navigate to **XDeploy → Operations** and
        run a **Deploy** or **Reconfigure** for the Block Storage service.

        <Check>Storage backends are configured and the volume service is operational.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Configure storage backends by editing `cinder.conf` directly at
    `/etc/xavs/config/cinder/cinder.conf`. See the individual backend guides below
    for detailed configuration parameters.
  </Tab>
</Tabs>

***

<p>Service Components</p>

| Component             | Role                                                                            |
| --------------------- | ------------------------------------------------------------------------------- |
| **Block Storage API** | RESTful endpoint on port 8776; sits behind the load balancer                    |
| **Scheduler**         | Selects the optimal backend for each volume operation using filters and weights |
| **Volume Service**    | Runs on each storage node; interfaces with the backend driver                   |
| **Backup Service**    | Manages backup creation and restoration to a separate backup target             |
| **Database**          | Stores volume, snapshot, backup, and attachment metadata                        |

***

<p>Quick Start</p>

For a new deployment, complete configuration in this order:

<Steps>
  <Step title="Review the architecture" icon="workflow">
    Understand the service components and request flows before configuring.
    See [Service Architecture](/services/storage/architecture).
  </Step>

  <Step title="Configure storage backends" icon="database">
    Register distributed storage (RBD), LVM, or NFS backends with the volume service.
    See [Storage Backends](/services/storage/storage-backends).
  </Step>

  <Step title="Create volume types" icon="layers">
    Define storage tiers and map them to backends. Configure QoS limits.
    See [Volume Types & QoS](/services/storage/volume-types-admin).
  </Step>

  <Step title="Configure storage tiers" icon="stack">
    Map NVMe, SSD, and HDD hardware classes to volume types and set the default.
    See [Storage Tiers](/services/storage/storage-tiers).
  </Step>

  <Step title="Configure the backup service" icon="archive">
    Point the backup service at an object storage or NFS target.
    See [Backup Configuration](/services/storage/backup-config).
  </Step>

  <Step title="Apply security hardening" icon="shield">
    Restrict volume type access, enforce minimums, and audit snapshots.
    See [Security Hardening](/services/storage/security).
  </Step>

  <Step title="Set project quotas" icon="gauge">
    Configure default and per-project storage limits.
    See [Quota Management](/services/storage/quotas).
  </Step>
</Steps>

***

<p>Administration Guides</p>

<CardGroup>
  <Card title="Service Architecture" icon="workflow" href="/services/storage/architecture">
    Components, request flows, and high-availability deployment model
  </Card>

  <Card title="Storage Backends" icon="database" href="/services/storage/storage-backends">
    Configure and verify RBD, LVM, and NFS backend drivers
  </Card>

  <Card title="Volume Types & QoS" icon="gauge" href="/services/storage/volume-types-admin">
    Create volume types, set backend associations, and configure QoS limits
  </Card>

  <Card title="Storage Tiers" icon="layers" href="/services/storage/storage-tiers">
    Map NVMe, SSD, and HDD hardware classes to volume types
  </Card>

  <Card title="Quota Management" icon="chart-bar" href="/services/storage/quotas">
    Set global defaults and per-project storage limits
  </Card>

  <Card title="Volume Migration" icon="move" href="/services/storage/migration">
    Migrate volumes between backends for rebalancing or hardware retirement
  </Card>

  <Card title="Backup Configuration" icon="archive" href="/services/storage/backup-config">
    Configure backup drivers and targets for production data protection
  </Card>

  <Card title="Volume Encryption" icon="lock" href="/services/storage/encryption">
    Enable LUKS at-rest encryption backed by the Key Management service
  </Card>

  <Card title="Security Hardening" icon="shield" href="/services/storage/security">
    Access controls, snapshot visibility, audit logging, and security baseline
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/storage/admin-troubleshooting">
    Service-level diagnostics for volume service, backends, and data operations
  </Card>
</CardGroup>

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Block Storage User Guide" icon="book-open" href="/services/storage/user-guide">
    End-user workflows for creating, attaching, and managing volumes
  </Card>

  <Card title="Compute Admin Guide" icon="server" href="/services/compute/admin-guide">
    Configure compute hosts, manage resources, and set compute quotas
  </Card>

  <Card title="CLI Setup" icon="terminal" href="/cli-setup">
    Install the CLI for administrative block storage operations
  </Card>

  <Card title="Authentication" icon="key" href="/cli-setup">
    Manage admin credentials and project access
  </Card>
</CardGroup>


# Block Storage Troubleshooting (Admin)
Source: https://docs.xloud.tech/services/storage/admin-troubleshooting

Diagnose block storage service issues — volume service failures, backend connectivity, stuck migrations, and encryption errors.

## Overview

This guide covers service-level troubleshooting for Xloud Block Storage administrators. It addresses issues with the volume service, scheduler, backend connectivity, and data operations that are not visible to or resolvable by end users.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Before troubleshooting**

  * Authenticate with admin credentials: `source openrc.sh`
  * Access service logs via XDeploy for detailed error messages
  * For critical production issues, contact [Xloud Support](mailto:support@xloud.tech)
    with the affected volume IDs and log excerpts
</Note>

***

## Service Health Checks

Run these commands first to establish the overall service state:

```bash title="Check all volume service states" theme={null}
openstack volume service list
```

```bash title="List all backend pools and capacity" theme={null}
openstack volume backend pool list --long
```

```bash title="Check API endpoint health" theme={null}
openstack volume list --all-projects --limit 1
```

All services should show `state = up` and `status = enabled`. Any service showing
`down` requires immediate investigation.

***

## Volume Service Issues

<AccordionGroup>
  <Accordion title="Volume service is 'down'" icon="circle-x">
    **Symptom**: `openstack volume service list` shows one or more services with
    state `down`.

    **Cause**: The volume service container has stopped, lost message queue connectivity,
    or the storage backend driver failed to initialize.

    **Resolution**:

    1. Access the affected node through XDeploy and check the volume service container:
       ```bash title="Check container status (on storage node)" theme={null}
       docker ps | grep cinder
       ```

    2. Review container logs for initialization errors:
       Access logs via XDeploy → **Logs → cinder-volume** on the affected node.

    3. Common causes:
       * Message queue (RabbitMQ) connectivity lost — check network and RabbitMQ status
       * Database connection failure — verify MariaDB/Galera cluster health
       * Backend driver error (keyring file missing, pool name wrong) — review driver-specific log entries

    4. After resolving the root cause, restart the volume service via XDeploy.

    <Tip>
      Run `openstack volume service list` 60 seconds after restarting to confirm
      the service re-registers with the scheduler.
    </Tip>
  </Accordion>

  <Accordion title="Scheduler fails to place volumes — 'No valid backend'" icon="triangle-alert">
    **Symptom**: Volume creation fails with "No valid host was found" or scheduler
    filter messages appear in the logs.

    **Cause**: All backends were eliminated by the scheduler filters. Common causes:

    * All backends are at capacity
    * The requested volume type's `volume_backend_name` does not match any active backend
    * The requested availability zone has no active backend

    **Resolution**:

    ```bash title="Verify backend capacity" theme={null}
    openstack volume backend pool list --long
    ```

    ```bash title="Verify volume type extra specs" theme={null}
    openstack volume type show <type-name> -c extra_specs
    ```

    Confirm that `volume_backend_name` in the type's extra specs matches the `name`
    column in the backend pool list.
  </Accordion>
</AccordionGroup>

***

## Backend Connectivity Issues

<AccordionGroup>
  <Accordion title="Backend reporting zero or unknown capacity" icon="database">
    **Symptom**: `openstack volume backend pool list` shows `free_capacity_gb = 0`
    or the backend pool does not appear in the list.

    **Cause**: The volume service cannot connect to the storage cluster to query capacity.

    **Resolution**:

    1. Verify storage cluster health from the storage administration interface.
    2. Verify the authentication keyring file is present on the volume service node:
       ```bash title="Check keyring file (on storage node)" theme={null}
       ls -la /etc/ceph/ceph.client.xloud-volume.keyring
       ```
    3. Verify the pool name matches the configured backend:
       ```bash title="List storage pools" theme={null}
       # Run on a node with storage admin access
       # (command varies by storage backend)
       ```
    4. Restart the volume service via XDeploy after resolving connectivity issues.
  </Accordion>

  <Accordion title="Volume creation succeeds but attachment fails" icon="link">
    **Symptom**: Volumes reach `available` status but fail when attaching to instances.
    Error typically references connection initialization or iSCSI/RBD target.

    **Cause**: The compute node cannot connect to the storage backend to initialize
    the volume attachment. Common causes:

    * Missing storage client package on the compute node
    * Authentication keyring not present on the compute node
    * Network routing between compute and storage nodes is blocked

    **Resolution**:

    1. Verify the storage client package is installed on the compute node
       (e.g., `librbd-dev`, `ceph-common`, or iSCSI initiator packages)
    2. Verify the keyring file is present on the compute node
    3. Test connectivity from the compute node to the storage cluster monitors
  </Accordion>
</AccordionGroup>

***

## Data Operation Issues

<AccordionGroup>
  <Accordion title="Volume migration stuck in 'migrating' status" icon="move">
    **Symptom**: Volume status remains `migrating` for more than 30 minutes with no
    completion.

    **Resolution**:

    ```bash title="Check migration status" theme={null}
    openstack volume show <volume-id> -c migration_status -c status
    ```

    Check volume service logs on both source and destination nodes via XDeploy for
    migration-related errors.

    If permanently stuck and data integrity has been verified:

    ```bash title="Reset volume state (admin only)" theme={null}
    openstack volume set --state available <volume-id>
    ```

    <Warning>
      Resetting the state does not undo a partial migration. Verify data integrity on
      both source and destination backends before resetting. Consult your storage
      backend documentation for checking partial migration state.
    </Warning>
  </Accordion>

  <Accordion title="Snapshot stuck in 'deleting' state" icon="clock">
    **Symptom**: A snapshot remains in `deleting` state for an extended period.

    **Cause**: The backend could not complete the deletion — typically because dependent
    volumes still reference the snapshot, or the storage cluster is degraded.

    **Resolution**:

    1. Check for volumes created from the snapshot:
       ```bash title="List dependent volumes" theme={null}
       openstack volume list --all-projects
       ```
       Look for volumes with `source_volid` matching the snapshot.
    2. Delete dependent volumes first, then retry the snapshot deletion.
    3. If the storage cluster is degraded, restore cluster health before retrying.
  </Accordion>

  <Accordion title="Encryption errors on volume attach" icon="lock">
    **Symptom**: Encrypted volume attachment fails with a key management or dm-crypt error.

    **Diagnosis**:

    1. Verify the Key Management service is running and accessible from the compute node:
       ```bash title="Check key manager connectivity" theme={null}
       openstack secret list
       ```
    2. Confirm the compute service on the affected node can reach the Key Management
       service API (network path, port 9311).
    3. Check compute service logs on the affected node via XDeploy for messages
       containing `barbican`, `secret`, or `crypt`.

    <Warning>
      Encryption key loss means the volume data is permanently inaccessible. Ensure
      the Key Management service is in a high-availability configuration and has a
      database backup before enabling volume encryption in production.
    </Warning>
  </Accordion>
</AccordionGroup>

***

## Recovering Orphaned Volumes

Volumes can become orphaned (in-use with no valid attachment) when compute instances
are force-deleted without detaching their volumes first:

```bash title="Find orphaned volumes (in-use with no valid instance)" theme={null}
openstack volume list --all-projects --status in-use
```

For each result, verify the attached instance still exists:

```bash title="Check attached instance" theme={null}
openstack server show <instance-id>
```

If the instance no longer exists, reset the volume state:

```bash title="Reset orphaned volume to available" theme={null}
openstack volume set --state available <volume-id>
```

***

## Diagnostic Commands Reference

| Command                                               | Purpose                           |
| ----------------------------------------------------- | --------------------------------- |
| `openstack volume service list`                       | Check all service states          |
| `openstack volume backend pool list --long`           | Verify backend capacity           |
| `openstack volume list --all-projects --status error` | Find volumes in error state       |
| `openstack volume snapshot list --all-projects`       | Audit all snapshots               |
| `openstack quota list --detail`                       | Check quota usage across projects |
| `openstack volume show <id> -c migration_status`      | Check migration state             |
| `openstack volume set --state <state> <id>`           | Force-reset volume state (admin)  |

***

## Next Steps

<CardGroup>
  <Card title="User Troubleshooting" href="/services/storage/troubleshooting">
    Common issues from the user perspective
  </Card>

  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Review backend configuration and connectivity requirements
  </Card>

  <Card title="Architecture" href="/services/storage/architecture">
    Understand service components to narrow down failure domains
  </Card>

  <Card title="Contact Support" href="mailto:support@xloud.tech">
    Open a support ticket for unresolved production issues
  </Card>
</CardGroup>


# Block Storage Service Architecture
Source: https://docs.xloud.tech/services/storage/architecture

Understand the Xloud Block Storage service architecture, principal components, message flow, and backend driver model for production deployments.

## Overview

Xloud Block Storage delivers persistent block volumes to compute instances through a distributed service architecture. The service separates its API layer, scheduling logic, and backend drivers — enabling flexible backend configurations, horizontal scaling, and multi-backend deployments where different hardware tiers serve distinct workload categories.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Architecture Diagram

```mermaid theme={null}
graph TD
    Client["Client\n(Dashboard / CLI / API)"] --> LB["Load Balancer\n:8776"]
    LB --> API["Block Storage API"]
    API --> Scheduler["Scheduler\n(Filter & Weigh)"]
    API --> DB["Database\n(Volume Metadata)"]
    Scheduler --> VS1["Volume Service\nNode A"]
    Scheduler --> VS2["Volume Service\nNode B"]
    VS1 --> BE1["Distributed Storage\n(RBD Pool — volumes)"]
    VS2 --> BE1
    VS2 --> BE2["LVM Backend\n(Local)"]
    API --> Backup["Backup Service"]
    Backup --> S3["Backup Target\n(Object Storage)"]
```

***

## Principal Components

| Component             | Default Port | Role                                                                                                            |
| --------------------- | ------------ | --------------------------------------------------------------------------------------------------------------- |
| **Block Storage API** | 8776         | RESTful endpoint for all volume operations; sits behind the load balancer on the VIP                            |
| **Scheduler**         | —            | Selects the appropriate backend for each volume create/migrate request using filter and weight algorithms       |
| **Volume Service**    | —            | Runs on each storage node; communicates with the backend driver to create, attach, snapshot, and delete volumes |
| **Backup Service**    | —            | Manages backup creation, restoration, and deletion to the configured backup target                              |
| **Database**          | 3306         | Stores volume, snapshot, backup, and attachment metadata (MariaDB/Galera cluster)                               |

***

## Request Flow

### Volume Create

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant API as Block Storage API
    participant Scheduler
    participant VS as Volume Service
    participant Backend as Storage Backend

    Client->>API: POST /v2/{project}/volumes
    API->>API: Validate request, check quota
    API->>Scheduler: Create volume (filter backends)
    Scheduler->>Scheduler: Apply filters + weights
    Scheduler->>VS: Create volume on selected backend
    VS->>Backend: Driver: create_volume()
    Backend-->>VS: Volume created (backend ID)
    VS-->>API: Volume metadata updated
    API-->>Client: 202 Accepted (volume ID, status=creating)
    Note over VS,Backend: Status transitions: creating → available
```

### Volume Attach

```mermaid theme={null}
sequenceDiagram
    participant Nova as Compute Service
    participant API as Block Storage API
    participant VS as Volume Service
    participant Backend as Storage Backend
    participant Hypervisor

    Nova->>API: POST /attachments (reserve volume)
    API-->>Nova: Attachment ID, connection info
    Nova->>VS: Initialize connection
    VS->>Backend: Driver: initialize_connection()
    Backend-->>VS: Connection properties (RADOS/iSCSI details)
    VS-->>Nova: Connection properties
    Nova->>Hypervisor: Attach device (virtio-blk/SCSI)
    Nova->>API: Finalize attachment (attachment_complete)
    API-->>Nova: Volume status = in-use
```

***

## Backend Driver Model

The backend driver is the component that communicates with the physical or virtual storage system. Xloud Block Storage supports multiple drivers that can be active simultaneously:

| Backend Driver                | Type               | HA                    | Production Ready                 |
| ----------------------------- | ------------------ | --------------------- | -------------------------------- |
| **RBD (Distributed Storage)** | Distributed        | Yes                   | Yes — recommended for production |
| **LVM**                       | Local block        | No                    | Development / single-node only   |
| **NFS**                       | Network filesystem | Depends on NFS server | Legacy integration               |

<Tip>
  Production deployments should use the distributed storage (RBD) driver. LVM is
  acceptable for single-node development environments and should not be used where
  data durability is required.
</Tip>

***

## Multi-Backend Deployment

Xloud Block Storage supports multiple concurrent backends. The scheduler filters backends
using configurable filters and selects the optimal backend using a weighting algorithm:

| Filter                   | Purpose                                                  |
| ------------------------ | -------------------------------------------------------- |
| `AvailabilityZoneFilter` | Only selects backends in the requested availability zone |
| `CapacityFilter`         | Eliminates backends without sufficient free capacity     |
| `CapabilitiesFilter`     | Matches backend capabilities to volume type extra specs  |
| `DriverFilter`           | Custom filter expressions in backend configuration       |

The `CapacityWeigher` (default) prioritizes backends with more free capacity to spread
load across the cluster.

***

## High Availability

In a high-availability deployment:

* The **Block Storage API** runs on all control plane nodes, load-balanced via the VIP
* The **Scheduler** runs on all control plane nodes (active-active)
* The **Volume Service** runs on each storage node (active-active per backend)
* The **Database** uses Galera multi-master replication across control plane nodes
* The **Backup Service** runs on one or more dedicated nodes

<Note>
  XDeploy automatically configures high-availability service placement based on your
  cluster topology. Manual service placement is not required for standard deployments.
</Note>

***

## Next Steps

<CardGroup>
  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Configure distributed storage, LVM, and NFS backend drivers
  </Card>

  <Card title="Volume Types & QoS" href="/services/storage/volume-types-admin">
    Create volume types and enforce I/O quality-of-service limits
  </Card>

  <Card title="Storage Tiers" href="/services/storage/storage-tiers">
    Configure NVMe, SSD, and HDD tier mappings for multi-tier deployments
  </Card>

  <Card title="Admin Guide" href="/services/storage/admin-guide">
    Return to the Block Storage administration overview
  </Card>
</CardGroup>


# Attach and Detach Volumes
Source: https://docs.xloud.tech/services/storage/attach-volume

Connect block storage volumes to compute instances, mount filesystems, and safely detach volumes. Includes formatting, mounting, and fstab persistence guidance.

## Overview

Attaching a volume connects a persistent block device to a running compute instance, making it accessible as a disk inside the guest operating system. A volume with status **Available** can be attached to any instance in the same availability zone. After attachment, the volume appears as a new block device (e.g., `/dev/vdb`) inside the instance — you must format and mount it before use if it is a new blank volume.

<Note>
  **Prerequisites**

  * A volume with status **Available**
  * A running compute instance in the same availability zone as the volume
  * SSH access to the instance for filesystem preparation
</Note>

***

## Attach a Volume

<Warning>
  Attaching a newly created blank volume does not create a filesystem. After attaching,
  format the device inside the instance before mounting. Attaching to a running instance
  does not require a reboot.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Open the Volumes list" icon="compass">
        Navigate to
        **Storage > Volumes**.
      </Step>

      <Step title="Attach to instance">
        Locate the volume with status **Available**. Click the **More** dropdown on the
        volume row, then select **Attach** under the **Instance Related** group.

        In the dialog, select the target instance from the instance table. The table
        shows available instances with their name, status, and attached volumes.
        Instances that are incompatible or already have this volume attached are disabled.

        Click **Confirm** to attach.
      </Step>

      <Step title="Confirm attachment" icon="circle-check">
        The volume status changes to **In-use** and the instance name appears in the
        **Attached To** column.

        <Check>Volume attached — status shows **In-use**.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Identify the instance and volume" icon="search">
        ```bash title="List instances" theme={null}
        openstack server list --column ID --column Name
        ```

        ```bash title="List available volumes" theme={null}
        openstack volume list --column ID --column Name --column Status
        ```
      </Step>

      <Step title="Attach the volume" icon="link">
        ```bash title="Attach volume to instance" theme={null}
        openstack server add volume <instance-name-or-id> <volume-name-or-id>
        ```

        Confirm the device path assigned:

        ```bash title="Show attachment details" theme={null}
        openstack volume show <volume-name-or-id> -c attachments
        ```
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="Confirm volume is in-use" theme={null}
        openstack volume show <volume-name-or-id> -c status
        ```

        Expected: `status = in-use`

        <Check>Volume attached to the instance successfully.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Format and Mount (First Use)

After attaching a new blank volume, prepare the filesystem inside the instance:

<Steps>
  <Step title="Identify the new device" icon="search">
    SSH into the instance and list block devices:

    ```bash title="List block devices" theme={null}
    lsblk
    ```

    The new volume appears as an unformatted disk — typically `/dev/vdb` or the next
    available device letter.
  </Step>

  <Step title="Format the volume" icon="hard-drive">
    ```bash title="Format with ext4" theme={null}
    sudo mkfs.ext4 /dev/vdb
    ```

    For XFS (preferred for databases and large files):

    ```bash title="Format with XFS" theme={null}
    sudo mkfs.xfs /dev/vdb
    ```

    <Warning>
      Only run `mkfs` on a **new, blank volume**. Running it on a volume that already
      contains data will permanently erase that data.
    </Warning>
  </Step>

  <Step title="Mount the volume" icon="folder-open">
    ```bash title="Create mount point and mount" theme={null}
    sudo mkdir -p /mnt/data
    sudo mount /dev/vdb /mnt/data
    ```
  </Step>

  <Step title="Persist the mount across reboots" icon="save">
    Add the mount to `/etc/fstab` so it survives instance reboots:

    ```bash title="Add to /etc/fstab" theme={null}
    echo '/dev/vdb /mnt/data ext4 defaults 0 2' | sudo tee -a /etc/fstab
    ```

    <Tip>
      Use the volume UUID instead of the device path for more reliable fstab entries —
      device names can change after reboot on some configurations:

      ```bash title="Find volume UUID" theme={null}
      sudo blkid /dev/vdb
      ```

      Then use: `UUID=<uuid> /mnt/data ext4 defaults 0 2`
    </Tip>
  </Step>

  <Step title="Verify" icon="circle-check">
    ```bash title="Confirm mount" theme={null}
    df -h /mnt/data
    ```

    <Check>Device mounted and accessible — filesystem capacity matches the volume size.</Check>
  </Step>
</Steps>

***

## Detach a Volume

<Warning>
  Always unmount the volume inside the instance **before** detaching. Detaching a
  mounted volume without unmounting first can cause filesystem corruption and data loss.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Unmount inside the instance" icon="terminal">
        SSH into the instance and unmount the volume:

        ```bash title="Unmount the volume" theme={null}
        sudo umount /mnt/data
        ```

        Remove or comment out the corresponding line in `/etc/fstab` to prevent boot
        errors after detachment.
      </Step>

      <Step title="Detach from the Dashboard">
        Navigate to **Storage > Volumes**. Click the **More** dropdown on the volume
        row, then select **Detach** under the **Instance Related** group.

        In the dialog, select the instance to detach from and click **Confirm**.
      </Step>

      <Step title="Confirm detachment" icon="circle-check">
        The volume status returns to **Available**.

        <Check>Volume detached — status shows **Available** and ready for reattachment.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Unmount inside the instance" icon="terminal">
        ```bash title="Unmount the volume" theme={null}
        sudo umount /mnt/data
        ```
      </Step>

      <Step title="Detach the volume" icon="unlink">
        ```bash title="Detach volume from instance" theme={null}
        openstack server remove volume <instance-name-or-id> <volume-name-or-id>
        ```
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="Confirm volume is available" theme={null}
        openstack volume show <volume-name-or-id> -c status
        ```

        Expected: `status = available`

        <Check>Volume detached and available for reattachment or deletion.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Volume fails to attach — availability zone mismatch" icon="triangle-alert">
    **Cause**: The volume and instance are in different availability zones. Volumes can
    only be attached to instances in the same zone.

    **Resolution**:

    ```bash title="Check volume availability zone" theme={null}
    openstack volume show <volume-id> -c availability_zone
    ```

    ```bash title="Check instance availability zone" theme={null}
    openstack server show <instance-id> -c OS-EXT-AZ:availability_zone
    ```

    If the zones do not match, create a new volume in the correct availability zone,
    or migrate the instance.
  </Accordion>

  <Accordion title="Device not visible inside instance after attach" icon="hard-drive">
    **Cause**: The guest OS did not detect the hot-plug event, or the virtio driver
    is not loaded.

    **Resolution**:

    ```bash title="Rescan SCSI bus inside instance" theme={null}
    sudo echo "- - -" | sudo tee /sys/class/scsi_host/host*/scan
    ```

    If the device still does not appear, verify that the instance's kernel supports
    virtio block devices. Modern Linux kernels include this by default.
  </Accordion>

  <Accordion title="Cannot unmount — device is busy" icon="lock">
    **Cause**: A process inside the instance still has files open on the mounted volume.

    **Resolution**:

    ```bash title="Find processes using the mount" theme={null}
    sudo lsof /mnt/data
    ```

    Stop or kill the listed processes, then retry `umount`. Alternatively:

    ```bash title="Force unmount (use with caution)" theme={null}
    sudo umount -l /mnt/data
    ```

    <Warning>
      Lazy unmount (`-l`) detaches the filesystem immediately but keeps references open
      until all processes close their file handles. Only use this when the process
      cannot be stopped gracefully.
    </Warning>
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Extend a Volume" href="/services/storage/extend-volume">
    Increase volume capacity online without detaching or stopping the instance
  </Card>

  <Card title="Volume Snapshots" href="/services/storage/snapshots">
    Create point-in-time snapshots for fast recovery and volume cloning
  </Card>

  <Card title="Create a Volume" href="/services/storage/create-volume">
    Provision a new volume with the appropriate size and storage tier
  </Card>

  <Card title="Volume Backups" href="/services/storage/backups">
    Create full and incremental backups for long-term data retention
  </Card>
</CardGroup>


# Backup Backends
Source: https://docs.xloud.tech/services/storage/backup-backends

Configure Cinder backup backends including Ceph, Swift, NFS, S3, Google Cloud Storage, and TSM for volume data protection.

## Overview

Xloud Block Storage backups are managed by the `cinder-backup` service, which stores volume backup data in a configurable backend. The backup backend is independent of the volume backend — backups from any volume backend (Ceph, NetApp, LVM) can be stored in any backup backend (Ceph, Swift, NFS, S3). Selecting the right backup backend depends on your recovery time objectives, storage capacity, and existing infrastructure.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * The `cinder-backup` service must be deployed and running
  * The backup backend must be network-accessible from the backup service nodes
  * Credentials and endpoint configuration must be available before deployment
</Note>

***

## Supported Backup Backends

| Backend                        | Driver   | Transport    | Use Case                                            |
| ------------------------------ | -------- | ------------ | --------------------------------------------------- |
| **Ceph RBD**                   | `ceph`   | RBD protocol | Native XAVS — incremental backups, fast restores    |
| **Swift**                      | `swift`  | HTTP/S       | Built-in object storage, multi-node replication     |
| **NFS**                        | `nfs`    | NFS          | Shared NAS appliances, on-premises tape replacement |
| **S3-compatible**              | `s3`     | HTTPS        | AWS S3, Ceph RGW, MinIO, Dell ECS, any S3 endpoint  |
| **Google Cloud Storage**       | `google` | HTTPS        | Off-site cloud backup to GCS buckets                |
| **IBM Spectrum Protect (TSM)** | `tsm`    | TSM API      | Mainframe-connected enterprise backup with TSM      |

***

## Backend Configuration

All backup backend settings are placed in the `[DEFAULT]` section of `cinder.conf`. Only one backup backend is active at a time.

<AccordionGroup>
  <Accordion title="Ceph RBD — recommended for XAVS" icon="database">
    The Ceph backup driver stores backups as RBD images in a dedicated backup pool. It supports **incremental backups** — only changed blocks are transferred after the first full backup, making subsequent backups fast and space-efficient.

    ```ini title="cinder.conf — Ceph backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
    backup_ceph_pool = backups
    backup_ceph_user = cinder-backup
    backup_ceph_conf = /etc/ceph/ceph.conf
    backup_ceph_keyring = /etc/ceph/ceph.client.cinder-backup.keyring
    backup_ceph_chunk_size = 134217728
    backup_ceph_stripe_unit = 0
    backup_ceph_stripe_count = 0
    restore_discard_excess_bytes = true
    ```

    <Tip>
      When both the volume backend and backup backend are Ceph, the backup driver can perform server-side RBD-to-RBD copies without moving data through the backup service node, reducing network load and backup time significantly.
    </Tip>
  </Accordion>

  <Accordion title="Swift (Object Storage)" icon="box">
    The Swift backup driver stores backup data as objects in a Swift container. Each backup chunk is stored as a separate object. This backend benefits from Swift's built-in replication for durability.

    ```ini title="cinder.conf — Swift backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver
    backup_swift_url = http://10.0.1.71:8080/v1/AUTH_
    backup_swift_auth = per_user
    backup_swift_auth_version = 3
    backup_swift_user = cinder
    backup_swift_key = <swift-key>
    backup_swift_container = volumes_backup
    backup_swift_object_size = 52428800
    backup_swift_retry_attempts = 3
    backup_swift_retry_backoff = 2
    backup_compression_algorithm = zlib
    ```
  </Accordion>

  <Accordion title="NFS" icon="hard-drive">
    The NFS backup driver mounts a remote NFS share and writes backup files to the mount point. It is suitable for organizations with existing NAS infrastructure.

    ```ini title="cinder.conf — NFS backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.nfs.NFSBackupDriver
    backup_mount_point_base = /var/lib/cinder/backup_nfs
    backup_share = 10.0.10.100:/export/cinder-backup
    backup_file_size = 1999999999
    backup_sha_block_size_bytes = 32768
    backup_enable_progress_timer = true
    ```

    The NFS share must be exported with write permissions for the cinder-backup service account. Add the export to `/etc/exports` on the NFS server:

    ```bash title="NFS server — /etc/exports" theme={null}
    /export/cinder-backup  10.0.0.0/8(rw,sync,no_subtree_check,no_root_squash)
    ```
  </Accordion>

  <Accordion title="S3-Compatible" icon="cloud">
    The S3 backup driver works with any S3-compatible endpoint including AWS S3, Ceph RADOS Gateway, MinIO, Dell ECS, and Wasabi. Set `backup_s3_endpoint_url` for non-AWS endpoints.

    ```ini title="cinder.conf — S3-compatible backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.s3.S3BackupDriver
    backup_s3_endpoint_url = https://s3.example.com
    backup_s3_store_access_key = <access-key>
    backup_s3_store_secret_key = <secret-key>
    backup_s3_store_bucket = cinder-backups
    backup_s3_object_size = 52428800
    backup_s3_retry_attempts = 3
    backup_s3_retry_backoff = 2
    backup_compression_algorithm = zlib
    backup_s3_ssl_insecure = false
    ```

    For AWS S3 without a custom endpoint, omit `backup_s3_endpoint_url` and set the region:

    ```ini title="cinder.conf — AWS S3 backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.s3.S3BackupDriver
    backup_s3_store_access_key = <aws-access-key>
    backup_s3_store_secret_key = <aws-secret-key>
    backup_s3_store_bucket = my-cinder-backups
    backup_s3_store_region = ap-south-1
    backup_compression_algorithm = zlib
    ```
  </Accordion>

  <Accordion title="Google Cloud Storage" icon="cloud">
    The GCS backup driver stores backups in a Google Cloud Storage bucket. Authentication uses a service account credentials file.

    ```ini title="cinder.conf — Google Cloud Storage backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.google.GoogleBackupDriver
    backup_gcs_bucket = my-cinder-backups
    backup_gcs_object_size = 52428800
    backup_gcs_block_size = 32768
    backup_gcs_retry_attempts = 3
    backup_gcs_retry_backoff = 2
    backup_gcs_credential_file = /etc/cinder/gcs-credentials.json
    backup_compression_algorithm = zlib
    ```

    The service account must have `storage.buckets.get`, `storage.objects.create`, `storage.objects.get`, and `storage.objects.delete` permissions on the target bucket.
  </Accordion>

  <Accordion title="IBM Spectrum Protect (TSM)" icon="server">
    The TSM backup driver integrates with IBM Spectrum Protect (formerly Tivoli Storage Manager) for enterprise tape and disk backup workflows.

    ```ini title="cinder.conf — TSM backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.tsm.TSMBackupDriver
    backup_tsm_volume_prefix = cinder
    backup_tsm_password = <dsm-password>
    backup_tsm_compression = true
    ```

    The TSM client (`dsmc`) must be installed and configured on the backup service node, and the `DSMI_CONFIG` environment variable must point to the active `dsm.sys` configuration file.
  </Accordion>
</AccordionGroup>

***

## Apply Backup Backend Configuration

In XAVS deployments, backup backend settings are configured through XDeploy and applied via Ansible. Do not edit `cinder.conf` directly.

```bash title="Apply backup configuration changes" theme={null}
xavs-ansible deploy --tags cinder
```

After deployment, verify the backup service is running:

```bash title="Check backup service status" theme={null}
openstack volume service list --service cinder-backup
```

***

## Verify a Backup

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    <Steps>
      <Step title="Create a test backup" icon="plus">
        Navigate to **Storage > Volumes**, select a volume, and click
        the **More** dropdown and select **Data Protection > Create Backup**.

        Set a name and leave the snapshot field empty for a full backup.
      </Step>

      <Step title="Monitor backup status" icon="activity">
        Navigate to **Storage > Volume Backups**. The backup transitions through
        `creating` to `available`.
      </Step>

      <Step title="Restore the backup" icon="rotate-ccw">
        Click the **More** dropdown and select **Restore** to restore to a new volume.
        Confirm the restored volume matches the original in size and status.

        <Check>Backup created and restored successfully — backend is operational.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Source your credentials file to authenticate with the Xloud platform:

    ```bash title="Load credentials" theme={null}
    source openrc.sh
    ```

    <Tip>
      Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
    </Tip>

    ```bash title="Create a backup" theme={null}
    openstack volume backup create \
      --name test-backup \
      <volume-id>
    ```

    ```bash title="Wait for backup to reach available state" theme={null}
    openstack volume backup show <backup-id> -c status
    ```

    ```bash title="Restore backup to a new volume" theme={null}
    openstack volume backup restore \
      --name restored-vol \
      <backup-id>
    ```

    ```bash title="Verify restored volume" theme={null}
    openstack volume show restored-vol -c status -c size
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Backups (User Guide)" href="/services/storage/backups">
    End-user guide for creating, managing, and restoring volume backups
  </Card>

  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Configure primary volume backends for block storage
  </Card>

  <Card title="External Storage" href="/services/storage/external-storage">
    Connect to NetApp, Pure Storage, Dell, and HPE enterprise arrays
  </Card>

  <Card title="Encryption" href="/services/storage/encryption">
    Encrypt volumes at rest — works independently of backup backend
  </Card>
</CardGroup>


# Backup Configuration
Source: https://docs.xloud.tech/services/storage/backup-config

Configure the block storage backup service with Ceph, Swift, NFS, or S3 targets. Enable and validate backups for production.

## Overview

The `cinder-backup` service writes volume data to a configurable backup target that is completely independent of the primary volume storage backend. Backups from any primary backend (Ceph, LVM, NetApp) can go to any backup target. For production deployments, a separate object storage target (Ceph, Swift, or S3) provides the isolation needed for backups to survive a primary storage failure.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Access to XDeploy for infrastructure-level configuration changes
  * A configured backup target reachable from all backup service nodes
  * Credentials and endpoint URL for the chosen backup target
</Note>

***

## Supported Backup Backends

| Backend                  | Driver                            | Best For                                     | Incremental | HA      |
| ------------------------ | --------------------------------- | -------------------------------------------- | ----------- | ------- |
| **Ceph RBD**             | `ceph.CephBackupDriver`           | XAVS deployments — fast, native, incremental | Yes         | Yes     |
| **Swift**                | `swift.SwiftBackupDriver`         | Built-in Xloud Object Storage                | Yes         | Yes     |
| **S3-compatible**        | `s3.S3BackupDriver`               | AWS S3, Ceph RGW, MinIO, Wasabi, Dell ECS    | Yes         | Yes     |
| **NFS**                  | `nfs.NFSBackupDriver`             | Existing NAS/NFS infrastructure              | Yes         | Depends |
| **Google Cloud Storage** | `gcs.GoogleBackupDriver`          | Off-site cloud backup to GCS                 | Yes         | Yes     |
| **GlusterFS**            | `glusterfs.GlusterfsBackupDriver` | Existing GlusterFS clusters                  | Yes         | Yes     |
| **POSIX (local path)**   | `posix.PosixBackupDriver`         | Dev/test — single node only                  | Yes         | No      |

<Warning>
  Only one backup backend is active at a time. All settings go in the `[DEFAULT]` section of `cinder.conf`. Apply changes through XDeploy — do not edit `cinder.conf` directly on production nodes.
</Warning>

***

## Backup Configuration

<Tabs>
  <Tab title="Ceph RBD" icon="database">
    The Ceph backup driver is the **recommended backend for XAVS deployments**. It stores backups as RBD images in a dedicated Ceph pool and supports differential (incremental) backups — only changed blocks transfer after the first full backup. When both the volume backend and backup backend are Ceph, data is copied server-side without passing through the backup service node, dramatically reducing backup time and network load.

    ```ini title="cinder.conf — Ceph RBD backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
    backup_ceph_pool = backups
    backup_ceph_user = cinder-backup
    backup_ceph_conf = /etc/ceph/ceph.conf
    backup_ceph_chunk_size = 134217728
    backup_ceph_stripe_unit = 0
    backup_ceph_stripe_count = 0
    backup_ceph_max_snapshots = 0
    backup_ceph_image_journals = false
    restore_discard_excess_bytes = true
    ```

    **Configuration options:**

    | Option                         | Default               | Description                                                      |
    | ------------------------------ | --------------------- | ---------------------------------------------------------------- |
    | `backup_ceph_pool`             | `backups`             | Ceph pool where backup images are stored                         |
    | `backup_ceph_user`             | `cinder`              | Ceph user to connect with                                        |
    | `backup_ceph_conf`             | `/etc/ceph/ceph.conf` | Path to Ceph configuration file                                  |
    | `backup_ceph_chunk_size`       | `134217728` (128 MiB) | Chunk size in bytes for transfer to Ceph                         |
    | `backup_ceph_stripe_unit`      | `0`                   | RBD stripe unit (0 = use pool default)                           |
    | `backup_ceph_stripe_count`     | `0`                   | RBD stripe count (0 = use pool default)                          |
    | `backup_ceph_max_snapshots`    | `0`                   | Max snapshots to retain per backup volume (0 = unlimited)        |
    | `backup_ceph_image_journals`   | `False`               | Enable JOURNALING and EXCLUSIVE\_LOCK features for RBD mirroring |
    | `restore_discard_excess_bytes` | `True`                | Pad restored volumes with zeroes to exact size                   |

    <Tip>
      Set `backup_ceph_max_snapshots` to a non-zero value (e.g., `5`) to automatically prune old incremental snapshots and reclaim Ceph pool space. Manually deleting incremental backups outside this mechanism forces the next backup to be a full copy.
    </Tip>

    **Create the backup pool and user:**

    ```bash title="Ceph — create backup pool and cephx user" theme={null}
    ceph osd pool create backups 32
    ceph auth get-or-create client.cinder-backup \
      mon 'profile rbd' \
      osd 'profile rbd pool=backups' \
      -o /etc/ceph/ceph.client.cinder-backup.keyring
    ```
  </Tab>

  <Tab title="Swift" icon="box">
    The Swift backup driver stores volume backup data as objects in an Xloud Object Storage container. Each backup is split into chunks stored as individual objects. Swift's built-in replication provides durability. Supports auth versions 1, 2, and 3.

    ```ini title="cinder.conf — Swift backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver
    backup_swift_url = http://10.0.1.70:8080/v1/AUTH_
    backup_swift_auth = per_user
    backup_swift_auth_version = 3
    backup_swift_user = cinder
    backup_swift_user_domain = Default
    backup_swift_key = <swift-password>
    backup_swift_project = service
    backup_swift_project_domain = Default
    backup_swift_container = volumebackups
    backup_swift_object_size = 52428800
    backup_swift_block_size = 32768
    backup_swift_retry_attempts = 3
    backup_swift_retry_backoff = 2
    backup_compression_algorithm = zlib
    ```

    **Configuration options:**

    | Option                               | Default             | Description                                       |
    | ------------------------------------ | ------------------- | ------------------------------------------------- |
    | `backup_swift_url`                   | `None`              | Swift endpoint URL                                |
    | `backup_swift_auth`                  | `per_user`          | Auth mechanism: `per_user` or `single_user`       |
    | `backup_swift_auth_version`          | `1`                 | Auth version: `1`, `2`, or `3`                    |
    | `backup_swift_auth_url`              | `None`              | Keystone endpoint URL (when `auth = single_user`) |
    | `backup_swift_user`                  | `None`              | Swift username                                    |
    | `backup_swift_user_domain`           | `None`              | Swift user domain (auth v3)                       |
    | `backup_swift_key`                   | `None`              | Swift password / API key                          |
    | `backup_swift_tenant`                | `None`              | Swift tenant name (auth v2)                       |
    | `backup_swift_project`               | `None`              | Swift project name (auth v3)                      |
    | `backup_swift_project_domain`        | `None`              | Swift project domain (auth v3)                    |
    | `backup_swift_container`             | `volumebackups`     | Container for backup objects                      |
    | `backup_swift_object_size`           | `52428800` (50 MiB) | Size of each backup chunk object                  |
    | `backup_swift_block_size`            | `32768` (32 KiB)    | Block size for incremental change tracking        |
    | `backup_swift_retry_attempts`        | `3`                 | Retry count for Swift operations                  |
    | `backup_swift_retry_backoff`         | `2`                 | Backoff in seconds between retries                |
    | `backup_swift_region_name`           | `None`              | Swift region name                                 |
    | `backup_swift_create_storage_policy` | `None`              | Storage policy when creating the container        |
    | `backup_swift_auth_insecure`         | `False`             | Skip SSL certificate verification                 |
    | `backup_swift_ca_cert_file`          | `None`              | CA bundle file for SSL verification               |
    | `backup_compression_algorithm`       | `zlib`              | Compression: `zlib`, `bz2`, or `none`             |

    <Info>
      `backup_swift_object_size` must be a multiple of `backup_swift_block_size`.
    </Info>
  </Tab>

  <Tab title="S3-Compatible" icon="cloud">
    The S3 backup driver works with AWS S3 and any S3-compatible endpoint: Ceph RADOS Gateway, MinIO, Wasabi, Dell ECS, and others. Set `backup_s3_endpoint_url` for non-AWS endpoints; omit it for native AWS S3.

    ```ini title="cinder.conf — S3 / Ceph RGW backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.s3.S3BackupDriver
    backup_s3_endpoint_url = https://rgw.example.com
    backup_s3_store_access_key = <access-key>
    backup_s3_store_secret_key = <secret-key>
    backup_s3_store_bucket = cinder-backups
    backup_s3_object_size = 52428800
    backup_s3_block_size = 32768
    backup_s3_retry_max_attempts = 4
    backup_s3_retry_mode = standard
    backup_compression_algorithm = zlib
    backup_s3_verify_ssl = true
    ```

    ```ini title="cinder.conf — Native AWS S3" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.s3.S3BackupDriver
    backup_s3_store_access_key = <aws-access-key>
    backup_s3_store_secret_key = <aws-secret-key>
    backup_s3_store_bucket = my-cinder-backups
    backup_compression_algorithm = zlib
    ```

    **Configuration options:**

    | Option                             | Default             | Description                                      |
    | ---------------------------------- | ------------------- | ------------------------------------------------ |
    | `backup_s3_endpoint_url`           | `None`              | S3-compatible endpoint URL (omit for native AWS) |
    | `backup_s3_store_access_key`       | `None`              | S3 access key                                    |
    | `backup_s3_store_secret_key`       | `None`              | S3 secret key                                    |
    | `backup_s3_store_bucket`           | `volumebackups`     | S3 bucket for backup data                        |
    | `backup_s3_object_size`            | `52428800` (50 MiB) | Size of each backup chunk                        |
    | `backup_s3_block_size`             | `32768` (32 KiB)    | Block size for incremental change tracking       |
    | `backup_s3_retry_max_attempts`     | `4`                 | Max retry attempts per request                   |
    | `backup_s3_retry_mode`             | `legacy`            | Retry mode: `legacy`, `standard`, or `adaptive`  |
    | `backup_s3_max_pool_connections`   | `10`                | Max connections in the connection pool           |
    | `backup_s3_timeout`                | `60`                | Request timeout in seconds                       |
    | `backup_s3_verify_ssl`             | `True`              | Enable SSL certificate verification              |
    | `backup_s3_ca_cert_file`           | `None`              | CA bundle file path                              |
    | `backup_s3_http_proxy`             | `""`                | HTTP proxy address                               |
    | `backup_s3_https_proxy`            | `""`                | HTTPS proxy address                              |
    | `backup_s3_md5_validation`         | `True`              | Validate MD5 checksums                           |
    | `backup_s3_sse_customer_algorithm` | `None`              | SSE-C algorithm (e.g., `AES256`)                 |
    | `backup_s3_sse_customer_key`       | `None`              | SSE-C encryption key                             |
    | `backup_compression_algorithm`     | `zlib`              | Compression: `zlib`, `bz2`, or `none`            |

    <Info>
      `backup_s3_object_size` must be a multiple of `backup_s3_block_size`.
    </Info>
  </Tab>

  <Tab title="NFS" icon="hard-drive">
    The NFS backup driver mounts a remote NFS share and writes backup files to the mount point. Suitable for environments with existing NAS infrastructure.

    ```ini title="cinder.conf — NFS backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.nfs.NFSBackupDriver
    backup_share = 10.0.10.100:/export/cinder-backup
    backup_mount_point_base = /var/lib/cinder/backup_nfs
    backup_mount_options = rw,sync,hard,intr
    backup_mount_attempts = 3
    backup_file_size = 1999994880
    backup_sha_block_size_bytes = 32768
    backup_enable_progress_timer = true
    ```

    **Configuration options:**

    | Option                         | Default                    | Description                                                                                                                                   |
    | ------------------------------ | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
    | `backup_share`                 | `None`                     | NFS share: `hostname:/path`, `ipv4addr:/path`, or `[ipv6addr]:/path`                                                                          |
    | `backup_mount_point_base`      | `$state_path/backup_mount` | Base directory for the NFS mount point                                                                                                        |
    | `backup_mount_options`         | `None`                     | NFS mount options (see `man nfs`)                                                                                                             |
    | `backup_mount_attempts`        | `3`                        | Mount retry attempts before raising an error                                                                                                  |
    | `backup_file_size`             | `1999994880` (\~1.86 GiB)  | Max backup file size in bytes — volumes larger than this are split across multiple files. Must be a multiple of `backup_sha_block_size_bytes` |
    | `backup_sha_block_size_bytes`  | `32768` (32 KiB)           | Block size for incremental change tracking                                                                                                    |
    | `backup_container`             | `None`                     | Custom subdirectory within the NFS share                                                                                                      |
    | `backup_enable_progress_timer` | `True`                     | Send periodic progress events to Ceilometer                                                                                                   |

    <Info>
      `backup_file_size` also determines the internal buffer size — scale it based on available RAM and the number of concurrent backup workers.
    </Info>

    **NFS server setup:**

    ```bash title="/etc/exports on the NFS server" theme={null}
    /export/cinder-backup  10.0.0.0/8(rw,sync,no_subtree_check,no_root_squash)
    ```

    ```bash title="Apply exports" theme={null}
    exportfs -ra
    ```
  </Tab>

  <Tab title="Google Cloud Storage" icon="cloud">
    The GCS backup driver stores backups in a Google Cloud Storage bucket. Authentication uses a GCS service account credential JSON file.

    ```ini title="cinder.conf — Google Cloud Storage backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.gcs.GoogleBackupDriver
    backup_gcs_bucket = my-cinder-backups
    backup_gcs_project_id = my-gcp-project
    backup_gcs_credential_file = /etc/cinder/gcs-credentials.json
    backup_gcs_object_size = 52428800
    backup_gcs_block_size = 32768
    backup_gcs_reader_chunk_size = 2097152
    backup_gcs_writer_chunk_size = 2097152
    backup_gcs_num_retries = 3
    backup_gcs_bucket_location = US
    backup_gcs_storage_class = NEARLINE
    backup_compression_algorithm = zlib
    ```

    **Configuration options:**

    | Option                             | Default             | Description                                                      |
    | ---------------------------------- | ------------------- | ---------------------------------------------------------------- |
    | `backup_gcs_bucket`                | `None`              | GCS bucket name                                                  |
    | `backup_gcs_project_id`            | `None`              | GCP project ID that owns the bucket                              |
    | `backup_gcs_credential_file`       | `None`              | Absolute path to GCS service account JSON key file               |
    | `backup_gcs_object_size`           | `52428800` (50 MiB) | Size of each backup chunk object                                 |
    | `backup_gcs_block_size`            | `32768` (32 KiB)    | Block size for incremental change tracking                       |
    | `backup_gcs_reader_chunk_size`     | `2097152` (2 MiB)   | Download chunk size; `-1` = single download                      |
    | `backup_gcs_writer_chunk_size`     | `2097152` (2 MiB)   | Upload chunk size; `-1` = single upload                          |
    | `backup_gcs_num_retries`           | `3`                 | Number of retry attempts                                         |
    | `backup_gcs_retry_error_codes`     | `[429]`             | HTTP error codes that trigger a retry                            |
    | `backup_gcs_bucket_location`       | `US`                | Bucket location (e.g., `US`, `EU`, `ASIA`, `us-central1`)        |
    | `backup_gcs_storage_class`         | `NEARLINE`          | GCS storage class: `STANDARD`, `NEARLINE`, `COLDLINE`, `ARCHIVE` |
    | `backup_gcs_proxy_url`             | `None`              | HTTP proxy URL                                                   |
    | `backup_gcs_user_agent`            | `gcscinder`         | HTTP user-agent string                                           |
    | `backup_gcs_enable_progress_timer` | `True`              | Send progress events to Ceilometer                               |
    | `backup_compression_algorithm`     | `zlib`              | Compression: `zlib`, `bz2`, or `none`                            |

    <Info>
      `backup_gcs_object_size` must be a multiple of `backup_gcs_block_size`.
    </Info>

    **Required GCS service account permissions:**

    | Permission               | Purpose                         |
    | ------------------------ | ------------------------------- |
    | `storage.buckets.get`    | Read bucket metadata            |
    | `storage.objects.create` | Write backup objects            |
    | `storage.objects.get`    | Read backup objects for restore |
    | `storage.objects.delete` | Delete backup objects           |
  </Tab>

  <Tab title="GlusterFS" icon="network">
    The GlusterFS backup driver mounts a GlusterFS volume and writes backup files to it. Suitable for environments already running GlusterFS distributed storage.

    ```ini title="cinder.conf — GlusterFS backup backend" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.glusterfs.GlusterfsBackupDriver
    glusterfs_backup_share = 10.0.10.100:backup_vol
    glusterfs_backup_mount_point = /var/lib/cinder/backup_glusterfs
    ```

    **Configuration options:**

    | Option                         | Default                    | Description                                                                                                                       |
    | ------------------------------ | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
    | `glusterfs_backup_share`       | `None`                     | GlusterFS volume in `hostname:vol_name` or `ipv4addr:vol_name` or `[ipv6addr]:vol_name` format. Example: `10.0.10.100:backup_vol` |
    | `glusterfs_backup_mount_point` | `$state_path/backup_mount` | Base directory for the GlusterFS mount point                                                                                      |

    <Note>
      The GlusterFS client (`glusterfs-fuse`) must be installed on all nodes running the backup service. The backup service account must have read/write access to the GlusterFS volume.
    </Note>
  </Tab>

  <Tab title="POSIX (local path)" icon="folder">
    The POSIX backup driver writes backups to a local filesystem path on the backup service node. It requires no external services and is straightforward to configure, but provides **no redundancy** — if the node fails, backups are lost.

    ```ini title="cinder.conf — POSIX local backup" theme={null}
    [DEFAULT]
    backup_driver = cinder.backup.drivers.posix.PosixBackupDriver
    backup_posix_path = /var/lib/cinder/backup
    backup_file_size = 1999994880
    backup_sha_block_size_bytes = 32768
    backup_enable_progress_timer = true
    ```

    **Configuration options:**

    | Option                         | Default                   | Description                                                                                                   |
    | ------------------------------ | ------------------------- | ------------------------------------------------------------------------------------------------------------- |
    | `backup_posix_path`            | `$state_path/backup`      | Local filesystem path where backups are stored                                                                |
    | `backup_file_size`             | `1999994880` (\~1.86 GiB) | Max backup file size; volumes larger than this are split. Must be a multiple of `backup_sha_block_size_bytes` |
    | `backup_sha_block_size_bytes`  | `32768` (32 KiB)          | Block size for incremental change tracking                                                                    |
    | `backup_container`             | `None`                    | Custom subdirectory within `backup_posix_path`                                                                |
    | `backup_enable_progress_timer` | `True`                    | Send progress events to Ceilometer                                                                            |

    <Warning>
      POSIX local backup provides **no redundancy**. A node failure loses both primary volumes and their backups simultaneously. Use this only for development or test environments. For any production workload, use Ceph, Swift, S3, or NFS.
    </Warning>
  </Tab>
</Tabs>

***

## Apply Backup Backend Configuration

In XAVS deployments, backup backend settings are configured through XDeploy and applied via Ansible. Do not edit `cinder.conf` directly on production nodes.

```bash title="Apply backup configuration changes" theme={null}
xavs-ansible deploy --tags cinder
```

After deployment, verify the backup service is running:

```bash title="Check backup service status" theme={null}
openstack volume service list --service cinder-backup
```

***

## Validate the Backup Backend

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create a test backup" icon="plus">
        Navigate to **Storage > Volumes**, select a volume, and click the **More** dropdown and select **Data Protection > Create Backup**. Set a name and leave the snapshot field empty for a full backup.
      </Step>

      <Step title="Monitor backup status" icon="activity">
        Navigate to **Storage > Volume Backups**. The backup transitions through `creating` → `available`.
      </Step>

      <Step title="Restore the backup" icon="rotate-ccw">
        Click the **More** dropdown and select **Restore** to restore to a new volume. Confirm the restored volume matches the original in size and status.

        <Check>Backup created and restored successfully — backend is operational.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Source your credentials file to authenticate with the Xloud platform:

    ```bash title="Load credentials" theme={null}
    source openrc.sh
    ```

    <Tip>
      Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
    </Tip>

    ```bash title="Create a test backup" theme={null}
    openstack volume backup create \
      --name config-validation-backup \
      <volume-id>
    ```

    ```bash title="Monitor backup status" theme={null}
    openstack volume backup show config-validation-backup -c status -c size
    ```

    ```bash title="Restore to a new volume" theme={null}
    openstack volume backup restore \
      --name restored-vol \
      config-validation-backup
    ```

    ```bash title="Verify and clean up" theme={null}
    openstack volume show restored-vol -c status -c size
    openstack volume backup delete config-validation-backup
    openstack volume delete restored-vol
    ```

    <Check>Backup service is operational and writing to the configured target.</Check>
  </Tab>
</Tabs>

***

## Backup Storage Sizing

| Workload Profile             | Recommended Backend | Storage Sizing                              |
| ---------------------------- | ------------------- | ------------------------------------------- |
| Development / test           | POSIX or NFS        | 2× total volume capacity                    |
| Small production (\< 10 TiB) | NFS or Swift        | 3× total volume capacity                    |
| Large production (> 10 TiB)  | Ceph or S3          | 5× total volume capacity (with incremental) |
| Off-site DR                  | GCS or S3           | Same as large production                    |

<Tip>
  With a weekly full + daily incremental strategy, effective backup storage is typically **1.2–1.5×** total volume capacity for a 30-day retention window. Ceph incremental backups are the most space-efficient — only changed blocks are stored after the first full backup.
</Tip>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Backup service is 'down'" icon="circle-xmark">
    **Symptom**: `openstack volume service list | grep backup` shows state `down`.

    **Common causes and fixes**:

    * **Authentication failure** — verify credentials (Swift/S3/GCS) are correct and the service user has access to the bucket/container
    * **NFS/GlusterFS mount failure** — verify the share is reachable and exported with write permission: `mount -t nfs <server>:<share> /mnt/test`
    * **Ceph connectivity** — verify the keyring and ceph.conf are correct: `ceph --user cinder-backup -s`
    * **Container not started** — restart the backup service: `xavs-ansible deploy --tags cinder`
  </Accordion>

  <Accordion title="Backup stuck in 'creating' status" icon="clock">
    **Cause**: Backup target is unreachable, slow, or the service is under load.

    Check backup service logs:

    ```bash title="Check cinder-backup logs" theme={null}
    docker logs cinder_backup | tail -50
    ```

    For large volumes (> 1 TiB), allow up to 60 minutes. If the backup stays in `creating` beyond that, check target connectivity and available space.
  </Accordion>

  <Accordion title="Incremental backup becomes a full backup unexpectedly" icon="database">
    **Cause** (Ceph): Manually deleting a Ceph incremental snapshot breaks the backup chain. The next backup must start a new full backup.

    **Cause** (Swift/S3/NFS): The base backup or a required incremental in the chain was deleted.

    **Fix**: Only delete backups through the Cinder API (`openstack volume backup delete`), never by directly removing objects from the storage backend. After a chain break, the next backup will automatically create a new full backup.
  </Accordion>

  <Accordion title="GCS authentication errors" icon="key">
    **Cause**: The service account credential file is missing, incorrect, or the service account lacks required permissions.

    **Fix**: Verify the credential file path in `cinder.conf` and test authentication:

    ```bash title="Test GCS credentials" theme={null}
    docker exec cinder_backup python3 -c "
    from google.oauth2 import service_account
    creds = service_account.Credentials.from_service_account_file('/etc/cinder/gcs-credentials.json')
    print('Credentials valid:', creds.valid)
    "
    ```

    Ensure the service account has `storage.objects.create`, `storage.objects.get`, and `storage.objects.delete` on the target bucket.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Backup Backends" href="/services/storage/backup-backends">
    Comparison of all backup backends with use-case guidance
  </Card>

  <Card title="Volume Backups (User)" href="/services/storage/backups">
    User guide for creating, managing, and restoring volume backups
  </Card>

  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Configure the primary volume storage backends
  </Card>

  <Card title="Quotas" href="/services/storage/quotas">
    Configure backup quota limits per project
  </Card>
</CardGroup>


# Volume Backups
Source: https://docs.xloud.tech/services/storage/backups

Create full and incremental backups of block storage volumes. Covers backup creation, incremental strategy, and restore.

## Overview

Volume backups write a complete copy of volume data to a separate backup target — typically an object storage bucket — providing protection against storage cluster failure. Unlike snapshots, backups are stored independently of the source volume and can survive the loss of the primary storage cluster. Backups support a full + incremental strategy to minimize storage usage and backup time after the initial full copy.

<Note>
  **Prerequisites**

  * The backup service must be enabled and configured by your administrator
  * Sufficient backup storage quota (`openstack quota show`)
  * The source volume must exist (attached or detached)
</Note>

***

## Snapshot vs Backup

| Feature                  | Snapshot                    | Backup                                 |
| ------------------------ | --------------------------- | -------------------------------------- |
| Storage location         | Same cluster as source      | Separate backup target                 |
| Creation time            | Seconds                     | Minutes (data copy)                    |
| Incremental support      | Implicit (copy-on-write)    | Yes — after first full backup          |
| Survives cluster failure | No                          | Yes                                    |
| Restore time             | Seconds                     | Minutes                                |
| Long-term retention      | Limited by cluster capacity | Yes                                    |
| Use case                 | Fast checkpoints, cloning   | Disaster recovery, compliance archival |

<Tip>
  Combine both strategies: use [snapshots](/services/storage/snapshots) for frequent
  short-term checkpoints and backups for weekly full copies plus daily incrementals
  to a separate target.
</Tip>

***

## Create a Full Backup

<iframe title="How to Create and Restore Volume Backups on Xloud" />

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Volumes" icon="compass">
        Navigate to
        **Storage > Volumes**.
      </Step>

      <Step title="Create backup" icon="archive">
        Click the **More** dropdown and select **Data Protection > Create Backup** on the target volume.
        Provide a name and select **Full Backup** from the backup mode radio buttons
        (default for the first backup). Click **Confirm**.

        <Note>
          A full backup copies all data from the volume. Duration depends on volume size
          and available backup bandwidth — expect 1–5 minutes per 10 GiB for typical
          workloads.
        </Note>
      </Step>

      <Step title="Monitor progress" icon="activity">
        Navigate to **Storage > Volume Backups**. The backup status transitions
        through `creating` → `available`.

        <Check>Backup status shows **Available** — full backup complete.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create a full backup" icon="archive">
        ```bash title="Create full backup" theme={null}
        openstack volume backup create \
          --name weekly-full-2026-03-18 \
          --description "Weekly full backup" \
          <volume-name-or-id>
        ```
      </Step>

      <Step title="Verify backup status" icon="circle-check">
        ```bash title="Check backup status" theme={null}
        openstack volume backup show weekly-full-2026-03-18 -c status -c size
        ```

        Expected: `status = available`

        <Check>Full backup created and available for restore.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Create an Incremental Backup

After a full backup exists, incremental backups copy only the blocks that changed since
the last backup — significantly reducing backup time and storage usage.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create incremental backup" icon="layers">
        Click the **More** dropdown and select **Data Protection > Create Backup** on the target volume.
        Provide a name and select **Increment Backup** from the backup mode
        radio buttons. Click **Confirm**.

        <Tip>
          Incremental backups depend on the existence of a previous backup for the same
          volume. If no previous backup exists, the backup service automatically creates
          a full backup instead.
        </Tip>
      </Step>

      <Step title="Verify" icon="circle-check">
        Navigate to **Storage > Volume Backups** and confirm the new backup appears
        with status **Available**.

        <Check>Incremental backup complete — only changed blocks were copied.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create incremental backup" icon="layers">
        ```bash title="Create incremental backup" theme={null}
        openstack volume backup create \
          --name daily-incr-2026-03-18 \
          --incremental \
          <volume-name-or-id>
        ```
      </Step>

      <Step title="List all backups for a volume" icon="list">
        ```bash title="List backups" theme={null}
        openstack volume backup list --volume <volume-name-or-id>
        ```

        <Check>Incremental backup available alongside the full backup chain.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Restore from a Backup

<Warning>
  Restoring a backup to an **existing volume** overwrites that volume's data. Select
  **New Volume** to restore to a fresh volume and verify data integrity before
  overwriting the original.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Backups" icon="compass">
        Navigate to **Storage > Volume Backups**.
      </Step>

      <Step title="Restore the backup">
        Click the **More** dropdown and select **Restore** on the target backup.

        The dialog shows the source volume name and status. The backup is restored
        to the **original volume** it was created from.

        <Warning>
          The source volume must be in `Available` or `In-use` status for restore.
          If the volume is in-use, all attached instances must be shut off before
          restoring.
        </Warning>

        Click **Confirm** to start the restore.
      </Step>

      <Step title="Verify" icon="circle-check">
        The restored volume appears in the Volumes list with status **Available**.
        [Attach it](/services/storage/attach-volume) and verify the data before
        making any changes.

        <Check>Volume restored from backup and ready for use.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="List available backups" icon="list">
        ```bash title="List backups" theme={null}
        openstack volume backup list
        ```
      </Step>

      <Step title="Restore to a new volume" icon="rotate-ccw">
        ```bash title="Restore backup to new volume" theme={null}
        openstack volume backup restore \
          --name restored-from-backup-2026-03-18 \
          <backup-id>
        ```
      </Step>

      <Step title="Verify the restored volume" icon="circle-check">
        ```bash title="Check restored volume" theme={null}
        openstack volume show restored-from-backup-2026-03-18 -c status -c size
        ```

        <Check>Status `available` — backup restored to a new volume.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Backup Retention Strategy

<AccordionGroup>
  <Accordion title="Recommended retention schedule" icon="calendar">
    For production volumes, Xloud recommends:

    | Frequency  | Type        | Retention                            |
    | ---------- | ----------- | ------------------------------------ |
    | Weekly     | Full backup | Keep 4 (1 month)                     |
    | Daily      | Incremental | Keep 7 (1 week)                      |
    | Pre-change | Full backup | Keep until change is verified stable |

    Adjust retention to match your RTO/RPO requirements and backup quota limits.
  </Accordion>

  <Accordion title="Cleaning up old backups" icon="trash">
    List and delete outdated backups to free quota:

    ```bash title="List all backups with dates" theme={null}
    openstack volume backup list --long
    ```

    ```bash title="Delete a backup" theme={null}
    openstack volume backup delete <backup-id>
    ```

    <Warning>
      Deleting a backup from a backup chain (full + incrementals) may invalidate
      dependent incremental backups. Delete incrementals before deleting the full backup
      they depend on.
    </Warning>
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Volume Snapshots" href="/services/storage/snapshots">
    Complement backups with fast point-in-time snapshots for short-term recovery
  </Card>

  <Card title="Backup Configuration (Admin)" href="/services/storage/backup-config">
    Configure the backup service driver and target for your environment
  </Card>

  <Card title="Attach / Detach" href="/services/storage/attach-volume">
    Attach a restored volume and verify its data
  </Card>

  <Card title="Quota Management (Admin)" href="/services/storage/quotas">
    Manage backup quota limits across projects
  </Card>
</CardGroup>


# Block Storage CLI Reference
Source: https://docs.xloud.tech/services/storage/cli-reference

Complete openstack volume CLI commands for managing Xloud Block Storage — create, attach, extend, snapshot, backup, and manage volume types.

## Overview

The `openstack volume` command group manages the full lifecycle of block storage volumes — creation, attachment, extension, snapshots, backups, and type management.

<Note>
  **Prerequisites**

  * CLI installed and authenticated — see [CLI Setup](/cli-setup)
  * Python cinderclient installed: `pip install python-cinderclient`
</Note>

***

## Volumes

### List and Inspect

<CodeGroup>
  ```bash title="List volumes" theme={null}
  openstack volume list
  openstack volume list --status available
  openstack volume list --all-projects
  ```

  ```bash title="Show volume details" theme={null}
  openstack volume show <volume>
  ```
</CodeGroup>

### Create and Delete

<CodeGroup>
  ```bash title="Create empty volume" theme={null}
  openstack volume create --size 50 my-volume
  ```

  ```bash title="Create from image" theme={null}
  openstack volume create \
    --size 50 \
    --image Ubuntu-22.04 \
    boot-volume
  ```

  ```bash title="Create from snapshot" theme={null}
  openstack volume create \
    --size 50 \
    --snapshot <snapshot-id> \
    restored-volume
  ```

  ```bash title="Create with volume type" theme={null}
  openstack volume create \
    --size 100 \
    --type ssd \
    fast-volume
  ```

  ```bash title="Delete volume" theme={null}
  openstack volume delete <volume>
  ```
</CodeGroup>

### Attach and Detach

<CodeGroup>
  ```bash title="Attach volume to instance" theme={null}
  openstack server add volume <server> <volume>
  ```

  ```bash title="Attach at specific device path" theme={null}
  openstack server add volume --device /dev/vdb <server> <volume>
  ```

  ```bash title="Detach volume" theme={null}
  openstack server remove volume <server> <volume>
  ```
</CodeGroup>

### Extend

```bash title="Extend volume size" theme={null}
openstack volume set --size 100 <volume>
```

***

## Snapshots

<CodeGroup>
  ```bash title="List snapshots" theme={null}
  openstack volume snapshot list
  ```

  ```bash title="Create snapshot" theme={null}
  openstack volume snapshot create --volume <volume> my-snapshot
  ```

  ```bash title="Show snapshot" theme={null}
  openstack volume snapshot show <snapshot>
  ```

  ```bash title="Delete snapshot" theme={null}
  openstack volume snapshot delete <snapshot>
  ```
</CodeGroup>

***

## Backups

<CodeGroup>
  ```bash title="List backups" theme={null}
  openstack volume backup list
  ```

  ```bash title="Create backup" theme={null}
  openstack volume backup create --name my-backup <volume>
  ```

  ```bash title="Create incremental backup" theme={null}
  openstack volume backup create --incremental --name incr-backup <volume>
  ```

  ```bash title="Restore backup" theme={null}
  openstack volume backup restore <backup> <volume>
  ```

  ```bash title="Delete backup" theme={null}
  openstack volume backup delete <backup>
  ```
</CodeGroup>

***

## Volume Types (Admin)

<CodeGroup>
  ```bash title="List volume types" theme={null}
  openstack volume type list
  ```

  ```bash title="Create volume type" theme={null}
  openstack volume type create \
    --property volume_backend_name=ceph \
    ssd
  ```

  ```bash title="Show volume type" theme={null}
  openstack volume type show ssd
  ```

  ```bash title="Set default volume type" theme={null}
  openstack volume type set --default ssd
  ```

  ```bash title="Delete volume type" theme={null}
  openstack volume type delete ssd
  ```
</CodeGroup>

***

## Transfers (Cross-Project)

<CodeGroup>
  ```bash title="Create transfer request" theme={null}
  openstack volume transfer request create --name my-transfer <volume>
  ```

  ```bash title="Accept transfer (recipient)" theme={null}
  openstack volume transfer request accept \
    --auth-key <auth-key> \
    <transfer-id>
  ```

  ```bash title="List pending transfers" theme={null}
  openstack volume transfer request list
  ```
</CodeGroup>

***

## Common Options

| Option                    | Description                                 |
| ------------------------- | ------------------------------------------- |
| `--format json`           | Output as JSON                              |
| `--format value -c <col>` | Extract a single field                      |
| `--all-projects`          | Show resources across all projects (admin)  |
| `--status <status>`       | Filter by status (available, in-use, error) |

***

## Next Steps

<CardGroup>
  <Card title="Volume User Guide" href="/services/storage/create-volume">
    Step-by-step guide to creating and managing volumes
  </Card>

  <Card title="Compute CLI" href="/services/compute/cli-reference">
    Server commands for attaching and managing instances
  </Card>
</CardGroup>


# Create a Volume
Source: https://docs.xloud.tech/services/storage/create-volume

Provision persistent block storage volumes in your Xloud project. Choose a data source, storage tier, and capacity using the Dashboard or CLI.

## Overview

Volumes are persistent block storage devices that can be attached to compute instances.
They persist independently of the instance lifecycle — detach, reattach, snapshot, or
transfer them between projects as needed.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

## Data Source Types

| Source           | Description                      | Use Case                        |
| ---------------- | -------------------------------- | ------------------------------- |
| **Blank Volume** | Empty disk, no pre-existing data | Data disks, scratch storage     |
| **Image**        | Pre-populated from an OS image   | Bootable root disks, appliances |
| **Snapshot**     | Restored from a volume snapshot  | Cloning data, test environments |

***

## Create a Volume

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Volumes">
        Navigate to **Storage > Volumes** in the sidebar. Click **Create Volume**.
        The create form opens as a full page.
      </Step>

      <Step title="Select the data source">
        Choose the **Data Source Type**:

        | Option           | Description                                        |
        | ---------------- | -------------------------------------------------- |
        | **Blank Volume** | Creates an empty volume (default)                  |
        | **Image**        | Select an OS image to populate the volume          |
        | **Snapshot**     | Select an existing volume snapshot to restore from |

        When **Image** is selected, an image selection table appears with tabs organized
        by OS distribution (CentOS, Ubuntu, Fedora, Windows, etc.).

        When **Snapshot** is selected, a snapshot selection table appears showing available
        snapshots with their size and status.
      </Step>

      <Step title="Select a volume type">
        Choose the **Volume Type** from the available storage tiers. The table shows
        all configured volume types with their properties.

        <Tip>
          If the volume type supports multi-attach (shared volumes), a note is displayed
          indicating that the volume can be attached to multiple instances simultaneously.
        </Tip>
      </Step>

      <Step title="Set the capacity">
        Enter the **Capacity (GiB)** using the slider or input field.

        * **Blank volume**: minimum 1 GiB
        * **Image source**: minimum is the image's `min_disk` requirement
        * **Snapshot source**: minimum is the snapshot's original size

        The slider range is limited by your project's remaining volume capacity quota.
        If quota is exhausted, the submit button is disabled.
      </Step>

      <Step title="Configure name and description">
        | Field           | Type      | Required | Description                     |
        | --------------- | --------- | -------- | ------------------------------- |
        | **Name**        | Text      | Yes      | Volume display name             |
        | **Description** | Text area | No       | Optional notes about the volume |
      </Step>

      <Step title="Set the availability zone (optional)">
        Select an **Available Zone** from the dropdown. If not specified, the system
        selects the default zone.

        <Note>
          The volume's availability zone should match the zone of the instance you
          plan to attach it to.
        </Note>
      </Step>

      <Step title="Create the volume">
        The footer shows the **Count** field (default: 1) to create multiple identical
        volumes in one operation. Real-time **quota usage** is displayed.

        Click **Confirm**. The volume appears in the list with status **Creating**,
        transitioning to **Available** when ready.

        <Check>Volume status is **Available** — ready to attach to an instance.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Source credentials" theme={null}
    source openrc.sh
    ```

    <CodeGroup>
      ```bash title="Create a blank volume" theme={null}
      openstack volume create \
        --size 100 \
        --type ssd \
        --availability-zone nova \
        my-data-volume
      ```

      ```bash title="Create a volume from image" theme={null}
      openstack volume create \
        --size 50 \
        --type ssd \
        --image <IMAGE_ID> \
        my-boot-volume
      ```

      ```bash title="Create a volume from snapshot" theme={null}
      openstack volume create \
        --size 100 \
        --snapshot <SNAPSHOT_ID> \
        my-restored-volume
      ```

      ```bash title="Create multiple volumes" theme={null}
      for i in $(seq 1 3); do
        openstack volume create --size 50 --type ssd data-disk-$i
      done
      ```
    </CodeGroup>

    ```bash title="Verify volume status" theme={null}
    openstack volume show my-data-volume -c status
    ```

    <Check>Status shows `available`.</Check>
  </Tab>
</Tabs>

***

## View Volumes

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Storage > Volumes**. The list shows:

    | Column          | Description                                   |
    | --------------- | --------------------------------------------- |
    | **ID/Name**     | Volume identifier (clickable to view details) |
    | **Size**        | Capacity in GiB                               |
    | **Status**      | Available, In-use, Error, etc.                |
    | **Type**        | Storage tier (volume type)                    |
    | **Disk Tag**    | Disk usage tag                                |
    | **Attached To** | Instance name and device path (if attached)   |
    | **Bootable**    | Whether the volume can be used as a boot disk |
    | **Shared**      | Whether multi-attach is enabled               |
    | **Created At**  | Creation timestamp                            |

    Filter by **Name** or **Status**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    <CodeGroup>
      ```bash title="List all volumes" theme={null}
      openstack volume list
      ```

      ```bash title="Show volume details" theme={null}
      openstack volume show <VOLUME_ID>
      ```
    </CodeGroup>
  </Tab>
</Tabs>

***

## Volume Detail

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Click a volume name to open the detail page. The header shows volume name,
    description, shared status, current status, size, creation time, type, and
    encryption status.

    Three tabs are available:

    * **Detail** — Base information, attachment details, source snapshot/image links
    * **Volume Backups** — All backups created from this volume
    * **Volume Snapshots** — All snapshots of this volume
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show full volume details" theme={null}
    openstack volume show <VOLUME_ID>
    ```
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="Attach a Volume" href="/services/storage/attach-volume">
    Connect this volume to a compute instance
  </Card>

  <Card title="Volume Snapshots" href="/services/storage/snapshots">
    Create point-in-time snapshots of your volumes
  </Card>

  <Card title="Volume Types" href="/services/storage/volume-types">
    Understand available storage tiers
  </Card>

  <Card title="Extend a Volume" href="/services/storage/extend-volume">
    Increase volume capacity without data loss
  </Card>
</CardGroup>


# Volume Encryption
Source: https://docs.xloud.tech/services/storage/encryption

Configure at-rest encryption for Xloud Block Storage volume types. Enable LUKS-based transparent encryption backed by the Xloud Key Management service.

## Overview

Volume encryption provides transparent at-rest protection for block storage data. Encryption is configured per volume type — all volumes created from an encrypted type are automatically encrypted without any additional action from you. Encryption keys are managed by the Xloud Key Management service and retrieved at volume attach time by the volume service, ensuring keys are never stored on the compute node's disk.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Tabs>
  <Tab title="XDeploy" icon="server">
    Disk encryption is enabled through the XDeploy Configuration panel:

    <Steps>
      <Step title="Enable KMS first" icon="key">
        Navigate to **XDeploy → Configuration → Advance Features** and ensure
        **Enable KMS** is set to **Yes**. Disk encryption requires the Xloud Key
        Management service to store and manage encryption keys.
      </Step>

      <Step title="Enable Disk Encryption" icon="lock">
        On the same **Advance Features** tab, set **Enable Disk Encryption** to **Yes**.
      </Step>

      <Step title="Save and deploy" icon="rocket">
        Click **Save Configuration**, then navigate to **XDeploy → Operations** and
        run a **Deploy** or **Reconfigure** for the Block Storage and Key Management
        services.

        <Check>Disk encryption is enabled. Create encrypted volume types to apply encryption to new volumes.</Check>
      </Step>
    </Steps>

    <Warning>
      KMS must be fully deployed and accessible from all compute nodes before enabling
      disk encryption. If KMS is unreachable, encrypted volume attach operations will fail.
    </Warning>
  </Tab>

  <Tab title="CLI" icon="terminal">
    Configure encryption by creating encrypted volume types via the CLI. The Key Management
    service must be deployed and accessible before proceeding. See the configuration steps below.
  </Tab>
</Tabs>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Xloud Key Management service deployed and accessible
  * At least one unencrypted volume type to apply encryption to (or create a new type)
  * All compute nodes must be able to reach the Key Management service API
</Note>

***

## Encryption Architecture

```mermaid theme={null}
graph TD
    User["User creates volume\nfrom encrypted type"]
    API["Block Storage API"]
    KMS["Key Management Service"]
    VS["Volume Service"]
    HV["Hypervisor\n(dm-crypt / LUKS)"]
    Storage["Storage Backend"]

    User --> API
    API --> KMS
    KMS --> API
    API --> VS
    VS --> Storage
    User -->|"Attach volume"| HV
    HV --> KMS
    KMS -->|"Encryption key"| HV
    HV -->|"Encrypted I/O"| Storage
```

| Component                  | Role                                                               |
| -------------------------- | ------------------------------------------------------------------ |
| **Block Storage API**      | Creates volume with encryption metadata; requests key from KMS     |
| **Key Management Service** | Generates and stores the encryption key; returns it at attach time |
| **Volume Service**         | Provisions the volume on the backend with encryption metadata      |
| **Hypervisor (dm-crypt)**  | Applies LUKS encryption/decryption at the block device layer       |

***

## Configure Volume Type Encryption

<Warning>
  Encryption settings cannot be added to or removed from a volume type that already has
  volumes. Create a new encrypted volume type and migrate existing volumes if encryption
  is required on previously unencrypted data.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Select the volume type to encrypt" icon="lock">
        Navigate to
        **Storage > Volume Types** (admin view). Click the volume type name to open its
        details page.
      </Step>

      <Step title="Create encryption settings" icon="key">
        Click **Create Encryption**. Configure the parameters:

        | Field            | Recommended Value | Description                                     |
        | ---------------- | ----------------- | ----------------------------------------------- |
        | Provider         | `LuksEncryptor`   | LUKS-based encryption via dm-crypt              |
        | Cipher           | `aes-xts-plain64` | AES-XTS — FIPS-compatible, hardware-accelerated |
        | Key Size         | `256`             | Key length in bits (256 for AES-256)            |
        | Control Location | `front-end`       | Encryption applied at the hypervisor layer      |

        Click **Confirm**.

        <Tip>
          `aes-xts-plain64` with a 256-bit key provides AES-256 encryption. This cipher
          is hardware-accelerated on all modern CPUs (AES-NI) and meets FIPS 140-2
          requirements when combined with a FIPS-validated key management service.
        </Tip>
      </Step>

      <Step title="Verify encryption is active" icon="circle-check">
        The volume type details page now shows an encryption configuration block.
        All new volumes created from this type will be encrypted automatically.

        <Check>Volume type encryption configured — all new volumes of this type are encrypted at rest.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create an encrypted volume type" icon="plus">
        Create a new type specifically for encrypted volumes:

        ```bash title="Create encrypted volume type" theme={null}
        openstack volume type create \
          --description "SSD — AES-256 encrypted at rest" \
          ssd-encrypted
        ```

        Associate with the correct backend:

        ```bash title="Set backend association" theme={null}
        openstack volume type set \
          --property volume_backend_name=ssd-backend \
          ssd-encrypted
        ```
      </Step>

      <Step title="Enable encryption on the type" icon="lock">
        ```bash title="Configure LUKS encryption" theme={null}
        openstack volume type set \
          --encryption-provider LuksEncryptor \
          --encryption-cipher aes-xts-plain64 \
          --encryption-key-size 256 \
          --encryption-control-location front-end \
          ssd-encrypted
        ```
      </Step>

      <Step title="Verify encryption configuration" icon="circle-check">
        ```bash title="Show encryption settings" theme={null}
        openstack volume type show ssd-encrypted -c encryption
        ```

        <Check>Encryption configured on the volume type. New volumes will be encrypted automatically.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Test Encryption

Verify that encryption is working end-to-end by creating and attaching a test volume:

<Steps>
  <Step title="Create an encrypted volume" icon="plus">
    ```bash title="Create test encrypted volume" theme={null}
    openstack volume create \
      --size 10 \
      --type ssd-encrypted \
      test-encrypted-volume
    ```
  </Step>

  <Step title="Attach to an instance" icon="link">
    ```bash title="Attach encrypted volume" theme={null}
    openstack server add volume <instance-id> test-encrypted-volume
    ```
  </Step>

  <Step title="Verify LUKS inside the instance" icon="lock">
    SSH into the instance and check the block device:

    ```bash title="Check LUKS header" theme={null}
    sudo cryptsetup isLuks /dev/vdb && echo "LUKS encrypted" || echo "Not encrypted"
    ```

    <Check>Device reports LUKS encryption — volume is encrypted at rest.</Check>
  </Step>

  <Step title="Clean up" icon="trash">
    ```bash title="Detach and delete test volume" theme={null}
    openstack server remove volume <instance-id> test-encrypted-volume
    openstack volume delete test-encrypted-volume
    ```
  </Step>
</Steps>

***

## Key Management Dependency

<Warning>
  Encryption key loss means the volume data is permanently inaccessible — there is no
  recovery path without the key. Before enabling volume encryption in production, ensure
  the Xloud Key Management service is:

  * Deployed in a high-availability configuration
  * Backed up regularly (key database backup)
  * Accessible from all compute nodes that may attach encrypted volumes
</Warning>

If the Key Management service is unavailable, attaching an encrypted volume will fail
with an authentication or connectivity error.

***

## Per-Volume Selective Encryption

<Info>**Xloud-Developed** -- This capability is developed by Xloud and ships with XAVS / XPCI.</Info>

Xloud supports **selective encryption** -- encrypted and unencrypted volume types coexist within the same deployment. This allows administrators to apply encryption only where compliance or data sensitivity requires it, avoiding the performance overhead of blanket encryption on non-sensitive workloads.

Key characteristics:

* **Per-tenant key isolation** -- each tenant's encryption keys are stored and managed independently in the Xloud Key Management service. Tenants cannot access each other's keys, even if they share the same storage backend.
* **Three independent encryption layers** -- Xloud provides encryption at three distinct levels that can be enabled independently or together:

| Layer                                 | Scope                               | Encryption Point                                  |
| ------------------------------------- | ----------------------------------- | ------------------------------------------------- |
| **Storage device encryption**         | Full disk on physical storage nodes | Hardware or dm-crypt on the OSD device            |
| **Block volume encryption**           | Individual persistent volumes       | LUKS at the hypervisor (dm-crypt) per volume type |
| **Compute ephemeral disk encryption** | Instance root and ephemeral disks   | LUKS at the hypervisor for ephemeral storage      |

<Tip>
  For most deployments, block volume encryption alone satisfies data-at-rest compliance
  requirements. Add storage device encryption for defense-in-depth on shared infrastructure,
  and ephemeral disk encryption for instances that process sensitive data without persistent volumes.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Key Manager User Guide" href="/services/key-manager/user-guide">
    Manage encryption keys and secrets in the Xloud Key Management service
  </Card>

  <Card title="Volume Types & QoS" href="/services/storage/volume-types-admin">
    Create and manage volume types with backend associations
  </Card>

  <Card title="Security Hardening" href="/services/storage/security">
    Additional security policies for Block Storage
  </Card>

  <Card title="Admin Guide" href="/services/storage/admin-guide">
    Return to the Block Storage administration overview
  </Card>
</CardGroup>


# Extend a Volume
Source: https://docs.xloud.tech/services/storage/extend-volume

Increase the capacity of an Xloud Block Storage volume online or offline. Covers resizing the block device and expanding the filesystem after extension.

## Overview

Volume extension increases the allocated capacity of an existing block storage volume. The operation grows the underlying block device — the filesystem inside the instance must then be resized separately to use the additional space. Xloud Block Storage supports online extension for volumes that are attached to running instances, depending on the volume type and backend configuration.

<Note>
  **Prerequisites**

  * The target volume must exist and have status **Available** (offline) or **In-use** (online)
  * The new size must be larger than the current size — volumes cannot be shrunk
  * For online extension, the instance must be running and the guest OS must support online resize
</Note>

***

## Online vs Offline Extension

| Mode        | Volume Status        | Instance Required | Filesystem Resize      | Downtime            |
| ----------- | -------------------- | ----------------- | ---------------------- | ------------------- |
| **Online**  | In-use (attached)    | Running           | Live, no reboot needed | None                |
| **Offline** | Available (detached) | Not required      | After re-attach        | Brief detach window |

<Note>
  Online extension is supported for most Linux filesystems (ext4, XFS) on volumes backed
  by distributed storage. The block device grows automatically — the filesystem does not
  resize automatically and must be expanded inside the instance after the volume is extended.
</Note>

***

## Extend a Volume

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Locate the volume" icon="compass">
        Navigate to
        **Storage > Volumes**.
      </Step>

      <Step title="Extend the volume" icon="expand">
        Click the **More** dropdown and select **Capacity & Type > Extend Volume** on the target volume. Enter the new total size
        in GiB — the value must be larger than the current size.

        Click **Confirm**.

        <Warning>
          Volume size can only be **increased**, never decreased. Verify the new size
          before confirming — the operation cannot be reversed.
        </Warning>
      </Step>

      <Step title="Verify the new size" icon="circle-check">
        The volume list refreshes. Confirm the **Size** column reflects the new capacity.

        <Check>Volume extended — new size visible in the Volumes list.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Check current size" icon="search">
        ```bash title="Show current volume size" theme={null}
        openstack volume show <volume-name-or-id> -c size -c status
        ```
      </Step>

      <Step title="Extend the volume" icon="expand">
        ```bash title="Extend volume to new size" theme={null}
        openstack volume set --size 200 <volume-name-or-id>
        ```

        Replace `200` with the new total size in GiB.
      </Step>

      <Step title="Verify the new size" icon="circle-check">
        ```bash title="Confirm new size" theme={null}
        openstack volume show <volume-name-or-id> -c size -c status
        ```

        <Check>Size field reflects the new capacity. Status remains `in-use` or `available`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Resize the Filesystem

After extending the block device, resize the filesystem inside the instance to use the
additional capacity. The correct command depends on the filesystem type.

<Tabs>
  <Tab title="ext4" icon="hard-drive">
    ext4 filesystems can be resized online while mounted:

    ```bash title="Resize ext4 filesystem (online)" theme={null}
    sudo resize2fs /dev/vdb
    ```

    ```bash title="Verify new capacity" theme={null}
    df -h /mnt/data
    ```

    <Check>Filesystem expanded to the full volume size with no downtime.</Check>
  </Tab>

  <Tab title="XFS" icon="hard-drive">
    XFS filesystems must be resized while mounted. The filesystem must be actively
    mounted at the target path:

    ```bash title="Resize XFS filesystem (online)" theme={null}
    sudo xfs_growfs /mnt/data
    ```

    ```bash title="Verify new capacity" theme={null}
    df -h /mnt/data
    ```

    <Check>XFS filesystem grown to the new volume size.</Check>
  </Tab>

  <Tab title="LVM (logical volume)" icon="layers">
    If the volume contains an LVM physical volume, resize the PV and LV separately:

    ```bash title="Resize physical volume" theme={null}
    sudo pvresize /dev/vdb
    ```

    ```bash title="Extend logical volume to 100% of available space" theme={null}
    sudo lvextend -l +100%FREE /dev/mapper/vg0-lv0
    ```

    ```bash title="Resize the filesystem on the logical volume" theme={null}
    sudo resize2fs /dev/mapper/vg0-lv0
    ```

    <Check>LVM logical volume and filesystem resized to the new capacity.</Check>
  </Tab>
</Tabs>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Extension fails with 'Invalid volume: Volume status must be available or in-use'" icon="triangle-alert">
    **Cause**: The volume is in an intermediate state (`creating`, `deleting`, `error`,
    or `migrating`) that does not permit size changes.

    **Resolution**:

    ```bash title="Check volume status" theme={null}
    openstack volume show <volume-id> -c status
    ```

    Wait for the operation to complete or contact your administrator if the volume is
    stuck in an error state.
  </Accordion>

  <Accordion title="Filesystem does not show new capacity after extension" icon="hard-drive">
    **Cause**: The block device was extended but the filesystem was not resized.

    **Resolution**: Run the appropriate filesystem resize command (`resize2fs` for ext4,
    `xfs_growfs` for XFS) as described in the **Resize the Filesystem** section above.
    Verify with `df -h` that the new capacity is visible.
  </Accordion>

  <Accordion title="resize2fs reports 'No space left on device'" icon="circle-x">
    **Cause**: The filesystem resize is being attempted on a device that has not yet
    been extended at the block level, or the extension has not propagated to the guest.

    **Resolution**:

    ```bash title="Check block device size inside instance" theme={null}
    sudo blockdev --getsize64 /dev/vdb
    ```

    Compare the reported bytes against the expected new size. If the block device still
    shows the old size, wait a few seconds and retry — the backend may still be
    completing the resize operation.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Volume Snapshots" href="/services/storage/snapshots">
    Create a snapshot before extending to preserve a recovery point
  </Card>

  <Card title="Attach / Detach" href="/services/storage/attach-volume">
    Connect volumes to compute instances and prepare filesystems
  </Card>

  <Card title="Volume Backups" href="/services/storage/backups">
    Back up volume data to a separate target for long-term retention
  </Card>

  <Card title="Create a Volume" href="/services/storage/create-volume">
    Provision a new volume with the correct initial capacity and storage tier
  </Card>
</CardGroup>


# External Storage Integration
Source: https://docs.xloud.tech/services/storage/external-storage

Connect Xloud Block Storage to enterprise storage backends including NetApp, Pure Storage, Dell, HPE, Hitachi, IBM, and Ceph RBD.

## Overview

Xloud Block Storage connects to a broad range of enterprise storage platforms through a pluggable driver architecture. Each backend is configured as a named section in `cinder.conf`, registered with the scheduler, and exposed to tenants as one or more volume types. Multiple backends can run simultaneously, enabling tiered storage, vendor-specific pools, and failover configurations within a single environment.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * The storage array must be network-accessible from all volume service nodes
  * Vendor-specific driver packages installed on the volume service nodes
  * Authentication credentials (username, password, API endpoint) for each backend
</Note>

***

## Supported Backends

| Backend                     | Protocol             | Use Case                                      | HA                |
| --------------------------- | -------------------- | --------------------------------------------- | ----------------- |
| **Ceph RBD**                | RBD                  | Production — native to XAVS, zero-copy clones | Yes               |
| **NetApp ONTAP**            | iSCSI / NFS / FC     | Enterprise NAS/SAN with ONTAP arrays          | Yes               |
| **Pure Storage FlashArray** | iSCSI / FC / NVMe-oF | All-flash arrays, high IOPS workloads         | Yes               |
| **Dell PowerStore**         | iSCSI / FC / NVMe-oF | Dell enterprise all-flash and hybrid arrays   | Yes               |
| **HPE 3PAR / Primera**      | iSCSI / FC           | HPE enterprise SAN arrays                     | Yes               |
| **Hitachi VSP**             | iSCSI / FC           | Hitachi enterprise storage systems            | Yes               |
| **IBM FlashSystem**         | iSCSI / FC / NVMe-oF | IBM all-flash enterprise arrays               | Yes               |
| **NFS**                     | NFS                  | Shared NAS appliances, legacy integrations    | Depends on server |
| **LVM**                     | iSCSI                | Development and single-node testing only      | No                |

***

## Multi-Backend Configuration

Cinder supports multiple backends in a single `cinder.conf`. Each backend is an independent section. The `[DEFAULT]` section lists all enabled backends by name.

```ini title="cinder.conf — multi-backend example" theme={null}
[DEFAULT]
enabled_backends = ceph-ssd,netapp-nas,pure-flash

[ceph-ssd]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph-ssd
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = <libvirt-secret-uuid>

[netapp-nas]
volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
volume_backend_name = netapp-nas
netapp_storage_family = ontap_cluster
netapp_transport_type = https
netapp_server_hostname = 10.0.10.50
netapp_server_port = 443
netapp_login = admin
netapp_password = <password>
netapp_vserver = svm0

[pure-flash]
volume_driver = cinder.volume.drivers.pure.PureISCSIDriver
volume_backend_name = pure-flash
san_ip = 10.0.10.60
pure_api_token = <api-token>
```

<Tip>
  In XAVS deployments, backend configuration is managed through XDeploy globals. The `cinder.conf` file is generated from Ansible templates and should not be edited directly on the host. Use `xavs-ansible deploy --tags cinder` to apply changes.
</Tip>

***

## Backend-Specific Configuration Examples

<AccordionGroup>
  <Accordion title="Ceph RBD" icon="database">
    Ceph RBD is the native XAVS production backend. It supports thin provisioning by default and enables zero-copy clones between the image pool and volume pool for fast instance launches.

    ```ini title="[ceph-ssd] section" theme={null}
    volume_driver = cinder.volume.drivers.rbd.RBDDriver
    volume_backend_name = ceph-ssd
    rbd_pool = volumes
    rbd_ceph_conf = /etc/ceph/ceph.conf
    rbd_user = cinder
    rbd_keyring_conf = /etc/ceph/ceph.client.cinder.keyring
    rbd_secret_uuid = <libvirt-secret-uuid>
    rbd_cluster_name = ceph
    rados_connect_timeout = 5
    ```
  </Accordion>

  <Accordion title="NetApp ONTAP" icon="server">
    NetApp ONTAP supports iSCSI, NFS, and Fibre Channel protocols. The `netapp_vserver` parameter specifies the storage virtual machine (SVM) on the array.

    ```ini title="[netapp-iscsi] section" theme={null}
    volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
    volume_backend_name = netapp-iscsi
    netapp_storage_family = ontap_cluster
    netapp_transport_type = https
    netapp_server_hostname = 10.0.10.50
    netapp_server_port = 443
    netapp_login = admin
    netapp_password = <password>
    netapp_vserver = svm0
    netapp_storage_protocol = iscsi
    ```
  </Accordion>

  <Accordion title="Pure Storage FlashArray" icon="zap">
    Pure Storage uses a REST API token for authentication. Both iSCSI and Fibre Channel drivers are available.

    ```ini title="[pure-flash] section (iSCSI)" theme={null}
    volume_driver = cinder.volume.drivers.pure.PureISCSIDriver
    volume_backend_name = pure-flash
    san_ip = 10.0.10.60
    pure_api_token = <api-token>
    use_multipath_for_image_xfer = true
    ```

    ```ini title="[pure-flash-fc] section (Fibre Channel)" theme={null}
    volume_driver = cinder.volume.drivers.pure.PureFCDriver
    volume_backend_name = pure-flash-fc
    san_ip = 10.0.10.60
    pure_api_token = <api-token>
    ```
  </Accordion>

  <Accordion title="Dell PowerStore" icon="box">
    Dell PowerStore supports iSCSI, NVMe-oF, and Fibre Channel. The management IP and credentials are required.

    ```ini title="[dell-powerstore] section" theme={null}
    volume_driver = cinder.volume.drivers.dell_emc.powerstore.driver.PowerStoreDriver
    volume_backend_name = dell-powerstore
    san_ip = 10.0.10.70
    san_login = admin
    san_password = <password>
    powerstore_appliances = Appliance-1
    storage_protocol = iSCSI
    ```
  </Accordion>

  <Accordion title="HPE 3PAR / Primera" icon="server">
    HPE 3PAR and Primera share the same driver. The CPG (Common Provisioning Group) name maps to a storage pool on the array.

    ```ini title="[hpe-3par] section" theme={null}
    volume_driver = cinder.volume.drivers.hpe.hpe_3par_iscsi.HPE3PARISCSIDriver
    volume_backend_name = hpe-3par
    san_ip = 10.0.10.80
    san_login = 3parrc
    san_password = <password>
    hpe3par_api_url = https://10.0.10.80:8080/api/v1
    hpe3par_username = 3parrc
    hpe3par_password = <password>
    hpe3par_cpg = SSD_r6
    ```
  </Accordion>

  <Accordion title="IBM FlashSystem" icon="cpu">
    IBM FlashSystem uses the Storwize/SVC driver. The `san_clustername` must match the system name on the array.

    ```ini title="[ibm-flash] section" theme={null}
    volume_driver = cinder.volume.drivers.ibm.storwize_svc.storwize_svc_iscsi.StorwizeSVCISCSIDriver
    volume_backend_name = ibm-flash
    san_ip = 10.0.10.90
    san_login = superuser
    san_password = <password>
    storwize_svc_volpool_name = Pool0
    san_clustername = IBM_Flash_1
    ```
  </Accordion>
</AccordionGroup>

***

## Register a Backend and Create a Volume Type

After configuring a backend, create a volume type that maps to it. Tenants select the volume type when provisioning volumes to route requests to a specific backend.

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    <Steps>
      <Step title="Verify backend registration" icon="activity">
        Navigate to **Storage > Storage Backends** (admin view) and confirm the volume service
        host for the backend shows state **Up** and status **Enabled**.
      </Step>

      <Step title="Create a volume type" icon="plus">
        Navigate to **Storage > Volume Types** (admin view) and click **Create Volume Type**.

        Enter a name that communicates the backend to users — e.g., `SSD-Flash` or `Enterprise-NAS`.
      </Step>

      <Step title="Set the backend name extra spec" icon="settings">
        After creating the type, click **View Extra Specs** and add:

        | Key                   | Value                                                         |
        | --------------------- | ------------------------------------------------------------- |
        | `volume_backend_name` | Must match the `volume_backend_name` in `cinder.conf` exactly |

        Click **Confirm** to save the extra spec.
      </Step>

      <Step title="Validate" icon="circle-check">
        Create a test volume using the new volume type and confirm it provisions successfully.

        <Check>Volume type mapped — tenants can now select this backend when creating volumes.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Verify backend is registered" icon="activity">
        ```bash title="List all backend pools" theme={null}
        openstack volume backend pool list --long
        ```

        Confirm the backend name appears with `free_capacity_gb` greater than 0.
      </Step>

      <Step title="Create volume type" icon="plus">
        ```bash title="Create a volume type" theme={null}
        openstack volume type create \
          --description "Pure Storage all-flash backend" \
          SSD-Flash
        ```
      </Step>

      <Step title="Map type to backend" icon="link">
        ```bash title="Set backend name extra spec" theme={null}
        openstack volume type set \
          --property volume_backend_name=pure-flash \
          SSD-Flash
        ```
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="Confirm extra spec" theme={null}
        openstack volume type show SSD-Flash
        ```

        ```bash title="Create a test volume" theme={null}
        openstack volume create \
          --type SSD-Flash \
          --size 10 \
          test-pure-vol
        ```

        <Check>Volume type registered and backend verified.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Failover Between Backends

When a backend becomes unavailable, volumes can be migrated to an alternative backend using the volume migration API. This is an admin operation and results in data movement.

```bash title="Migrate a volume to a different backend" theme={null}
openstack volume migrate \
  --host <cinder-volume-host>@<target-backend>#<pool> \
  <volume-id>
```

```bash title="Check migration status" theme={null}
openstack volume show <volume-id> -c migration_status
```

<Warning>
  Volume migration copies all data from the source backend to the destination. Migration time depends on volume size and network throughput. The volume remains available (online migration) but may experience reduced I/O performance during the transfer.
</Warning>

For backends that support replication (such as Pure Storage and NetApp ONTAP), use the replication failover API instead of data migration:

```bash title="Failover to replication target" theme={null}
cinder failover-host <cinder-volume-host> \
  --backend_id <replication-target-backend-id>
```

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Backend does not appear in pool list" icon="triangle-alert">
    **Cause**: The volume service failed to initialize the driver, or the backend configuration section is missing from `cinder.conf`.

    **Resolution**:

    * Verify the backend name is listed in `enabled_backends` in `[DEFAULT]`
    * Check volume service logs for driver initialization errors
    * Confirm network connectivity from the volume service node to the storage array management IP

    ```bash title="Check volume service status" theme={null}
    openstack volume service list
    ```
  </Accordion>

  <Accordion title="Volume creation fails with 'No valid host was found'" icon="x-circle">
    **Cause**: No backend matches the scheduler filters for the requested volume type, or the backend is reporting no free capacity.

    **Resolution**:

    * Verify the `volume_backend_name` extra spec on the volume type matches the backend name exactly (case-sensitive)
    * Check backend capacity: `openstack volume backend pool list --long`
    * Confirm the volume service is in state `up` and status `enabled`
  </Accordion>

  <Accordion title="Authentication error connecting to storage array" icon="lock">
    **Cause**: Incorrect credentials or the management IP is unreachable.

    **Resolution**:

    * Verify the `san_ip`, `san_login`, and `san_password` (or equivalent) values in `cinder.conf`
    * Test network connectivity: `curl -k https://<san_ip>/api/v1` from the volume service node
    * Check if a firewall or ACL is blocking access from the volume service node
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Volume Types Admin" href="/services/storage/volume-types-admin">
    Create and manage volume types that expose backends to tenants
  </Card>

  <Card title="Thin Provisioning" href="/services/storage/thin-provisioning">
    Configure thin and thick provisioning per backend
  </Card>

  <Card title="Storage Tiers" href="/services/storage/storage-tiers">
    Configure multi-tier storage pools across different hardware classes
  </Card>

  <Card title="Backup Backends" href="/services/storage/backup-backends">
    Configure backup destinations for volume data protection
  </Card>
</CardGroup>


# File-Level Restore
Source: https://docs.xloud.tech/services/storage/file-level-restore

Recover individual files from a volume or instance snapshot without restoring the whole volume — download the files you need directly.

## Overview

File-Level Restore lets you reach into an existing snapshot and pull out a single file, a
handful of files, or a folder — without restoring the entire volume first. Pick a
snapshot, open its filesystem in the Dashboard, navigate the directory tree, and download
the files you need as individual files or a single ZIP archive.

This is the fastest way to recover from accidental deletions, configuration mistakes, or
corruption of a specific file, without spinning up a new volume or attaching a recovery
VM.

<Note>
  **Prerequisites**

  * An existing volume snapshot or instance snapshot
  * The snapshot must be in a usable state — **Available** for a volume snapshot,
    **Active** for an instance snapshot
</Note>

***

## Video Walkthrough

<iframe title="How to Recover Files from a Snapshot on Xloud" />

***

## When to Use File-Level Restore

| Scenario                                               | Full Restore | File-Level Restore |
| ------------------------------------------------------ | :----------: | :----------------: |
| Recover a single deleted config file                   |   Overkill   |      Best fit      |
| Recover a folder of documents                          |   Overkill   |      Best fit      |
| Recover from ransomware that encrypted a whole disk    |   Best fit   |          —         |
| Roll back after a failed OS upgrade                    |   Best fit   |          —         |
| Pull a log file from last week's snapshot for auditing |   Overkill   |      Best fit      |
| Compare old vs current configuration                   |   Overkill   |      Best fit      |

<Tip>
  File-Level Restore is non-destructive — it opens the snapshot read-only. You cannot
  accidentally modify the snapshot's contents while browsing it.
</Tip>

***

## Where to Find It

The **Browse Files** action appears on the row actions of every eligible snapshot:

| Dashboard location                                        | Source               | Eligible when                    |
| --------------------------------------------------------- | -------------------- | -------------------------------- |
| **Storage → Volume Snapshots** → row → **Browse Files**   | A volume snapshot    | Snapshot status is **Available** |
| **Compute → Instance Snapshots** → row → **Browse Files** | An instance snapshot | Snapshot status is **Active**    |

<Info>
  If you do not see **Browse Files**, either the snapshot is in a non-ready state, or the
  File-Level Restore feature is not enabled on your cluster. Ask your administrator.
</Info>

***

## Recover Files from a Snapshot

<Steps>
  <Step title="Find your snapshot" icon="search">
    Open the appropriate page — **Storage → Volume Snapshots** or
    **Compute → Instance Snapshots** — and locate the snapshot you want to recover
    from.
  </Step>

  <Step title="Click Browse Files" icon="folder-open">
    In the row actions, click **Browse Files**. The Dashboard opens a full-page file
    browser tagged with the snapshot type and ID.

    <Info>
      The first open of a snapshot can take **20-30 seconds** as the Dashboard opens the
      snapshot's filesystem. Subsequent opens of the same snapshot are much faster.
    </Info>
  </Step>

  <Step title="Pick the right partition" icon="hard-drive">
    At the top of the page, a **partition dropdown** lists every partition on the
    snapshot with its size and detected OS or filesystem label. The Dashboard picks the
    most likely default — usually the OS partition. Change it if your target files live
    on a different partition.
  </Step>

  <Step title="Navigate the filesystem" icon="folder-tree">
    Use the file table to browse:

    * Click a **folder** row to enter it
    * Click a breadcrumb segment at the top to jump back to a parent folder
    * Click the **Up** button to move one folder up
    * Click **Reload** to refresh the current folder

    Each row shows the name, type (file or directory), size, and last-modified time.
  </Step>

  <Step title="Download what you need" icon="download">
    Two ways to download:

    | Method                       | Use for                                                                                                 |
    | ---------------------------- | ------------------------------------------------------------------------------------------------------- |
    | **Per-row Download icon**    | A single file — downloads directly to your browser                                                      |
    | **Download Selected as ZIP** | Multiple files at once — tick the checkboxes, then click the primary button to get a single ZIP archive |

    <Check>Files arrive in your browser's default download folder.</Check>
  </Step>

  <Step title="Close or continue browsing" icon="arrow-left">
    Click the **Back** button in the top-left when you are done, or continue browsing to
    recover additional files from the same snapshot.
  </Step>
</Steps>

***

## Common Tasks

<AccordionGroup>
  <Accordion title="Recover a single deleted file" icon="file">
    1. Go to the snapshot that was taken before the file was deleted
    2. Click **Browse Files**
    3. Navigate to the folder the file was in (e.g. `/etc`, `/home/you/Documents`)
    4. Click the per-row **Download** icon

    The file is restored to your laptop — copy it back into the running VM to finish the
    recovery.
  </Accordion>

  <Accordion title="Recover a folder of files" icon="folder">
    1. **Browse Files** into the snapshot
    2. Navigate to the folder
    3. Tick the checkboxes for the files you want
    4. Click **Download Selected as ZIP**

    Extract the ZIP on your workstation and copy the files back where they belong.
  </Accordion>

  <Accordion title="Compare an old config against the current one" icon="clipboard">
    1. **Browse Files** into an older snapshot
    2. Download the config file (for example `/etc/nginx/nginx.conf`)
    3. Download the current version from the running VM
    4. Run your favorite diff tool on the two files

    No downtime, no extra volume provisioning.
  </Accordion>

  <Accordion title="Find a file when you do not know the partition" icon="search">
    Many VMs have multiple partitions (EFI, system, data). If the OS partition is picked
    by default and your file is not there, open the **partition dropdown** and try the
    next partition — typically the largest one or the one labeled **data**.
  </Accordion>
</AccordionGroup>

***

## Good to Know

* **Read-only by design** — browsing a snapshot never modifies its contents.
* **Project-scoped** — you can only browse snapshots in projects you have access to.
* **First open is the slowest** — opening a snapshot takes 20-30 seconds the first time.
  Subsequent navigation inside the same snapshot is near-instant.
* **Best for recovery, not discovery** — File-Level Restore shines when you know roughly
  which snapshot and folder to look in. For wide searches, consider restoring the whole
  volume to a rescue VM.

***

## Related Topics

<CardGroup>
  <Card title="Volume Snapshots" href="/services/storage/snapshots">
    Create snapshots you can later browse for file-level recovery
  </Card>

  <Card title="Volume Backups" href="/services/storage/backups">
    Full volume protection with off-site retention for disaster recovery
  </Card>

  <Card title="Storage Troubleshooting" href="/services/storage/troubleshooting">
    Diagnose snapshot and volume issues
  </Card>
</CardGroup>


# Block Storage
Source: https://docs.xloud.tech/services/storage/index

Persistent block storage volumes for Xloud Compute instances.

Durable, high-performance block storage volumes that attach to your Xloud Compute instances and persist independently of instance lifecycle.

<Card title="XSDS — Software-Defined Storage" icon="external-link" href="https://xloud.tech/ceph">
  Product details, hardware compatibility, and datasheet on xloud.tech
</Card>

***

<p>Block Storage</p>

<CardGroup>
  <Card title="User Guide" icon="book-open" href="/services/storage/user-guide">
    Create, attach, detach, and resize volumes. Manage snapshots, backups, and volume transfers. Step-by-step workflows for both Dashboard and CLI.
  </Card>

  <Card title="Admin Guide" icon="settings" href="/services/storage/admin-guide">
    Configure storage backends, define volume types and QoS policies, manage storage tiers, set quotas, and maintain the block storage service.
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/services/storage/cli-reference">
    `openstack volume` commands for creating, managing, and querying volumes and snapshots from the command line.
  </Card>

  <Card title="XSDS Product Page" icon="external-link" href="https://xloud.tech/ceph">
    Distributed storage architecture, supported hardware, performance benchmarks, and licensing information.
  </Card>
</CardGroup>

***

<p>Key Capabilities</p>

<CardGroup>
  <Card title="Persistent Volumes" icon="hard-drive" href="/services/storage/create-volume">
    Volumes exist independently of instances — data survives instance deletion, restart, and migration.
  </Card>

  <Card title="Live Resize" icon="expand" href="/services/storage/extend-volume">
    Extend volume capacity online without detaching or rebooting the attached instance.
  </Card>

  <Card title="Snapshots & Backups" icon="copy" href="/services/storage/snapshots">
    Point-in-time crash-consistent snapshots and incremental backups to protect critical data.
  </Card>

  <Card title="Storage Tiers" icon="layers" href="/services/storage/storage-tiers">
    NVMe, SSD, and HDD-backed volume types with defined IOPS and throughput guarantees for every workload class.
  </Card>

  <Card title="Volume Encryption" icon="lock" href="/services/storage/encryption">
    Transparent at-rest encryption backed by the Xloud Key Management service — no application changes required.
  </Card>

  <Card title="Cross-Project Transfers" icon="arrow-right-arrow-left" href="/services/storage/volume-transfers">
    Transfer volume ownership between projects with a secure token-based handoff workflow.
  </Card>
</CardGroup>

***

<p>Related Services</p>

<CardGroup>
  <Card title="Xloud Compute" icon="server" href="/services/compute">
    Virtual machine instances that consume block storage volumes as persistent disks
  </Card>

  <Card title="Xloud Networking" icon="network">
    Virtual networks and floating IPs used alongside compute and storage resources
  </Card>

  <Card title="Xloud Distributed Storage" icon="database" href="https://xloud.tech/ceph">
    The distributed storage backend that powers block, object, and file storage
  </Card>
</CardGroup>


# Volume Migration
Source: https://docs.xloud.tech/services/storage/migration

Migrate block storage volumes between backends for capacity rebalancing, hardware retirement, or tier changes.

## Overview

Volume migration copies all volume data to a different backend while preserving volume identity, metadata, and attachment state. Migration is used to rebalance capacity across backends, retire storage hardware, or move volumes to a higher or lower performance tier. Xloud Block Storage supports two migration modes: backend-assisted migration (when backends share a storage layer) and host-copy migration (a full data copy between independent backends).

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * The destination backend must be registered and have sufficient free capacity
  * Identify the destination host with `openstack volume service list`
</Note>

***

## Migration Modes

| Mode                 | When Used                                             | Duration                                  | Impact                   |
| -------------------- | ----------------------------------------------------- | ----------------------------------------- | ------------------------ |
| **Backend-assisted** | Source and destination share the same storage cluster | Seconds to minutes (metadata only)        | Minimal — no data copy   |
| **Host-copy**        | Independent backends (different clusters)             | Minutes to hours depending on volume size | I/O overhead during copy |

<Warning>
  Volume migration copies all data to the destination backend. Duration depends on
  volume size and available network/storage bandwidth. Schedule large migrations during
  off-peak hours to minimize impact on running workloads.
</Warning>

***

## Migrate a Volume

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Locate the volume" icon="search">
        Navigate to
        **Storage > Volumes** (admin view). Use the search or filter to find the target volume.
      </Step>

      <Step title="Initiate migration" icon="move">
        Click the **More** dropdown and select **Migrate**. Select the destination backend from the
        **Destination Host** dropdown.

        Enable **Force Host Copy** only if the backends do not share a storage layer
        and backend-assisted migration is not possible.

        Click **Confirm**.
      </Step>

      <Step title="Monitor progress" icon="activity">
        The volume status changes to `migrating`. Refresh the volume list periodically
        to track progress. On completion, status returns to `available` and the
        **Hosted At** field updates to the destination backend.

        <Check>Migration complete — volume is now on the destination backend.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="List available destination backends" icon="list">
        ```bash title="List volume service backends" theme={null}
        openstack volume service list
        ```

        Note the host name in the format `<service>@<backend-name>#<pool>` from the
        `Host` column — use this value for the `--host` parameter.
      </Step>

      <Step title="Check source volume location" icon="search">
        ```bash title="Show current backend for volume" theme={null}
        openstack volume show <volume-id> -c os-vol-host-attr:host
        ```
      </Step>

      <Step title="Initiate migration" icon="move">
        ```bash title="Migrate volume to new backend" theme={null}
        openstack volume migrate \
          --host <destination-host> \
          <volume-id>
        ```

        To force a host-level data copy (bypassing backend-assisted migration):

        ```bash title="Force host-copy migration" theme={null}
        openstack volume migrate \
          --host <destination-host> \
          --force-host-copy \
          <volume-id>
        ```
      </Step>

      <Step title="Monitor migration status" icon="activity">
        ```bash title="Check migration progress" theme={null}
        openstack volume show <volume-id> -c migration_status -c status
        ```

        Expected final values: `migration_status = success`, `status = available`

        <Check>Volume migrated successfully to the new backend.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Migrate Multiple Volumes (Backend Retirement)

To retire a backend, migrate all volumes off it before decommissioning:

<Steps>
  <Step title="List volumes on the backend to retire" icon="list">
    ```bash title="List volumes on a specific backend" theme={null}
    openstack volume list --all-projects \
      --host <backend-host-to-retire>
    ```
  </Step>

  <Step title="Disable the backend service" icon="ban">
    Prevent new volumes from being scheduled to the retiring backend:

    ```bash title="Disable backend service" theme={null}
    openstack volume service set \
      --disable \
      --disable-reason "Scheduled for retirement" \
      <backend-host> \
      cinder-volume
    ```
  </Step>

  <Step title="Migrate all volumes" icon="move">
    Run migration for each volume listed in Step 1:

    ```bash title="Migrate each volume" theme={null}
    openstack volume migrate --host <destination-host> <volume-id>
    ```

    For large numbers of volumes, use a loop:

    ```bash title="Batch migration" theme={null}
    for vid in $(openstack volume list --all-projects --host <old-host> -f value -c ID); do
      openstack volume migrate --host <new-host> "$vid"
    done
    ```
  </Step>

  <Step title="Verify all migrations complete" icon="circle-check">
    ```bash title="Check for remaining volumes on old backend" theme={null}
    openstack volume list --all-projects --host <old-host>
    ```

    <Check>No volumes remaining on the retired backend — safe to decommission.</Check>
  </Step>
</Steps>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Migration stuck in 'migrating' status" icon="clock">
    **Symptom**: Volume status remains `migrating` for an extended period with no progress.

    **Resolution**:

    ```bash title="Check migration status and logs" theme={null}
    openstack volume show <volume-id> -c migration_status -c status
    ```

    Check volume service logs on the source and destination nodes via XDeploy for
    migration-related errors.

    If the migration is permanently stuck, reset the volume state:

    ```bash title="Reset volume state (admin only)" theme={null}
    openstack volume set --state available <volume-id>
    ```

    <Warning>
      Resetting the volume state does not undo a partial migration. Verify data integrity
      on both the source and destination backends before resetting. In some cases,
      partial data may exist on both backends.
    </Warning>
  </Accordion>

  <Accordion title="Migration fails with 'destination backend has insufficient capacity'" icon="database">
    **Cause**: The destination backend does not have enough free capacity to accommodate
    the volume.

    **Resolution**:

    ```bash title="Check destination backend capacity" theme={null}
    openstack volume backend pool list --long
    ```

    Select a different destination with sufficient free capacity, or free up space on
    the intended destination backend.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Configure and manage backend drivers and capacity
  </Card>

  <Card title="Storage Tiers" href="/services/storage/storage-tiers">
    Configure multi-tier storage to give users tier selection at volume creation
  </Card>

  <Card title="Quotas" href="/services/storage/quotas">
    Monitor and manage storage quota allocation across projects
  </Card>

  <Card title="Admin Guide" href="/services/storage/admin-guide">
    Return to the Block Storage administration overview
  </Card>
</CardGroup>


# Storage Quality of Service
Source: https://docs.xloud.tech/services/storage/qos

Define IOPS limits, throughput caps, and burst allowances for block storage volumes. Create and associate QoS specifications.

## Overview

Storage QoS (Quality of Service) specifications define performance boundaries for block storage volumes — capping IOPS, throughput, and burst behavior at the hypervisor-to-storage layer. QoS specs are attached to volume types, so every volume created from that type automatically inherits the performance constraints. This prevents high-throughput workloads from starving adjacent volumes on shared storage.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Admin credentials sourced from `openrc.sh`
  * Block Storage service deployed and healthy
  * Volume types configured ([Volume Types Admin Guide](/services/storage/volume-types-admin))
  * QoS support enabled on the target storage backend (Ceph, NetApp, or Pure Storage)
</Note>

***

## QoS Specification Parameters

Storage QoS specs define performance limits using key-value pairs. The specific keys supported depend on the storage backend driver.

<AccordionGroup>
  <Accordion title="Standard QoS Keys (all backends)" icon="settings">
    | Key                   | Unit    | Description                                 |
    | --------------------- | ------- | ------------------------------------------- |
    | `total_iops_sec`      | IOPS    | Maximum combined read + write IOPS          |
    | `read_iops_sec`       | IOPS    | Maximum read IOPS                           |
    | `write_iops_sec`      | IOPS    | Maximum write IOPS                          |
    | `total_bytes_sec`     | bytes/s | Maximum combined read + write throughput    |
    | `read_bytes_sec`      | bytes/s | Maximum read throughput                     |
    | `write_bytes_sec`     | bytes/s | Maximum write throughput                    |
    | `total_iops_sec_max`  | IOPS    | Burst IOPS ceiling                          |
    | `total_bytes_sec_max` | bytes/s | Burst throughput ceiling                    |
    | `size_iops_sec`       | IOPS/GB | IOPS scaling factor (per GB of volume size) |

    <Info>
      `size_iops_sec` scales limits proportionally to volume size. A spec of `size_iops_sec=10` grants a 100 GB volume 1,000 IOPS and a 500 GB volume 5,000 IOPS automatically.
    </Info>
  </Accordion>

  <Accordion title="Ceph RBD QoS Keys" icon="database">
    Ceph RBD enforces QoS through its `rbd_qos_*` parameters, configured in `ceph.conf` and overridable per volume via extra specs.

    | Key                        | Unit    | Description                            |
    | -------------------------- | ------- | -------------------------------------- |
    | `rbd_qos_iops_limit`       | IOPS    | Combined read+write IOPS cap           |
    | `rbd_qos_read_iops_limit`  | IOPS    | Read IOPS cap                          |
    | `rbd_qos_write_iops_limit` | IOPS    | Write IOPS cap                         |
    | `rbd_qos_bps_limit`        | bytes/s | Combined throughput cap                |
    | `rbd_qos_read_bps_limit`   | bytes/s | Read throughput cap                    |
    | `rbd_qos_write_bps_limit`  | bytes/s | Write throughput cap                   |
    | `rbd_qos_iops_burst`       | IOPS    | Burst IOPS above the sustained limit   |
    | `rbd_qos_bps_burst`        | bytes/s | Burst throughput above sustained limit |

    <Tip>
      Ceph QoS is enforced at the librbd layer — inside the QEMU process on the hypervisor. It does not require OSD-level configuration changes and applies per-volume independently.
    </Tip>
  </Accordion>

  <Accordion title="NetApp ONTAP QoS Keys" icon="server">
    | Key                                | Unit        | Description                               |
    | ---------------------------------- | ----------- | ----------------------------------------- |
    | `netapp:qos_policy_group`          | Policy name | Reference to an ONTAP QoS policy group    |
    | `netapp:qos_adaptive_policy_group` | Policy name | Reference to an ONTAP adaptive QoS policy |
    | `total_iops_sec`                   | IOPS        | Enforced via ONTAP QoS policy             |
    | `total_bytes_sec`                  | bytes/s     | Enforced via ONTAP QoS policy             |

    NetApp adaptive QoS adjusts limits automatically based on volume size (TB) — `expected` and `peak` IOPS/TB values are set in the ONTAP adaptive policy group.
  </Accordion>

  <Accordion title="Pure Storage QoS Keys" icon="zap">
    | Key                      | Unit    | Description                              |
    | ------------------------ | ------- | ---------------------------------------- |
    | `pure_qos_max_bandwidth` | bytes/s | Maximum combined read + write throughput |
    | `pure_qos_max_iops`      | IOPS    | Maximum combined IOPS                    |

    Pure Storage enforces QoS at the array level — limits apply to the volume regardless of which host or hypervisor is accessing it.
  </Accordion>
</AccordionGroup>

***

## Create QoS Specifications

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to QoS Specs" icon="compass">
        Navigate to **Storage > Volume Types** (QoS Specs tab, admin view). Click **Create QoS Spec**.
      </Step>

      <Step title="Define the spec" icon="settings">
        | Field        | Value                   | Description                                                                                |
        | ------------ | ----------------------- | ------------------------------------------------------------------------------------------ |
        | **Name**     | e.g., `bronze-1000iops` | Descriptive name                                                                           |
        | **Consumer** | `front-end`             | Where limits are enforced: `front-end` (hypervisor), `back-end` (storage array), or `both` |

        Click **Confirm** to save the spec.
      </Step>

      <Step title="Add key-value pairs" icon="plus">
        Open the newly created spec and click **Edit Keys**. Add the desired performance parameters:

        | Key                  | Example Value          |
        | -------------------- | ---------------------- |
        | `total_iops_sec`     | `1000`                 |
        | `total_bytes_sec`    | `104857600` (100 MB/s) |
        | `total_iops_sec_max` | `2000`                 |

        <Check>Click **Confirm** to apply the key-value pairs to the spec.</Check>
      </Step>

      <Step title="Associate with a volume type" icon="link">
        Navigate to **Storage > Volume Types** (admin view), select a volume type, click **Manage QoS Spec**, and select the QoS spec from the dropdown.

        <Check>All new volumes created from this volume type inherit the QoS limits automatically.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the QoS spec" icon="plus">
        ```bash title="Create bronze QoS spec (1,000 IOPS, 100 MB/s)" theme={null}
        openstack volume qos create bronze-1000iops \
          --consumer front-end \
          --property total_iops_sec=1000 \
          --property total_bytes_sec=104857600 \
          --property total_iops_sec_max=2000
        ```

        ```bash title="Create gold QoS spec (10,000 IOPS, 1 GB/s)" theme={null}
        openstack volume qos create gold-10000iops \
          --consumer front-end \
          --property total_iops_sec=10000 \
          --property total_bytes_sec=1073741824 \
          --property total_iops_sec_max=15000 \
          --property total_bytes_sec_max=2147483648
        ```

        The `--consumer` flag controls enforcement location:

        * `front-end` — enforced by the hypervisor (libvirt QoS)
        * `back-end` — enforced by the storage array
        * `both` — enforced at both layers
      </Step>

      <Step title="Associate with a volume type" icon="link">
        ```bash title="Associate QoS spec with a volume type" theme={null}
        openstack volume qos associate bronze-1000iops standard-ssd
        ```

        ```bash title="Associate gold spec with premium volume type" theme={null}
        openstack volume qos associate gold-10000iops performance-ssd
        ```
      </Step>

      <Step title="Verify the association" icon="circle-check">
        ```bash title="Show volume type with QoS association" theme={null}
        openstack volume type show standard-ssd -c qos_specs_id
        ```

        ```bash title="List all QoS specs" theme={null}
        openstack volume qos list
        ```

        <Check>The volume type displays the associated QoS spec ID. New volumes created from this type are QoS-limited automatically.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Backend-Specific QoS Examples

<Tabs>
  <Tab title="Ceph RBD" icon="database">
    For Ceph RBD backends, use the `rbd_qos_*` keys. These are enforced by the QEMU librbd driver on the hypervisor.

    ```bash title="Create Ceph-specific QoS spec" theme={null}
    openstack volume qos create ceph-standard-qos \
      --consumer front-end \
      --property rbd_qos_iops_limit=1000 \
      --property rbd_qos_bps_limit=104857600 \
      --property rbd_qos_iops_burst=2000 \
      --property rbd_qos_bps_burst=209715200
    ```

    ```bash title="Associate with Ceph volume type" theme={null}
    openstack volume qos associate ceph-standard-qos ceph-ssd
    ```

    <Tip>
      Set `rbd_qos_iops_burst` to 2× the sustained `rbd_qos_iops_limit` to absorb short IO spikes (e.g., application startup, database checkpoint) without violating the sustained limit.
    </Tip>
  </Tab>

  <Tab title="Standard (libvirt)" icon="server">
    For standard backends where QoS is enforced at the hypervisor by libvirt:

    ```bash title="Tiered QoS specs — bronze, silver, gold" theme={null}
    # Bronze: 500 IOPS, 50 MB/s
    openstack volume qos create bronze \
      --consumer front-end \
      --property total_iops_sec=500 \
      --property total_bytes_sec=52428800

    # Silver: 2,000 IOPS, 200 MB/s
    openstack volume qos create silver \
      --consumer front-end \
      --property total_iops_sec=2000 \
      --property total_bytes_sec=209715200

    # Gold: 10,000 IOPS, 1 GB/s with burst
    openstack volume qos create gold \
      --consumer front-end \
      --property total_iops_sec=10000 \
      --property total_bytes_sec=1073741824 \
      --property total_iops_sec_max=15000 \
      --property total_bytes_sec_max=2147483648
    ```
  </Tab>

  <Tab title="Size-Scaled QoS" icon="trending-up">
    Use `size_iops_sec` to scale IOPS proportionally with volume size — larger volumes receive more IOPS automatically.

    ```bash title="Create size-scaled QoS spec (10 IOPS per GB)" theme={null}
    openstack volume qos create scaled-10iops-per-gb \
      --consumer front-end \
      --property size_iops_sec=10
    ```

    Effective IOPS by volume size with this spec:

    * 100 GB volume → 1,000 IOPS
    * 500 GB volume → 5,000 IOPS
    * 2,000 GB volume → 20,000 IOPS

    <Tip>
      Size-scaled QoS is ideal for database volume types where larger databases justifiably require more IO capacity, and smaller test databases should receive proportionally less.
    </Tip>
  </Tab>
</Tabs>

***

## QoS Capability Comparison

| Capability            | Ceph RBD            | NetApp ONTAP   | Pure Storage | Standard (libvirt)   |
| --------------------- | ------------------- | -------------- | ------------ | -------------------- |
| IOPS limit            | Yes                 | Yes            | Yes          | Yes                  |
| Throughput limit      | Yes                 | Yes            | Yes          | Yes                  |
| Burst IOPS            | Yes                 | Yes (adaptive) | No           | Yes                  |
| Burst throughput      | Yes                 | Yes (adaptive) | No           | Yes                  |
| Per-read/write limits | Yes                 | No             | No           | Yes                  |
| Size-scaled IOPS      | No                  | Yes (adaptive) | No           | Yes                  |
| Enforcement layer     | Hypervisor (librbd) | Array          | Array        | Hypervisor (libvirt) |
| Live update           | Yes (libvirt)       | Yes            | Yes          | Yes (libvirt)        |

***

## Manage QoS Specs

```bash title="List all QoS specifications" theme={null}
openstack volume qos list
```

```bash title="Show a QoS spec and its properties" theme={null}
openstack volume qos show bronze-1000iops
```

```bash title="Update a QoS spec property" theme={null}
openstack volume qos set bronze-1000iops \
  --property total_iops_sec=1500
```

```bash title="Remove a property from a QoS spec" theme={null}
openstack volume qos unset bronze-1000iops --property total_iops_sec_max
```

```bash title="Disassociate QoS spec from a volume type" theme={null}
openstack volume qos disassociate bronze-1000iops standard-ssd
```

```bash title="Delete a QoS spec" theme={null}
openstack volume qos delete bronze-1000iops
```

<Warning>
  Deleting a QoS spec that is still associated with a volume type will fail. Disassociate
  from all volume types before deleting the spec.
</Warning>

***

## Monitoring QoS Compliance

<Tabs>
  <Tab title="Xloud Dashboard (XIMP)" icon="activity">
    Navigate to **Monitoring → Storage → Volume Metrics** in Xloud XIMP. Per-volume IOPS and throughput graphs show whether volumes are hitting their QoS limits:

    * IOPS at or near the `total_iops_sec` limit — the volume is QoS-constrained
    * Sustained IOPS below limit — the workload does not require the full allocation
    * Burst IOPS exceeding sustained limit but below burst cap — normal burst behavior

    <Tip>
      Set XIMP alerts for volumes that sustain QoS-constrained behavior for more than 5 minutes — this signals that the volume type's QoS limit may be too low for the workload.
    </Tip>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Show volume details including QoS association" theme={null}
    openstack volume show <volume-id> -c volume_type -c metadata
    ```

    ```bash title="Show current QoS spec for a volume type" theme={null}
    openstack volume type show <volume-type> -c qos_specs_id
    ```

    For Ceph-backed volumes, verify QoS enforcement on the hypervisor:

    ```bash title="Check RBD QoS settings on running volume (on compute node)" theme={null}
    virsh domblkstat <domain-name> <disk-device>
    ```
  </Tab>
</Tabs>

***

## Best Practices

<CardGroup>
  <Card title="Align with Volume Types" icon="layers">
    Define one QoS spec per volume tier (bronze, silver, gold). Attach specs to volume
    types so QoS is automatically applied without per-volume manual steps.
  </Card>

  <Card title="Set Burst at 2x Sustained" icon="gauge">
    Configure burst limits at 2× the sustained limit to absorb application startup
    and checkpoint IO spikes without permanently raising the sustained cap.
  </Card>

  <Card title="Use Size-Scaled for Databases" icon="database">
    Apply `size_iops_sec` to database volume types. Larger database volumes
    automatically receive proportionally more IOPS without manual reconfiguration.
  </Card>

  <Card title="Monitor Before Limiting" icon="activity">
    Baseline actual workload IOPS using XIMP before applying QoS limits. Setting limits
    below actual peak workload demand causes application performance degradation.
  </Card>
</CardGroup>

***

## Tiered QoS Profiles

<Info>**Xloud-Developed** -- This capability is developed by Xloud and ships with XAVS / XPCI.</Info>

Xloud provides pre-defined tiered QoS profiles that align with common workload classes. Each profile sets per-instance IOPS limits, bandwidth caps, and burst allowances with separate read and write thresholds.

| Profile      | IOPS (sustained) | Throughput | Burst IOPS | Target Workload                             |
| ------------ | ---------------- | ---------- | ---------- | ------------------------------------------- |
| **Bronze**   | 500              | 50 MB/s    | 1,000      | Development, test environments              |
| **Silver**   | 2,000            | 200 MB/s   | 4,000      | General-purpose applications                |
| **Gold**     | 10,000           | 1 GB/s     | 15,000     | Transactional databases, analytics          |
| **Platinum** | 50,000           | 3 GB/s     | 75,000     | High-frequency trading, real-time ingestion |

Key capabilities of tiered QoS profiles:

* **Per-instance enforcement** -- limits apply at the individual instance level, not shared across a tenant
* **Read/write-differentiated limits** -- separate IOPS and bandwidth caps for read and write operations allow tuning for read-heavy or write-heavy workloads independently
* **Burst support** -- short IO spikes (application startup, checkpoints) are absorbed up to the burst ceiling without throttling

<Tip>
  Assign profiles to volume types so every volume inherits the correct QoS automatically.
  See [Storage Tiers](/services/storage/storage-tiers) for aligning QoS profiles with hardware tiers.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Volume Types Admin" href="/services/storage/volume-types-admin">
    Create and manage volume types to associate with QoS specs
  </Card>

  <Card title="Storage Tiers" href="/services/storage/storage-tiers">
    Configure tiered storage pools aligned with QoS tiers
  </Card>

  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Review backend-specific capabilities and QoS driver support
  </Card>

  <Card title="Block Storage Architecture" href="/services/storage/architecture">
    Understand how QoS is enforced across the storage service stack
  </Card>
</CardGroup>


# Quota Management
Source: https://docs.xloud.tech/services/storage/quotas

Configure and manage Xloud Block Storage quotas per project. Set global defaults, override per-project limits, and monitor quota usage across your deployment.

## Overview

Storage quotas limit the total volumes, snapshots, backups, and capacity that each project can consume. Quotas prevent runaway resource consumption in multi-tenant deployments and ensure fair capacity distribution. Default limits apply to all new projects; per-project overrides allow specific projects to receive higher or lower limits.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Project IDs for per-project quota adjustments
</Note>

***

## Default Quota Values

| Quota                  | Default | Description                                       |
| ---------------------- | ------- | ------------------------------------------------- |
| `volumes`              | 10      | Maximum number of volumes per project             |
| `gigabytes`            | 1,000   | Maximum total volume capacity (GiB) per project   |
| `snapshots`            | 10      | Maximum number of snapshots per project           |
| `backups`              | 10      | Maximum number of backups per project             |
| `backup_gigabytes`     | 1,000   | Maximum backup storage capacity (GiB) per project |
| `per_volume_gigabytes` | -1      | Maximum size of a single volume (-1 = unlimited)  |

***

## View Quota Usage

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="View global defaults" icon="compass">
        Navigate to
        **Storage > Volume Types** (admin view). The **Volume Quotas** section shows the global
        default limits applied to all new projects.
      </Step>

      <Step title="View project-specific usage" icon="gauge">
        Navigate to **Identity > Projects** (admin view). Select a project, then click
        **Edit Quota** to see the current limits and usage for that project.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Show quota for a specific project" icon="gauge">
        ```bash title="Show quota usage for a project" theme={null}
        openstack quota show <project-id-or-name>
        ```

        The output shows current limits and usage for all resource types including
        `volumes`, `gigabytes`, `snapshots`, `backups`, and `backup_gigabytes`.
      </Step>

      <Step title="Show quota for the current project" icon="gauge">
        ```bash title="Show current project quota" theme={null}
        openstack quota show
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Update Project Quotas

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Projects" icon="compass">
        Navigate to **Identity > Projects** (admin view) and select the target project.
      </Step>

      <Step title="Modify quotas" icon="edit">
        Click **Edit Quota**. Adjust the volume, gigabytes, snapshot, and backup
        limits as required. Click **Confirm**.

        <Warning>
          Reducing a quota below the project's current usage does not reclaim resources.
          Existing volumes and snapshots are unaffected, but no new resources can be
          created until usage drops below the new limit.
        </Warning>
      </Step>

      <Step title="Verify" icon="circle-check">
        Return to **Edit Quota** for the project and confirm the new limits are applied.

        <Check>Quota updated — new limits take effect immediately for future operations.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Update storage quotas for a project" icon="edit">
        ```bash title="Set volume quotas for a project" theme={null}
        openstack quota set \
          --volumes 50 \
          --gigabytes 5000 \
          --snapshots 50 \
          --backups 20 \
          --backup-gigabytes 2000 \
          <project-id-or-name>
        ```
      </Step>

      <Step title="Set per-volume size limit" icon="ruler">
        ```bash title="Enforce minimum volume size" theme={null}
        openstack quota set \
          --per-volume-gigabytes 10 \
          <project-id-or-name>
        ```

        Setting `per-volume-gigabytes=10` requires every volume to be at least 10 GiB,
        reducing fragmentation on the backend.
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="Confirm new quota values" theme={null}
        openstack quota show <project-id-or-name>
        ```

        <Check>New quota limits visible and active for the project.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Update Global Default Quotas

Global defaults apply to all new projects. Updating them does not affect projects that
already have explicit quota overrides.

```bash title="Set global default quotas" theme={null}
openstack quota set \
  --volumes 20 \
  --gigabytes 2000 \
  --snapshots 20 \
  --class default
```

***

## Remove Per-Project Quota Override

To revert a project to the global defaults, delete its quota override:

```bash title="Delete project quota override" theme={null}
openstack quota delete <project-id-or-name>
```

The project reverts to the global default values immediately.

***

## Quota Planning Guidelines

<AccordionGroup>
  <Accordion title="Calculating appropriate gigabyte quotas" icon="calculator">
    To size quotas appropriately, estimate:

    1. Average volume size per instance (typically 50–200 GiB for boot + data)
    2. Number of instances expected in the project
    3. Snapshot overhead (typically 10–30% of volume data for active snapshots)
    4. Growth buffer (20–50% headroom)

    Example: A project running 10 instances with 100 GiB average volumes, 20% snapshot
    overhead, and 30% growth buffer:
    `(10 × 100) + (1000 × 0.20) + (1200 × 0.30) = 1000 + 200 + 360 = 1,560 GiB`
  </Accordion>

  <Accordion title="Monitoring quota consumption across projects" icon="gauge">
    Regularly audit quota usage to identify projects approaching their limits:

    ```bash title="List all project quotas" theme={null}
    openstack quota list --detail
    ```

    Review projects approaching 80% of their `gigabytes` or `volumes` quota — either clean up unused resources or increase the quota proactively.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Volume Types & QoS" href="/services/storage/volume-types-admin">
    Control per-volume IOPS and throughput with QoS policies
  </Card>

  <Card title="Security Hardening" href="/services/storage/security">
    Restrict volume type access and enforce security policies
  </Card>

  <Card title="Admin Guide" href="/services/storage/admin-guide">
    Return to the Block Storage administration overview
  </Card>

  <Card title="Troubleshooting (Admin)" href="/services/storage/admin-troubleshooting">
    Diagnose and resolve service-level Block Storage issues
  </Card>
</CardGroup>


# Block Storage Security Hardening
Source: https://docs.xloud.tech/services/storage/security

Harden Xloud Block Storage with volume type access controls, minimum volume size policies, snapshot visibility management, and audit logging configuration.

## Overview

Xloud Block Storage provides several hardening controls that restrict unauthorized access to sensitive storage resources, limit resource abuse, and maintain an auditable record of storage operations. Apply these controls after completing the initial service configuration to meet compliance and security baseline requirements.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Volume types and backends already configured
  * CLI authenticated via `source openrc.sh`
</Note>

***

## Security Controls Overview

| Control                            | Purpose                                                                  | Impact                                             |
| ---------------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------- |
| **Volume type access restriction** | Limit sensitive types (encrypted, high-performance) to specific projects | Prevents unauthorized access to premium tiers      |
| **Minimum volume size**            | Require a minimum GiB per volume                                         | Reduces storage fragmentation                      |
| **Snapshot visibility**            | Prevent public snapshots                                                 | Stops data leakage across project boundaries       |
| **Volume encryption**              | At-rest protection via LUKS                                              | Protects data on stolen or decommissioned hardware |
| **Audit logging**                  | API request logging                                                      | Provides evidence for compliance audits            |

***

## Restrict Volume Type Access

By default, volume types are accessible to all projects. Restrict sensitive or
high-performance types to specific authorized projects:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Volume Types" icon="compass">
        Navigate to
        **Storage > Volume Types** (admin view).
      </Step>

      <Step title="Make the type private">
        Create the volume type with the **Public** checkbox unchecked, or use the
        CLI to change an existing type to private.
      </Step>

      <Step title="Grant access to authorized projects">
        On the volume type row, click the **More** dropdown and select **Manage Access**.
        Select the authorized projects and click **Confirm**.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Make the type private" icon="lock">
        ```bash title="Restrict volume type to specific projects" theme={null}
        openstack volume type set --private <type-name>
        ```
      </Step>

      <Step title="Grant access to specific projects" icon="users">
        ```bash title="Add project access" theme={null}
        openstack volume type access add \
          --project <project-id> \
          <type-name>
        ```
      </Step>

      <Step title="Verify access list" icon="circle-check">
        ```bash title="List projects with access" theme={null}
        openstack volume type access list <type-name>
        ```

        <Check>Volume type restricted — only authorized projects can create volumes of this type.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Enforce Minimum Volume Size

Prevent creation of many small volumes that fragment storage capacity and inflate volume
counts against quota:

```bash title="Set minimum volume size for a project" theme={null}
openstack quota set \
  --per-volume-gigabytes 10 \
  <project-id>
```

A value of `10` requires every volume to be at least 10 GiB. Adjust based on your
workload profile. Set globally with `--class default` to apply to all new projects:

```bash title="Set global minimum volume size" theme={null}
openstack quota set \
  --per-volume-gigabytes 5 \
  --class default
```

***

## Manage Snapshot Visibility

Snapshots can be set to `public`, making them visible and accessible to all projects.
Audit and restrict public snapshots to prevent unintended data exposure:

```bash title="List public snapshots across all projects" theme={null}
openstack volume snapshot list --all-projects \
  --property visibility=public
```

```bash title="Make a public snapshot private" theme={null}
openstack volume snapshot set --private <snapshot-id>
```

<Warning>
  Public snapshots expose volume data to all projects in your deployment. Audit public
  snapshots regularly and make them private unless cross-project sharing is explicitly
  required.
</Warning>

***

## Enable Volume Encryption

Configure at-rest encryption on sensitive volume types to protect data on storage media.
Encryption is transparent to users — all I/O is encrypted/decrypted at the hypervisor
layer without application changes.

Refer to the [Volume Encryption](/services/storage/encryption) guide for full
configuration instructions.

***

## Audit Logging

The Block Storage service logs all API requests including volume creation, deletion,
attachment, and snapshot operations. Review service logs via XDeploy for security audits.

<AccordionGroup>
  <Accordion title="Key events to audit" icon="eye">
    | Event                | Log Pattern                                    | Security Significance   |
    | -------------------- | ---------------------------------------------- | ----------------------- |
    | Volume deletion      | `DELETE /volumes/<id>`                         | Data destruction        |
    | Volume type creation | `POST /types`                                  | New storage tier added  |
    | Snapshot made public | `PUT /snapshots/<id>` with `visibility=public` | Potential data exposure |
    | Quota increase       | `PUT /os-quota-sets/<project>`                 | Resource limit change   |
    | Volume migration     | `POST /volumes/<id>/action`                    | Data movement           |
  </Accordion>

  <Accordion title="Enable debug logging (temporary)" icon="terminal">
    Enable verbose API logging on the Block Storage service for detailed audit trails:

    Access the volume service configuration via XDeploy and add `debug = True` to the
    `[DEFAULT]` section. Apply configuration changes and restart the service.

    <Warning>
      Debug logging is verbose and generates significant log volume. Enable only during
      active security audits and disable promptly when finished.
    </Warning>
  </Accordion>
</AccordionGroup>

***

## Security Checklist

<AccordionGroup>
  <Accordion title="Deployment security baseline" icon="shield">
    Verify the following after initial Block Storage deployment:

    * [ ] All encrypted volume types are set to private and scoped to authorized projects
    * [ ] Public snapshot count is zero (or justified exceptions documented)
    * [ ] Minimum volume size configured per project to prevent fragmentation
    * [ ] Volume encryption enabled for any compliance-sensitive data tiers
    * [ ] Key Management service deployed with HA for encrypted volume availability
    * [ ] Service logs reviewed for anomalous patterns after first week of operation
    * [ ] Storage backend credentials (keyring files) have `0600` permissions on disk
    * [ ] Backup target credentials stored in `passwords.yml` (not in configuration files)
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="Volume Encryption" href="/services/storage/encryption">
    Configure LUKS-based at-rest encryption for volume types
  </Card>

  <Card title="Quota Management" href="/services/storage/quotas">
    Enforce storage limits to prevent resource exhaustion
  </Card>

  <Card title="Volume Types & QoS" href="/services/storage/volume-types-admin">
    Restrict volume type access to authorized projects
  </Card>

  <Card title="Admin Guide" href="/services/storage/admin-guide">
    Return to the Block Storage administration overview
  </Card>
</CardGroup>


# Volume Snapshots
Source: https://docs.xloud.tech/services/storage/snapshots

Create, restore, and manage point-in-time snapshots of Xloud Block Storage volumes. Use snapshots for fast recovery, volume cloning, and pre-change checkpoints.

## Overview

A snapshot captures the state of a volume at a specific point in time. Snapshots are <Tooltip>crash-consistent</Tooltip> and stored within the same storage cluster as the source volume. Creating a snapshot takes seconds — only metadata is recorded at creation time, with changed blocks tracked incrementally. Snapshots are ideal for pre-change checkpoints, test environment cloning, and fast volume recovery.

<Note>
  **Prerequisites**

  * An active project with snapshot quota available (`openstack quota show`)
  * The source volume must exist (it can be attached or detached)
  * For application-consistent snapshots of databases, quiesce writes or use a
    maintenance window before creating the snapshot
</Note>

***

## Snapshot vs Backup

| Feature                  | Snapshot                           | Backup                                 |
| ------------------------ | ---------------------------------- | -------------------------------------- |
| Storage location         | Same cluster as source volume      | Separate backup target                 |
| Creation time            | Seconds (metadata + COW tracking)  | Minutes (full data copy)               |
| Incremental support      | Implicit (copy-on-write)           | Yes — after first full backup          |
| Survives cluster failure | No                                 | Yes                                    |
| Restore time             | Seconds (new volume from snapshot) | Minutes (data restore)                 |
| Use case                 | Fast recovery points, cloning      | Long-term retention, disaster recovery |

<Tip>
  Use snapshots for frequent, short-term checkpoints. Use [backups](/services/storage/backups)
  for long-term retention and disaster recovery scenarios.
</Tip>

***

## Create a Snapshot

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Volumes" icon="compass">
        Navigate to
        **Storage > Volumes**.
      </Step>

      <Step title="Create snapshot" icon="camera">
        Locate the target volume and click the **More** dropdown and select **Data Protection > Create Snapshot**.

        Provide a descriptive name, then click **Confirm**.

        <Tip>
          Name snapshots with a timestamp and purpose for easy identification —
          e.g., `db-primary-2026-03-18-pre-migration`.
        </Tip>
      </Step>

      <Step title="Verify" icon="circle-check">
        Navigate to **Storage > Volume Snapshots**. The new snapshot appears with
        status **Available**.

        <Check>Snapshot created — status shows **Available**.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the snapshot" icon="camera">
        ```bash title="Create volume snapshot" theme={null}
        openstack volume snapshot create \
          --volume <volume-name-or-id> \
          --description "Pre-deployment checkpoint" \
          my-snapshot-2026-03-18
        ```
      </Step>

      <Step title="List and verify" icon="list">
        ```bash title="List all snapshots" theme={null}
        openstack volume snapshot list
        ```

        ```bash title="Show snapshot details" theme={null}
        openstack volume snapshot show my-snapshot-2026-03-18
        ```

        <Check>Snapshot status shows `available` — ready for restore or volume creation.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Restore from a Snapshot

Restoring creates a new independent volume pre-populated with the snapshot data. The
original source volume and snapshot are unaffected.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Snapshots" icon="compass">
        Navigate to **Storage > Volume Snapshots**.
      </Step>

      <Step title="Create volume from snapshot" icon="rotate-ccw">
        Click the **More** dropdown and select **Create Volume** on the desired snapshot. Configure the new
        volume name, type, and availability zone, then click **Confirm**.

        <Note>
          The new volume size defaults to the snapshot size. You can specify a larger
          size, but not smaller.
        </Note>
      </Step>

      <Step title="Attach the restored volume" icon="link">
        The restored volume appears in the Volumes list with status **Available**.
        [Attach it to an instance](/services/storage/attach-volume) and verify the data.

        <Check>Volume restored from snapshot and ready for attachment.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create volume from snapshot" icon="rotate-ccw">
        ```bash title="Restore snapshot to a new volume" theme={null}
        openstack volume create \
          --snapshot my-snapshot-2026-03-18 \
          --size 100 \
          restored-volume
        ```
      </Step>

      <Step title="Verify the restored volume" icon="circle-check">
        ```bash title="Check restored volume status" theme={null}
        openstack volume show restored-volume -c status -c size
        ```

        <Check>Status `available` — volume restored and ready to attach.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Delete a Snapshot

<Warning>
  Snapshot deletion is permanent and cannot be undone. Ensure no volumes are currently
  being created from the snapshot before deleting it. Volumes that depend on a snapshot
  (created from it) must be deleted first if the backend requires it.
</Warning>

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Storage > Volume Snapshots**. Click the **More** dropdown and select **Delete**
    on the target snapshot and confirm the deletion.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Delete a snapshot" theme={null}
    openstack volume snapshot delete <snapshot-name-or-id>
    ```

    ```bash title="Verify deletion" theme={null}
    openstack volume snapshot list
    ```

    <Check>Snapshot deleted. The source volume is unaffected.</Check>
  </Tab>
</Tabs>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Snapshot stuck in 'creating' status" icon="clock">
    **Cause**: The storage backend did not complete the snapshot operation within the
    expected timeout, or the volume is in an inconsistent state.

    **Resolution**:

    ```bash title="Check snapshot status" theme={null}
    openstack volume snapshot show <snapshot-id> -c status
    ```

    Wait a few minutes for backends under load. If the snapshot remains in `creating`
    for more than 10 minutes, contact your administrator — the backend may require
    manual cleanup.
  </Accordion>

  <Accordion title="Snapshot in 'error' status" icon="triangle-alert">
    **Cause**: The backend encountered an error during snapshot creation. Common causes
    include insufficient storage capacity or backend driver issues.

    **Resolution**:

    ```bash title="Delete the failed snapshot" theme={null}
    openstack volume snapshot delete <snapshot-id>
    ```

    Check available capacity on the storage cluster and retry. Do not repeatedly retry
    on a degraded backend — resolve the underlying storage issue first.
  </Accordion>

  <Accordion title="Cannot delete snapshot — 'snapshot has dependent volumes'" icon="link">
    **Cause**: A volume was created from this snapshot and the backend tracks the
    dependency. Some backends require the child volume to be deleted before the snapshot.

    **Resolution**: Delete the volumes created from the snapshot first, then delete
    the snapshot.
  </Accordion>
</AccordionGroup>

***

## Application-Consistent and File-Level Restore

For application-consistent snapshots with XAVS Guest Agent coordination (filesystem freeze, VSS provider for Windows, custom hooks) and agentless file-level restore, see the Compute snapshots guide. The file-level restore feature lets you browse and download individual files from any snapshot.

<CardGroup>
  <Card title="Instance Snapshots and Backups" href="/services/compute/snapshots">
    Application-consistent snapshots, XAVS Guest Agent integration, and agentless file-level restore from the Dashboard
  </Card>
</CardGroup>

***

## Next Steps

<CardGroup>
  <Card title="Volume Backups" href="/services/storage/backups">
    Create full and incremental backups for long-term retention and disaster recovery
  </Card>

  <Card title="Extend a Volume" href="/services/storage/extend-volume">
    Increase volume capacity before or after restoring from a snapshot
  </Card>

  <Card title="Attach / Detach" href="/services/storage/attach-volume">
    Attach a restored volume to a compute instance
  </Card>

  <Card title="Create a Volume" href="/services/storage/create-volume">
    Create a new volume from an image, snapshot, or blank
  </Card>
</CardGroup>


# Storage Backend Configuration
Source: https://docs.xloud.tech/services/storage/storage-backends

Configure block storage backends including RBD, LVM, and NFS. Manage backend registration and status verification.

## Overview

A storage backend is the combination of a driver and a physical storage system that serves Xloud Block Storage volumes. Backends are configured at deploy time through XDeploy and registered with the scheduler. Each backend must have a unique `volume_backend_name` that maps to at least one volume type, making it accessible to users.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * Access to XDeploy for infrastructure-level configuration
  * The storage system (distributed storage cluster, LVM volume group, or NFS server) must
    be provisioned before configuring the backend
</Note>

***

## Supported Backends

| Backend                       | Driver                                                                             | Protocol             | Use Case                                               | HA                |
| ----------------------------- | ---------------------------------------------------------------------------------- | -------------------- | ------------------------------------------------------ | ----------------- |
| **Distributed Storage (RBD)** | `cinder.volume.drivers.rbd.RBDDriver`                                              | RBD                  | Production — full replication, zero-copy clones        | Yes               |
| **LVM**                       | `cinder.volume.drivers.lvm.LVMVolumeDriver`                                        | iSCSI                | Development / single-node testing                      | No                |
| **NFS**                       | `cinder.volume.drivers.nfs.NfsDriver`                                              | NFS                  | Shared NAS appliances, legacy NFS integration          | Depends on server |
| **NetApp ONTAP**              | `cinder.volume.drivers.netapp.common.NetAppDriver`                                 | iSCSI / NFS / FC     | Enterprise NAS/SAN — FlexVol, thin provisioning, dedup | Yes               |
| **Pure Storage FlashArray**   | `cinder.volume.drivers.pure.PureISCSIDriver`                                       | iSCSI / FC / NVMe-oF | All-flash arrays, high IOPS, thin by default           | Yes               |
| **Dell PowerStore**           | `cinder.volume.drivers.dell_emc.powerstore.driver.PowerStoreDriver`                | iSCSI / FC / NVMe-oF | Dell enterprise all-flash and hybrid arrays            | Yes               |
| **HPE 3PAR / Primera**        | `cinder.volume.drivers.hpe.hpe_3par_iscsi.HPE3PARISCSIDriver`                      | iSCSI / FC           | HPE enterprise SAN — CPG-based provisioning            | Yes               |
| **Hitachi VSP**               | `cinder.volume.drivers.hitachi.hbsd_iscsi.HBSDISCSIDriver`                         | iSCSI / FC           | Hitachi enterprise storage systems                     | Yes               |
| **IBM FlashSystem**           | `cinder.volume.drivers.ibm.storwize_svc.storwize_svc_iscsi.StorwizeSVCISCSIDriver` | iSCSI / FC / NVMe-oF | IBM all-flash and Storwize arrays                      | Yes               |
| **iSCSI (generic)**           | `cinder.volume.drivers.iscsi.ISCSIDriver`                                          | iSCSI                | Generic iSCSI targets not covered by vendor drivers    | Depends on array  |
| **Fibre Channel (generic)**   | `cinder.volume.drivers.fibre_channel.FibreChannelDriver`                           | FC                   | Generic Fibre Channel targets                          | Depends on array  |

***

## Configure a Distributed Storage (RBD) Backend

The distributed storage driver is the recommended backend for all production deployments. It uses RADOS Block Devices backed by a Xloud Distributed Storage cluster.

<Tabs>
  <Tab title="XDeploy" icon="package">
    <Steps>
      <Step title="Open Block Storage configuration" icon="settings">
        Log in to XDeploy and navigate to **Configuration → Block Storage**.
      </Step>

      <Step title="Select the distributed storage driver" icon="database">
        Select **Distributed Storage (RBD)** from the **Backend Driver** dropdown.

        <Tip>
          Ensure the storage pool is created and the authentication keyring is available
          on all volume service nodes before completing this step.
        </Tip>
      </Step>

      <Step title="Configure RBD parameters" icon="sliders">
        | Parameter     | Description                                      | Example                                      |
        | ------------- | ------------------------------------------------ | -------------------------------------------- |
        | Pool Name     | Name of the storage pool for volumes             | `volumes`                                    |
        | Auth Username | Storage authentication user                      | `xloud-volume`                               |
        | Keyring Path  | Path to the keyring file on volume service nodes | `/etc/ceph/ceph.client.xloud-volume.keyring` |
        | Cluster Name  | Storage cluster identifier                       | `ceph`                                       |
        | Backend Name  | Unique name for this backend                     | `rbd-ssd`                                    |
      </Step>

      <Step title="Apply and deploy" icon="play">
        Click **Save Configuration**, then click **Deploy Block Storage**. XDeploy
        restarts the volume service and validates backend connectivity.

        <Check>Backend configured — volume service restarted and backend registered with scheduler.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Verify volume service status" icon="activity">
        ```bash title="List volume service endpoints" theme={null}
        openstack volume service list
        ```

        All services should show state `up` and status `enabled`.
      </Step>

      <Step title="List configured backends" icon="database">
        ```bash title="List storage backends and capacity" theme={null}
        openstack volume backend pool list --long
        ```

        This shows all active backend pools registered with the scheduler, including
        free capacity, reserved capacity, and driver version.
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Configure Enterprise Storage Backends

For NetApp, Pure Storage, Dell, HPE, Hitachi, and IBM backends, configuration is applied through XDeploy globals and Ansible. The `cinder.conf` sections are generated from templates. See [External Storage Integration](/services/storage/external-storage) for full configuration examples, driver parameters, and volume type mapping for each enterprise backend.

<Note>
  Enterprise backend drivers require vendor-specific Python packages installed on the volume service nodes. XDeploy manages driver installation for supported backends during deployment.
</Note>

***

## Configure an LVM Backend

LVM backends are suitable for development and single-node testing environments only.

<Tabs>
  <Tab title="XDeploy" icon="package">
    <Steps>
      <Step title="Select LVM driver" icon="database">
        In XDeploy, navigate to **Configuration → Block Storage** and select
        **LVM** from the Backend Driver dropdown.
      </Step>

      <Step title="Configure LVM parameters" icon="sliders">
        | Parameter     | Description                  | Example         |
        | ------------- | ---------------------------- | --------------- |
        | Volume Group  | LVM volume group name        | `xloud-volumes` |
        | Target Helper | iSCSI helper binary          | `lioadm`        |
        | Backend Name  | Unique name for this backend | `lvm-local`     |
      </Step>

      <Step title="Apply" icon="play">
        Click **Save Configuration** and **Deploy Block Storage**.

        <Warning>
          LVM provides no replication or redundancy. Do not use LVM backends for
          production workloads where data durability is required.
        </Warning>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Backend Health Verification

After configuration, verify that the backend is registered and reporting capacity:

```bash title="List backends with capacity details" theme={null}
openstack volume backend pool list --long
```

Key columns to verify:

| Column              | Expected Value                                          |
| ------------------- | ------------------------------------------------------- |
| `free_capacity_gb`  | Greater than 0 — confirms backend is reporting capacity |
| `total_capacity_gb` | Matches expected pool size                              |
| `driver_version`    | Populated — confirms driver initialized correctly       |
| `storage_protocol`  | `rbd`, `iSCSI`, or `NFS` depending on backend type      |

***

## Enable or Disable a Backend Service

<Warning>
  Disabling a volume service prevents the scheduler from sending new volume operations
  to that backend. Existing volumes on the backend are unaffected. Re-enable the service
  to restore normal scheduling.
</Warning>

```bash title="Disable a volume service (maintenance)" theme={null}
openstack volume service set \
  --disable \
  --disable-reason "Scheduled maintenance" \
  <volume-service-host> \
  cinder-volume
```

```bash title="Re-enable a volume service" theme={null}
openstack volume service set \
  --enable \
  <volume-service-host> \
  cinder-volume
```

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Backend pool does not appear in list" icon="database">
    **Cause**: The volume service has not registered the backend with the scheduler, or
    the service is not running.

    **Resolution**:

    ```bash title="Check service status" theme={null}
    openstack volume service list
    ```

    Look for the volume service on the affected storage node. If state is `down`,
    check service logs via XDeploy for driver initialization errors.
  </Accordion>

  <Accordion title="Backend reporting zero free capacity" icon="triangle-alert">
    **Cause**: The storage pool is full, or the backend driver cannot reach the storage
    cluster to report capacity.

    **Resolution**:

    * Verify distributed storage cluster health: check cluster status from the storage
      administration interface
    * Verify the authentication keyring is present on the volume service node
    * Verify the pool name matches the configured backend pool
    * Increase pool capacity or add OSDs to the storage cluster
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup>
  <Card title="External Storage Integration" href="/services/storage/external-storage">
    Full configuration guide for NetApp, Pure Storage, Dell, HPE, and IBM backends
  </Card>

  <Card title="Volume Types & QoS" href="/services/storage/volume-types-admin">
    Map volume types to backends and configure QoS policies
  </Card>

  <Card title="Thin Provisioning" href="/services/storage/thin-provisioning">
    Configure thin and thick provisioning with overcommit ratios
  </Card>

  <Card title="Storage Tiers" href="/services/storage/storage-tiers">
    Configure multi-tier storage with NVMe, SSD, and HDD backends
  </Card>

  <Card title="Backup Backends" href="/services/storage/backup-backends">
    Configure Ceph, Swift, S3, or NFS as the volume backup destination
  </Card>

  <Card title="Migration" href="/services/storage/migration">
    Migrate volumes between backends for rebalancing or hardware retirement
  </Card>
</CardGroup>


# Storage Tiers
Source: https://docs.xloud.tech/services/storage/storage-tiers

Configure NVMe, SSD, and HDD storage tiers in Xloud Block Storage. Map hardware device classes to volume types and set the default tier for your deployment.

## Overview

Storage tiering exposes different hardware performance classes as distinct volume types, allowing users to select the appropriate tier for each workload. Xloud Block Storage supports NVMe, SSD, and HDD tiers within a single distributed storage cluster by mapping CRUSH device classes to dedicated pools and volume types. Configure one volume type per hardware tier and set a default to ensure users without a specific preference land on the most appropriate tier.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Distributed storage cluster with at least one device class configured (e.g., `nvme`, `ssd`, `hdd`)
  * Pools created for each device class in the storage cluster
  * Storage backends registered in Block Storage (one per pool)
  * Volume types created for each tier via the [Volume Types & QoS](/services/storage/volume-types-admin) guide
</Note>

***

## Tier Performance Reference

| Tier               | Hardware       | IOPS (4K Random)    | Throughput        | Recommended For                             |
| ------------------ | -------------- | ------------------- | ----------------- | ------------------------------------------- |
| **NVMe**           | PCIe NVMe SSDs | 100,000 – 1,000,000 | 3 – 14 GB/s       | Transactional databases, low-latency caches |
| **SSD**            | SATA/SAS SSDs  | 10,000 – 100,000    | 500 MB/s – 2 GB/s | Web application data, instance boot disks   |
| **Standard (HDD)** | 7.2K/10K RPM   | 150 – 500           | 100 – 300 MB/s    | Archival, bulk storage, infrequent access   |

***

## Configure Tier Mappings

<Tabs>
  <Tab title="XDeploy" icon="package">
    <Steps>
      <Step title="Open Storage Tiers in XDeploy" icon="layers">
        Log in to XDeploy and navigate to **Configuration → Storage → Storage Tiers**.
        XDeploy displays the detected hardware device classes from the storage cluster.
      </Step>

      <Step title="Configure tier mappings" icon="settings">
        For each detected hardware tier, specify:

        | Field             | Description                                                        |
        | ----------------- | ------------------------------------------------------------------ |
        | Storage Pool Name | The pool in the distributed storage cluster (e.g., `volumes-nvme`) |
        | Volume Type Name  | The volume type to expose to users (e.g., `nvme`)                  |
        | QoS Spec          | Optional QoS policy to attach to this tier                         |

        Click **Save Tier Configuration**.
      </Step>

      <Step title="Set the default volume type" icon="star">
        In the **Default Volume Type** field, select the tier to use when users create
        volumes without specifying a type.

        Xloud recommends setting `ssd` as the default for most deployments — you can
        explicitly request `nvme` or `standard` when needed.

        Click **Apply**.

        <Check>Tier mappings saved and default type configured.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="List backend pools and device classes" icon="list">
        ```bash title="List backend pools with capabilities" theme={null}
        openstack volume backend pool list --long
        ```

        Review `volume_backend_name`, `free_capacity_gb`, and `total_capacity_gb`
        for each pool to understand available capacity per tier.
      </Step>

      <Step title="Create volume types for each tier" icon="plus">
        ```bash title="Create NVMe volume type" theme={null}
        openstack volume type create \
          --description "NVMe SSD — high-performance, low-latency" \
          nvme
        openstack volume type set --property volume_backend_name=nvme-backend nvme
        ```

        ```bash title="Create SSD volume type" theme={null}
        openstack volume type create \
          --description "SSD — balanced performance and cost" \
          ssd
        openstack volume type set --property volume_backend_name=ssd-backend ssd
        ```

        ```bash title="Create HDD volume type" theme={null}
        openstack volume type create \
          --description "Standard HDD — archival and bulk storage" \
          standard
        openstack volume type set --property volume_backend_name=hdd-backend standard
        ```
      </Step>

      <Step title="Set the default volume type" icon="star">
        ```bash title="Set default volume type" theme={null}
        openstack volume type set --default ssd
        ```
      </Step>

      <Step title="Verify tier configuration" icon="circle-check">
        ```bash title="List all volume types with properties" theme={null}
        openstack volume type list --long
        ```

        Confirm each type shows the correct `volume_backend_name` and that one type
        is marked as default.

        <Check>All storage tiers configured and visible to users.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Single-Tier Deployments

If your cluster has only one hardware class (e.g., all SSD), configure a single volume
type pointing to the existing pool. There is no need to create additional pools or types.

<Tip>
  In a single-tier cluster, Xloud recommends still naming the type explicitly (e.g.,
  `ssd`) rather than using a generic name — this makes future tier additions easier
  without requiring users to update their volume creation workflows.
</Tip>

***

## Adding a New Tier to an Existing Deployment

When adding new hardware (e.g., NVMe drives to an existing SSD cluster):

<Steps>
  <Step title="Add hardware and create a new pool" icon="database">
    Add the new OSDs to the distributed storage cluster with the appropriate device
    class. Create a dedicated pool using a CRUSH rule that targets only the new
    device class.
  </Step>

  <Step title="Configure a new backend in XDeploy" icon="settings">
    Add a new backend entry in XDeploy pointing to the new pool. Deploy the configuration
    to register the new backend with the volume service.
  </Step>

  <Step title="Create a new volume type" icon="plus">
    ```bash title="Create the new tier volume type" theme={null}
    openstack volume type create \
      --description "NVMe PCIe — ultra-low latency" \
      nvme
    openstack volume type set --property volume_backend_name=nvme-pool nvme
    ```
  </Step>

  <Step title="Verify availability" icon="circle-check">
    ```bash title="Test the new tier" theme={null}
    openstack volume create --size 1 --type nvme test-nvme
    openstack volume show test-nvme -c status
    openstack volume delete test-nvme
    ```

    <Check>New tier operational and accessible to users.</Check>
  </Step>
</Steps>

***

## Automated Storage Tiering

<Info>**Xloud-Developed** -- This capability is developed by Xloud and ships with XAVS / XPCI.</Info>

The Xloud Resource Optimizer can automatically move volumes between NVMe, SSD, and HDD pools based on observed IOPS activity -- eliminating manual tier management for dynamic workloads.

| Parameter                 | Description                                                                                                             |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Promotion threshold**   | Sustained high IOPS triggers migration to a faster tier (e.g., HDD to SSD, SSD to NVMe)                                 |
| **Demotion threshold**    | Sustained low IOPS combined with volume age triggers migration to a cheaper tier                                        |
| **Max retypes per cycle** | Limits the number of volume migrations per optimization cycle to prevent thundering-herd effects on the storage cluster |

All thresholds are configurable per deployment. The Resource Optimizer evaluates volumes periodically, generates a migration plan, and executes volume retypes through the Block Storage API without downtime to attached instances.

<Tip>
  Start with conservative thresholds and monitor the Resource Optimizer audit logs before tightening promotion or demotion criteria. Aggressive thresholds can cause volumes to bounce between tiers repeatedly.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Volume Types & QoS" href="/services/storage/volume-types-admin">
    Configure QoS policies to enforce per-tier IOPS and throughput limits
  </Card>

  <Card title="Migration" href="/services/storage/migration">
    Move volumes between tiers to rebalance capacity or meet new requirements
  </Card>

  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Configure backend drivers for each storage tier
  </Card>

  <Card title="Volume Types (User)" href="/services/storage/volume-types">
    User guide for selecting the right storage tier for a workload
  </Card>
</CardGroup>


# Thin and Thick Provisioning
Source: https://docs.xloud.tech/services/storage/thin-provisioning

Configure thin and thick provisioning for Xloud Block Storage backends. Understand overcommit ratios, backend-specific settings, and capacity monitoring.

## Overview

Xloud Block Storage supports both thin and thick provisioning models. Thin provisioning allocates storage capacity on demand — physical space is consumed only as data is written, not when the volume is created. Thick provisioning reserves the full volume capacity immediately at creation time. The provisioning behavior is controlled by the backend driver and the scheduler overcommit configuration.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

***

## Thin vs Thick Comparison

| Attribute                       | Thin Provisioning                                   | Thick Provisioning                                 |
| ------------------------------- | --------------------------------------------------- | -------------------------------------------------- |
| **Space allocated at creation** | No — space consumed as data is written              | Yes — full size reserved immediately               |
| **Overcommit support**          | Yes — allocate more than physical capacity          | No — bounded by actual available space             |
| **Performance**                 | Comparable to thick once space is allocated         | Predictable from the start                         |
| **Risk**                        | Out-of-space if usage exceeds physical capacity     | No overcommit risk                                 |
| **Supported backends**          | Ceph RBD, LVM thin, NetApp, Pure Storage, HPE, Dell | LVM (default), some SAN arrays                     |
| **Recommended for**             | Multi-tenant clouds, variable-usage workloads       | Latency-sensitive databases, predictable workloads |

***

## Backend Provisioning Support

<AccordionGroup>
  <Accordion title="Ceph RBD — thin by default" icon="database">
    Ceph RBD images are sparse by design. When a volume is created, no physical storage is consumed until data is written. No configuration is required to enable thin provisioning on the Ceph backend.

    ```ini title="cinder.conf — Ceph RBD backend" theme={null}
    [ceph-ssd]
    volume_driver = cinder.volume.drivers.rbd.RBDDriver
    volume_backend_name = ceph-ssd
    rbd_pool = volumes
    rbd_ceph_conf = /etc/ceph/ceph.conf
    rbd_user = cinder
    rbd_secret_uuid = <libvirt-secret-uuid>
    ```

    <Tip>
      Because Ceph RBD is thin by default, the `max_over_subscription_ratio` setting in the `[DEFAULT]` section controls how aggressively the scheduler overcommits capacity. The default value is 20.0 (20x overcommit).
    </Tip>
  </Accordion>

  <Accordion title="LVM — configure thin pools" icon="layers">
    LVM uses thick provisioning by default. To enable thin provisioning, create a thin pool in the volume group before configuring Cinder, then set `lvm_type = thin` in the backend section.

    **Step 1: Create the thin pool**

    ```bash title="Create LVM thin pool" theme={null}
    pvcreate /dev/sdX
    vgcreate cinder-volumes /dev/sdX
    lvcreate --type thin-pool \
      --size 500G \
      --name thin-pool \
      cinder-volumes
    ```

    **Step 2: Configure Cinder**

    ```ini title="cinder.conf — LVM thin backend" theme={null}
    [lvm-thin]
    volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
    volume_backend_name = lvm-thin
    volume_group = cinder-volumes
    lvm_type = thin
    target_helper = lioadm
    target_protocol = iscsi
    ```

    **Step 3: Apply the configuration**

    ```bash title="Deploy block storage configuration" theme={null}
    xavs-ansible deploy --tags cinder
    ```

    **Step 4: Verify thin volumes are created**

    ```bash title="Check LVM thin volume allocation" theme={null}
    lvs --options +lv_layout
    ```

    Volumes created from this backend will show `Twi` in the `Attr` column, confirming thin layout.
  </Accordion>

  <Accordion title="NetApp ONTAP — FlexVol thin provisioning" icon="server">
    NetApp ONTAP supports thin provisioning through FlexVol volumes. Set `netapp_lun_space_reservation = disabled` to enable thin provisioning on the LUN.

    ```ini title="cinder.conf — NetApp thin provisioning" theme={null}
    [netapp-thin]
    volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
    volume_backend_name = netapp-thin
    netapp_storage_family = ontap_cluster
    netapp_transport_type = https
    netapp_server_hostname = 10.0.10.50
    netapp_login = admin
    netapp_password = <password>
    netapp_vserver = svm0
    netapp_storage_protocol = iscsi
    netapp_lun_space_reservation = disabled
    ```
  </Accordion>

  <Accordion title="Pure Storage — thin by default" icon="zap">
    Pure Storage FlashArray volumes are thin-provisioned by default. The array reports the provisioned size to the scheduler, not physical consumption. No additional configuration is needed.

    ```ini title="cinder.conf — Pure Storage thin provisioning" theme={null}
    [pure-flash]
    volume_driver = cinder.volume.drivers.pure.PureISCSIDriver
    volume_backend_name = pure-flash
    san_ip = 10.0.10.60
    pure_api_token = <api-token>
    use_multipath_for_image_xfer = true
    ```
  </Accordion>
</AccordionGroup>

***

## Scheduler Overcommit Configuration

The Cinder scheduler uses `max_over_subscription_ratio` to determine how much capacity to advertise to tenants above physical availability. This applies to backends that report `thin_provisioning_support = True`.

```ini title="cinder.conf — scheduler overcommit settings" theme={null}
[DEFAULT]
max_over_subscription_ratio = 20.0
reserved_percentage = 5
```

| Parameter                     | Description                                                          | Default |
| ----------------------------- | -------------------------------------------------------------------- | ------- |
| `max_over_subscription_ratio` | Maximum ratio of provisioned capacity to physical capacity           | `20.0`  |
| `reserved_percentage`         | Percentage of backend capacity reserved and excluded from scheduling | `0`     |

<Warning>
  Setting `max_over_subscription_ratio` too high without monitoring physical usage can result in volumes that cannot be written to once the physical backend is full. Monitor actual consumption against provisioned capacity and set alerts at 70–80% physical utilization.
</Warning>

***

## Monitor Overcommit and Capacity

<Tabs>
  <Tab title="Dashboard" icon="monitor">
    Navigate to **Storage > Storage Backends** (admin view) to view backend capacity statistics. The **Free Capacity** and **Total Capacity** columns show the physical pool state reported by each backend driver.

    For provisioned-vs-physical overcommit visibility, use the XIMP monitoring dashboard to track Cinder backend pool utilization over time.
  </Tab>

  <Tab title="CLI" icon="terminal">
    Source your credentials file to authenticate with the Xloud platform:

    ```bash title="Load credentials" theme={null}
    source openrc.sh
    ```

    <Tip>
      Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
    </Tip>

    ```bash title="List backend capacity and provisioning" theme={null}
    openstack volume backend pool list --long
    ```

    Key columns:

    | Column                        | Description                                               |
    | ----------------------------- | --------------------------------------------------------- |
    | `free_capacity_gb`            | Physical free space remaining on the backend              |
    | `total_capacity_gb`           | Total physical capacity of the backend pool               |
    | `provisioned_capacity_gb`     | Total provisioned (allocated) capacity across all volumes |
    | `thin_provisioning_support`   | `True` if the backend supports thin provisioning          |
    | `max_over_subscription_ratio` | Configured overcommit ratio for this backend              |

    ```bash title="Check volume actual allocation (Ceph)" theme={null}
    rbd du volumes
    ```

    The `rbd du` command shows the difference between provisioned size and actual data written for each RBD volume, providing a direct view of thin provisioning savings.
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup>
  <Card title="External Storage Integration" href="/services/storage/external-storage">
    Connect to enterprise storage backends including NetApp, Pure Storage, and Dell
  </Card>

  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Configure and register backend drivers with the scheduler
  </Card>

  <Card title="Volume Types Admin" href="/services/storage/volume-types-admin">
    Expose provisioning mode to tenants through volume type extra specs
  </Card>

  <Card title="Storage Tiers" href="/services/storage/storage-tiers">
    Configure multi-tier storage with NVMe, SSD, and HDD backends
  </Card>
</CardGroup>


# Third-Party Backup Support
Source: https://docs.xloud.tech/services/storage/third-party-backup

Use enterprise backup vendors alongside the native Xloud backup — agentless and agent-based options for compliance and application-aware protection.

## Overview

In addition to the **Xloud Native Backup** for block storage volumes, the Xloud Platform
is fully open to enterprise third-party backup solutions. Customers who need
application-aware protection, long-term off-site retention, regulatory archive depths,
or a single backup pane across multi-cloud and on-premises infrastructure can plug in
the backup vendor of their choice.

Third-party backup tools integrate with Xloud through standard APIs and snapshot
mechanisms — no proprietary backend changes are required.

<Note>
  Xloud Native Backup is the recommended starting point for most workloads. Adopt a
  third-party solution when you need capabilities Xloud Native Backup doesn't cover —
  such as application-consistent quiescing for databases, indexed file-level recovery
  across thousands of guests, or cross-cluster / cross-cloud lifecycle policies.
</Note>

***

## Native vs Third-Party Backup

| Concern                                             | Xloud Native Backup      | Third-Party Backup   |
| --------------------------------------------------- | ------------------------ | -------------------- |
| **Volume-level full + incremental backup**          | Yes                      | Yes                  |
| **Restore to original or new volume**               | Yes                      | Yes                  |
| **Off-cluster retention**                           | Yes (S3/NFS/SMB targets) | Yes (vendor-managed) |
| **Application-aware backup (DB quiesce)**           | No                       | Yes (with agent)     |
| **Indexed file-level search across many VMs**       | Limited                  | Yes                  |
| **Multi-cluster / multi-cloud catalog**             | No                       | Yes                  |
| **Long-term archive (years, tape, glacier)**        | Limited                  | Yes                  |
| **Per-VM auto-enrollment via tags / policies**      | Manual scheduling        | Yes (most vendors)   |
| **Compliance certifications (SOC2, HIPAA archive)** | Underlying storage       | Vendor-supplied      |

For most general-purpose workloads, the native backup is sufficient. Third-party tools
shine when you have complex application stacks, multi-cluster fleets, or audit/archive
requirements that need a dedicated backup product.

***

## Two Integration Approaches

Third-party backup tools integrate with Xloud in one of two architectural patterns.

<CardGroup>
  <Card title="Agentless" icon="server">
    The backup tool talks to Xloud APIs (Block Storage, Image, Compute) and reads VM
    disks via snapshots. **No software is installed inside the guest.** Best for
    standardized fleets and snapshot-consistent backups.
  </Card>

  <Card title="With Agent" icon="cpu">
    A small agent runs inside each guest VM. The backup tool coordinates with the agent
    to quiesce applications, capture file-level metadata, and stream changed data.
    Best for application-aware and granular file-level recovery.
  </Card>
</CardGroup>

***

## Agentless Backup

Agentless backup uses Xloud's snapshot APIs to read VM disks. The backup tool calls
Xloud, asks for a snapshot, reads the snapshot's data through standard storage APIs,
and stores it in its own catalog.

### How agentless backup works

<Steps>
  <Step title="Discovery" icon="search">
    The backup tool authenticates with Xloud Identity and lists projects, instances,
    and volumes the backup user can see.
  </Step>

  <Step title="Snapshot" icon="camera">
    The tool calls Xloud Block Storage to take a snapshot of the target volume. The
    snapshot is crash-consistent (the same as a power-off + power-on).
  </Step>

  <Step title="Read snapshot" icon="download">
    The tool streams the snapshot data through the storage API and stores it in its
    own backup repository (object storage, NAS, dedup appliance, tape, etc.).
  </Step>

  <Step title="Catalog and retain" icon="archive">
    The tool indexes the backup, applies the retention policy, and is ready to
    restore when needed.
  </Step>
</Steps>

### Strengths of agentless

<CardGroup>
  <Card title="No guest footprint" icon="check">
    Nothing runs inside the VM — no extra package to install, patch, or troubleshoot.
    Works for any guest OS, including hardened or appliance images.
  </Card>

  <Card title="Fleet-wide auto-discovery" icon="layers">
    The backup tool sees every VM through the Xloud API. New VMs are picked up
    automatically by tag-based or project-based policies.
  </Card>

  <Card title="Lower operational overhead" icon="users">
    No agent rollouts, no agent-version drift across the fleet. Upgrades happen on
    the backup-tool side only.
  </Card>

  <Card title="Snapshot-fast" icon="zap">
    Backup windows are short — the snapshot takes seconds, and reading happens
    asynchronously without locking the running VM.
  </Card>
</CardGroup>

### Limitations of agentless

* **Crash-consistent only** by default — a snapshot captures disk state without
  flushing application buffers. Databases may need replay on restore.
* **Limited application awareness** — without an agent, the backup tool cannot quiesce
  Microsoft SQL, Oracle, Exchange, or similar transactional applications cleanly.
* **File-level restore needs extra work** — the backup tool must mount and parse the
  guest filesystem to expose individual files (most agentless tools do support this).

### Vendors that integrate with Xloud APIs (agentless)

<Info>
  **Recommended — Commvault** has explicit, link-backed documentation for the four
  capabilities most commonly required of a third-party backup integration:
  deep VM-provisioning integration, rule-based auto-enrollment of newly created VMs,
  schedule + retention count, and on-demand backup. The full per-capability matrix
  is below the table.
</Info>

| Vendor                    | Xloud-native | Notes                                                                                                                                                        |
| ------------------------- | :----------: | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Commvault IntelliSnap** |      Yes     | VM Groups with rule-based selection (project, name pattern, power state); schedule + retention on assigned Server Backup Plan; on-demand backup per VM Group |
| **Trilio**                |      Yes     | Workload Manager dashboard plugin, policy-based protection (hourly / daily / weekly / monthly / yearly), workload-level restores                             |
| **Storware vProtect**     |      Yes     | Dashboard plugin, SLA Policy with schedule + retention per policy, full and incremental                                                                      |
| **Veeam**                 |      Yes     | Cluster-level API integration, image-based backup, GFS retention schemes                                                                                     |
| **Cohesity**              |    Partial   | Protection Policy with min 12 h interval + retention; Extended Retention for longer cycles                                                                   |

#### Commvault capability matrix (with dedicated URLs)

| #      | Capability                                                                              | Commvault feature                                                                                                                                           | Reference                                                                                                                                         |
| ------ | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **1**  | Third-party backup software named with OEM documentation                                | Commvault IntelliSnap (agentless) and Commvault (with-agent)                                                                                                | [Commvault for Xloud — overview](https://documentation.commvault.com/saas/openstack.html)                                                         |
| **2**  | Deeply integrated into VM provisioning so newly created VMs are protected automatically | **VM Group** with rule-based content (Project / Instance name pattern / Power state) — every new VM matching the rule is picked up on the next backup cycle | [Adding a VM Group for Xloud](https://documentation.commvault.com/saas/adding_vm_group_for_openstack.html)                                        |
| **3a** | Backup scheduling                                                                       | **Server Backup Plan** controls the schedule (granularity from minutes to weekly / monthly / yearly via secondary copies)                                   | [Performing Backups](https://documentation.commvault.com/2024e/software/performing_backups_for_openstack_instances_or_vm_groups.html)             |
| **3b** | Backup retention counts                                                                 | **Days-Based Retention** (keep N days) or count-based for IntelliSnap; secondary copies have separate counts for monthly-full / yearly-full                 | [Days-Based Retention](https://documentation.commvault.com/v11/commcell-console/days_based_retention.html)                                        |
| **3c** | On-demand backup from the same workflow                                                 | Manual backup action triggered against any VM Group or instance, independent of the schedule                                                                | [Performing Backups (on-demand)](https://documentation.commvault.com/2024e/software/performing_backups_for_openstack_instances_or_vm_groups.html) |

<Tip>
  Other strong agentless choices: **Trilio** and **Storware** also ship rule-based
  policy engines that pick up newly created VMs, but their public docs are slightly
  less explicit about "auto-enroll on VM create" enforcement. See their accordions
  below for the exact docs.
</Tip>

### Schedule + retention reference docs (agentless)

<AccordionGroup>
  <Accordion title="Commvault — VM Group + Server Backup Plan (recommended)" icon="server">
    Commvault organizes Xloud VMs into **VM Groups** with rule-based content selection.
    The schedule + retention live on the **Server Backup Plan** assigned to the group.

    Available VM-Group rule types:

    * **Project** — every VM in a chosen project
    * **Instance name or pattern** — name `Contains` operator
    * **Power state** — On / Off / Other
    * **Browse / Instance** — explicit single-VM selection

    **Retention count** — configured on the Server Backup Plan as **Days-Based
    Retention** (keep backups for N days) or as the **number of snapshots to retain**
    for IntelliSnap. Secondary copies have separate retention counts for monthly-full
    and yearly-full tiers.

    **On-demand backup** — yes. The Performing Backups page documents the manual
    backup action that can be triggered against any VM Group or instance independent
    of the schedule.

    * [Commvault for Xloud (overview)](https://documentation.commvault.com/saas/openstack.html)
    * [Adding a VM Group for Xloud](https://documentation.commvault.com/saas/adding_vm_group_for_openstack.html)
    * [Performing Backups (on-demand)](https://documentation.commvault.com/2024e/software/performing_backups_for_openstack_instances_or_vm_groups.html)
    * [Days-Based Retention](https://documentation.commvault.com/v11/commcell-console/days_based_retention.html)
  </Accordion>

  <Accordion title="Trilio — Workload Policies, Schedulers, Workloads" icon="shield">
    Trilio's protection model has two parts:

    * **Workload Policies** — admins define schedule + retention and assign policies to projects.
    * **Workloads** — group VMs to be backed up together with one of those policies.

    **Retention count** — configured as the `retention` parameter on the policy
    (e.g. `--hourly interval=1,retention=30,snapshot_type=incremental` keeps 30
    snapshots; default for hourly is 30, incremental). For one-off jobs the syntax is
    `--manual retention=<count>,retention_days_to_keep=<days>`.

    **On-demand backup** — yes. From the **Workloads** page in the dashboard, you can
    trigger a snapshot manually for any workload at any time, independent of the
    scheduled policy.

    For *fully automatic* discovery of newly created VMs, confirm the exact behavior in
    the current Trilio release notes — public docs describe policy-driven workload
    backup, not turnkey auto-enrollment of every new VM.

    * [Workload Policies](https://docs.trilio.io/openstack/admin-guide/workload-policies)
    * [Schedulers](https://docs.trilio.io/openstack/user-guide/schedulers)
    * [Workloads](https://docs.trilio.io/openstack/user-guide/workloads)
    * [Snapshots (on-demand)](https://docs.trilio.io/openstack/user-guide/snapshots)
  </Accordion>

  <Accordion title="Storware vProtect — SLA Policy" icon="tag">
    Storware's **SLA Policy** combines schedule (daily / weekly / etc.) and retention
    in one policy. New VMs matching the policy's project / tag selector are picked up
    on the next protection cycle. Full and incremental backups both supported.

    **Retention count** — configured per policy as a count of backups to keep, or as
    a time window. Storware moved retention from backup destination to the policy
    itself, so different policies can have different retention without changing the
    backup target.

    **On-demand backup** — yes. The Storware OpenStack plugin dashboard exposes a
    direct backup action per VM, independent of the SLA Policy schedule.

    * [Storware Xloud integration docs](https://docs.storware.eu/protecting-virtual-machines/virtual-machines/openstack)
    * [Solution overview](https://storware.eu/solutions/virtual-machine-backup-and-recovery/openstack-backup-and-recovery/)
  </Accordion>

  <Accordion title="Veeam — Backup Retention + GFS Schedules" icon="rotate-cw">
    Veeam uses a job-level schedule combined with retention policies; GFS
    (Grandfather-Father-Son) schemes give you long-term tiered retention.

    * [Backup Retention](https://helpcenter.veeam.com/docs/vbr/userguide/sch_backup_retention.html)
    * [Retention Policies](https://helpcenter.veeam.com/docs/vbr/userguide/sch_retention_policy.html)
    * [GFS Policy Schedules](https://helpcenter.veeam.com/docs/vbr/userguide/sch_backup_job_create_gfs.html)
  </Accordion>

  <Accordion title="Cohesity — Protection Policies + Extended Retention" icon="layers">
    Cohesity Protection Policies define backup interval (minimum 12 h) and primary
    retention; Extended Retention adds longer hourly / daily / weekly / monthly /
    yearly snapshots beyond the primary policy.

    * [Policies](https://docs.cohesity.com/baas/data-protect/policies.htm)
    * [Extended Retention](https://docs.cohesity.com/baas/data-protect/extended-retention.htm)
  </Accordion>
</AccordionGroup>

***

## With-Agent Backup

Agent-based backup runs a small piece of software inside each guest VM. The agent
coordinates with the backup tool over the network to capture data with
application-aware consistency and full file-level granularity.

### How agent-based backup works

<Steps>
  <Step title="Install the agent" icon="download">
    A backup agent package is installed inside the guest OS (Linux .deb / .rpm,
    Windows .msi). The agent registers with the backup server.
  </Step>

  <Step title="Quiesce the application" icon="pause">
    Before each backup, the agent calls application hooks (VSS on Windows, scripts on
    Linux) to flush database buffers, lock log writers, and ensure the on-disk state
    is consistent.
  </Step>

  <Step title="Stream changed data" icon="upload">
    The agent identifies changed blocks or files since the last backup and streams
    them to the backup server over the network.
  </Step>

  <Step title="Catalog with file-level index" icon="list">
    The backup tool catalogs every file, registry key, or database object captured —
    enabling granular restore by name, not just full-volume restore.
  </Step>
</Steps>

### Strengths of with-agent

<CardGroup>
  <Card title="Application-consistent backup" icon="database">
    Microsoft SQL, Oracle, MySQL, PostgreSQL, Exchange, Active Directory, SAP, and
    other transactional apps are backed up cleanly with no replay needed at restore.
  </Card>

  <Card title="Granular file-level restore" icon="file">
    Recover a single email, table row, registry key, or document from a backup —
    without restoring the whole volume.
  </Card>

  <Card title="Cross-platform consistency" icon="globe">
    The same backup tool protects VMs on Xloud, on-prem hypervisors, public clouds,
    and physical machines — one console, one policy framework.
  </Card>

  <Card title="In-guest encryption + dedup" icon="shield">
    Source-side deduplication and encryption reduce network traffic and protect
    backup data in transit and at rest.
  </Card>
</CardGroup>

### Limitations of with-agent

* **Operational overhead** — agent installation, version upgrades, and patching across
  the fleet require lifecycle management.
* **Guest OS coupling** — agents are OS-specific (Linux distro, Windows version) and
  may not be available for some appliance images or hardened guests.
* **In-guest resource use** — the agent consumes a small amount of guest CPU, memory,
  and network capacity during backup windows.
* **Doesn't protect non-running VMs** — if the VM is powered off, the agent isn't
  running, so the backup tool falls back to agentless or snapshot mode.

### Vendors that support agent-based backup on Xloud guests

| Vendor                    | Notes                                                                                                                                         |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **Veritas NetBackup**     | Comprehensive enterprise agent suite; very strong app-aware coverage (SQL, Oracle, Exchange, SAP). Schedule + retention live on the policy    |
| **Commvault**             | Agent-based or agentless. Schedule + retention managed on the assigned Server Backup Plan                                                     |
| **Veeam**                 | Veeam Agent for Linux / Windows. Often paired with the agentless plugin                                                                       |
| **Acronis Cyber Protect** | Agent + cloud catalog. Strong endpoint + workload protection. Backup Plan defines schedule and retention together                             |
| **Bacula Enterprise**     | Open-source-rooted agent, Xloud-aware module available. Per-Job File/Job Retention + per-Pool Volume Retention                                |
| **IBM Storage Protect**   | Enterprise agent for large heterogeneous fleets (formerly IBM Spectrum Protect). Retention defined in Management Class within a Policy Domain |

### Schedule + retention reference docs (with-agent)

<AccordionGroup>
  <Accordion title="Veritas NetBackup — Policy schedules + Retention Periods" icon="shield">
    Schedule + retention live on the NetBackup policy.

    * [Admin Guide for Xloud](https://www.veritas.com/support/en_US/doc/147269993-149461653-0/index)
    * [Edit a protection (schedules)](https://www.veritas.com/support/en_US/doc/147269993-165972820-0/v148941437-165972820)
    * [Retention Periods](https://www.veritas.com/support/en_US/doc/18716246-126559472-0/v40229538-126559472)
  </Accordion>

  <Accordion title="Commvault — Server Backup Plan + Days-Based Retention" icon="server">
    Schedule + retention are managed on the Server Backup Plan assigned to a VM Group.

    * [Adding a VM Group for Xloud](https://documentation.commvault.com/saas/adding_vm_group_for_openstack.html)
    * [Days-Based Retention](https://documentation.commvault.com/v11/commcell-console/days_based_retention.html)
  </Accordion>

  <Accordion title="Veeam — Backup Retention + Retention Policies" icon="rotate-cw">
    * [Backup Retention](https://helpcenter.veeam.com/docs/vbr/userguide/sch_backup_retention.html)
    * [Retention Policies](https://helpcenter.veeam.com/docs/vbr/userguide/sch_retention_policy.html)
  </Accordion>

  <Accordion title="Acronis Cyber Protect — Backup Plan schedule + retention" icon="cloud">
    Schedule and retention live in the Backup Plan together.

    * [Set up backup schedule](https://kb.acronis.com/content/59296)
    * [Retention rules — how & when they work](https://kb.acronis.com/content/68304)
    * [Retention of Backups](https://kb.acronis.com/content/12366)
  </Accordion>

  <Accordion title="Bacula Enterprise — Schedules + Retentions" icon="cog">
    Bacula uses per-Job File/Job Retention and per-Pool Volume Retention.

    * [Schedules and Retentions](https://docs.baculasystems.com/BEPlanningAndPreparation/BackupPolicy/SchedulesAndRetentions/index.html)
    * [Retention Period Configuration](https://docs.baculasystems.com/BEPlanningAndPreparation/BackupPolicy/Configuration/RetentionPeriod/index.html)
    * [Xloud Backup & Restore Strategies](https://docs.baculasystems.com/BEDedicatedBackupSolutions/Virtualization/Hypervisors/openstack/BackupAndRestoreStrategies/index.html)
  </Accordion>

  <Accordion title="IBM Storage Protect — Policy Domain + Management Class" icon="database">
    Retention is defined in a **Management Class** within a **Policy Domain**.

    * [Define Policy Domain](https://www.ibm.com/docs/en/storage-protect/8.1.25?topic=commands-define-objectdomain-define-policy-domain-object-clients)
    * [Storage Protect product page](https://www.ibm.com/products/storage-protect)
  </Accordion>
</AccordionGroup>

***

## Hybrid: Agentless + Agent

Many production environments use **both** approaches together:

* **Agentless** for the bulk of the fleet — fast snapshot-based backups of stateless
  workloads, web tiers, and standard application servers.
* **With-agent** for specific high-value VMs — databases, mail servers, ERP/SAP,
  domain controllers, and any workload where a few minutes of replay-on-restore is
  unacceptable.

A single backup tool typically supports both modes from the same console — pick the
backup method per workload class via policy, not per-tool.

***

## Choosing the Right Approach

<AccordionGroup>
  <Accordion title="Standard application servers and web tiers" icon="server">
    **Pick agentless.** Snapshot-based backups are fast, fleet-wide, and good enough
    for stateless or session-replicated workloads.
  </Accordion>

  <Accordion title="Database VMs (SQL, Oracle, MySQL, PostgreSQL)" icon="database">
    **Pick with-agent.** Application-aware quiescing prevents database replay or
    corruption on restore. Trilio also offers app-aware workload types for databases
    via in-guest helper scripts.
  </Accordion>

  <Accordion title="Microsoft Exchange / SharePoint / SQL Always-On" icon="mail">
    **Pick with-agent.** VSS integration is essential. Veeam, Commvault, and Veritas
    all have strong Exchange and SQL Always-On support.
  </Accordion>

  <Accordion title="Hardened or appliance VMs (no shell access)" icon="lock">
    **Pick agentless.** You cannot install an agent into a sealed appliance image.
    Snapshot-based backup is the only viable option.
  </Accordion>

  <Accordion title="Long-term compliance archive (HIPAA, SOC2, financial)" icon="archive">
    **Pick whatever your audit framework prefers** — most enterprise backup vendors
    offer immutable retention, write-once tiers, and audit reporting designed for
    these frameworks.
  </Accordion>

  <Accordion title="Multi-cluster, multi-cloud fleet under one console" icon="globe">
    **Pick agentless** for the Xloud side and use the same vendor's connectors
    for AWS / Azure / GCP / on-prem. Veeam, Commvault, Cohesity, Trilio, and Veritas
    all support multi-cloud.
  </Accordion>
</AccordionGroup>

***

## How to Integrate a Third-Party Backup Tool

<Steps>
  <Step title="Provision a backup service account" icon="user">
    Create a dedicated user in Xloud Identity with read access to all projects you
    want backed up. Grant the minimum roles required by the vendor (typically
    `member` plus `volume:create_snapshot` on Block Storage).
  </Step>

  <Step title="Open Xloud APIs to the backup server" icon="network">
    Ensure the backup tool can reach Xloud's Identity, Compute, Block Storage, and
    Image API endpoints. Open required network paths, configure TLS trust, and (for
    agent-based mode) ensure guest VMs can reach the backup server on its data port.
  </Step>

  <Step title="Configure the vendor's Xloud connector" icon="plug">
    Each vendor ships an Xloud connector module. Point it at your Xloud Identity
    endpoint, supply the service account credentials, and select the projects to
    discover.

    <Info>
      For Trilio, this is the **Workload Manager** Dashboard panel. For Veeam and
      Commvault, it's a connector wizard inside the vendor console.
    </Info>
  </Step>

  <Step title="Define protection policies" icon="shield">
    Create backup policies — typically by tag, project, or VM list. Most vendors
    let you specify schedule, retention, target repository, and (for agent-based)
    application-quiesce settings.
  </Step>

  <Step title="Verify the first backup and restore" icon="check">
    Run a test backup of a single VM, then perform a test restore to a new VM in a
    sandbox project. Confirm boot, network, and application functionality before
    enabling the policy fleet-wide.

    <Check>The test VM restores successfully and applications start cleanly.</Check>
  </Step>
</Steps>

<Warning>
  Always test restores on a regular cadence — quarterly at minimum. A backup you
  cannot restore is not a backup.
</Warning>

***

## What Xloud Provides for Backup Vendors

Xloud exposes the standard cloud APIs that every modern backup vendor expects:

| API                | Used for                                                                          |
| ------------------ | --------------------------------------------------------------------------------- |
| **Identity**       | Authentication, project scoping, role-based access for the backup service account |
| **Compute**        | List instances, query state, attach/detach volumes, snapshot instances            |
| **Block Storage**  | Create/list/delete volume snapshots, mount snapshots for backup reads             |
| **Image**          | List images, create/delete snapshot-type images                                   |
| **Object Storage** | Direct backup target for native and many third-party tools                        |

For low-level integration, Xloud also exposes:

* **File-level read of snapshots** through the [File-Level Restore](/services/storage/file-level-restore)
  service — useful for backup tools that want to verify snapshot contents or build
  a file index.
* **Standard storage protocols** — RBD for storage running on XSDS, NFS, iSCSI, and
  Fibre Channel for external arrays — so backup tools that prefer direct storage
  access can also integrate.

***

## Related Topics

<CardGroup>
  <Card title="Xloud Native Backup" href="/services/storage/backups">
    Volume backup with full + incremental modes — the built-in option
  </Card>

  <Card title="File-Level Restore" href="/services/storage/file-level-restore">
    Recover individual files from a snapshot without a full restore
  </Card>

  <Card title="Disaster Recovery (XDR)" href="/services/disaster-recovery">
    Replication, failover, and continuous protection beyond backup
  </Card>
</CardGroup>


# Block Storage Troubleshooting
Source: https://docs.xloud.tech/services/storage/troubleshooting

Diagnose common block storage issues — volumes stuck in error, attachment failures, snapshot errors, and quota limits.

## Overview

This guide covers the most common issues encountered when working with Xloud Block Storage volumes, snapshots, backups, and attachments. Each section identifies the symptom, explains the most likely cause, and provides step-by-step resolution commands.

<Note>
  **Before troubleshooting**

  * Authenticate with a valid project token: `source openrc.sh`
  * For persistent or recurring issues, contact [Xloud Support](mailto:support@xloud.tech)
    with the volume ID and the error message
</Note>

***

## Volume Issues

<AccordionGroup>
  <Accordion title="Volume stuck in 'creating' status" icon="clock">
    **Symptom**: A newly created volume remains in `creating` status for more than
    5 minutes.

    **Cause**: The storage backend did not respond within the expected timeout, or
    the scheduler could not find a suitable backend for the requested volume type.

    **Resolution**:

    ```bash title="Check volume status and error reason" theme={null}
    openstack volume show <volume-id> -c status -c error_reason
    ```

    Common causes and fixes:

    * **No backend available** — the requested volume type has no active backend. Try
      a different volume type or availability zone.
    * **Capacity exhausted** — the backend has no free capacity. Check with your
      administrator.
    * **Backend unreachable** — contact your administrator to verify storage service health.

    <Tip>Try a different volume type or availability zone if the issue persists.</Tip>
  </Accordion>

  <Accordion title="Volume in 'error' status after creation" icon="circle-x">
    **Symptom**: Volume creation completes but status shows `error`.

    **Cause**: The backend driver encountered an error allocating the volume. Common
    causes include pool capacity exhaustion, backend authentication failure, or
    configuration issues.

    **Resolution**:

    ```bash title="Show volume details" theme={null}
    openstack volume show <volume-id>
    ```

    Delete the failed volume and retry with a different type:

    ```bash title="Delete failed volume" theme={null}
    openstack volume delete <volume-id>
    ```

    If the error persists on all volume types, contact your administrator.
  </Accordion>

  <Accordion title="Cannot delete volume — 'volume is in use'" icon="lock">
    **Symptom**: `openstack volume delete` fails with "volume is in use" or similar error.

    **Cause**: The volume is still attached to an instance, or a snapshot was taken
    while the volume was in a non-available state.

    **Resolution**:

    ```bash title="Check volume attachments" theme={null}
    openstack volume show <volume-id> -c status -c attachments
    ```

    [Detach the volume](/services/storage/attach-volume) from the instance first,
    then retry deletion.
  </Accordion>

  <Accordion title="Quota exceeded when creating volume" icon="gauge">
    **Symptom**: Volume creation fails with "Quota exceeded for volumes" or
    "Quota exceeded for gigabytes".

    **Cause**: The project has reached the storage quota limit.

    **Resolution**:

    ```bash title="Check current quota usage" theme={null}
    openstack quota show
    ```

    Review `volumes`, `gigabytes`, and `per_volume_gigabytes` limits. Either:

    * Delete unused volumes and snapshots to free quota
    * Contact your administrator to request a quota increase
  </Accordion>
</AccordionGroup>

***

## Attachment Issues

<AccordionGroup>
  <Accordion title="Cannot attach volume — availability zone mismatch" icon="triangle-alert">
    **Symptom**: Attachment fails with an error referencing availability zone.

    **Cause**: The volume and instance are in different availability zones. Volumes
    can only be attached to instances in the same zone.

    **Resolution**:

    ```bash title="Check volume availability zone" theme={null}
    openstack volume show <volume-id> -c availability_zone
    ```

    ```bash title="Check instance availability zone" theme={null}
    openstack server show <instance-id> -c OS-EXT-AZ:availability_zone
    ```

    If the zones differ, create a new volume in the correct availability zone, or
    restore a snapshot of the volume in the correct zone.
  </Accordion>

  <Accordion title="Volume shows 'In-use' but is not attached" icon="link">
    **Symptom**: Volume status is `in-use` but the `attachments` field is empty, or
    the instance it was attached to no longer exists.

    **Cause**: The compute instance was force-deleted without detaching the volume,
    leaving the volume in a stale `in-use` state.

    **Resolution** (admin only):

    ```bash title="Reset volume state" theme={null}
    openstack volume set --state available <volume-id>
    ```

    <Warning>
      Only reset the state if you are certain the volume is not actually in use. Verify
      the instance no longer exists before resetting.
    </Warning>
  </Accordion>

  <Accordion title="Block device not visible inside instance after attach" icon="hard-drive">
    **Symptom**: The volume shows `in-use` in the Dashboard/CLI, but `lsblk` inside
    the instance does not show the new device.

    **Cause**: The guest OS did not detect the hot-plug event.

    **Resolution**: Rescan the SCSI bus inside the instance:

    ```bash title="Rescan SCSI bus" theme={null}
    sudo echo "- - -" | sudo tee /sys/class/scsi_host/host*/scan
    ```

    Verify the device is now visible:

    ```bash title="List block devices" theme={null}
    lsblk
    ```
  </Accordion>
</AccordionGroup>

***

## Snapshot Issues

<AccordionGroup>
  <Accordion title="Snapshot failed or stuck in 'error' status" icon="triangle-alert">
    **Cause**: The volume was in an inconsistent state, or the backend encountered
    an error during the snapshot operation.

    **Resolution**:

    ```bash title="List snapshots with status" theme={null}
    openstack volume snapshot list
    ```

    ```bash title="Delete a failed snapshot" theme={null}
    openstack volume snapshot delete <snapshot-id>
    ```

    For snapshots stuck in `deleting` state, contact your administrator — the backend
    may require manual cleanup.

    <Warning>
      Do not repeatedly retry snapshot creation on a degraded backend. Resolve the
      underlying storage issue first.
    </Warning>
  </Accordion>

  <Accordion title="Snapshot stuck in 'deleting' state" icon="clock">
    **Cause**: The backend could not complete the deletion — common with distributed
    storage when the cluster is degraded or the snapshot has dependent volumes.

    **Resolution**: Check for volumes created from the snapshot:

    ```bash title="List volumes from snapshot" theme={null}
    openstack volume list --property source_volid=<snapshot-id>
    ```

    Delete dependent volumes first, then retry snapshot deletion. If the snapshot
    remains stuck, contact your administrator.
  </Accordion>
</AccordionGroup>

***

## Backup Issues

<AccordionGroup>
  <Accordion title="Backup creation fails with 'backup service unavailable'" icon="circle-x">
    **Cause**: The backup service is not running or is not configured for this
    environment.

    **Resolution**: Contact your administrator to verify the backup service is enabled
    and the backup target is accessible.

    ```bash title="Check backup service status (admin)" theme={null}
    openstack volume service list | grep backup
    ```

    State should be `up`.
  </Accordion>

  <Accordion title="Restore takes longer than expected" icon="clock">
    **Cause**: Backup restoration is a full data copy operation. Duration depends on
    the volume size, the backup target throughput, and current system load.

    For large volumes (100+ GiB), restores may take 15–60 minutes. Monitor progress:

    ```bash title="Check restore status" theme={null}
    openstack volume show <restored-volume-id> -c status
    ```

    Status transitions: `restoring-backup` → `available`
  </Accordion>
</AccordionGroup>

***

## Extension Issues

<AccordionGroup>
  <Accordion title="Filesystem does not reflect new capacity after extension" icon="expand">
    **Cause**: The block device was extended but the filesystem was not resized.
    The filesystem resize must be performed manually inside the instance.

    **Resolution**:

    ```bash title="Resize ext4 filesystem" theme={null}
    sudo resize2fs /dev/vdb
    ```

    ```bash title="Resize XFS filesystem" theme={null}
    sudo xfs_growfs /mnt/data
    ```

    Verify with:

    ```bash title="Check new capacity" theme={null}
    df -h /mnt/data
    ```
  </Accordion>
</AccordionGroup>

***

## Still Having Issues?

<CardGroup>
  <Card title="Contact Support" icon="headset" href="mailto:support@xloud.tech">
    Open a support ticket with your volume ID and error message
  </Card>

  <Card title="Admin Troubleshooting" icon="shield" href="/services/storage/admin-guide">
    Administrators: service-level diagnostics and backend troubleshooting
  </Card>

  <Card title="User Guide" icon="book-open" href="/services/storage/user-guide">
    Return to the Block Storage user guide for operational procedures
  </Card>

  <Card title="Quota Management (Admin)" icon="gauge" href="/services/storage/quotas">
    Request quota increases from your administrator
  </Card>
</CardGroup>


# Block Storage User Guide
Source: https://docs.xloud.tech/services/storage/user-guide

Create, attach, snapshot, backup, and manage persistent block storage volumes in your Xloud project. Find guides for every volume lifecycle operation.

<p>Overview</p>

Xloud Block Storage provides persistent, high-performance block devices that attach to your
[Compute instances](/services/compute/user-guide) as virtual disks. Unlike ephemeral instance
storage, block volumes retain data across reboots, migrations, and instance deletion —
making them the standard choice for databases, application state, and any workload where
data durability is required.

<Note>
  **Prerequisites**

  * An active Xloud account with appropriate permissions
  * Access to the **Xloud Dashboard** or CLI configured with credentials
  * API credentials sourced (`source openrc.sh`)
</Note>

***

<p>Key Concepts</p>

| Term            | Description                                                                                          |
| --------------- | ---------------------------------------------------------------------------------------------------- |
| **Volume**      | A persistent block device that can be attached to one compute instance at a time                     |
| **Snapshot**    | A point-in-time, crash-consistent copy of a volume stored within the same storage cluster            |
| **Backup**      | A full or incremental copy of a volume written to a separate backup target for long-term retention   |
| **Volume Type** | A named storage profile defining the backend, performance tier, IOPS limits, and encryption policy   |
| **Attachment**  | The binding between a volume and a compute instance, represented as a device path (e.g., `/dev/vdb`) |
| **Transfer**    | A token-based mechanism to move volume ownership between projects without copying data               |

***

<p>Volume Lifecycle</p>

```mermaid theme={null}
graph LR
    Create["Create Volume"] --> Attach["Attach to Instance"]
    Attach --> Use["Format & Mount\nInside Instance"]
    Use --> Detach["Unmount & Detach"]
    Detach --> Attach
    Use --> Snapshot["Take Snapshot"]
    Snapshot --> Restore["Restore to New Volume"]
    Use --> Backup["Create Backup"]
    Backup --> RestoreB["Restore from Backup"]
    Detach --> Extend["Extend Volume"]
    Extend --> Attach
    Detach --> Transfer["Transfer to Another Project"]
    Detach --> Delete["Delete Volume"]
```

***

<p>Guides</p>

<CardGroup>
  <Card title="Create a Volume" icon="plus" href="/services/storage/create-volume">
    Provision a persistent volume — blank, from an image, or from a snapshot
  </Card>

  <Card title="Attach / Detach Volumes" icon="link" href="/services/storage/attach-volume">
    Connect volumes to instances, format filesystems, mount, and safely detach
  </Card>

  <Card title="Extend a Volume" icon="expand" href="/services/storage/extend-volume">
    Increase volume capacity online without detaching or stopping the instance
  </Card>

  <Card title="Volume Snapshots" icon="camera" href="/services/storage/snapshots">
    Create point-in-time snapshots for fast recovery and volume cloning
  </Card>

  <Card title="Volume Backups" icon="archive" href="/services/storage/backups">
    Full and incremental backups to a separate target for disaster recovery
  </Card>

  <Card title="File-Level Restore" icon="folder-open" href="/services/storage/file-level-restore">
    Browse a snapshot and download individual files or folders — no full-volume restore needed
  </Card>

  <Card title="Third-Party Backup" icon="shield-check" href="/services/storage/third-party-backup">
    Use Trilio, Veeam, Commvault, and other vendors alongside the native backup
  </Card>

  <Card title="Volume Transfers" icon="send" href="/services/storage/volume-transfers">
    Move volume ownership between projects using a token-based transfer
  </Card>

  <Card title="Volume Types" icon="layers" href="/services/storage/volume-types">
    Understand NVMe, SSD, and Standard storage tiers and choose the right one
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/services/storage/troubleshooting">
    Diagnose and resolve common volume, snapshot, and attachment issues
  </Card>
</CardGroup>

***

<p>Next Steps</p>

<CardGroup>
  <Card title="Block Storage Admin Guide" icon="shield" href="/services/storage/admin-guide">
    Configure backends, volume types, QoS policies, and storage tiers
  </Card>

  <Card title="Compute User Guide" icon="server" href="/services/compute/user-guide">
    Launch instances and attach volumes as persistent boot or data disks
  </Card>

  <Card title="CLI Setup" icon="terminal" href="/cli-setup">
    Install and configure the CLI for volume management from the terminal
  </Card>

  <Card title="Authentication" icon="key" href="/cli-setup">
    Configure Dashboard access and CLI credentials
  </Card>
</CardGroup>


# Volume Transfers Between Projects
Source: https://docs.xloud.tech/services/storage/volume-transfers

Transfer ownership of Xloud Block Storage volumes between projects using a token-based mechanism. No data is copied — ownership changes instantly.

## Overview

Volume transfers move ownership of a volume from one project to another without copying any data. The operation is token-based: the source project creates a transfer request that generates a one-time authorization key, then the destination project accepts the transfer using the ID and key. The volume changes ownership instantly — no data movement occurs.

<Note>
  **Prerequisites**

  * The volume to transfer must have status **Available** (not attached to an instance)
  * The source project user must have the `member` or `admin` role
  * The destination project user must have the `member` or `admin` role
  * Both users must have access to their respective project credentials
</Note>

***

## How Volume Transfers Work

```mermaid theme={null}
sequenceDiagram
    participant Source as Source Project
    participant API as Block Storage API
    participant Dest as Destination Project

    Source->>API: Create transfer request (volume ID)
    API-->>Source: Transfer ID + Authorization Key
    Source->>Dest: Share Transfer ID + Auth Key (out-of-band)
    Dest->>API: Accept transfer (Transfer ID + Auth Key)
    API-->>Dest: Volume ownership updated
    API-->>Source: Volume removed from project
```

<Warning>
  The authorization key is displayed **only once** at the time of transfer creation.
  Store it securely before closing the dialog or terminal session. If the key is lost,
  cancel the transfer and create a new one.
</Warning>

***

## Transfer a Volume

### Step 1 — Create the Transfer Request (Source Project)

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Volumes" icon="compass">
        Navigate to
        **Storage > Volumes** as the **source project**.
      </Step>

      <Step title="Create transfer" icon="send">
        Locate the target volume (status must be **Available**). Click
        the **More** dropdown and select **Create Transfer**.

        Provide a descriptive transfer name and click **Confirm**.
      </Step>

      <Step title="Record the credentials" icon="key">
        A dialog displays the **Transfer ID** and **Authorization Key**. Copy both
        values immediately and share them securely with the recipient project
        (e.g., via an encrypted message or password manager).

        <Warning>
          The authorization key is shown only once. It cannot be retrieved after
          closing this dialog. If lost, delete this transfer and create a new one.
        </Warning>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate as source project" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create transfer request" icon="send">
        ```bash title="Create transfer (source project)" theme={null}
        openstack volume transfer request create \
          --name transfer-to-project-b \
          <volume-name-or-id>
        ```

        The output includes `auth_key` — copy this value immediately. It is shown
        only once.

        ```bash title="List pending transfers" theme={null}
        openstack volume transfer request list
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Step 2 — Accept the Transfer (Destination Project)

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Switch to destination project" icon="arrows-rotate">
        Log out and log in as a user in the **destination project**, or switch projects
        using the project selector in the Dashboard.
      </Step>

      <Step title="Accept the transfer" icon="inbox">
        Navigate to **Storage > Volumes** and click **Accept Transfer**.

        Enter the **Transfer ID** and **Authorization Key** provided by the source
        project, then click **Confirm**.
      </Step>

      <Step title="Verify" icon="circle-check">
        The volume now appears in the destination project's Volumes list with
        status **Available**.

        <Check>Volume ownership transferred — the volume is now in the destination project.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate as destination project" icon="key">
        Source credentials for the destination project:

        ```bash title="Load destination project credentials" theme={null}
        source project-b-openrc.sh
        ```
      </Step>

      <Step title="Accept the transfer" icon="inbox">
        ```bash title="Accept transfer (destination project)" theme={null}
        openstack volume transfer request accept \
          --auth-key <auth-key> \
          <transfer-id>
        ```

        <Check>Volume ownership transferred. It now appears in the destination project.</Check>
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="Confirm volume is in new project" theme={null}
        openstack volume list
        ```

        The transferred volume appears in the destination project's volume list.
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Cancel a Pending Transfer

If the transfer has not been accepted, the source project can cancel it:

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Navigate to **Storage > Volumes**. Find the volume with a pending transfer
    indicator and click the **More** dropdown and select **Cancel Transfer**.
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="List pending transfers" theme={null}
    openstack volume transfer request list
    ```

    ```bash title="Cancel a transfer" theme={null}
    openstack volume transfer request delete <transfer-id>
    ```

    <Check>Transfer cancelled. The volume remains in the source project with status **Available**.</Check>
  </Tab>
</Tabs>

***

## Important Limitations

| Constraint          | Detail                                                                                                                |
| ------------------- | --------------------------------------------------------------------------------------------------------------------- |
| **Volume status**   | Must be **Available** — cannot transfer attached volumes                                                              |
| **Snapshots**       | Snapshots are not transferred — they remain in the source project                                                     |
| **Encryption keys** | Encrypted volumes can be transferred, but the destination project must have access to the same key management service |
| **Auth key expiry** | Transfer requests do not expire automatically — cancel unused transfers to maintain security hygiene                  |

***

## Next Steps

<CardGroup>
  <Card title="Create a Volume" href="/services/storage/create-volume">
    Provision a new volume in the destination project after transfer
  </Card>

  <Card title="Volume Snapshots" href="/services/storage/snapshots">
    Create a snapshot before transferring to preserve a copy in the source project
  </Card>

  <Card title="Attach / Detach" href="/services/storage/attach-volume">
    Attach the transferred volume to an instance in the destination project
  </Card>

  <Card title="Volume Types" href="/services/storage/volume-types">
    Understand storage tiers and verify the transferred volume type is available
  </Card>
</CardGroup>


# Volume Types and Storage Tiers
Source: https://docs.xloud.tech/services/storage/volume-types

Understand Xloud Block Storage volume types, storage performance tiers, and how to select the right tier for your workload requirements.

## Overview

A <Tooltip>volume type</Tooltip> is the mechanism through which administrators expose storage tiers to users. Each type maps to a backend driver and optionally enforces I/O quality-of-service limits. When creating a volume, selecting the appropriate type ensures the data lands on the correct hardware and receives the expected performance characteristics.

<Note>
  **Prerequisites**

  * Volume types are defined by your administrator
  * Contact your administrator if the required type is not available in your project
</Note>

***

## Storage Tiers

Xloud Block Storage supports tiered storage configurations. Each tier corresponds to a
hardware class with distinct performance characteristics:

| Tier         | Hardware          | IOPS (4K Random)    | Throughput        | Best For                                                             |
| ------------ | ----------------- | ------------------- | ----------------- | -------------------------------------------------------------------- |
| **NVMe**     | PCIe NVMe SSDs    | 100,000 – 1,000,000 | 3 – 14 GB/s       | Transactional databases, low-latency caches, real-time analytics     |
| **SSD**      | SATA/SAS SSDs     | 10,000 – 100,000    | 500 MB/s – 2 GB/s | Web application data, instance boot disks, general-purpose workloads |
| **Standard** | 7.2K/10K RPM HDDs | 150 – 500           | 100 – 300 MB/s    | Archival storage, bulk data, infrequent access                       |

<Tip>
  Select the tier based on your workload's I/O profile — not just raw capacity.
  Placing a high-transaction database on a Standard (HDD) type will result in
  significant latency and throughput bottlenecks. Use NVMe for latency-sensitive
  workloads and SSD for most application data.
</Tip>

***

## View Available Volume Types

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    Volume types are visible when creating a volume. Navigate to **Storage > Volumes**
    and click **Create Volume**. The **Volume Type** selection table in the create form
    shows all types available to your project with their properties.

    <Note>
      There is no standalone Volume Types listing page for regular users. Volume types
      are managed by administrators through the admin view at **Storage > Volume Types**.
    </Note>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="List volume types" icon="list">
        ```bash title="List all available volume types" theme={null}
        openstack volume type list --long
        ```

        ```bash title="Show details and extra specs for a type" theme={null}
        openstack volume type show <type-name>
        ```

        The `extra_specs` field shows backend-specific properties such as:

        * `volume_backend_name` — routes volumes to the correct backend
        * `capabilities:replication_enabled` — whether replication is enabled
        * `qos` — associated QoS spec name
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Select a Volume Type for Your Workload

<AccordionGroup>
  <Accordion title="Databases (MySQL, PostgreSQL, MongoDB)" icon="database">
    **Recommended tier**: NVMe or SSD

    Databases require consistent low-latency random I/O. NVMe is the preferred choice
    for production databases with high transaction rates. SSD provides an acceptable
    balance for development and staging environments.

    | Database                     | Recommended Type | Rationale                                     |
    | ---------------------------- | ---------------- | --------------------------------------------- |
    | MySQL / MariaDB (production) | `nvme`           | High write IOPS for WAL and data pages        |
    | PostgreSQL (production)      | `nvme`           | Random read/write for shared\_buffers and WAL |
    | MongoDB                      | `ssd`            | Mixed read/write, benefits from SSD caching   |
    | Development / staging        | `ssd`            | Cost-effective for lower traffic              |
  </Accordion>

  <Accordion title="Web and Application Servers" icon="server">
    **Recommended tier**: SSD

    Web application data — configuration, assets, application state — requires moderate
    IOPS with good throughput. SSD types deliver an optimal balance of performance and
    cost for these workloads.
  </Accordion>

  <Accordion title="Archival and Bulk Storage" icon="archive">
    **Recommended tier**: Standard (HDD)

    Log archives, data warehouses, and infrequently accessed bulk storage do not
    require high IOPS. Standard HDD-backed volumes provide large capacity at lower
    cost per GiB.
  </Accordion>

  <Accordion title="Boot Disks (Instance Root Volumes)" icon="hard-drive">
    **Recommended tier**: SSD

    Instance boot volumes are read-intensively at startup and write-intensively during OS
    operations. SSD provides the performance needed for fast boot times and responsive
    OS operations.
  </Accordion>
</AccordionGroup>

***

## Encryption-Enabled Types

Some volume types have encryption configured by your administrator. Volumes created
from an encrypted type are automatically encrypted at creation — no additional action
is required from you.

<Note>
  Encrypted volume types require the Xloud Key Management service to be accessible.
  If encryption is required for compliance, confirm with your administrator that an
  encrypted type is available and that key management is configured for your environment.
</Note>

```bash title="Check if a volume type has encryption configured" theme={null}
openstack volume type show <type-name> -c encryption
```

***

## Next Steps

<CardGroup>
  <Card title="Create a Volume" href="/services/storage/create-volume">
    Create a volume using the appropriate type for your workload
  </Card>

  <Card title="Volume Types Admin" href="/services/storage/volume-types-admin">
    Administrators: create types, set backend associations, and configure QoS
  </Card>

  <Card title="Storage Tiers (Admin)" href="/services/storage/storage-tiers">
    Configure NVMe, SSD, and HDD tier mappings for your storage cluster
  </Card>

  <Card title="Encryption (Admin)" href="/services/storage/encryption">
    Enable at-rest encryption on volume types using the key management service
  </Card>
</CardGroup>


# Volume Types and QoS (Admin)
Source: https://docs.xloud.tech/services/storage/volume-types-admin

Create and manage Xloud Block Storage volume types, set backend associations via extra specs, and configure QoS policies to enforce I/O limits per storage tier.

## Overview

Volume types are the mechanism through which administrators expose storage tiers to users. Each volume type maps to a specific backend through the `volume_backend_name` extra spec and optionally enforces I/O quality-of-service limits. Well-defined volume types are essential for predictable workload performance and efficient backend utilization.

<Warning>
  **Administrator Access Required** — This operation requires the `admin` role. Contact your
  Xloud administrator if you do not have sufficient permissions.
</Warning>

<Note>
  **Prerequisites**

  * Administrator credentials with the `admin` role
  * At least one storage backend configured and registered with the scheduler
  * Backend names available via `openstack volume backend pool list --long`
</Note>

***

## Create a Volume Type

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Navigate to Volume Types" icon="compass">
        Navigate to
        **Storage > Volume Types** (admin view).
      </Step>

      <Step title="Create the type" icon="plus">
        Click **Create Volume Type**. Enter a descriptive name (e.g., `nvme`, `ssd`,
        `standard`) and an optional description. Click **Confirm**.
      </Step>

      <Step title="Set the backend association" icon="link">
        In the volume type list, click the type name to open its details page. Under
        **Extra Specs**, click **Create** and add:

        | Key                   | Value            | Purpose                               |
        | --------------------- | ---------------- | ------------------------------------- |
        | `volume_backend_name` | `<backend-name>` | Routes volumes to the correct backend |

        <Tip>
          The value of `volume_backend_name` must exactly match the `volume_backend_name`
          configured in the backend driver settings. Use `openstack volume backend pool list`
          to confirm the correct name.
        </Tip>
      </Step>

      <Step title="Verify" icon="circle-check">
        Create a test volume using the new type and confirm it reaches `available` status:

        ```bash title="Test volume type" theme={null}
        openstack volume create --size 1 --type <type-name> test-vol
        openstack volume show test-vol -c status -c type
        openstack volume delete test-vol
        ```

        <Check>Volume type created and validated against the backend.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Authenticate" icon="key">
        Source your credentials file to authenticate with the Xloud platform:

        ```bash title="Load credentials" theme={null}
        source openrc.sh
        ```

        <Tip>
          Your administrator provides the RC (credentials) file for your project. See [CLI Setup](/cli-setup) for configuration details.
        </Tip>
      </Step>

      <Step title="Create the volume type" icon="plus">
        ```bash title="Create volume type" theme={null}
        openstack volume type create \
          --description "NVMe-backed high-performance storage" \
          nvme
        ```
      </Step>

      <Step title="Set the backend association" icon="link">
        ```bash title="Set extra spec for backend routing" theme={null}
        openstack volume type set \
          --property volume_backend_name=nvme-backend \
          nvme
        ```
      </Step>

      <Step title="Set as default (optional)" icon="star">
        ```bash title="Set default volume type" theme={null}
        openstack volume type set --default ssd
        ```

        The default type is used when users create volumes without specifying a type.
      </Step>

      <Step title="Verify" icon="circle-check">
        ```bash title="List volume types with properties" theme={null}
        openstack volume type list --long
        ```

        <Check>Volume type visible with correct backend association.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Configure QoS Policies

Quality-of-Service specs enforce I/O limits on volumes associated with a type. QoS
policies are applied at the hypervisor (`front-end`), at the storage backend (`back-end`),
or at both layers.

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps>
      <Step title="Create a QoS Spec" icon="gauge">
        Navigate to **Storage > Volume Types** (QoS Specs tab, admin view). Click **Create QoS Spec**.
        Enter a name (e.g., `ssd-standard-qos`) and set the consumer:

        | Consumer    | Effect                                                  |
        | ----------- | ------------------------------------------------------- |
        | `front-end` | Limits enforced at the hypervisor (recommended for RBD) |
        | `back-end`  | Limits enforced at the storage backend                  |
        | `both`      | Limits enforced at both layers                          |
      </Step>

      <Step title="Add I/O limit specs" icon="sliders">
        After creation, open the QoS Spec and add limits:

        | Key               | Example Value | Description                             |
        | ----------------- | ------------- | --------------------------------------- |
        | `total_iops_sec`  | `5000`        | Maximum combined read + write IOPS      |
        | `read_iops_sec`   | `3000`        | Read IOPS limit                         |
        | `write_iops_sec`  | `2000`        | Write IOPS limit                        |
        | `total_bytes_sec` | `524288000`   | Throughput cap in bytes/sec (500 MiB/s) |
        | `read_bytes_sec`  | `314572800`   | Read throughput cap (300 MiB/s)         |
        | `write_bytes_sec` | `209715200`   | Write throughput cap (200 MiB/s)        |
      </Step>

      <Step title="Associate with a volume type">
        Switch to the **Volume Types** tab. On the target volume type row, click
        **Manage QoS Spec** (the first row action). In the dialog, select the
        QoS spec from the table and click **Confirm**.

        <Check>QoS spec associated with volume type — limits active on all new volumes.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps>
      <Step title="Create a QoS spec" icon="gauge">
        ```bash title="Create QoS spec for SSD tier" theme={null}
        openstack volume qos create \
          --consumer front-end \
          --property total_iops_sec=10000 \
          --property total_bytes_sec=1073741824 \
          ssd-standard-qos
        ```
      </Step>

      <Step title="Associate with volume type" icon="link">
        ```bash title="Associate QoS spec with volume type" theme={null}
        openstack volume qos associate ssd-standard-qos ssd
        ```
      </Step>

      <Step title="Verify association" icon="circle-check">
        ```bash title="Show volume type with QoS" theme={null}
        openstack volume type show ssd
        ```

        <Check>Volume type `ssd` is now enforcing QoS limits for all new volumes.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Restrict Volume Type Access

By default, volume types are available to all projects. Sensitive types (e.g., encrypted
or high-performance tiers) can be restricted to specific projects:

```bash title="Make a volume type private" theme={null}
openstack volume type set --private <type-name>
```

```bash title="Grant access to a specific project" theme={null}
openstack volume type access add \
  --project <project-id> \
  <type-name>
```

```bash title="List projects with access" theme={null}
openstack volume type access list <type-name>
```

***

## Recommended QoS Values by Tier

| Tier         | total\_iops\_sec | total\_bytes\_sec      | Consumer    |
| ------------ | ---------------- | ---------------------- | ----------- |
| **NVMe**     | 50,000 – 200,000 | 4,294,967,296 (4 GB/s) | `front-end` |
| **SSD**      | 5,000 – 20,000   | 1,073,741,824 (1 GB/s) | `front-end` |
| **Standard** | 500 – 2,000      | 314,572,800 (300 MB/s) | `front-end` |

<Tip>
  Start with conservative limits and adjust based on observed workload behavior. Overly
  restrictive QoS on shared backends can throttle legitimate I/O and cause application
  timeouts.
</Tip>

***

## Next Steps

<CardGroup>
  <Card title="Storage Tiers" href="/services/storage/storage-tiers">
    Configure NVMe, SSD, and HDD tier mappings and set the default volume type
  </Card>

  <Card title="Encryption" href="/services/storage/encryption">
    Enable at-rest encryption on volume types
  </Card>

  <Card title="Storage Backends" href="/services/storage/storage-backends">
    Configure and manage the underlying backend drivers
  </Card>

  <Card title="Volume Types (User)" href="/services/storage/volume-types">
    User guide for selecting the correct volume type for a workload
  </Card>
</CardGroup>


# Compute Videos
Source: https://docs.xloud.tech/videos/compute

Video tutorials for creating, migrating, and tagging compute instances on the Xloud Dashboard.

## Create an Instance

4-step wizard walkthrough — Base Config, Network Config, System Config, and Confirm Config. Pick image or boot source, attach networks and security groups, set login credentials, and review quota before submit.

<iframe title="How to Create an Instance on Xloud" />

[Read the full guide](/services/compute/launch-instance)

***

## Live Migrate an Instance

Move a running VM to another compute host via the Live Migrate action. Pick the target host from the list or leave it empty to let the scheduler decide. Available when the instance is Active or Paused.

<iframe title="How to Live Migrate an Instance on Xloud" />

[Read the full guide](/services/compute/live-migration)

***

## Modify Instance Tags

Add or remove tags on a running instance for organization, filtering, and automation. Supported when the VM is in active, paused, suspended, or stopped state — up to 50 tags per instance.

<iframe title="How to Tag Instances on Xloud" />

[Read the full guide](/services/compute/instance-tagging)

***

## Enable vTPM and Secure Boot

Harden an instance at launch with a Virtual TPM and UEFI Secure Boot — for Windows 11, BitLocker, LUKS disk encryption, and measured boot. Configure both from the Create Instance wizard's System Config → Advanced Options. Requires Xloud KMS.

<iframe title="How to Enable vTPM and Secure Boot on Xloud" />

[Read the full guide](/services/compute/vtpm-secure-boot)


# Video Tutorials
Source: https://docs.xloud.tech/videos/index

Step-by-step video walkthroughs for deploying and managing the Xloud Platform — XDEPLOY, instances, storage, networking, security, and more.

Video tutorials demonstrating key workflows on the Xloud Platform.

***

## Featured — Full XDEPLOY Walkthrough

<div>
  <div>
    End-to-end deployment of Xloud with XDEPLOY — from first login after XOS installation to a live multi-node production cluster running XAVS, XSDS, and XMS. Covers activation, host setup, XSDS storage, cluster configuration, load balancer, monitoring, advanced features, cluster licensing, and the full Operations workflow.

    [Read the full XDEPLOY guide](/deployment)
  </div>

  <iframe title="How to Deploy Xloud with XDEPLOY — Full Walkthrough" />
</div>

***

## Browse by Category

<CardGroup>
  <Card title="Compute" icon="server" href="/videos/compute">
    Create instances, live migrate, clone, tag, and scale VMs
  </Card>

  <Card title="Storage" icon="hard-drive" href="/videos/storage">
    Create volumes, backups, and restore from backup
  </Card>

  <Card title="Networking & DNS" icon="network" href="/videos/networking">
    DNS zones, record sets, and network configuration
  </Card>

  <Card title="Security & Identity" icon="shield" href="/videos/security">
    RBAC permissions, secrets management, and image uploads
  </Card>

  <Card title="Monitoring" icon="activity" href="/videos/monitoring">
    Activity Log auditing, dashboards, and alerts in Monitor Center
  </Card>
</CardGroup>


# Monitoring Videos
Source: https://docs.xloud.tech/videos/monitoring

Video tutorials for observability and activity auditing on the Xloud Platform — XIMP dashboards, alerts, and Activity Log.

## Audit Activity in Monitor Center

See who did what, when, and whether it succeeded. The Activity Log timeline in Monitor Center shows every action across Xloud services with live updates, filters, and a detail panel for each event.

<iframe title="How to Audit Activity on Xloud" />

[Read the full guide](/services/monitoring/user-guide/activity-log)


# Networking & DNS Videos
Source: https://docs.xloud.tech/videos/networking

Video tutorials for DNS zone management and record sets on the Xloud Dashboard.

## Manage DNS Zones and Record Sets

Create a DNS zone (primary or secondary) with a name, email, and TTL, then add record sets. Supported record types include A, AAAA, CAA, CNAME, MX, NS, PTR, SOA, SPF, SRV, SSHFP, and TXT.

<iframe title="How to Manage DNS Zones on Xloud" />

[Read the full guide](/services/dns/create-zone)


# Security & Identity Videos
Source: https://docs.xloud.tech/videos/security

Video tutorials for Xloud SIEM, RBAC, secrets, image uploads, User Center, and 2FA on the Xloud Dashboard.

## Review Security Posture with Xloud SIEM

Walk through the unified Security Posture dashboard — Wazuh threats, Lynis hardening, OpenSCAP compliance, volume encryption, vulnerabilities, microsegmentation, and live alerts, all in Monitor Center. Admin-only view.

<iframe title="How to Review Security Posture on Xloud" />

[Read the full guide](/security/xloud-siem)

***

## Edit Permissions on a Custom Role (Extended RBAC)

Walk through the RBAC Permissions page in the admin view — pick a Custom role, toggle privileges in the service-tabbed matrix, attach tag conditions, preview effective rules, and save. Built-in system roles (admin, member, reader, service) are protected from edits.

<iframe title="How to Edit Permissions on a Custom Role with Extended RBAC on Xloud" />

[Read the full guide](/services/identity/extended-rbac)

***

## Create a Secret

Store passwords, API tokens, encryption keys, and certificates in the Key Manager. Supports Opaque, Symmetric, Public, Private, Certificate, and Passphrase secret types with configurable algorithms and payload content types.

<iframe title="How to Create a Secret on Xloud" />

[Read the full guide](/services/key-manager/store-secrets)

***

## Upload an Image

Upload an OS image to the Image Service for VM deployment. Supports RAW, QCOW2, and ISO disk formats with Public, Private, or Shared visibility — upload a file directly or provide a file URL.

<iframe title="How to Upload an Image on Xloud" />

[Read the full guide](/services/images/upload-image)

***

## Set Up Two-Factor Authentication

Protect your Xloud account with Two-Factor Authentication from User Center → Security. Scan the QR in any authenticator app (Google Authenticator, Microsoft Authenticator, Authy, 1Password, Aegis), verify the 6-digit code, and save the 10 one-time recovery codes — every sign-in from then on needs a code from your phone.

<iframe title="How to Set Up 2FA and User Center on Xloud" />

[Read the full guide](/services/dashboard/user-guide/user-center)


# Storage Videos
Source: https://docs.xloud.tech/videos/storage

Video tutorials for volume backups, restores, and file-level recovery on the Xloud Dashboard.

## Back Up and Restore a Volume

Create a volume backup (full or incremental mode) and restore it to the original volume or a new one. Backups can be taken from in-use volumes and restored any time from the Backup detail page.

<iframe title="How to Back Up and Restore Volumes on Xloud" />

[Read the full guide](/services/storage/backups)

***

## Recover Files from a Snapshot

Pull individual files out of a volume or instance snapshot — no full-volume restore needed. Open the snapshot's filesystem in the Dashboard, navigate folders, and download a single file or a ZIP of selected files.

<iframe title="How to Recover Files from a Snapshot on Xloud" />

[Read the full guide](/services/storage/file-level-restore)