> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xloud.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# Cluster Upgrades

> Upgrade Xloud K8SaaS clusters to newer Kubernetes versions — rolling upgrades, sequential version progression, and post-upgrade validation.

## Overview

Xloud K8SaaS supports rolling Kubernetes version upgrades. Master nodes are upgraded
first, followed by sequential worker node replacement — draining each node before
removing it and adding a replacement with the new version. This approach maintains
cluster availability throughout the upgrade process.

<Warning>
  Kubernetes does not support skipping minor versions. Upgrade sequentially through
  each minor version (e.g., 1.27 → 1.28 → 1.29). Attempting to skip a version
  (e.g., 1.27 → 1.29 directly) is not supported and will fail.
</Warning>

<Note>
  **Prerequisites**

  * A cluster in `CREATE_COMPLETE` or `UPDATE_COMPLETE` status
  * A cluster template with the target Kubernetes version available in your project
  * Sufficient compute quota to temporarily run extra nodes during the rolling replacement
</Note>

***

## Pre-Upgrade Checklist

<Steps titleSize="h3">
  <Step title="Verify the target template exists" icon="file-text">
    Confirm a cluster template with the target Kubernetes version is available:

    ```bash title="List available templates" theme={null}
    openstack coe cluster template list
    ```

    If the target version template does not exist, ask your administrator to create
    a public template or create a private one in your project.
  </Step>

  <Step title="Check cluster health" icon="heart-pulse">
    ```bash title="Verify cluster is healthy" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status -c health_status
    ```

    Proceed only when `status = CREATE_COMPLETE` or `UPDATE_COMPLETE` and
    `health_status = HEALTHY`.
  </Step>

  <Step title="Verify node readiness" icon="server">
    ```bash title="Check all nodes are Ready" theme={null}
    kubectl get nodes
    ```

    All nodes must show `STATUS: Ready` before starting the upgrade.
  </Step>

  <Step title="Notify application teams" icon="bell">
    Inform application owners that the cluster will undergo a rolling upgrade.
    While the upgrade is non-destructive, pods will be evicted during node replacement.
    Ensure workloads use multiple replicas and have Pod Disruption Budgets configured.
  </Step>
</Steps>

***

## Perform the Upgrade

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    <Steps titleSize="h3">
      <Step title="Navigate to the cluster" icon="compass">
        Navigate to
        **Container > Clusters**. Click your cluster name.
      </Step>

      <Step title="Start upgrade" icon="arrow-up">
        Click **Actions → Upgrade Cluster**. Select the target cluster template
        with the desired Kubernetes version and confirm.
      </Step>

      <Step title="Monitor progress" icon="clock">
        The cluster enters `UPDATE_IN_PROGRESS` status. The upgrade progresses through:

        1. Master nodes upgraded first (one at a time for HA clusters)
        2. Worker nodes drained and replaced sequentially

        <Tip>
          Use `kubectl get nodes -w` to watch node status changes in real time
          during the rolling replacement.
        </Tip>
      </Step>

      <Step title="Verify completion" icon="circle-check">
        The cluster returns to `UPDATE_COMPLETE` when all nodes are on the new version.

        <Check>All nodes show the new Kubernetes version in `kubectl get nodes`.</Check>
      </Step>
    </Steps>
  </Tab>

  <Tab title="CLI" icon="terminal">
    <Steps titleSize="h3">
      <Step title="Initiate upgrade" icon="arrow-up">
        ```bash title="Upgrade cluster to new template" theme={null}
        openstack coe cluster upgrade prod-cluster-01 \
          k8s-1.30-prod
        ```
      </Step>

      <Step title="Monitor progress" icon="clock">
        ```bash title="Check upgrade status" theme={null}
        watch -n 15 "openstack coe cluster show prod-cluster-01 \
          -f value -c status -c status_reason"
        ```

        ```bash title="Watch node replacement in kubectl" theme={null}
        kubectl get nodes -w
        ```
      </Step>

      <Step title="Verify all nodes on new version" icon="circle-check">
        ```bash title="Check Kubernetes version on all nodes" theme={null}
        kubectl get nodes \
          -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion'
        ```

        All nodes should show the same target version.

        <Check>All nodes show the target Kubernetes version and `STATUS: Ready`.</Check>
      </Step>
    </Steps>
  </Tab>
</Tabs>

***

## Post-Upgrade Validation

<Tabs>
  <Tab title="Dashboard" icon="gauge">
    After upgrade completes, verify:

    * Cluster shows `Health Status: HEALTHY`
    * All nodes listed in the cluster detail show `STATUS: Ready`

    <Check>Cluster is healthy and all nodes are at the target version.</Check>
  </Tab>

  <Tab title="CLI" icon="terminal">
    ```bash title="Verify cluster health" theme={null}
    openstack coe cluster show prod-cluster-01 \
      -f value -c status -c health_status
    ```

    ```bash title="Verify node versions" theme={null}
    kubectl get nodes -o wide
    ```

    ```bash title="Check system pod health" theme={null}
    kubectl get pods -n kube-system
    ```

    All system pods should show `STATUS: Running` or `Completed`.

    <Check>Cluster is `UPDATE_COMPLETE`, all nodes are `Ready`, and system pods are running.</Check>
  </Tab>
</Tabs>

***

## Next Steps

<CardGroup cols={2}>
  <Card title="Troubleshooting" href="/services/kubernetes/user-guide/troubleshooting" color="#197560">
    Resolve upgrade failures and post-upgrade issues.
  </Card>

  <Card title="Node Groups" href="/services/kubernetes/user-guide/node-groups" color="#197560">
    Manage specialized node pools after an upgrade.
  </Card>

  <Card title="Kubernetes Admin Guide — Certificates" href="/services/kubernetes/admin-guide/certificates" color="#197560">
    Rotate cluster CA certificates after major version upgrades.
  </Card>

  <Card title="Access Cluster" href="/services/kubernetes/user-guide/access-cluster" color="#197560">
    Refresh kubeconfig after a cluster upgrade if the CA was rotated.
  </Card>
</CardGroup>