kubevirt-community-stack

Kubevirt Community Stack

Create Kubevirt VMs via Helm
for use with ArgoCD, Argo Workflows, KEDA, ClusterAPI, Tekton etc...

Who is this for:

The Kubevirt-Community-Stack may be of interest if you:

operate one or more physical computers which you would like to split into smaller virtual machiens.
are already running kubernetes to orchestrate container workloads
are already in the ArgoCD or Tekton ecosystem and/or work primarily with some other Helm-based tooling.
want/need fully-featured VMs for hardware emulation, hardware-passthrough, Virtual Desktops, vGPU which are not suppoted by Micro-VMs like Firecracker
want to integrate Kubevirt into your existing infrastructure without needing to adopt a full platform like OpenShift Virtuazation, HarvesterHCI, StarlingX, or KubeSphere etc…
want to install and operate Kubevirt on an existing system withhout needing to re-image it with an installer ISO.

Component charts

Kubervirt

Kubevirt is a Kubernetes Virtualization API and runtime which controls QEMU/KVM virtual machine instances and provides the CRDs that define them. It's distrubuted as a Kubernetes Operator which is install via the kubevirt chart.

Kubevirt CDI

The Containerized Data Importer can pull virtual machine images, ISO files, and other types of bootable media from sources like S3, HTTP, or OCI images. This data is then written to PVCs which are mounted as disks. For examples of various ways to use the CDI, see the notes in Argocd-Apps

Cloud-Init

The Cloud-init helm chart allows the user to define the specification of a linux-based vm's operating system as code. In addition to basec cloud-init functions, his chart provides some extra functionality via an initjob that makes cloud-init more GitOps friendly.

Additional Features: - Regex values using existing secrets or environmental variables via envsubst - Create random user passwords or use an existing secret - Download files from a URL - Base64 encode + gzip your `write_files` content - Populate Wireguard configuration values from an existsing secret - Track the total size of user-data and check file for valid syntax

Kubevirt VM

The Kubevirt-VM Chart allows a user to easily template a Kubevirt VirtualMachine or VirtualMachinePool and its associated resources sudch as Disks, DataVolumes, Horizontal Pod Autoscaler, Network Policies, Service, Ingres, Probes, and Cloud-init data (via bundled cloud-init subchart).

Kubevirt Manager

This is a community-developed web-ui which allows users to create, manage, and interact with virtual machines running in Kubevirt. See their official docs at kubevirt-manager.io

Cluster API Operator & Addons

Cluster API provides a standardised kubernetes-native interface for creating k8s clusters using a wide variety of providers. The combined chart can install the Cluster API Operator as well as bootstrap the Cluster API Kubevirt Provider which allows creating k8s clusters from the CLI or as YAML using Kubevirt VMs. Cluster-api-provider-kubevirt also includes cloud-provider-kubevirt which enables the exposeure of LoadBalancer type services within tenant clusters to the host cluster. This negates the need for a dedicated loadbalancer such as MetalLB inside the tenant cluster.

See CAPI.md for a basic walkthrough of creating a CAPI-based tenant cluster.

CAPI Cluster

The CAPI Cluster helm chart provides a way to create workload clusters using the Kubevirt infrastructure, Kubeadm Bootstrap + ControlPlane, and Helm providers.

Dependencies

libvirt-clients

This utility will audit a host machine and report what virtualisation capabilities are available - Installation ```bash sudo apt-get install -y libvirt-clients ``` - Usage ```console $ virt-host-validate qemu QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS QEMU: Checking if device /dev/vhost-net exists : PASS QEMU: Checking if device /dev/net/tun exists : PASS ```

virtctl

virtctl is the command-line utility for managing Kubevirt resources. It can be installed as a standalone CLI or as a Kubectl plugin via krew. - Standalone ```bash export VERSION=v0.41.0 wget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-linux-amd64 ``` - Plugin ```bash kubectl krew install virt ```

clusterctl

The clusterctl CLI tool handles the lifecycle of a Cluster API management cluster. ```bash curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.2/clusterctl-linux-amd64 -o clusterctl sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl ```

Install The Kubevirt-Community-Stack

Install the combined chart (kubevirt-stack).

  helm repo add kubevirt https://cloudymax.github.io/kubevirt-community-stack
  helm install kubevirt-stack kubevirt/kubevirt-stack \
    --namespace kubevirt \
    --create-namespace

Expand to see individual chart installation

- kubevirt: Installs the Kubevirt Operator. ```bash helm repo add kubevirt https://cloudymax.github.io/kubevirt-community-stack helm install kubevirt kubevirt/kubevirt \ --namespace kubevirt \ --create-namespace ``` - Cluster API Operator: Installs the Cluster API Operator. ```bash Work in progress. ``` - kubevirt-cdi: Install the Containerized Data Importer. ```bash helm repo add kubevirt https://cloudymax.github.io/kubevirt-community-stack helm install kubevirt-cdi kubevirt/kubevirt-cdi \ --namespace cdi \ --create-namespace ``` - kubevirt-manager: Deploy the Kubevirt-Manager UI ```bash # Customize your own values.yaml before deploying helm repo add kubevirt https://cloudymax.github.io/kubevirt-charts helm install kubevirt-manager kubevirt/kubevirt-manager \ --fnamespace kubevirt-manager \ --create-namespace ```

Creating a VM

This is a qucik walkthrough of how I create VMs using kubevirt-community-stack. All the configuration for the VM happens in the values.yaml file of the Kubevirt-VM Chart chart.

From this file we can configure the VM, Disks, Cloudinit config, services, probes and more.

With the command or file below we will:

Create a new VM named example with with 2 cores and 2Gi of RAM.

Create a 16Gi PVC named harddrive which holds a debian12 cloud-image.

Define a user named example and assign the user some groups and a random password which will be stored in a secret.

Save our user-data as a secret named example-user-data

Update apt-packes and install docker.

Run the nginx docker container with port 8080 exposed from the container to the VM

Define a service over which to expose port 8080 from the VM to the host.

Requirements

- you are running on bare-metal, not inside a VM - you set `cpuManagerPolicy: static` in your kubelet config - you have `yq` and either `virtctl` or `krew virt` installed - your host system passes all `virt-host-validate qemu` checks for KVM ```console QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS ```

Command Line method:

```bash helm repo add kubevirt https://cloudymax.github.io/kubevirt-community-stack helm install example kubevirt/kubevirt-vm \ --namespace kubevirt \ --set virtualMachine.name="example" \ --set virtualMachine.namespace="kubevirt" \ --set virtualMachine.machine.vCores=2 \ --set virtualMachine.machine.memory.base="2Gi" \ --set disks[0].name="harddrive" \ --set disks[0].type="disk" \ --set disks[0].bus="virtio" \ --set disks[0].bootorder=2 \ --set disks[0].readonly="false" \ --set disks[0].pvsize="16Gi" \ --set disks[0].pvstorageClassName="fast-raid" \ --set disks[0].pvaccessMode="ReadWriteOnce" \ --set disks[0].source="url" \ --set disks[0].url="https://buildstars.online/debian-12-generic-amd64-daily.qcow2" \ --set cloudinit.hostname="example" \ --set cloudinit.namespace="kubevirt" \ --set cloudinit.users[0].name="example" \ --set cloudinit.users[0].groups="users\, admin\, docker\, sudo\, kvm" \ --set cloudinit.users[0].sudo="ALL=(ALL) NOPASSWD:ALL" \ --set cloudinit.users[0].shell="/bin/bash" \ --set cloudinit.users[0].lock_passwd="false" \ --set cloudinit.users[0].password.random="true" \ --set cloudinit.secret_name="example-user-data" \ --set cloudinit.package_update="true" \ --set cloudinit.packages[0]="docker.io" \ --set cloudinit.runcmd[0]="docker run -d -p 8080:80 nginx" \ --set service[0].name="example" \ --set service[0].type="NodePort" \ --set service[0].externalTrafficPolicy="Cluster" \ --set service[0].ports[0].name="nginx" \ --set service[0].ports[0].port="8080" \ --set service[0].ports[0].targePort="8080" \ --set service[0].ports[0].protocol="TCP" \ --create-namespace ```

Values File method:

```bash helm repo add kubevirt https://cloudymax.github.io/kubevirt-community-stack cat < example.yaml --- virtualMachine: name: example namespace: kubevirt machine: vCores: "2" memory: base: "2Gi" disks: - name: harddrive type:disk bus: virtio bootorder: 2 readonly: false pvsize: 16Gi pvstorageClassName: fast-raid pvaccessMode: ReadWriteOnce source: url url: "https://buildstars.online/debian-12-generic-amd64-daily.qcow2" cloudinit: hostname: example namespace: kubevirt users: - name: example groups: "users, admin, docker, sudo, kvm" sudo: "ALL=(ALL) NOPASSWD:ALL" shell:"/bin/bash" lock_passwd:"false" password: random: "true" secret_name: "example-user-data" package_update: "true" packages: - docker.io runcmd: - "docker run -d -p 8080:80 nginx" service: name: example type: ClusterIP externalTrafficPolicy: Cluster ports: - name: "nginx" port: "8080" targePort: "8080" protocol: "TCP" EOF ``` - Install VM as a helm-chart (or template it out as manifests): ```bash helm install example kubevirt/kubevirt-vm \ --namespace kubevirt \ --create-namespace \ -f example.yaml ``` </details> 1. Find the secret create to hold our user's password: ```bash kubectl get secret example-password -n kubevirt -o yaml \ |yq '.data.password' |base64 -d ``` 2. Connect to the vm over console & login as user "example": ```console kubectl virt console example -n kubevirt Successfully connected to example console. The escape sequence is ^] example login: example Password: ``` 3. Port-forward the nginx service and vistit in your browser: ```bash kubectl port-forward service/example -n kubevirt 8080:8080 --address 0.0.0.0 ``` 4. Uninstall/Delete the VM ```bash helm uninstall example ``` ## Uninstall In the event that Kubevirt does not uninstall gracefully, you may need to perform the following steps: ```bash export RELEASE=v0.17.0 # --wait=true should anyway be default kubectl delete -n kubevirt kubevirt kubevirt --wait=true # this needs to be deleted to avoid stuck terminating namespaces kubectl delete apiservices v1.subresources.kubevirt.io # not blocking but would be left over kubectl delete mutatingwebhookconfigurations virt-api-mutator # not blocking but would be left over kubectl delete validatingwebhookconfigurations virt-operator-validator # not blocking but would be left over kubectl delete validatingwebhookconfigurations virt-api-validator kubectl delete -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml --wait=false # Find hanging resources kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n kubevirt # If namespace is stuck kubectl get namespace "kubevirt" -o json | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" | kubectl replace --raw /api/v1/namespaces/kubevirt/finalize -f - ```

This site is open source. Improve this page.