A look into our use of Kubernetes within the VMware VPS platform

Kubernetes at GleSYS

For the past few years, we at GleSYS have assisted customers in running their applications on production-ready Kubernetes clusters. These clusters are built according to industry best practices and fully supported by VMware, who provide the underlying technology for the service.

Our customers use the managed Kubernetes service to run various types of applications, for example:

• AI/ML applications
• Scalable web applications
• Stateful databases
• Cloud-native applications
• Event-driven applications

In this article, we explore how we use our Kubernetes offering to unlock the potential of event-driven automation.

The 10,000 foot view

We leverage Kubernetes to trigger serverless functions in response to events from our VMware VPS platform. This enables us to automate manual tasks and streamline operations.

Here are some common use cases:

• Trigger an alarm in the monitoring system when a login attempt is registered with a break glass account (vsphere.local\administrator).
• Disable/enable alerts for a host in the monitoring system when it has been automatically shut down or powered on by vSphere Distributed Power Management (DPM).
• Send an automated email to VM owners when their VM has been affected by a host outage.
• Send a Slack notification when a VM has been running at 100% utilization for an extended period.

How does this work under the hood?

Our Kubernetes cluster consists of 3 worker nodes. The Kubernetes nodes are virtual machines (VMs) running on the VMware platform. As a result, we automatically benefit from vSphere's features, including High Availability, Distributed Resource Scheduler, and built-in security features such as instruction isolation and memory isolation.

To interact with the Kubernetes cluster, we are using kubectl (k as an alias):

$ k get nodes
NAME              STATUS   ROLES           AGE   VERSION
k8s-worker-01     Ready    <none>          82d   v1.27.10+vmware.1-fips.1
k8s-worker-02     Ready    <none>          82d   v1.27.10+vmware.1-fips.1
k8s-worker-03     Ready    <none>          82d   v1.27.10+vmware.1-fips.1   

On top of the cluster, we have installed Knative which enables us to build an event driven architecture. In this article, we won't describe the installation steps, but you can see the core Knative components running in our cluster:

$ k get pods -n knative-eventing
NAME                                          READY   STATUS    RESTARTS   AGE
eventing-controller-5955fd4d-7cwqp            1/1     Running   0          81d
eventing-webhook-5546b48456-7txqt             1/1     Running   0          81d
rabbitmq-broker-controller-75d6487bf8-8jg69   1/1     Running   0          81d
rabbitmq-broker-webhook-764b6bc9f8-9lb2q      1/1     Running   0          81d

$ k get pods -n knative-serving
NAME                                      READY   STATUS    RESTARTS   AGE
activator-574844cbbf-gdbkr                1/1     Running   0          81d
autoscaler-9ddbcfcc5-lj4cr                1/1     Running   0          81d
controller-6997848b67-pc974               1/1     Running   0          81d
net-kourier-controller-76c68964f7-nhwts   1/1     Running   0          81d
webhook-6c59d8bf9b-h6kqc                  1/1     Running   0          81d

Now that the building blocks are in place, we must deploy an event source. This source will be responsible for receiving events from the VMware environment and delivering them to a function called "kn-glesys-demo" which we will create in the next step:

$ k apply -f - <<EOF
apiVersion: sources.tanzu.vmware.com/v1alpha1
kind: VSphereSource
metadata:
  name: vcenter-lab
  namespace: vmware-events
spec:
  address: https://<vcenter_address>
  secretRef:
    name: vcenter-credentials
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: kn-glesys-demo
EOF

Next, we will deploy a function called "kn-glesys-demo" that prints out any event data it receives in JSON format:

$ k apply -f - <<EOF
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: kn-glesys-demo
  namespace: vmware-events
spec:
  template:
    spec:
      containers:
        - image: ghcr.io/vmware-samples/vcenter-event-broker-appliance/kn-go-echo:1.1
EOF

$ k get ksvc -n vmware-events
NAME             URL                                                      LATESTCREATED           LATESTREADY            READY   REASON
kn-glesys-demo   http://kn-glesys-demo.vmware-events.svc.cluster.local    kn-glesys-demo-00001    kn-glesys-demo-00001   True

To confirm that the function is working as expected, check the container logs to verify that event data is being printed to stdout:

$ k logs -l serving.knative.dev/service=kn-glesys-demo -c user-container -n vmware-events

Context Attributes,
  specversion: 1.0
  type: com.vmware.vsphere.VmPoweredOnEvent.v0.echo
  source: https://<vcenter_address>
  id: 9665046
  time: 2024-09-03T08:59:00.438Z
  datacontenttype: application/json
Extensions,
  eventclass: event
  vsphereapiversion: 8.0.3.0
Data,
  {
    "Key": 9665046,
    "ChainId": 9665044,
    "CreatedTime": "2024-09-03T08:59:00.438Z",
    ...
    "FullFormattedMessage": "wps******* on esx01 in GleSYS Lab has powered on",
  }

To demonstrate a more realistic example, let's deploy a new version of the function. This latest version will perform a more practical task of sending an email to the owner of a virtual machine if their VM has been restarted due to a host outage:

$ k apply -f - <<EOF
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: kn-glesys-demo
  namespace: vmware-events
spec:
  template:
    spec:
      containers:
        - image: <private-registry>/events/kn-go-vm-restarted-by-ha:1.0
EOF

$ k get ksvc -n vmware-events
NAME             URL                                                      LATESTCREATED           LATESTREADY            READY   REASON
kn-glesys-demo   http://kn-glesys-demo.vmware-events.svc.cluster.local    kn-glesys-demo-00002    kn-glesys-demo-00002   True

To ensure the new function is working correctly, simulate a host outage and verify if any emails are received:

[root@esx01:~] vsish -e set /reliability/crashMe/Panic 1

Bingo!

Subject: VMware HA has restarted your server (wps*******) on a new node
From: info@glesys.com

Hi,

The hardware node where your server (wps*******) is running has experienced a host failure. VMware HA (High Availability) has detected the failure and restarted your server on an alternative node in the cluster. Your workload should now be fully accessible and functional again.

We apologize for any problems this short outage may have caused you.

Best regards,
Your friends at GleSYS

One of the benefits of using Knative is the out-of-the-box auto-scaling functionality, where a function will scale to zero when there are no events that trigger it and scale it back up when needed:

$ k get pod -l serving.knative.dev/service=kn-glesys-demo -n vmware-events -w

NAME                                               READY   STATUS              RESTARTS   AGE
kn-glesys-demo-00002-deployment-56f9c87f9b-9qggw   2/2     Running             0          63s
kn-glesys-demo-00002-deployment-56f9c87f9b-9qggw   2/2     Terminating         0          2m3s
kn-glesys-demo-00002-deployment-56f9c87f9b-9qggw   1/2     Terminating         0          2m31s
kn-glesys-demo-00002-deployment-56f9c87f9b-9qggw   0/2     Terminating         0          2m33s

kn-glesys-demo-00002-deployment-56f9c87f9b-9qggw   0/2     Pending             0          0s
kn-glesys-demo-00002-deployment-56f9c87f9b-9qggw   0/2     ContainerCreating   0          0s
kn-glesys-demo-00002-deployment-56f9c87f9b-9qggw   1/2     Running             0          1s
kn-glesys-demo-00002-deployment-56f9c87f9b-9qggw   2/2     Running             0          1s

Summary

This article provides an overview of how we utilize our Kubernetes offering to enable event-driven automation by triggering functions based on events from our VMware VPS platform.

Of course, we have demonstrated one of the many potential use cases for running applications on a Kubernetes cluster. If you're interested in learning more about our managed Kubernetes offering for hosting your applications, please contact our sales team, and we would be delighted to schedule a meeting.