It is essential to understand how it differs from Serverless technologies and even the traditional way of deploying on Kubernetes.

1. Serverless vs. Cloud Run: While both are serverless in nature, Cloud Run specifically focuses on running containerized applications. Unlike traditional serverless platforms (e.g., AWS Lambda, Google Cloud Functions), which are limited to specific runtimes and code formats, Cloud Run allows you to deploy any containerized application, giving you more flexibility.

2. Traditional Kubernetes vs. Cloud Run: Kubernetes (e.g., GKE) requires you to manage clusters, nodes, and scaling policies. Cloud Run, on the other hand, is fully managed, eliminating the need for cluster management. It automatically scales to zero when idle and scales up instantly during traffic spikes, making it more cost-effective and easier to use for lightweight, event-driven workloads.

Before we start with the deployment of our first application on Google Cloud Run, I want to show case the pricing plan and review that up to 2M requests are already free for the month. Even after that, it is only $0.04 / million request. Review detailed pricing here.

Now that we understand the basics of Google Cloud Run and how cheap is it to host our applications on Cloud Run, lets try creating a “Hello World“ application.

Pre-requisites:

1. Google Cloud Account

2. Billing Enabled

3. Google Cloud Run API Enabled

4. Gcloud CLI installed on local machine (you can also use cloud-shell)

Agenda:

During this elmental exploration journey of Deployment on Google Cloud Run we will go through the following concepts:

1. Enable Cloud Run API using CLI

2. Create a simple Node.js Application

3. Create a Dockerfile to containerize the application

4. Create and Push the Image to Google Artifact Repository

5. Deploy the Application

6. Cleanup by deleting service and images.

Cloud Run API setup

If you are already logged in with your gcloud cli, then list your account to verify the right account

gcloud auth list

You should see an output like:

Credentialed accounts:
 - myaccount@mydomain.com (active)

Now list the projects in your account to verify the right project is selected as current project or if required then create a project:

gcloud config list project

This will give an output like below:

[core]
project = nodejs-gcp-66776654433

For more information on gcloud CLI, review the cli documentation here.

Now that we have selected the right account and project, lets start by enabling the Google Cloud Run API. You can do it from the API and Services Section of the Google Cloud Console also by lets do it gcloud cli, run the following command:

gcloud services enable run.googleapis.com

It can take a moment to enable the API, once done, then let us set the right compute region and location environment variable to use later for docker commands.

export LOCATION="us-east-1
export GOOGLE_CLOUD_PROJECT="nodejs-gcp-66776654433"
gcloud config set compute/region $LOCATION

Here, the region will be your preferred region.

Write a Nodejs Hello World Application

To write an express-based nodejs application, you will need two files, a package.json to list all the dependencies and an index.js file with the application logic.

1. Create a package.json file and write the following content into it. (You can also use npm init, if you have npm installed on your machine)

    vi package.json
   

    {
      "name": "helloworld",
      "description": "Simple hello world sample in Node",
      "version": "1.0.0",
      "main": "index.js",
      "scripts": {
        "start": "node index.js"
      },
      "author": "Google LLC",
      "license": "Apache-2.0",
      "dependencies": {
        "express": "^4.17.1"
      }
    }
   

2. Create index.js file and write the following content into it:

    const express = require('express');
    const app = express();
    const port = process.env.PORT || 8080;
   
    app.get('/', (req, res) => {
      const name = process.env.NAME || 'World';
      res.send(`Hello ${name}!`);
    });
   
    app.listen(port, () => {
      console.log(`helloworld: listening on port ${port}`);
    });
   

   You may try to run it locally by first downloading the package dependencies using npm install and then running the command node index.js

Containerize the Application using Dockerfile

The Docker deamon uses a Dockerfile to create a docker image, which can be used later to run its containers. Create a file named Dockerfile in the same folder with the following content:

# Use the official lightweight Node.js 12 image.
# https://hub.docker.com/_/node
FROM node:12-slim

# Create and change to the app directory.
WORKDIR /usr/src/app

# Copy application dependency manifests to the container image.
# A wildcard is used to ensure copying both package.json AND package-lock.json (when available).
# Copying this first prevents re-running npm install on every code change.
COPY package*.json ./

# Install production dependencies.
# If you add a package-lock.json, speed your build by switching to 'npm ci'.
# RUN npm ci --only=production
RUN npm install --only=production

# Copy local code to the container image.
COPY . ./

# Run the web service on container startup.
CMD [ "npm", "start" ]

Now you need to build your docker image using the either the local Docker daemon, or if you are running these commands on your Google Cloud Shell then you can also use gcloud cli to build it on cloud. here are both the procedures:

1. Build Docker Image using Local Docker Daemon (Docker Desktop in most cases)

    docker build . -t gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld:1.0.0
    docker push gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld:1.0.0
   

   Remember, we created the GOOGLE_CLOUD_PROJECT environment variable already. In you cloud shell, you will not need to create it as it is available by default.

   The first command will build the image, where as the second command will push it to your GCR (Google Container Registry)

2. Build and push Docker image using gcloud cli by running the following command

    gcloud builds submit --tag gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld
   

   The above command will build the Image in cloud and even push it to the respective repository as well.

3. You may list the images by using the following commands

    gcloud container images list
    docker images
   

   You may use the first command to list the images on GCR, where as second command will list the images on your local machine.

4. Lets test the application by running the container locally

    docker run -d -p 8080:8080 gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld
   

   The above command will pull the Image from the GCR if not already available locally. In the second Step it will run a container using the pulled image. To test the application, do curl localhost:8080 you should see hello world message as a response from the container.

Deploy the Image as a Cloud Run Service

Deploying any Docker Image as a docker container on Google Cloud Run is fairly simple and straight forward, use the following command:

gcloud run deploy --image gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld 
--allow-unauthenticated --region=$LOCATION

We used the —allow-unauthenticated flag as we want to keep the application public. We used the same $LOCATION environment variable we created earlier.

On Success, you will get a service URL and other details as follows:

Service [helloworld] revision [helloworld-00001-xit] has been deployed
and is serving 100 percent of traffic.

Service URL: https://helloworld-abc1234-uc.a.run.app

Congratulations! you have successfully deployed your first Cloud Run Application. You can also deploy applications directly from public repositories like docker hub and other GCR.io registries.

To verify your deployment, use the Service URL to open it in your browser. You can also verify and manage the deployed services by navigating to Cloud Run in your Google Cloud Console as follows:

Cleanup

Run the following command to delete the docker image from the GCR registry:

gcloud container images delete gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld

Also, run the following commands to delete the local docker image:

docker rmi hellowworld:1.0.0
docker image prune -f

delete running Cloud Run service by using the cli as follows:

gcloud run services delete helloworld --region=$LOCATION

Conclusion:

Deploying a container on Google Cloud Run is simple, straightforward, and cost-effective, making it an excellent choice for a wide range of applications. By leveraging its serverless architecture, you can focus on building and scaling your applications without worrying about infrastructure management. Here are some key use cases where Google Cloud Run shines, offering both convenience and cost efficiency:

1. Deploying a Static Website Using NGINX:
   Hosting a static website is a perfect fit for Cloud Run. With its ability to scale to zero, you only pay when users access your site. This is ideal for portfolios, documentation sites, or small business websites that don’t require constant uptime.

2. Deploying a Front-End Application (React, Angular, or Vue):
   Cloud Run seamlessly handles single-page applications (SPAs) built with modern frameworks. Its automatic scaling ensures your app remains responsive during traffic spikes, while the pay-as-you-go model keeps costs low during periods of low activity.

3. Deploying an API:
   Whether it’s a RESTful API or a GraphQL endpoint, Cloud Run is an excellent platform for backend services. Its ability to handle concurrent requests and scale instantly makes it ideal for APIs with fluctuating traffic, such as those used in mobile apps or microservices architectures.

4. Deploying Machine Learning Models (e.g., Deepseek LLM):
   Cloud Run is a great choice for deploying machine learning models or AI-powered applications. For instance, you can containerize a large language model (LLM) like Deepseek and deploy it as a scalable, cost-effective service. Since Cloud Run scales to zero, you avoid paying for idle resources when the model isn’t in use.

5. Event-Driven Applications:
   Cloud Run integrates seamlessly with event-driven architectures. For example, you can deploy a service that processes data from Pub/Sub, triggers workflows in response to Cloud Storage events, or handles webhooks from third-party services. This makes it ideal for batch processing, data pipelines, or automation tasks.

6. Microservices and Lightweight Workloads:
   If you’re building a microservices-based application, Cloud Run allows you to deploy each service independently. Its lightweight nature and fast cold-start times make it perfect for small, focused services that don’t require the overhead of a full Kubernetes cluster.

7. Prototyping and Development:
   For developers and startups, Cloud Run is a cost-effective way to prototype and test new ideas. You can quickly deploy and iterate on your applications without worrying about infrastructure costs or complexity.

In summary, Google Cloud Run offers a versatile, cost-efficient platform for deploying a wide variety of applications. Its serverless nature, automatic scaling, and pay-as-you-go pricing make it an attractive option for everything from static websites and APIs to advanced AI models like Deepseek LLM. By choosing Cloud Run, you not only simplify deployment but also optimize costs, ensuring you only pay for what you use. Whether you’re a startup, a developer, or an enterprise, Cloud Run empowers you to focus on innovation while leaving the infrastructure management to Google Cloud.

After 3 years of playing around with Kubernetes, in 2020, a recruiter wanted to discuss further a consulting requirement and said “It would have been great if you had a Kubernetes Certification”. I told him to schedule an interview with the client and I will ensure he will have it before that. Indeed I passed it in less than a week but with a score of 71% only. Scores were not important to me then, I got the offer letter but the offer was pulled with a regret letter because of the 2020 Pandemic.

Fast forward to December 2024, our company Low Code Solutions got a Lead to set up an on-prem capability of Openshift Multi-Cluster for a Telco. This is when I planned to support the company’s Pre-Qualification in front of the Customer and started preparing for Kubestronaut Training on Dec 13 2024.

Due to my busy schedule, I was not able to spend more than a couple of hours a day, which is why I am going to share how smartly it will be possible for you to achieve your certifications in even less time. I will discuss each one in the sequence that I attempted:

CKA

Starting off with CKA, as I thought it would have been the hardest, however, it turned out to be the easier one compared to CKAD and CKS. If you have a reasonable knowledge of Linux basics, Docker and Linux Package Managers then you can prepare for it in less than a week.

Important Topics:

Following are the most important topics, where you will need to spend most of your time:

1. Imperative Commands to create, delete and update Kubernetes Resources, e.g: Pods, Deployments, Services etc.

2. Cluster Installation, Upgrade and Basic Troubleshooting

3. Understanding of Kubernetes Services Architecture (ETCD, Kube-Scheduler, Kube-Controller-Manager, Kube-ApiServer, etc.)

4. Understanding of Deployment, and Management of Nodes into a Cluster, understanding the role of Kubelet and Kube-Proxy

5. Understanding Static Pods and Important Config file Locations e.g: /etc/kubernetes, /var/lib/kubelet etc.

6. Understanding of TLS and Certificate Management.

7. Backup and Restore of ETCD

8. RBAC, how to create roles, rolebindings, clusterroles, clusterrolebindings, and service accounts. If you understand these 5 elements in depth then RBAC will be a piece of cake for you. (tip: it will also help you with your CKS exam, so spend more time with it.)

CKAD

CKAD is more about deploying and managing cloud-native applications on Kubernetes. Remember if you have done CKA, then it will help you in preparing for CKAD, and even for CKS, all you need to do is prepare some additional concepts, here are the important topics that I think you would need to emphasize more than others:

1. Imperative Commands as mentioned for CKA also.

k create --help 
k run --help
k expose --help
k edit --help
k replace -f file.yaml -n namespace
k delete --help
k scale --help

2. Understand the concept of Ingress Resource and practice the following command:

k create ingress NAME --rule=host/path=service:port[,tls[=secret]]

3. Practice the creation, updation and mounting of secrets, configmaps, service accounts, volumes and volume claims in Pods/Containers.

4. Taints, Tolerations, Node Affinity, Node Selector and nodeName for controlling how we schedule pods on specific nodes. Also understand why we need node affinity when we have taints and tolerations.

5. Init and Sidecar Containers

6. Logging, Monitoring and Probes.

7. Green-Blue and Canary Deployments

8. Pod SecurityContext and RBAC as mentioned above also.

9. Custom Resource Definitions and HELM

CKS

The CKS exam is where the real challenge begins. If you excelled in the CKA and CKAD exams, it’s easy to underestimate the difficulty of the CKS. I made the same mistake — despite my extensive Kubernetes experience and proficiency with the complex commands and vim, I failed on my first attempt. So before I tell you what to prepare most, let me share the mistakes:

Mistake #1:

I spent my first hour on just 3 questions, which left me with no chance to attempt all the questions and I was not able to even view the last 3 questions. So time management is important. Cluster Upgrade, Setting Up Audit Policy, and ImagePolicyWebhook are the most time-consuming questions, attempt them wisely.

Mistake #2:

I had clear and vivid core concepts of all the topics, so I thought I will take help from the documentation and solve all the questions. But Practice is the key, you can’t switch between the documentation and expect that you will finish in time.

Mistake #3:

I did not practice troubleshooting cluster crashes. So practice troubleshooting Api-Server or Kubelet issues by utilizing docker/crictl logs, displaying logs from /var/log folders for pods and containers, and other interactive ways. There are specific scenarios of such troubleshooting in KillerCoda Playgrounds.

CKS Exam TIPs:

1. Falco questions are tricky, most of my peers were unable to solve it, so do not spend too much time on it.

2. Make sure you know the difference between Layer 3, Layer 4, and Layer 7 Cillium Policies.

3. Some questions have supplementary tasks at the end of the questions. Read the whole scenario, do not assume that you are done.

4. Verify your answers. For every scenario, practice the verification process. Spare at least 15 to 20 min for your answer verification, which leaves you with 90 to 100 minutes for solving the scenarios.

Important Topics for CKS which require extreme practice:

1. BOM, Benchmarking and Vulnerability Scanning CLI Tools e.g: BOM, Kube-bench, Trivy, lsof, strace, netstat -plnt etc.

2. Api-Server Audit Log, ImagePolicyWebhook, Cluster Upgrade, Securing and Encrypting ETCD data and Setting up Network Policy (both native and cilium)

3. Sandboxing the Containers, Immutability and Runtime Security with Falco etc.

4. Setting proper Pod Security Context, AppArmor/Seccomp profiles, readOnlyRootFilesystem, allowPrivilegeEsclation etc. in context of Pod Security Standards.

5. Ingress Resource with TLS Secrets and its important annotations. (Hint: Imperitive Commands will help save time)

General Exam Tips

KCNA and KCSA are relatively easier exams, I recommend you give them at the end as after you prepare the above three exams you will have enough general knowledge to attempt the MCQs effectively, however there are a lot of topics that you might have to prepare e.g: Compliance Standards like CIS, OWASP, NIST etc. Here are some generic exam tips for CKAD, CKA and CKS:

1. Practice is the key, you may know and understand the concepts in depth, but remember CKA, CKAD and CKS are performance-based exams, and you will get 16 to 18 scenarios to solve. In our consulting world, we take at least a day or two to solve even one.

2. Solve at least 2 to 3 times the KillerCoda Playgrounds thanks to Kim Wustkamp, the founder, for keeping it free. However, I would recommend taking at least a 1 month pro subscription because this way you will get the exam desktop which will help you familiarize yourself with the exam environment. I wasted 5 minutes finding how to do basic functionalities like copy/pasting and keeping notes etc.

3. Get a KodeKloud Subscription, it is a gold mine of learning material for Infrastructure and DevOps. If you are validating

4. VIM proficiency: you need to start using some shortcuts in vim, some of the most handy are the ones which will help you with: Copy, Past, Delete Line, replace-in-place, Indentation, set number to show line numbers, visual tool to copy/duplicate multiple lines, find text.

5. Grep proficiency: Grep can help you in unimaginable ways e.g: finding a particular vulnerability in a BOM scan, finding a file containing a specific text in a folder of multiple such files (hint: grep -r), greping two options (hint: grep -E “one|two”).

Profession Tips

Even though The Linux Foundation and the CNCF has done a great job at creating and crafting these exams and certifications programs where they have made sure that only the exceptional comes out shining. However, it is important to know that real-world scenarios are even more complex and worse than the ones you will encounter in these exams. So, getting the certification does not guarantee that you are suitable for some challenging role. The certification will help you to land an interview though, but to get a great offer you will need a lot more than the certifications. So after hiring 100s of individual I am going to share some of the most important traits that you also need to land your dream job:

1. Prepare for these certifications keeping your professional goals in mind. Prepare yourself for the industry challenges not only for the exam, have your previous job scenarios in mind and relate every topic with your current or upcoming role in business.

2. Work on your soft-skills, make sure you know how to express your learning and experiences effectively. Understand and make use of industry and technology buzzwords and their concepts e.g: cloud-native, cluster-hardening, encryption at rest, zero-trust, software security compliance and standards etc.

3. Make yourself vocal on platforms like LinkedIn and X by sharing your thoughts on generic technology discussions. (even I am struggling to do that often)

4. Express your ideas in Diagrams and Visuals. Make every presentation or discussion more meaningful with your visual skills.

5. Learn how to explain a complex idea or architecture to multiple audiences, tech, non-tech, business or a layman.

6. Write blogs/articles while your are learning or even after accomplishing a goal. Focus on spreading the knowledge, tips and tricks rather than publicity or marketing.

7. Start Contributing in Kubernetes and/or other CNCF Projects. Start by attending the weekly or monthly meetings and help yourself understand the working group processes until you think you are ready to take on the challenge of taking an assignment.

8. Remember the motive, we are working to help make the world a better place.

By establishing such an externalized architecture, PEGA can now be deployed in over a dozen different configurations. In this article, we will discuss some of the most important ones.

PEGA CLASSIC DEPLOYMENT

A classic Pega deployment entails deploying all components for every environment. For example, consider an insurance company that requires Pega to be installed in multiple Kubernetes clusters, with each cluster representing a distinct environment.

Each cluster must have its own database, Kafka, Elasticsearch, Cassandra, and more. In addition to third-party services, Pega services are also deployed individually within each cluster. This type of deployment is recommended for various reasons, but it has its drawbacks. On the one hand, it can be advantageous since each environment is entirely isolated from the others, eliminating the need to worry about micro-configurations for each service to manage environments simultaneously. On the other hand, it necessitates at least twice the hardware resources to run everything and will be twice as expensive to maintain each environment, as each one must be managed separately.

PEGA CONNECT DEPLOYMENT

A connected deployment is one in which the customer already has running clusters of one or more third-party services, such as Elastic Search, Kafka, or Cassandra, etc. We often encounter customer requirements where they mention having existing clusters of any of the aforementioned third-party services and wish to utilize them for PEGA as well.

Generally, it is not recommend sharing these services with other Pega applications; however, it is always a design decision to make based on the Cloud Topology Architecture. We see no harm in implementing this kind of design, where we only deploy the PEGA platform and utilize the customer-provided clusters for third-party applications. This setup reduces the friction in rolling out the PEGA platform, uses significantly fewer resources compared to the classic model, and sometimes is more efficient and performant as well.

PEGA SHARED DEPLOYMENT

A shared deployment is when we deploy and share many different shareable PEGA Services across different environments and platforms. Most of the services are highly recommended to be shared as it serves the reasons of breaking down Pega into microservices. Lets drill down this deployment in more detail by understanding different scenarios.

1. Shared CDN-ONPREM
   CDN is something that you can not only share among all of your environments without any hesitation but you may also utilize the PEGA-provided CDN for this reason (only if your pods have access to internet).

2. Shared Constellation App Static

   This service keeps track of your custom components and serves these components to your PEGA Applications which use constellation UI instead of Traditional UI. It is highly recommended that you share your Constellation App Static service among your environments. One of the major reasons is that you will get consistent Static Custom Components across all environments which means once deployed and tested a component will never be required to test again even after a feature update by DM in the Pega Environment. One scenario where I might recommend using a separate Constellation App Static is when you are using a Third-Party Pipeline Manager like Jenkins to trigger DM Pipelines along with other third-party deployment pipelines like React App, Mobile Apps etc, this way you can trigger a Constellation App Static Custom Component Publish in each environment as a Deployment Pipeline Task.

3. Shared Search and Reporting Service

   SRS is a multi-tenant service, which means many environments with different environment/tenent IDs can connect to a single SRS service. In a Kubernetes cluster, we can deploy it along with a single Elastic Search Cluster serving many different environments.

4. Deployment Manager and PDC

   PDC and Deployment Manager are designed as a shared service for Multiple Environments, DM requires multiple environments to perform code promotions and feature deployment from one environment to the other, however, the the PDC is also a multi-tenant multi-environment service. This means you can use Single PDC for numerous Pega environments for multiple customers/business units.

The following figure shows a fully shared model, where all multi-tenant applications and services of Pega as well as other shareable third-party services are shown:

Deploying Pega for enterprises has long been a challenge for Infrastructure and DevOps teams. While finding skilled Infra/DevOps professionals is common, however, individuals with both Pega Admin and Kubernetes DevOps experience are rare. This scarcity is one of the primary reasons many Pega customers face difficulties in managing and maintaining their enterprise Pega deployments.

With the introduction of Pega Cloud, enterprise customers can significantly reduce the complexity and burden of implementation and ongoing maintenance once migrated to Pega Cloud. However, the transition can still be a complex and time-consuming process for organizations with multiple integrations between Pega and other on-prem applications.

As a consultant, I have had the privilege of working with numerous enterprise customers who faced complex application integration challenges. With over two decades of experience in software engineering and cloud technologies, I specialize in designing intricate, scalable architectures tailored to unique business needs. My expertise extends to creating comprehensive communication matrices well in advance, ensuring seamless coordination and faster deployments. I have consistently delivered projects in record time, helping organizations save millions of dollars while achieving their digital transformation goals efficiently.

So feel free to contact me for any design, implementation, or infrastructure management reviews or consulting requirements.

So before we explore the descheduler, we might need to recap how the scheduler works. The scheduling decision is based on 4 stages or extension points, these are:

1. Scheduling Queue

2. Filtering

3. Scoring

4. Binding

There can be multiple plugins installed on these extension points, e.g: PrioritySort plugin on the queue, NodeResourceFit, and NodeName plugin on the filtering extension point. Because of the highly extensible nature of Kubernetes, it makes it possible first to customize which plugin goes where and also allows us to write our custom plugins. It also gives us the ability to add a plugin in the post and pre-stages of the extension points.

Now since the scheduling decision is based on a decision taken place by multiple plugins and their placement in the extension points at the time of scheduling, so it is highly possible that the original scheduling decision is not valid anymore.

When do you need a Descheduler

Due to the dynamic nature of the Kubernetes Cluster, there could be several reasons why you may want to evict (deschedule) a Pod from a node:

1. To improve cluster performance and availability by redistributing pods to optimize resource usage and reduce contention for resources.

2. To minimize downtime by automatically rescheduling pods on healthy nodes when a node fails.

3. To help with scaling by removing underutilized pods and redistributing them to nodes where they can be better utilized.

4. To improve security by ensuring that only authorized pods are running on the cluster.

5. To enforce policies such as inter-pod anti-affinity, where it can detect and deschedule the pods that don't conform to the policy and redistribute the pods to other nodes.

6. To improve cost-efficiency by reducing wastage of resources by identifying and removing duplicate pods and rescheduling them.

Below is the list of different scenarios where the use of a descheduler is unavoidable:

1. A new node is introduced in a cluster

You have just introduced a new node in the cluster and want to distribute the workload evenly. Without descheduling, your pods may reside on the original nodes for ages, and due to this adding new nodes will not have any immediate performance benefits. Descheduling pods and redistributing them on new nodes can help improve resource usage and ensure that resources are distributed evenly across the cluster. By spreading the pods across different nodes, the load on individual nodes is reduced resulting in improved performance and stability of the cluster. It will also help the default scheduler and auto-scaler to adjust the number of replicas to match the new capacity and resource available in the cluster.

2. Node labels are updated

A node label update can affect different scenarios and the original scheduling decision may not be appropriate for certain pods. Here are some of the important ones:

1. Node Affinity: Node affinity allows for pods to be scheduled based on the labels assigned to a node. If the labels of a node is changed, it may no longer match the node affinity rules of a pod, which can lead to an undesired state.

2. Node Selector: Node selector allows to schedule pods on specific nodes based on the node labels, if a label is updated then these decisions are no longer valid and require eviction.

3. Failure Domain: Node labels can be used to indicate the failure domain of a node like a region, rack, or zone, which can be used to schedule the pods to spread across multiple failure domains. An intelligent deschduler will ensure the high availability of services by taking optimum descheduling decisions.

3. Node failure requires Pods to be moved

It is important to deschedule pods on a failed node because:

1. High availability: When a node fails, the pods running on that node can become unavailable, and this can have a significant impact on the availability of the applications and services running on the cluster. By descheduling the pods on a failed node, the cluster can automatically reschedule the pods on healthy nodes, which can help to minimize downtime and improve availability.

2. Resource Utilization: A failed node can cause a significant drain on resources such as CPU and memory. By descheduling the pods on a failed node, the cluster can free up these resources, which can be used more efficiently by other pods, improving overall cluster performance.

3. Auto Scaling: A failed node can impact the scaling of pods. By descheduling the pods running on a failed node, the auto-scaler can automatically adjust the number of pods running on healthy nodes to maintain the desired number of replicas.

4. Networking: Descheduling pods on a failed node can help prevent networking issues, such as IP conflicts or service outages, which may be caused by pods running on a failed node.

5. Security: Descheduling pods running on a failed node can help prevent security risks. A failed node can be compromised and running malicious pods on a compromised node can pose a significant risk to the cluster.

Descheduling pods on failed nodes are important to ensure that the cluster remains highly available, that resources are used efficiently, and that the auto-scaler can adjust the number of replicas running, avoiding networking issues and maintaining security.

4. Remove Duplicates

Duplicate pods in a Kubernetes cluster can cause several issues that can negatively impact the performance and availability of the cluster. Some reasons why it's important to remove duplicate pods from a node running in Kubernetes include:

1. Resource Utilization: Duplicate pods consume resources such as CPU and memory that could be used more efficiently by other pods. This can cause resource contention, which can lead to delays in container startup times and negatively impact the overall performance of the cluster.

2. Networking: Each pod consumes network resources such as IP addresses, and having multiple pods with the same IP address can cause networking issues such as IP conflicts, which can cause communication problems between pods and services.

3. Scalability: Duplicate pods can make it difficult to scale the number of pods running in a cluster. For example, if a Deployment controller creates multiple replicas of a pod, each replica will have a different replica number and the same pod name which can confuse when trying to scale the number of replicas.

4. Security: Having duplicate pods can make it difficult to keep track of what is running on a cluster and can open security vulnerabilities. It is important to ensure that all running pods are authorized and that no rogue pods are running.

5. Cost: Running duplicate pods can result in a waste of resources, which can result in higher costs.

   Removing duplicate pods from a node can help to improve the overall performance and availability of a Kubernetes cluster by optimizing resource utilization, reducing network conflicts, improving scalability, security, and reducing costs.

5. Low/High Node Utilization

A descheduler can help distribute load across different nodes in the cluster in several ways:

1. Balancing resource usage: A descheduler can identify pods that are consuming a disproportionate amount of resources on a node, such as CPU or memory, and move them to other nodes where resources are more available. This can help to balance the resource usage across the cluster and improve overall cluster performance.

2. Reducing node overcommitment: A descheduler can identify nodes that have a high number of pods running on them and redistribute the pods to other nodes to reduce the number of pods running on the overcommitted node. This can help to reduce contention for resources and improve the overall performance of the cluster.

3. Improving node utilization: A descheduler can help to identify and remove underutilized pods on a node, and redistribute them to nodes where they can be utilized better. This can help to improve the utilization of resources across the cluster.

6. Pods Violating Inter Pod AntiAffinity

Inter-pod anti-affinity is a feature that allows you to specify rules for how pods should be scheduled in relation to one another. These rules can be used to ensure that pods that belong to the same application or service are spread across different nodes in a cluster, in order to improve availability and reduce the risk of single points of failure.

One important reason to use inter-pod anti-affinity is to ensure that pods that need to be highly available, such as database pods, are not scheduled on the same node. If multiple pods that need to be highly available are scheduled on the same node and that node goes down, multiple pods will become unavailable at the same time. Spreading these pods across different nodes can help to mitigate this risk.

Another reason to use inter-pod anti-affinity is to ensure that pods are spread across different zones or regions to improve resiliency in the event of a zone or region failure.

It is important to use a descheduler in this scenario because, despite the best efforts of Kubernetes scheduler, sometimes pods can violate inter-pod anti-affinity rules due to various reasons like over-commitment of resources or other factors. A descheduler can help identify and remove these pods, ensuring they are rescheduled to comply with the specified anti-affinity rules. This can help to improve cluster availability, reduce the risk of single points of failure, and optimize resource utilization.

10. Pods Violating Topology Spread Constraint

Topology spread constraints allow to spread of pods evenly across different nodes, zones, regions, or racks. By descheduling pods that violate these constraints, the cluster can ensure that the resources are utilized more efficiently, pods are spread out to reduce contention of resources, it can guarantee high availability by running the services on multiple nodes, zones, and regions and it can also help minimize the impact of an infrastructure failure by failure domain awareness.

11. Pods Having Too Many Restarts

When a pod has too many restarts, it can indicate that there is an issue with the pod or the node it is running on. Pods that are continuously restarting can destabilize the cluster, can delay container startup times, and negatively impact the overall performance of the cluster. Descheduling such a pod can help the operation teams to understand the issue behind the continuous restarts and stabilize the application and cluster performance.

Conclusion

In summary, a descheduler plays an important role in managing and optimizing the distribution of pods within a Kubernetes cluster. It can help to ensure that the cluster is running at optimal performance, that resources are being used efficiently, and that the cluster is secure and available.

Deploying a static website on AWS Amplify is a straightforward process that can be broken down into the following steps:

1. Create a new Amplify app.

Log in to the AWS Amplify Console and create a new app by selecting the Host Web App option and by providing a name, environment, and repository for your website.

2.Connect your repository

Connect your repository to the app by linking it to your GitHub, GitLab, or Bitbucket account, or by manually uploading your website files. Here are the steps to connect your GitHub repository in AWS Amplify:

1. Go to the Amplify Console and select the app that you want to connect to your GitHub repository.

2. Click on the "Connect branch" button.

3. Select "GitHub" from the list of repository providers.

4. Use your GitHub account to sign in, and authorize Amplify to access your GitHub repositories.

5. Select the repository that contains your website or app, and choose the branch that you want to connect.

6. Click on the "Next" button and configure the build settings, environment variables, and other options as needed.

7. Click on the "Save and Deploy" button to start the build and deploy process.

8. After the deployment is complete, you can monitor the build and deploy process, and make updates to your app through the Amplify console.

Heres the screen shot of the options available for Code Repository options:

It's important to note that in order to connect to your GitHub repository, you need to have a GitHub account, and your repository should be public or accessible by AWS Amplify.

3. Build and Deploy your App

Once your repository is connected, Amplify will automatically build and deploy your app. You can also configure custom build settings and environment variables if needed. Please comment if you need any support in your custom build process.

4. Configure custom domains

Amplify will provide a default domain for your app, but you can also configure custom domains if needed. Here are the steps to configure custom domains for a deployed app on AWS Amplify:

1. Go to the Amplify Console and select the app that you want to configure a custom domain for.

2. Click on the "Domain settings" tab and then click on the "Connect domain" button.

3. Enter the domain name that you want to use for your app. Amplify will automatically check the availability of the domain and suggest a domain if it's not available.

4. If the domain is available, Amplify will provide you with a set of instructions to verify that you own the domain. This typically involves adding a CNAME or A record to your domain's DNS settings.

5. Once the domain is verified, Amplify will automatically create a certificate for your custom domain.

6. Once the certificate is ready, you can associate the custom domain with your app by clicking on the "Associate" button.

7. Now, your custom domain is configured and should be active within a few minutes. You can test it by visiting the custom domain in your browser.

Here is a screenshot of how I have added my custom domain and mapped the branches to each of the sub-domains:

It's important to note that you will need to have access to your domain's DNS settings to configure custom domains in Amplify. Also, Amplify require a valid SSL certificate for custom domains, it will create one for you automatically but you can also use your own certificate.

5. Monitor and update your app

After your app is deployed, you can monitor it's performance, view analytics, and make updates through the Amplify console. You can also monitor the application's access logs and set alarms based on different metrics like 40x and 50x errors etc.

Below are the screen shots of the access logs and the site analytics of my personal portfolio website I deployed on Amplify earlier.

6. Enable CloudFront distribution

By default CloudFront distribution is enabled for you application if you are using Route 53 to configure your DNS.

According to the State of Kubernetes Security Report 2022, security is one of the biggest concerns with container adoption, and security issues continue to cause delays in deploying applications into production.

In the last 1 year, 93% said that they have experienced at least one major security incident. More than 30% of them have experienced customer or revenue losses due to these incidents. According to a recent study, 95% of the breaches were due to human errors.

Source (Red Hat State of Kubernetes Security Report - 2022)

Impact

1. Data Compromise
An attacker with access to business or infrastructure data can leak or destroy the data.

2. Resource Hijacking
If an attacker gets access to a node, or any compute resource in a cluster; he can easily run resource-hungry scripts like crypto mining (crypto-jacking), AI model processing, etc.

3. Denial of Service
DoS can be achieved using buffer overflow (by flooding general requests), ICMP flooding (by sending spoofed packets), or SYN flooding (by sending false requests to the server without a handshake). DoS attack is meant to shut down a computer or make a network inaccessible.

4. Ransom
An attacker can take over the Management Layer or even remove the (important) data and ask for a ransom in return for the data or control.

5. Loss in Customers and/or revenue
One may incur major losses once a service is down because of any reason mentioned above.

AIOps to the Rescue

KaiOps is an AI/ML SaaS-based tool, that connects with your clusters using a secure connection with the KubeAPI server. You may need to update the default security profile to enable the KaiOps security agents to scan and flag different vulnerabilities as low, medium, or high on threats. Once a threat is found, it will send you notifications on your preferred communication channels and inform you of the possible implication, remediation, and/or prevention strategies.

KaiOps scans the cluster every few seconds at runtime and observes the most important Kube events related to network, storage, and workload. It also watches all whitelisted Kube objects for any change and runs AI inferences using GNN (Graph Neural network) to detect misconfigurations, policy violations, and potential threats.
Cluster monitoring methods are divided into 10 classes:

1. Access
Exposing cluster Nodes publicly may give the same access to containers with potential vulnerabilities and may lead to attackers penetrating the cluster.
2. Execution
Execution of commands is monitored very closely as an SSH server or a bash script running in a container can be compromised using brute force attacks.
3. Persistence
Writeable hostPaths in containers or persistence volumes, backdoor containers, or exposed/compromised CronJobs are also monitored to reduce the risk of penetration.
4. Privilege
The security contexts of all running containers are continuously monitored along with the RBAC context.
5. Defense Evasion
Kube objects deletion, clearing of container logs, or connections through proxy server are some of the defense evasive events its monitors according to security policy.
6. Credentials Leak
Leakage of Kube CA data of an exposed ApiServer or cloud credential files on a hosted Kubernetes service. KaiOps continuously monitors Kube's internal network and even the object calls data, which is fed into the GNN for AI model training for anomaly detection.
7. Discovery
Exposed Observability platforms, K8s dashboard, ApiServer, and similar other services may lead an attacker to penetrate the cluster.
8. Lateral Movement
Writeable volumes mounted on hosts, exposed access to cloud resources, privileged service accounts assigned to a container, exposed application configuration through environment variables, core DNS poisoning, arp poisoning, IP spoofing, or even public IPs assigned to a container may compromise the security of a cluster. KaiOps flags its vulnerabilities and whitelists some of them based on a user-defined security policy.
9. Collection
Images from unknown or public sources and compromised container images can jeopardize cluster security. KaiOps continuously scans for Image and Package Vulnerabilities using static image scanners and feed the data into the AI inference engine.
10. Custom K-Tactics
There is some custom patented K-Tactics that KaiOps uses to help its GNN Models detect errors, ambiguities, vulnerabilities, and even anomalies. K-Tactics are also used to reduce nuisance alarms and notifications.

Conclusion

With almost One Million Kubernetes clusters found to be exposed, it is evident that most of the above security vulnerabilities are present in the vast majority of Kubernetes clusters, making remediation difficult. KaiOps uses its patented AI/ML agents by feeding the telemetry data into a GNN model to predict and detect security vulnerabilities, misconfigurations, and anomalies. KaiOps SaaS is free for a cluster with up to 3 nodes for 14 days, and even includes live support in setting up their service for your clusters. KaiOps currently provide support for Native Kubernetes Clusters on almost all hosted Kubernetes services. Use this link to register and get an extended 1 month of a free trial.

References

• Red Had State of Security Report
• AIOps use cases
• Common Security Issues found in almost all Kubernetes Deployments
• MITRE ATT&CK Matrix
• Kubernetes Security Workshop

• Over 900,000 Kubernetes Instances are found Exposed Online
• Tesla Clusters on Amazon Cloud were Hacked with Cryptojackers
• Exposed Prometheus exploit Kubecon EU

Now that we know that a single container vulnerability may compromise the overall security of the cluster, let us review a few aspects.

Public Nodes

A public cluster is one whose Nodes are accessible from the internet, technically they have one or more networks configured with ExternalIP. You can test it using the following command

kubectl get node <node-name> -o jsonpath='{.status.addresses}' | grep ExternalIP

Replace with your node name in the cluster. following the sample output

If any of your nodes have an ExternalIP then it is exposed, which means all your containers/pods running on this node are also exposed to the public. It is highly recommended to keep your nodes private and route the outgoing traffic using a Cloud NAT.

Public Kubernetes Dashboard

Kubernetes Dashboard is an open-source deployment used to manage and monitor cluster deployments and other objects. It is available on all clusters and can be deployed or installed with a few simple commands. Here is how you can also deploy it on your cluster:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.1/aio/deploy/recommended.yaml

Once installed you can proxy it to access it from your remote machine (machine with cluster access) using kubectl proxy command. Now you can access your dashboard using the following url: http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Note: The kubeconfig authentication method does not support external identity providers or X.509 certificate-based authentication. The UI can only be accessed from the machine where the command is executed. See kubectl proxy --help for more options.

Now, this is your standard access method, if one of your vulnerable (or compromised) containers has the access to your dashboard then the hacker will have access to the whole Kubernetes cluster.

Exposed Observability Platform

If you are using Prometheus, Grafana, or Splunk, chances are that you are exposing them by using an external load balancer. These and other observability platforms are often used to monitor and prevent performance, security, and other issues in a cluster. Once exposed to public traffic either through an ingress or through a vulnerable (or compromised) container may lead to a brute force attack. A hacker can take control of the whole cluster once getting access to such a platform.

Privileged Container

A privileged container is one that has full access to itself and through the reverse shell technique it can also get full control over its encapsulating Node. Full access to a node exposes all the images stored on it and it may lead to access to other nodes in the network exposing all available resource objects on the cluster. Hackers may deploy a cron job or hijack the resources of the nodes without even letting it be noticed from the Kubernetes platform.

Conclusion

In this part, I have just scratched the surface and mentioned some of the most common security vulnerabilities. An attacker can use impact techniques to destroy, abuse, or disrupt the normal behavior of an environment. I will discuss some more security vulnerabilities in my upcoming articles.