More than half of all enterprises consider security as their biggest challenge when publishing their microservice workloads in production. 50% require developers to use validated images only, around 80% want to have a DevSecOps initiative, more than 40% consider DevOps as the role most responsible for Kubernetes security, and most importantly, more than half have delayed application deployment due to security concerns.

According to the State of Kubernetes Security Report 2022, security is one of the biggest concerns with container adoption, and security issues continue to cause delays in deploying applications into production.

In the last 1 year, 93% said that they have experienced at least one major security incident. More than 30% of them have experienced customer or revenue losses due to these incidents. According to a recent study, 95% of the breaches were due to human errors.

Source (Red Hat State of Kubernetes Security Report - 2022)

Impact

1. Data Compromise
An attacker with access to business or infrastructure data can leak or destroy the data.

2. Resource Hijacking
If an attacker gets access to a node, or any compute resource in a cluster; he can easily run resource-hungry scripts like crypto mining (crypto-jacking), AI model processing, etc.

3. Denial of Service
DoS can be achieved using buffer overflow (by flooding general requests), ICMP flooding (by sending spoofed packets), or SYN flooding (by sending false requests to the server without a handshake). DoS attack is meant to shut down a computer or make a network inaccessible.

4. Ransom
An attacker can take over the Management Layer or even remove the (important) data and ask for a ransom in return for the data or control.

5. Loss in Customers and/or revenue
One may incur major losses once a service is down because of any reason mentioned above.

AIOps to the Rescue

KaiOps is an AI/ML SaaS-based tool, that connects with your clusters using a secure connection with the KubeAPI server. You may need to update the default security profile to enable the KaiOps security agents to scan and flag different vulnerabilities as low, medium, or high on threats. Once a threat is found, it will send you notifications on your preferred communication channels and inform you of the possible implication, remediation, and/or prevention strategies.

KaiOps scans the cluster every few seconds at runtime and observes the most important Kube events related to network, storage, and workload. It also watches all whitelisted Kube objects for any change and runs AI inferences using GNN (Graph Neural network) to detect misconfigurations, policy violations, and potential threats.
Cluster monitoring methods are divided into 10 classes:

1. Access
Exposing cluster Nodes publicly may give the same access to containers with potential vulnerabilities and may lead to attackers penetrating the cluster.
2. Execution
Execution of commands is monitored very closely as an SSH server or a bash script running in a container can be compromised using brute force attacks.
3. Persistence
Writeable hostPaths in containers or persistence volumes, backdoor containers, or exposed/compromised CronJobs are also monitored to reduce the risk of penetration.
4. Privilege
The security contexts of all running containers are continuously monitored along with the RBAC context.
5. Defense Evasion
Kube objects deletion, clearing of container logs, or connections through proxy server are some of the defense evasive events its monitors according to security policy.
6. Credentials Leak
Leakage of Kube CA data of an exposed ApiServer or cloud credential files on a hosted Kubernetes service. KaiOps continuously monitors Kube's internal network and even the object calls data, which is fed into the GNN for AI model training for anomaly detection.
7. Discovery
Exposed Observability platforms, K8s dashboard, ApiServer, and similar other services may lead an attacker to penetrate the cluster.
8. Lateral Movement
Writeable volumes mounted on hosts, exposed access to cloud resources, privileged service accounts assigned to a container, exposed application configuration through environment variables, core DNS poisoning, arp poisoning, IP spoofing, or even public IPs assigned to a container may compromise the security of a cluster. KaiOps flags its vulnerabilities and whitelists some of them based on a user-defined security policy.
9. Collection
Images from unknown or public sources and compromised container images can jeopardize cluster security. KaiOps continuously scans for Image and Package Vulnerabilities using static image scanners and feed the data into the AI inference engine.
10. Custom K-Tactics
There is some custom patented K-Tactics that KaiOps uses to help its GNN Models detect errors, ambiguities, vulnerabilities, and even anomalies. K-Tactics are also used to reduce nuisance alarms and notifications.

Conclusion

With almost One Million Kubernetes clusters found to be exposed, it is evident that most of the above security vulnerabilities are present in the vast majority of Kubernetes clusters, making remediation difficult. KaiOps uses its patented AI/ML agents by feeding the telemetry data into a GNN model to predict and detect security vulnerabilities, misconfigurations, and anomalies. KaiOps SaaS is free for a cluster with up to 3 nodes for 14 days, and even includes live support in setting up their service for your clusters. KaiOps currently provide support for Native Kubernetes Clusters on almost all hosted Kubernetes services. Use this link to register and get an extended 1 month of a free trial.

References

• Red Had State of Security Report
• AIOps use cases
• Common Security Issues found in almost all Kubernetes Deployments
• MITRE ATT&CK Matrix
• Kubernetes Security Workshop