Kubernetes Security for Exposed Workloads

A successful breach of an exposed workload carries the risk of spreading laterally, identifying targets, and ultimately gaining access to the organization’s crown jewels Kubernetes security framework provides default security policies and guidelines from pod level, with Pod Security Policies (PSP), to cloud level. These policies need to be configured, but even perfectly calibrated security policies do not provide optimal security at runtime. The number of system workloads, both proprietary and imported, can quickly run into thousands. This makes detecting workload vulnerabilities in real-time, and promptly mitigating them crucial to maintain the system’s integrity. With the myriad of microservices and external data and service sources, exhaustive mapping and constant monitoring of every workload is an absolute necessity. Workloads with unnecessary exposure to unvalidated sources are susceptible to attacks that can compromise workloads and platforms.

How to tighten Kubernetes security against exposed workloads

Cisco Cloud Native Security’s workload protection solution can identify vulnerabilities that endanger your application and clusters. Taking a cluster running the Kubeflow toolkit, for example. Kubeflow toolkit includes a dashboard accessible through a browser on the Istio ingress gateway. Leaving the dashboard configured to allow access to the Internet can create a vulnerability inviting malicious threat actors? to gain entry to the cluster. Cisco Cloud Native Security Solution can detect, identify, and block, mitigate, or monitor that attack. The dashboard’s one-time view displays what is running on your cluster. The new security threat indicator shows the risk the vulnerable workload poses to the cluster. This signal that this dashboard app is exposed to the Internet. Digging deeper into the type of risk detected reveals stacks of additional information. The Kubeflow toolkit, used as an example above, includes a dashboard accessible from external sources through the Istio ingress gateway. Failure to properly define access policies for the Kubeflow workload is creating a critical vulnerability. This example is selected as a reminder that even reputable external resources such as Istio service mesh can generate vulnerabilities in your system and need to be actively monitored. The main risk is that the dashboard is accessible from a browser. The runtime connection monitor has captured an unprotected connection to the Kubeflow dashboard that provides a potential entry point to attackers. More granular information is available at a click, in this case showing that the exposed workload can run as root and can escalate its privileges, potentially exposing the entire environment. Having captured the connection to the Kubeflow dashboard, the one-time monitor automatically ends the connection. Even though it stems from Istio controlled Kubeflow dashboard, this vulnerability can be blocked with a one-time policy rule. The extent of the access limitation is easily determined from the New Connection Rule options. This enables a granular definition of the scope of the source to block, from unique ID to entire environments. The New Connection Rules section provides options for precisely selecting the source’s scope and the destination for which the rule is defined. '''The source''': The rule of thumb to achieve optimal Kubernetes security is to apply the least privilege rule and limit access as narrowly as possible without interfering with the smooth running of the upstream and downstream processes. However, there are many exceptions to this default rule, such as, for example, attack monitoring for identification of the attack source, allowing additional flexibility for further development. '''Destination''': The detected vulnerability relates to a specific destination. However, the operator could, if deemed beneficial, select a narrower or wider destination. In this Kubeflow vulnerability example, the new connection rule defines external connections as the source and the dashboard workload as the destination. This selection covers all communications to the central dashboard emanating from an external source. After defining the communication scope, it is time to select the required behavior from the Rule Properties section. The operator selects one of the three following options, each with its own consequences.

• '''Allow only the connection between source and destination''' – This option is used to provide the narrowest possible privileged access, where access is limited exclusively to the named source.
• '''Detect the connection between the source and the destination''' – This option lets the operator closely monitor all communications between various sources to a specific destination. Typically used to monitor traffic to a target destination, such information provides valuable insights into penetration attempts, from hackers’ identification to the techniques used to attempt a breach.
• '''Block the connection between the source and destination''' – This option effectively eliminates the detected vulnerability by plugging the hole that provides the defined source a point of entry into the cluster.

In this example, the third option is selected. The operator directly blocked access to the Kubeflow dashboard to external sources, without requiring any interaction with Istio. Once the new connection rule is applied, Cisco Cloud Native Security Solution’s one-time viewer dashboard is updated in real-time and the operator can continue monitoring the overall security without distraction from solved issues. In this example, the unsafe Kubeflow connection status is now “Ended,” and the connection itself appears as blocked. The volume and complexity of attacks keeps on increasing, so constant vigilance is no longer optional. Lack of preventative detection and mitigation exposes organizations to the rising cost of undetected attacks, from direct loss of business revenue and tarnished reputation to lack of compliance fines and legal settlements, or loss of intellectual property. In these days of rapid change where the threat landscape is in constant evolution, exhaustive monitoring and short reaction time upon vulnerability detection are crucial. Whether the selected action upon detection is to tackle the threat or monitor it for information gathering purposes, active monitoring coupled with immediate intervention tools is key to maintain Kubernetes security at all times. Cisco Cloud Native Security Solution’s single-pane one-view dashboard provides a comprehensive solution to monitor and mitigate threats to your Kubernetes security in real-time.