![\"\"\/](\"https:\/\/engineering.atspotify.com\/wp-content\/themes\/theme-spotify\/images\/icon.png\"){kind=icon}
<\/p>\n
<\/p>\n
<\/p>\n TL:DR<\/strong> At Spotify, we run containerized workloads in manufacturing throughout our total group in 5 areas the place our essential manufacturing workloads are in Google Kubernetes Engine (GKE) on Google Cloud Platform (GCP). If we detect suspicious habits in our workloads, we’d like to have the ability to shortly analyze it and decide if one thing malicious has occurred. Today we leverage business options to observe them, however we additionally do our personal analysis to find choices and various strategies. This new technique empowers us and different organizations to make use of an open supply various if we should not have a business answer in place or if we wish to evaluate our present monitoring to the open supply one.<\/p>\n In this weblog put up, I\u2019ll clarify intimately how reminiscence evaluation works and the way this new technique can be utilized on any GKE node in manufacturing as we speak.\u00a0<\/p>\n Spotify is a heavy consumer of GKE on GCP, and we run most of our manufacturing workloads as we speak in GKE. We\u2019re current in 5 GCP areas and run a couple of hundred thousand pods in manufacturing on the similar time throughout greater than 3,000 GKE namespaces.\u00a0<\/p>\n In abstract, it\u2019s secure to say that we\u2019re a giant consumer of GKE and have a must each scale our manufacturing workloads and in addition monitor what is going on in our manufacturing.<\/p>\n Although Google has its personal technique for implementing Kubernetes in its cloud atmosphere, being GKE, there are a couple of normal phrases<\/a> to bear in mind:<\/p>\n Below, you may see a high-level structure of a GKE cluster on GCP (Source: https:\/\/cloud.google.com\/kubernetes-engine\/docs\/concepts\/cluster-architecture<\/a>).\u00a0<\/p>\n The kernel<\/a> is the principle layer between the working system (OS) of the GKE node and the underlying server assets. It helps with vital duties like course of and reminiscence administration<\/a>, file programs, machine management, and networking. Below is an summary of the kernel format:<\/p>\n If we wish to perceive what is going on on a GKE node and what processes are working on it in reminiscence \u2014 the kernel is the optimum place to search out it. Many business options as we speak leverage the prolonged Berkeley Packet Filter (eBPF)<\/a> and its sandbox method to entry the kernel. This, nevertheless, requires that you simply purchase a business answer that makes use of eBPF otherwise you construct your individual answer on prime of it. As my analysis confirmed, there’s one other method we will take.<\/p>\n So how can we entry the kernel on a GKE node and analyze the reminiscence? My analysis boiled it all the way down to the next three steps:<\/p>\n In order to exhibit the next steps, I created the beneath structure utilizing Terraform and a Python script that built-in with the GCP API.<\/p>\n By taking a kernel reminiscence dump, we will get a \u201csnapshot\u201d of all of the kernel exercise at a selected time that we then can analyze.<\/p>\n Since GKE nodes are working the hardened working system COS<\/a>, we will\u2019t use a kernel module<\/a> or comparable answer. However, by quickly including a privileged container to the GKE node with privileged permissions, we will entry the kernel area within the file path: \/proc\/kcore<\/em><\/strong><\/a>.<\/em><\/strong><\/p>\n Once now we have entry and may learn from this file path, we will use the open supply instrument AVML to take a kernel reminiscence dump. The code beneath reveals a Terraform instance of a privileged container in GKE.<\/p>\n In order to interpret the kernel reminiscence dump, we have to construct an Intermediate Symbol File (ISF) of the particular kernel model of the GKE node. This will be accomplished by accessing the vmlinux<\/em><\/a> file, which is the uncompressed model of the kernel picture, after which utilizing an open supply instrument referred to as dwarf2json<\/em> to construct the image file. With the image file, we will now interpret the kernel reminiscence dump code into the working software program and processes.\u00a0<\/p>\n In our case, the issue was looking for the place Google Cloud hosts the vmlinux<\/em> file of the COS model of a GKE node. After a lot analysis and interplay with a few of Google\u2019s engineers who construct GKE and COS, we found an undocumented API that permits you to entry the vmlinux<\/em> file if you recognize the build_id<\/em><\/a> of the COS model working in your GKE node.<\/p>\n As the build_id<\/em> is current within the GKE picture title, we will discover it and use it to entry the API by way of the next hyperlink: https:\/\/storage.googleapis.com\/cos-tools\/$build_id\/vmlinux<\/em><\/strong>.<\/p>\n In the instance beneath, you\u2019ll see that the GKE picture has the build_id = 16919.235.1.<\/em><\/p>\n With this data, we will entry the vmlinux<\/em> file by way of:<\/p>\n https:\/\/storage.googleapis.com\/cos-tools\/16919.235.1\/vmlinux<\/a><\/p>\n and construct the image file utilizing dwarf2json<\/em>.<\/p>\n Now that we lastly have each the kernel reminiscence dump and the image file to interpret that kernel model, we will now analyze it with Volatility 3<\/em>. Using Volatility 3 permits us to see all working processes on each the privileged pod and one other take a look at pod on the identical GKE node. This \u201cattacker\u201d pod is working a sequence of take a look at processes to create some examples for us to investigate (for instance, a Netcat<\/a> listener, a watch command that queries the native IP and at last a Python script). Below, you may see the entire course of output from the kernel reminiscence dump evaluation.<\/p>\n In abstract, we will now see all of the processes on the complete GKE node for all working pods.<\/p>\n Using the three instruments talked about above has offered us with free and open supply alternate options to preexisting business options for monitoring containerized workloads. Although this method supplies a snapshot of the method exercise, it may be used both as a place to begin for reminiscence evaluation in GKE or as a complement to present business options.\u00a0<\/p>\n All the code used on this analysis challenge is accessible right here on <\/em>GitHub<\/em><\/a> and was additionally introduced at <\/em>BSidesNYC 2023<\/em><\/a>.\u00a0<\/em><\/p>\n Kubernetes is a registered trademark of the Linux Foundation within the United States and different nations.<\/em><\/p>\n <\/p>\n
\n June 22, 2023<\/span>
\n
\n Published by Marcus Hallberg, Security Engineer <\/span>
\n <\/p>\n<\/p><\/div>\n
\n ![\"Analyzing](\"https:\/\/storage.googleapis.com\/production-eng\/1\/2023\/06\/EN196_Analyzing-Volatile-Memory-final-590.png\")
<\/a>
\n \n <\/div>\n
One such analysis challenge led to the invention of a brand new technique for conducting reminiscence evaluation on GKE by combining three open supply instruments, AVML<\/a>, dwarf2json<\/a>, and Volatility 3<\/a>, the end result being a snapshot of all of the processes and reminiscence actions on a GKE node.<\/p>\n\n
![\"Example](\"https:\/\/storage.googleapis.com\/production-eng\/1\/2023\/06\/GKE-Cluster.png\")
![\"Application](\"https:\/\/storage.googleapis.com\/production-eng\/1\/2023\/06\/Kernel.png\")
\n
![\"A](\"https:\/\/storage.googleapis.com\/production-eng\/1\/2023\/06\/GKE-Architecture.png\")
Step 1: Create a kernel reminiscence dump<\/h2>\n
![\"Terraform](\"https:\/\/storage.googleapis.com\/production-eng\/1\/2023\/06\/Terraform-700x308.png\")
Step 2: Build a logo file of the kernel\u00a0<\/h2>\n
![\"GKE](\"https:\/\/storage.googleapis.com\/production-eng\/1\/2023\/06\/GKE-Image.png\")
Step 3: Analyze the kernel reminiscence dump\u00a0<\/h2>\n
![\"Example](\"https:\/\/storage.googleapis.com\/production-eng\/1\/2023\/06\/Process-Output.jpg\")
Conclusion\u00a0<\/h2>\n