![\"Netflix](\"https:\/\/miro.medium.com\/v2\/resize:fill:88:88\/1*BJWRqfSMf9Da9vsXG9EBRQ.jpeg\")
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/a><\/p>\n <\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n How Netflix\u2019s Container Platform Connects Linux Kernel Panics to Kubernetes Pods<\/p>\n By Kyle Anderson<\/em><\/p>\n With a latest effort to scale back buyer (engineers, not finish customers) ache on our container platform Titus<\/a>, I began investigating \u201corphaned\u201d pods. There are pods that by no means acquired to complete and needed to be rubbish collected with no actual passable remaining standing. Our Service job (suppose ReplicatSet<\/a>) house owners don\u2019t care an excessive amount of, however our Batch customers care loads. Without an actual return code, how can they know whether it is protected to retry or not?<\/p>\n These orphaned pods symbolize actual ache for our customers, even when they’re a small share of the whole pods within the system. Where are they going, precisely? Why did they go away?<\/p>\n This weblog submit exhibits the right way to join the dots from the worst case situation (a kernel panic) by to Kubernetes (k8s) and finally as much as us operators in order that we are able to observe how and why our k8s nodes are going away.<\/p>\n Orphaned pods get misplaced as a result of the underlying k8s node object goes away. Once that occurs a GC<\/a> course of deletes the pod. On Titus we run a customized controller to retailer the historical past of Pod and Node objects, in order that we are able to avoid wasting clarification and present it to our customers. This failure mode appears like this in our UI:<\/p>\n This is an <\/em>clarification, but it surely wasn\u2019t very satisfying to me or to our customers. Why<\/em> was the agent misplaced?<\/p>\n Nodes can go away for any motive, particularly in \u201cthe cloud\u201d. When this occurs, normally a k8s cloud-controller offered by the cloud vendor will detect that the precise server, in our case an EC2 Instance, has truly gone away, and can in flip delete the k8s node object. That nonetheless doesn\u2019t actually reply the query of why<\/em>.<\/p>\n How can we ensure that each occasion that goes away has a motive, account for that motive, and bubble it up all the best way to the pod? It all begins with an annotation:<\/p>\n Just making a spot to place this information is a superb begin. Now all we’ve got to do is make our GC controllers conscious of this annotation, after which sprinkle it into any course of that would probably make a pod or node go away unexpectedly. Adding an annotation (versus patching the standing) preserves the remainder of the pod as-is for historic functions. (We additionally add annotations for what did the terminating, and a brief The But wait, how are we going to annotate a pod for a node that kernel panicked?<\/p>\n When the Linux kernel panics, there may be simply not a lot you are able to do. But what in the event you may ship out some type of \u201cwith my final breath, I curse Kubernetes!\u201d UDP packet?<\/p>\n Inspired by this Google Spanner paper<\/a>, the place Spanner nodes ship out a \u201clast gasp\u201d UDP packet to launch leases & locks, you can also configure your servers to do the identical upon kernel panic utilizing a inventory Linux module: The indisputable fact that the Linux kernel may even ship out UDP packets with the string \u2018kernel panic\u2019, whereas it’s panicking<\/em>, is form of wonderful. This works as a result of netconsole must be configured with nearly the complete IP header crammed out already beforehand. That is true, it’s important to inform Linux precisely what your supply MAC, IP, and UDP Port are, in addition to the vacation spot MAC, IP, and UDP ports. You are virtually setting up the UDP packet for the kernel. But, with that prework, when the time comes, the kernel can simply assemble<\/a> the packet and get it out the (preconfigured) community interface as issues come crashing down. Luckily the Once that is setup, kernel messages will begin flowing proper after With The final piece is to attach is Kubernetes (k8s). We want a k8s controller to do the next:<\/p>\n Parts 1&2 may appear like this:<\/p>\n And then components 3&4 may appear like this:<\/p>\n With that code in place, as quickly as a kernel panic is detected, the pods and nodes instantly go away. No want to attend for any GC course of. The annotations assist doc what occurred to the node & pod:<\/p>\n Marking {that a} job failed due to a kernel panic is probably not that<\/em> passable to our prospects. But they’ll take satisfaction in figuring out that we now have the required observability instruments to begin fixing these kernel panics!<\/p>\n![\"Netflix](\"https:\/\/miro.medium.com\/v2\/resize:fill:48:48\/1*ty4NvNrGg4ReETxqU2N3Og.png\")
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/picture><\/div>\n<\/div>
{reason-code<\/code> for tagging)<\/p>\npod-termination-reason<\/code> annotation is helpful to populate human readable messages like:<\/p>\n\n
netconsole<\/a><\/code>.<\/p>\nnetconsole-setup<\/a><\/code> command makes the setup fairly simple. All the configuration choices may be set dynamically<\/a> as nicely, in order that when the endpoint modifications one can level to the brand new IP.<\/p>\nmodprobe<\/code>. Imagine the entire thing working like a dmesg | netcat -u $vacation spot 6666<\/code>, however in kernel house.<\/p>\nnetconsole<\/code> setup, the final gasp from a crashing kernel appears like a set of UDP packets precisely like one may count on, the place the info of the UDP packet is just the textual content of the kernel message. In the case of a kernel panic, it can look one thing like this (one UDP packet per line):<\/p>\nKernel panic - not syncing: buffer overrun at 0x4ba4c73e73acce54\n
for {func deal withKernelPanicOnNode(ctx context.Context, addr *internet.UDPAddr, nodeInformer cache.SharedIndexInformer, podInformer cache.SharedIndexInformer, kubeClient kubernetes.Interface, line string) {![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/format:webp\/1*cjClRuyUQ67lu2shmjCObQ.png)