![\"Netflix](\"https:\/\/miro.medium.com\/v2\/resize:fill:88:88\/1*BJWRqfSMf9Da9vsXG9EBRQ.jpeg\")
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/a><\/p>\n <\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n Tycho Andersen<\/a><\/p>\n The Compute staff at Netflix is charged with managing all AWS and containerized workloads at Netflix, together with autoscaling, deployment of containers, situation remediation, and many others. As a part of this staff, I work on fixing unusual issues that customers report.<\/p>\n This specific situation concerned a customized inner FUSE filesystem<\/a>: ndrive<\/a>. It had been festering for a while, however wanted somebody to take a seat down and have a look at it in anger. This weblog submit describes how I poked at We had a caught docker API name:<\/p>\n Here, our administration engine has made an HTTP name to the Docker API\u2019s unix socket asking it to kill a container. Our containers are configured to be killed by way of Hmm. Seems prefer it\u2019s alive, however Ok, so the container\u2019s init course of remains to be alive, nevertheless it has one zombie baby. What might the container\u2019s init course of presumably be doing?<\/p>\n It is within the means of exiting, nevertheless it appears caught. The solely baby is the ndrive course of in Z (i.e. \u201czombie\u201d) state, although. Zombies are processes which have efficiently exited, and are ready to be reaped by a corresponding Ah ha, there are two threads within the thread group. One of them is a zombie, possibly the opposite one isn\u2019t:<\/p>\n Indeed it’s not a zombie. It is making an attempt to grow to be one as onerous as it may well, nevertheless it\u2019s blocking inside FUSE for some cause. To discover out why, let\u2019s have a look at some kernel code. If we have a look at which is the place we’re caught, however earlier than that, it has achieved:<\/p>\n which is why docker can\u2019t Ok, so we\u2019re ready for an occasion (on this case, that userspace has replied to the FUSE flush request). But Viewing course of standing this fashion, you may see Well that’s bizarre. The wait code (i.e. So it loops eternally, doing spin_lock_irqsave(&wq_head->lock, flags); return ret; It seems to be like the one approach we are able to get away of this with a non-zero exit code is that if return (state & TASK_INTERRUPTIBLE) <\/p><\/span><\/pre>\n Our process will not be interruptible, so the primary if fails. Our process ought to have a sign pending, although, proper?<\/p>\n As the remark notes, Hmm. Seems like we should always have that flag set, proper? To determine that out, let\u2019s have a look at how sign supply works. When we\u2019re shutting down the pid namespace in which finally will get to Using But it seems it is a little bit of a pink herring (though<\/a> it<\/a> took<\/a> a<\/a> whereas<\/a> for me to know that).<\/p>\n To perceive what\u2019s actually occurring right here, we have to have a look at however why doesn\u2019t it work? At the highest of the perform we’ve:<\/p>\n however as Eric Biederman described<\/a>, mainly each thread can deal with a So\u2026 if a thread is already exiting (i.e. it has 1. a process opens a FUSE file, and doesn\u2019t shut it, then exits. During that exit, the kernel dutifully calls 2. 3. the pid namespace exits, and enters 4. this kills the FUSE daemon within the pid ns so it may well by no means reply.<\/p>\n 5. 6. Deadlock. Without manually aborting the FUSE connection, issues will cling eternally.<\/p>\n It doesn\u2019t actually make sense to attend for flushes on this case: the duty is dying, so there\u2019s no one to inform the return code of Individual filesystems will should be patched within the meantime, for instance the repair for FUSE is right here<\/a>, which was launched on April 23 in Linux 6.3.<\/p>\n![\"Netflix](\"https:\/\/miro.medium.com\/v2\/resize:fill:48:48\/1*ty4NvNrGg4ReETxqU2N3Og.png\")
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\/proc<\/code>to get a way of what was occurring, earlier than posting the problem to the kernel mailing record and getting schooled on how the kernel\u2019s wait code truly works!<\/p>\ngoroutine 146 [select, 8817 minutes]:SIGKILL<\/code>. But that is unusual. kill(SIGKILL)<\/code> needs to be comparatively deadly, so what’s the container doing?<\/p>\n$ docker exec -it 6643cd073492 bashsetns(2)<\/code> fails. Why would that be? If we have a look at the method tree by way of ps awwfux<\/code>, we see:<\/p>\n_ containerd-shim -namespace moby -workdir \/var\/lib\/containerd\/io.containerd.runtime.v1.linux\/moby\/6643cd073492ba9166100ed30dbe389ff1caef0dc3d35# cat \/proc\/1528591\/stackwait()<\/code> syscall from their dad and mom. So how might the kernel be caught ready on a zombie?<\/p>\n# ls \/proc\/1544450\/process# cat \/proc\/1544574\/stackzap_pid_ns_processes()<\/a><\/code>, it does:<\/p>\n\/*\/* Don't enable any extra processes into the pid namespace *\/setns()<\/code> \u2014 the namespace<\/em> is a zombie. Ok, so we are able to\u2019t setns(2)<\/code>, however why are we caught in kernel_wait4()<\/code>? To perceive why, let\u2019s have a look at what the opposite thread was doing in FUSE\u2019s request_wait_answer()<\/a><\/code>:<\/p>\n\/*zap_pid_ns_processes()<\/code>despatched a SIGKILL<\/code>! SIGKILL<\/code> needs to be very deadly to a course of. If we have a look at the method, we are able to certainly see that there\u2019s a pending SIGKILL<\/code>:<\/p>\n# grep Pnd \/proc\/1544574\/standing0x100<\/code> (i.e. the ninth bit is ready) underneath ShdPnd<\/code>, which is the sign quantity comparable to SIGKILL<\/code>. Pending alerts are alerts which have been generated by the kernel, however haven’t but been delivered to userspace. Signals are solely delivered at sure instances, for instance when getting into or leaving a syscall, or when ready on occasions. If the kernel is at present doing one thing on behalf of the duty, the sign could also be pending. Signals may also be blocked by a process, in order that they’re by no means delivered. Blocked alerts will present up of their respective pending units as properly. However, man 7 sign<\/code> says: \u201cThe signals SIGKILL<\/code> and SIGSTOP<\/code> cannot be caught, blocked, or ignored.\u201d But right here the kernel is telling us that we’ve a pending SIGKILL<\/code>, aka that it’s being ignored even whereas the duty is ready!<\/p>\nembody\/linux\/wait.h<\/code>) is used in all places within the kernel: semaphores, wait queues, completions, and many others. Surely it is aware of to search for SIGKILL<\/code>s. So what does wait_event()<\/code> truly do? Digging by way of the macro expansions and wrappers, the meat of it’s:<\/p>\n#outline ___wait_event(wq_head, situation, state, unique, ret, cmd) prepare_to_wait_event()<\/code>, checking the situation, then checking to see if we have to interrupt. Then it does cmd<\/code>, which on this case is schedule()<\/code>, i.e. \u201cdo something else for a while\u201d. prepare_to_wait_event()<\/code> seems to be like:<\/p>\nlengthy prepare_to_wait_event(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state)signal_pending_state()<\/code> is true. Since our name website was simply wait_event()<\/code>, we all know that state right here is TASK_UNINTERRUPTIBLE<\/code>; the definition of signal_pending_state()<\/code> seems to be like:<\/p>\nstatic inline int signal_pending_state(unsigned int state, struct task_struct *p)static inline int signal_pending(struct task_struct *p)TIF_NOTIFY_SIGNAL<\/code> isn\u2019t related right here, regardless of its identify, however let\u2019s have a look at task_sigpending()<\/code>:<\/p>\nstatic inline int task_sigpending(struct task_struct *p)zap_pid_ns_processes()<\/code>, it does:<\/p>\ngroup_send_sig_info(SIGKILL, SEND_SIG_PRIV, process, PIDTYPE_MAX);<\/span><\/pre>\n__send_signal_locked()<\/code>, which has:<\/p>\npending = (sort != PIDTYPE_PID) ? &t->signal->shared_pending : &t->pending;PIDTYPE_MAX<\/code> right here as the sort is slightly bizarre, nevertheless it roughly signifies \u201cthis is very privileged kernel stuff sending this signal, you should definitely deliver it\u201d. There is a little bit of unintended consequence right here, although, in that __send_signal_locked()<\/code> finally ends up sending the SIGKILL<\/code> to the shared set, as an alternative of the person process\u2019s set. If we have a look at the __fatal_signal_pending()<\/code> code, we see:<\/p>\nstatic inline int __fatal_signal_pending(struct task_struct *p)complete_signal()<\/code>, because it unconditionally provides a SIGKILL<\/code> to the duty\u2019s pending set:<\/p>\nsigaddset(&t->pending.sign, SIGKILL);<\/span><\/pre>\n\/*SIGKILL<\/code> at any time. Here\u2019s wants_signal()<\/code>:<\/p>\nstatic inline bool wants_signal(int sig, struct task_struct *p)PF_EXITING<\/code>), it doesn\u2019t need a sign. Consider the next sequence of occasions:<\/p>\ndo_exit()<\/code>, which does the next:<\/p>\nexit_signals(tsk); \/* units PF_EXITING *\/<\/span><\/pre>\ndo_exit()<\/code> continues on to exit_files(tsk);<\/code>, which flushes all recordsdata which are nonetheless open, ensuing within the stack hint above.<\/p>\nzap_pid_ns_processes()<\/code>, sends a SIGKILL<\/code> to everybody (that it expects to be deadly), after which waits for everybody to exit.<\/p>\ncomplete_signal()<\/code> for the FUSE process that was already exiting ignores the sign, because it has PF_EXITING<\/code>.<\/p>\nflush()<\/code> to. It additionally seems that this bug can occur with a number of filesystems (something that calls the kernel\u2019s wait code in flush()<\/code>, i.e. mainly something that talks to one thing exterior the native kernel).<\/p>\n