{"id":133934,"date":"2024-12-10T17:28:17","date_gmt":"2024-12-10T17:28:17","guid":{"rendered":"https:\/\/showbizztoday.com\/index.php\/2024\/12\/10\/investigation-of-a-workbench-ui-latency-issue-by-netflix-technology-blog-oct-2024\/"},"modified":"2024-12-10T17:28:18","modified_gmt":"2024-12-10T17:28:18","slug":"investigation-of-a-workbench-ui-latency-issue-by-netflix-technology-blog-oct-2024","status":"publish","type":"post","link":"https:\/\/showbizztoday.com\/index.php\/2024\/12\/10\/investigation-of-a-workbench-ui-latency-issue-by-netflix-technology-blog-oct-2024\/","title":{"rendered":"Investigation of a Workbench UI Latency Issue | by Netflix Technology Blog | Oct, 2024"},"content":{"rendered":"

[ad_1]
\n<\/p>\n

\n
\n
\n
\n
\n
\n
\n
\n
\n
<\/p>\n
\n
\"Netflix<\/div>\n<\/div>\n

<\/a><\/div>\n<\/div>\n

\n
\n
<\/p>\n
\n
\"Netflix<\/div>\n<\/div>\n

<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\n
<\/p>\n
<\/p>\n
12 min learn<\/span><\/p>\n

\u00b7<\/span><\/span><\/p>\n

Oct 14, 2024<\/span><\/div>\n

<\/span><\/div>\n

<\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

By: Hechao Li<\/a> and Marcelo Mayworm<\/a><\/p>\n

With particular due to our gorgeous colleagues Amer Ather<\/a>, Itay Dafna<\/a>, Luca Pozzi<\/a>, Matheus Le\u00e3o<\/a>, and Ye Ji<\/a>.<\/p>\n

At Netflix, the Analytics and Developer Experience group, a part of the Data Platform, presents a product known as Workbench. Workbench is a distant growth workspace primarily based on Titus<\/a> that permits information practitioners to work with large information and machine studying use instances at scale. A standard use case for Workbench is operating JupyterLab<\/a> Notebooks.<\/p>\n

Recently, a number of customers reported that their JupyterLab UI turns into sluggish and unresponsive when operating sure notebooks. This doc particulars the intriguing strategy of debugging this difficulty, all the best way from the UI right down to the Linux kernel.<\/p>\n

Machine Learning engineer Luca Pozzi<\/a> reported to our Data Platform staff that their JupyterLab UI on their workbench turns into sluggish and unresponsive when operating a few of their Notebooks.<\/strong> Restarting the ipykernel<\/em> course of, which runs the Notebook, may quickly alleviate the issue, however the frustration persists as extra notebooks are run.<\/p>\n

While we noticed the problem firsthand, the time period \u201cUI being slow\u201d is subjective and tough to measure. To examine this difficulty, we would have liked a quantitative evaluation of the slowness<\/strong>.<\/p>\n

Itay Dafna<\/a> devised an efficient and easy technique to quantify the UI slowness. Specifically, we opened a terminal by way of JupyterLab and held down a key (e.g., \u201cj\u201d) for 15 seconds whereas operating the person\u2019s pocket book. The enter to stdin is shipped to the backend (i.e., JupyterLab) by way of a WebSocket, and the output to stdout is shipped again from the backend and displayed on the UI. We then exported the .har <\/em>file recording all communications from the browser and loaded it right into a Notebook for evaluation.<\/p>\n

\n
\n
\"\"<\/picture><\/div>\n<\/div>\n<\/figure>\n

Using this method, we noticed latencies starting from 1 to 10 seconds, averaging 7.4 seconds.<\/p>\n

\n
\n
\"\"<\/picture><\/div>\n<\/div>\n<\/figure>\n

Now that now we have an goal metric for the slowness, let\u2019s formally begin our investigation. If you may have learn the symptom rigorously, you will need to have seen that the slowness solely happens when the person runs sure<\/strong> notebooks however not others.<\/p>\n

Therefore, step one is scrutinizing the precise Notebook experiencing the problem. Why does the UI at all times decelerate after operating this explicit Notebook? Naturally, you’d assume that there should be one thing incorrect with the code operating in it.<\/p>\n

Upon carefully inspecting the person\u2019s Notebook, we seen a library known as pystan<\/em> , which offers Python bindings to a local C++ library known as stan, seemed suspicious. Specifically, pystan<\/em> makes use of asyncio<\/em>. However, as a result of there’s already an present asyncio<\/em> occasion loop operating within the Notebook course of and asyncio<\/em> can’t be nested by design, to ensure that pystan<\/em> to work, the authors of pystan<\/em> <\/strong>suggest<\/strong><\/a> injecting pystan<\/em> into the present occasion loop through the use of a package deal known as <\/strong>nest_asyncio<\/em><\/strong><\/a>, a library that turned unmaintained as a result of the writer sadly handed away<\/a>.<\/p>\n

Given this seemingly hacky utilization, we naturally suspected that the occasions injected by pystan<\/em> into the occasion loop have been blocking the dealing with of the WebSocket messages used to speak with the JupyterLab UI. This reasoning sounds very believable. However, the person claimed that there have been instances when a Notebook not utilizing pystan<\/em> runs, the UI additionally turned sluggish<\/strong>.<\/p>\n

Moreover, after a number of rounds of debate with ChatGPT, we discovered extra in regards to the structure and realized that, in principle, the utilization of pystan<\/em> and nest_asyncio<\/em> mustn’t trigger the slowness in dealing with the UI WebSocket<\/strong> for the next causes:<\/p>\n

Even although pystan<\/em> makes use of nest_asyncio<\/em> to inject itself into the primary occasion loop, the Notebook runs on a toddler course of (i.e.<\/strong>, the ipykernel<\/em> course of) of the jupyter-lab<\/em> server course of<\/strong>, which suggests the primary occasion loop being injected by pystan<\/em> is that of the ipykernel<\/em> course of, not the jupyter-server<\/em> course of. Therefore, even when pystan<\/em> blocks the occasion loop, it shouldn\u2019t impression the jupyter-lab<\/em> most important occasion loop that’s used for UI websocket communication. See the diagram under:<\/p>\n

\n
\n
\"\"<\/picture><\/div>\n<\/div>\n<\/figure>\n

In different phrases, pystan<\/em> occasions are injected to the occasion loop B on this diagram as an alternative of occasion loop A<\/strong>. So, it shouldn\u2019t block the UI WebSocket occasions.<\/p>\n

You may additionally assume that as a result of occasion loop A handles each the WebSocket occasions from the UI and the ZeroMQ socket occasions from the ipykernel<\/em> course of, a excessive quantity of ZeroMQ occasions generated by the pocket book might block the WebSocket. However, after we captured packets on the ZeroMQ socket whereas reproducing the problem, we didn\u2019t observe heavy site visitors on this socket that might trigger such blocking<\/strong>.<\/p>\n

A stronger piece of proof to rule out pystan<\/em> was that we have been finally in a position to reproduce the problem even with out it, which I\u2019ll dive into later.<\/p>\n

The Workbench occasion runs as a Titus container<\/a>. To effectively make the most of our compute sources, Titus employs a CPU oversubscription function<\/strong>, that means the mixed digital CPUs allotted to containers exceed the variety of out there bodily CPUs on a Titus agent. If a container is unlucky sufficient to be scheduled alongside different \u201cnoisy\u201d containers \u2014 people who devour lots of CPU sources \u2014 it might endure from CPU deficiency.<\/strong><\/p>\n

However, after inspecting the CPU utilization of neighboring containers on the identical Titus agent because the Workbench occasion, in addition to the general CPU utilization of the Titus agent, we shortly dominated out this speculation. Using the highest command on the Workbench, we noticed that when operating the Notebook, the Workbench occasion makes use of solely 4 out of the 64 CPUs allotted to it<\/strong>. Simply put, this workload just isn’t CPU-bound.<\/strong><\/p>\n

\n
\n
\"\"<\/picture><\/div>\n<\/div>\n<\/figure>\n

The subsequent principle was that the community between the online browser UI (on the laptop computer) and the JupyterLab server was sluggish. To examine, we captured all of the packets between the laptop computer and the server<\/strong> whereas operating the Notebook and repeatedly urgent \u2018j\u2019 within the terminal.<\/p>\n

When the UI skilled delays, we noticed a 5-second pause in packet transmission from server port 8888 to the laptop computer. Meanwhile, site visitors from different ports, akin to port 22 for SSH, remained unaffected<\/strong>. This led us to conclude that the pause was attributable to the applying operating on port 8888 (i.e., the JupyterLab course of) slightly than the community.<\/p>\n

\n
\n
\"\"<\/picture><\/div>\n<\/div>\n<\/figure>\n

As beforehand talked about, one other robust piece of proof proving the innocence of pystan was that we might reproduce the problem with out it. By regularly stripping down the \u201cbad\u201d Notebook, we finally arrived at a minimal snippet of code that reproduces the problem with none third-party dependencies or complicated logic:<\/p>\n

import timeimport osfrom multiprocessing import Process
N = os.cpu_count()<\/p>
def launch_worker(worker_id):time.sleep(60)<\/p>
if __name__ == '__main__':with open('\/root\/2GB_file', 'r') as file:information = file.learn()processes = []for i in vary(N):p = Process(goal=launch_worker, args=(i,))processes.append(p)p.begin()<\/p>
for p in processes:p.be a part of()<\/p><\/span><\/pre>\n
The code does solely two issues:<\/p>\n
    \n
  1. Read a 2GB file into reminiscence (the Workbench occasion has 480G reminiscence in complete so this reminiscence utilization is sort of negligible).<\/li>\n
  2. Start N processes the place N is the variety of CPUs. The N processes do nothing however sleep.<\/li>\n<\/ol>\n
    There is little question that that is probably the most foolish piece of code I\u2019ve ever written. It is neither CPU certain nor reminiscence certain. Yet it may well trigger the JupyterLab UI to stall for as many as 10 seconds!<\/strong><\/p>\n
    There are a few attention-grabbing observations that elevate a number of questions:<\/p>\n