![\"Netflix](\"https:\/\/miro.medium.com\/v2\/resize:fill:64:64\/1*BJWRqfSMf9Da9vsXG9EBRQ.jpeg\")
<\/div>\n<\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n
<\/div>\n
\u00b7<\/span><\/span><\/p>\n Jul 15, 2025<\/span><\/div>\n <\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n By Sergey Fedorov<\/a>, Chris Pham<\/a>, Flavio Ribeiro<\/a>, Chris Newton<\/a>, and Wei Wei<\/a><\/p>\n Many nice concepts at Netflix start with a query, and three years in the past, we requested considered one of our boldest but: if we had been to entertain the world by way of Live \u2014 a format nearly as outdated as tv itself \u2014 how would we<\/em> do it?<\/p>\n What started with an engineering plan to pave the trail in the direction of our first Live comedy particular, Chris Rock: Selective Outrage<\/a>, has since led to a whole lot of Live occasions starting from the most important comedy reveals<\/a> and NFL Christmas Games<\/a> to record-breaking boxing fights<\/a> and changing into the house of WWE<\/a>.<\/p>\n In our sequence Behind the Streams<\/em> \u2014 the place we take you thru the technical journey of our greatest bets \u2014 we are going to do a a number of half deep-dive into the structure of Live and what we discovered whereas constructing it. Part one begins with the muse we set for Live, and the crucial choices we made that influenced our strategy.<\/p>\n While Live as a tv format shouldn’t be new, the streaming expertise we supposed to construct required capabilities we didn’t have on the time. Despite 15 years of on-demand streaming beneath our belt, Live launched new concerns influencing structure and expertise decisions:<\/p>\n This implies that we had loads to construct to be able to make Live work effectively on Netflix. That begins with making the precise decisions relating to the basics of our Live Architecture.<\/p>\n Our Live Technology wanted to increase the identical promise to members that we\u2019ve made with on-demand streaming: nice high quality<\/strong> on as many gadgets<\/strong> as attainable with out interruptions<\/strong>. Live is considered one of many leisure codecs on Netflix, so we additionally wanted to seamlessly mix Live occasions into the consumer expertise, all whereas scaling to over 300 million world subscribers.<\/p>\n When we began, we had 9 months<\/strong> till the primary launch. While we wanted to execute shortly, we additionally wished to architect for future progress<\/strong> in each magnitude<\/strong> and multitude<\/strong> of occasions. As a key precept, we leveraged our distinctive place of constructing help for a single product \u2014 Netflix \u2014 and having management over the complete Live lifecycle, from Production to Screen.<\/p>\n Dedicated Broadcast Facilities to Ingest Live Content from Production<\/strong><\/p>\n Live occasions can occur wherever on the planet, however not each location has Live amenities or nice connectivity. To guarantee safe and dependable stay sign transport, we leverage distributed and extremely related broadcast operations facilities, with specialised gear for sign ingest and inspection, closed-captioning, graphics and commercial administration. We prioritized repeatability<\/strong>, conditioning engineering to launch stay occasions constantly, reliably, and cost-effectively, leaning into automation<\/strong> wherever attainable. As a consequence, we have now been in a position to cut back the event-specific setup to the transmission between manufacturing and the Broadcast Operations Center, reusing the remainder throughout occasions.<\/p>\n Cloud-based Redundant Transcoding and Packaging Pipelines<\/strong><\/p>\n The feed acquired on the Broadcast Center accommodates a completely produced program, however nonetheless must be encoded and packaged for streaming on gadgets. We selected a Cloud-based strategy to permit for dynamic scaling<\/strong>, flexibility<\/strong> in configuration, and ease of integration<\/strong> with our Digital Rights Management (DRM), content material administration, and content material supply providers already deployed within the cloud. We leverage AWS Elemental MediaConnect and AWS Elemental MediaLive to amass feeds within the cloud and transcode them into varied video high quality ranges with bitrates tailor-made per present. We constructed a customized packager<\/strong> to higher combine with our supply and playback techniques. We additionally constructed a customized Live Origin<\/strong> to make sure strict learn and write SLAs for Live segments.<\/p>\n Scaling Live Content Delivery to thousands and thousands of viewers with Open Connect CDN<\/strong><\/p>\n In order for the produced media belongings to be streamed, they should be transferred from a couple of AWS areas, the place Live Origin is deployed, to a whole lot of thousands and thousands of gadgets worldwide. We leverage Netflix\u2019s CDN, Open Connect<\/a>, to scale Live asset supply. Open Connect servers are positioned near the viewers at over 6K areas<\/strong> and related to AWS areas through a devoted Open Connect Backbone community<\/strong>.<\/p>\n By enabling Live supply on Open Connect, we construct on high of $1B+ in Netflix investments over the past 12 years targeted on scaling the community and optimizing the efficiency of supply servers. By sharing capability throughout on-demand and Live viewership we enhance utilization, and by caching previous Live content material on the identical servers used for on-demand streaming, we will simply allow catch-up viewing.<\/p>\n Optimizing Live Playback for Device Compatibility, Scale, Quality, and Stability<\/strong><\/p>\n To make Live accessible to the vast majority of our prospects with out upgrading their streaming gadgets, we settled on utilizing HTTPS<\/strong>-based Live Streaming. While UDP-based protocols can present extra options like ultra-low latency, HTTPS has ubiquitous help amongst gadgets and compatibility with supply and encoding techniques. Furthermore, we use AVC<\/strong> and HEVC<\/strong> video codecs, transcode with a number of high quality ranges up from SD to 4K<\/strong>, and use a 2-second phase<\/strong> period to stability compression effectivity, infrastructure load, and latency. While prioritizing streaming high quality and playback stability, we have now additionally achieved business commonplace latency from digicam to system, and proceed to enhance it.<\/p>\n To configure playback, the system participant receives a playback manifest on the play begin. The manifest accommodates gadgets just like the encoding bitrates and CDN servers gamers ought to use. We ship the manifest from the cloud as a substitute of the CDN, because it permits us to personalize the configuration for every system. To reference segments of the stream, the manifest features a phase template that’s utilized by gadgets to map a wall-clock time to URLs on the CDN. Using a phase template vs periodic polling for manifest updates minimizes community dependencies, CDN server load, and overhead on resource-constrained gadgets, like sensible TVs, thus bettering each scalability and stability of our system. While streaming, the participant screens community efficiency and dynamically chooses the bitrate and CDN server, maximizing streaming high quality whereas minimizing rebuffering.<\/p>\n Run Discovery and Playback Control Services within the Cloud<\/strong><\/p>\n So far, we have now coated the streaming path from Camera to Device. To make the stream absolutely work, we additionally must orchestrate throughout all techniques, and guarantee viewers can discover and begin the Live occasion. This performance is carried out by dozens of Cloud providers<\/strong>, with capabilities like playback configuration, personalization, or metrics assortment. These providers are inclined to obtain disproportionately increased masses round Live occasion begin time<\/strong>, and Cloud deployment gives flexibility in dynamically scaling compute sources. Moreover, as Live demand tends to be localized, we’re in a position to stability load throughout a number of AWS areas<\/strong>, higher using our world footprint. Deployment within the cloud additionally permits us to construct a consumer expertise the place we embed Live content material right into a broader number of leisure choices within the UI, like on-demand titles or Games.<\/p>\n Centralize Real-time Metrics within the Cloud with Specialized Tools and Facilities<\/strong><\/p>\n With management over ingest, encoding pipelines, the Open Connect CDN, and system gamers, we have now almost end-to-end observability<\/strong> into the Live workflow. During Live, we gather system and consumer metrics in real-time (e.g., the place members see the title on Netflix and their high quality of expertise), alerting us to poor consumer experiences or degraded system efficiency. Our real-time monitoring is constructed utilizing a mixture of internally developed instruments, equivalent to Atlas<\/a>, Mantis<\/a>, and Lumen<\/a>, and open-source applied sciences, equivalent to Kafka and Druid<\/a>, processing as much as 38 million occasions per second<\/strong> throughout a few of our largest stay occasions whereas offering crucial metrics and operational insights in a matter of seconds. Furthermore, we arrange devoted \u201cControl Center\u201d amenities<\/strong>, which deliver key metrics collectively to the operational group that screens the occasion in real-time.<\/p>\n Building new performance all the time brings contemporary challenges and alternatives to be taught, particularly with a system as complicated as Live. Even after three years, we\u2019re nonetheless studying daily find out how to ship Live occasions extra successfully. Here are a couple of key highlights:<\/p>\n Extensive testing: <\/strong>Prior to Live we closely relied on the predictable circulation of on-demand visitors for pre-release canaries or A\/B checks to validate deployments. But Live visitors was not all the time obtainable, particularly not on the scale consultant of an enormous launch. As a consequence, we spent appreciable effort to:<\/p>\n Regular apply: <\/strong>Despite rigorous pre-release testing, nothing beats a manufacturing surroundings, particularly when working at scale. We discovered that having a daily schedule with numerous Live content material is important to creating enhancements whereas balancing the dangers of member affect. We run A\/B checks<\/a>, carry out chaos testing<\/a>, operational workout routines, and prepare operational groups for upcoming launches.<\/p>\n Viewership predictions:<\/strong> We use prediction-based strategies to pre-provision Cloud and CDN capability, and share forecasts with our ISP and Cloud companions forward of time to allow them to plan community and compute sources. Then we complement them with reactive scaling of cloud techniques powering sign-up, log-in, title discovery, and playback providers to account for viewership exceeding our predictions. We have discovered success with forward-looking real-time viewership predictions throughout<\/em> a stay occasion, permitting us to take steps to mitigate dangers earlier, earlier than extra members are impacted.<\/p>\n Graceful degradation: <\/strong>Despite our greatest efforts, we will (and did!) discover ourselves in a scenario the place viewership exceeded our predictions and provisioned capability. In this case, we developed a lot of levers to proceed streaming, even when it means steadily eradicating some nice-to-have options. For instance, we use service-level prioritized load shedding<\/a> to prioritize stay visitors over non-critical visitors (like pre-fetch). Beyond that, we will lighten the expertise, like dialing down personalization, disabling bookmarks, or reducing the utmost streaming high quality. Our load checks embody situations the place we under-scale techniques to validate desired conduct.<\/p>\n Retry storms: <\/strong>When techniques attain capability, our key focus is to keep away from cascading points or additional overloading techniques with retries. <\/strong>Beyond system retries, customers might retry manually \u2014 we\u2019ve seen a 10x enhance in visitors load resulting from stream restarts after viewing interruptions of as little as 30 seconds. We spent appreciable time understanding system retry conduct within the presence of points like community timeouts or lacking segments. As a consequence, we applied methods like server-guided backoff for system retries, absorbing spikes through prioritized visitors shedding at Cloud Edge Gateway, and re-balancing visitors between cloud areas.<\/p>\n Contingency planning: <\/strong>\u201cEveryone has a plan until they get punched in the mouth<\/em>\u201d could be very related for Live. When one thing breaks, there may be virtually no time for troubleshooting. For massive occasions, we arrange in-person launch rooms<\/strong> with engineering house owners of crucial techniques. For fast detection and response, we developed a small set of metrics as early indicators of points, and have in depth runbooks for frequent operational points. We don\u2019t be taught on launch day; as a substitute, launch groups apply failure response through Game Day workout routines forward of time<\/strong>. Finally, our runbooks prolong past engineering, masking escalation to government management and coordination throughout capabilities like Customer Service, Production, Communications, or Social.<\/p>\n Our dedication to enhancing the member expertise doesn\u2019t finish on the \u201cThanks for Watching!\u201d display screen. Shortly after every stay stream, we dive into metrics to determine areas for enchancment. Our Data & Insights group conducts complete analyses, A\/B checks, and client analysis to make sure the subsequent occasion is much more pleasant for our members. We leverage insights on member conduct, preferences, and expectations to refine the Netflix product expertise and optimize our Live expertise \u2014 like lowering latency by ~10 seconds by way of A\/B checks, with out affecting high quality or stability.<\/p>\n<\/picture><\/div>\n<\/div>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/format:webp\/1*drjqrhSVFS7jfQzOBoG7dA.png)
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/format:webp\/1*KWo6BzotdnxqoveWCvY12Q.png)
\n
\u2014 Capture, modify, and replay previous Live manufacturing visitors, representing a range of consumer gadgets and request patterns.
\u2014 Virtualize Netflix gadgets and generate visitors in opposition to CDN or Cloud endpoints to check the affect of the newest adjustments throughout all techniques.<\/li>\n