Maximizing immersion for our members is a vital objective for the Netflix product and engineering groups to maintain our members entertained and absolutely engaged in our content material. Leveraging a superb mixture of mature and cutting-edge shopper machine applied sciences to ship a clean playback expertise with glitch-free in-app transitions is a vital step in direction of attaining this objective. In this text we clarify our journey in direction of productizing a greater viewing expertise for our members by using options and capabilities in shopper streaming units.
If you have got a streaming machine linked to your TV, resembling a Roku Set Top Box (STB) or an Amazon FireTV Stick, you will have come throughout an choice within the machine show setting pertaining to content material body fee. Device producers usually name this characteristic “Match Content Frame Rate”, “Auto adjust display refresh rate” or one thing comparable. If you’ve ever puzzled what these options are and the way they’ll enhance your viewing expertise, hold studying — the next sections cowl the fundamentals of this characteristic and clarify the main points of how the Netflix software makes use of it.
Netflix’s content material catalog consists of video captured and encoded in one in every of varied body charges starting from 23.97 to 60 frames per second (fps). When a member chooses to observe a film or a TV present on a supply machine (ex. Set-top field, Streaming stick, Game Console, and so forth…) the content material is delivered after which decoded at its native body fee, which is the body fee it was captured and encoded in. After the decode step, the supply machine converts it to the HDMI output body fee which was configured based mostly on the capabilities of the HDMI enter port of the linked sink machine (TV, AVR, Monitor and so forth). In normal, the output body fee over HDMI is robotically set to 50fps for PAL areas and 60fps for NTSC areas.
Netflix gives restricted excessive body fee content material (50fps or 60fps), however the majority of our catalog and viewing hours may be attributed to members watching 23.97 to 30fps content material. This primarily implies that more often than not, our content material goes by means of a course of known as body fee conversion (aka FRC) on the supply machine which converts the content material from its native body fee to match the HDMI output body fee by replicating frames. Figure 1 illustrates a easy FRC algorithm that converts 24fps content material to 60fps.
Converting the content material and transmitting it over HDMI on the output body fee sounds logical and easy. In truth, FRC works properly when the output body fee is an integer a number of of the native body fee ( ex. 24→48, 25→50, 30→60, 24→120, and so forth…). On the opposite hand, FRC introduces a visible artifact known as Judder when non-integer a number of conversion is required (ex. 24→60, 25→60, and so forth…), which manifests as uneven video playback as illustrated under:
It is essential to notice that the severity of the judder depends upon the replication sample. For this purpose, judder is extra outstanding in PAL areas due to the method of changing 24fps content material to 50fps over HDMI (see Figure 2):
- Total of fifty frames should be transmitted over HDMI per second
- Source machine should replicate the unique 24 frames to fill within the lacking 26 frames
- 50 output frames from 24 unique frames are derived as follows:
- 22 frames are duplicated ( whole of 44 frames )
- 2 frames are repeated 3 times ( whole of 6 frames )
As a evaluation, judder is extra pronounced when the frequency of the variety of repeated frames is inconsistent and unfold out e.g. within the situation talked about above, the body replication issue varies between 2 and three leading to a extra outstanding judder.
Now that we have now a greater understanding of the problem, let’s evaluation the options that Netflix has invested in. Due to the fragmented nature of machine capabilities within the ecosystem, we explored a number of options to handle this concern for as many units as attainable. Each distinctive resolution leverages present or new supply machine capabilities and comes with varied tradeoffs.
The first resolution we explored and just lately enabled leverages the potential of present supply & sink units to alter the outgoing body fee on the HDMI hyperlink. Once this characteristic is enabled within the system settings, units will match the HDMI output body fee with the content material body fee, both precisely or an integer a number of, with out consumer intervention.
While this feels like the right resolution, units that assist older HDMI applied sciences e.g. HDMI v<2.1, can’t change the body fee with out additionally altering the HDMI knowledge fee. This leads to what is commonly referred as an “HDMI bonk” which causes the TV to show a clean display screen momentarily. Not solely is that this a disruptive expertise for members, however the length of the clean display screen varies relying on how briskly the supply and sink units can resynchronize. Figure 3 under is an instance of how this transition appears to be like:
Improvements within the current HDMI requirements (HDMI 2.1+) now enable a supply machine to ship the video content material at its native body fee while not having an HDMI resynchronization. This is feasible by means of an modern know-how known as Quick Media Switching (QMS) which is an extension of Variable Refresh Rate (VRR) focused for content material playback situations. QMS permits a supply machine to keep up a continuing knowledge fee on the HDMI hyperlink even throughout transmission of content material with totally different body charges. It does so by adjusting the quantity of non-visible padding knowledge whereas maintaining the quantity of seen video knowledge fixed. Due to the fixed HDMI knowledge fee, the HDMI transmitter and receiver don’t must resynchronize, resulting in a seamless/glitch-free transition as illustrated in Figure 4.
HDMI QMS is positioned to be the best resolution to handle the issue we’re presenting. Unfortunately, at current, this know-how is comparatively new and adoption into supply and sink units will take time.
Apart from the above HDMI specification dependent options, it’s attainable for an software like Netflix to control the presentation time stamp worth of every video body to reduce the impact of judder i.e. the appliance can current video frames to the underlying supply machine platform at a cadence that may assist the supply machine to reduce the judder related to FRC on the HDMI output hyperlink.
Let us perceive this concept with the assistance of an instance. Let’s return to the identical 24 to 50 fps FRC situation that was lined earlier. But, as a substitute of desirous about the FRC fee per second (24 ⇒ 50 fps), let’s develop the FRC calculation time interval to three seconds (24*3 = 72 ⇒50*3 = 150 fps). For content material with a local body fee of 24 fps, the supply machine must get 72 frames from the streaming software in a interval of three seconds. Now as a substitute of sending 24 frames per second at an everyday per second cadence, for every 3 second interval the Netflix software can determine to ship 25 frames within the first 2 seconds (25 x 2 = 50) and 22 frames within the third second thereby nonetheless sending a complete of 72 (50+22) frames in 3 seconds. This method creates an excellent FRC within the first 2 seconds (25 frames replicated twice evenly) and within the third second the supply machine can do a 22 to 50 fps FRC which is able to create much less visible judder in comparison with the 24->50 fps FRC given a extra even body replication sample. This idea is illustrated in Figure 5 under.
NOTE: This resolution was developed by David Zheng within the Partner Experience Technology staff at Netflix. Watch out for an upcoming article going into additional particulars of this resolution.
Given the attainable options out there to make use of and the related advantages and limitations, the Netflix software operating on a supply machine adapts to make use of one in every of these approaches based mostly on elements resembling supply and sink machine capabilities, consumer preferences and the particular use case inside the Netflix software. Let’s stroll by means of every of those points briefly.
Every supply machine that integrates the Netflix software is required to let the appliance know if it and the linked sink machine have the power to ship and obtain video content material at its native body fee. In addition, a supply machine is required to tell whether or not it may well assist QMS and carry out a seamless playback begin of any content material at its native body fee on the linked HDMI hyperlink.
As mentioned within the introduction part, the presence of a system setting like “Match Content Frame Rate” sometimes signifies {that a} supply machine is able to this characteristic.
Even if a supply machine and the linked sink can assist Native content material body fee streaming (seamless or non-seamless), a consumer might need chosen not to do that by way of the supply machine system settings e.g. “Match Content Frame Rate” set to “Never”. Or they could have indicated a desire of doing this solely when the native content material body fee play begin can occur in a seamless method e.g. “Match Content Frame Rate” set to “Seamless”.
The Netflix software must know this consumer choice as a way to honor their desire. Hence, supply units are anticipated to relay this consumer desire to the Netflix software to assist with this run-time choice making.
In spite of supply machine functionality and the consumer preferences collectively indicating that the Native Content Frame Rate streaming needs to be enabled, the Netflix software can determine to disable this characteristic for particular member experiences. As an instance, when the consumer is searching Netflix content material within the house UI, we can’t play Netflix trailers of their Native body fee because of the following causes:
- If utilizing Solution # 1, when the Netflix trailers are encoded in various content material body charges, switching between trailers will end in display screen blanking, thereby making the UI searching unusable.
- If utilizing Solution # 2, sending Netflix trailers of their Native body fee would imply that the related UI parts (motion of cursor, asset choice and so forth) would even be displayed on the lowered body fee and it will end in a sluggish UI searching expertise. This is as a result of on HDMI output from the supply machine, each graphics (Netflix software UI) and video parts will exit on the similar body fee (native content material body fee of the trailer) after being blended collectively on the supply machine.
To deal with these points we comply with an method as proven in Figure 6 under the place we allow the Native Frame Rate playback expertise solely when the consumer selects a title and watches it in full display screen with minimal graphical UI parts.
This article offered options that intention to enhance the content material playback expertise on HDMI supply units. The breadth of accessible technical options, consumer selectable preferences, machine capabilities and the appliance of every of those permutations within the context of assorted in-app member journeys characterize a typical engineering and product choice framework at Netflix. Here at Netflix, our objective is to maximise immersion for our members by means of introduction of latest options that may enhance their viewing expertise and hold them absolutely engaged in our content material.
We wish to acknowledge the arduous work of plenty of groups that got here collectively to ship the options being mentioned on this doc. These embody Core UI and JS Player growth, Netflix Application Software growth, AV Test and Tooling (earlier article from this staff), Partner Engineering and Product groups within the Consumer Engineering group and our knowledge science pals within the Data Science and Engineering group at Netflix. Diagrams on this article are courtesy of our Partner Enterprise Platform XD staff.