We are excited to share our work on the way to be taught good proxy metrics from historic experiments at KDD 2024. This work addresses a basic query for know-how firms and educational researchers alike: how can we set up {that a} remedy that improves short-term (statistically delicate) outcomes additionally improves long-term (statistically insensitive) outcomes? Or, confronted with a number of short-term outcomes, how can we optimally commerce them off for long-term profit?
For instance, in an A/B check, you might observe {that a} product change improves the click-through charge. However, the check doesn’t present sufficient sign to measure a change in long-term retention, leaving you at nighttime as as to whether this remedy makes customers extra glad along with your service. The click-through charge is a proxy metric (S, for surrogate, in our paper) whereas retention is a downstream enterprise final result or north star metric (Y). We might even have a number of proxy metrics, similar to different kinds of clicks or the size of engagement after click on. Taken collectively, these type a vector of proxy metrics.
The objective of our work is to know the true relationship between the proxy metric(s) and the north star metric — in order that we are able to assess a proxy’s capacity to face in for the north star metric, discover ways to mix a number of metrics right into a single finest one, and higher discover and examine completely different proxies.
Several intuitive approaches to understanding this relationship have stunning pitfalls:
- Looking solely at user-level correlations between the proxy S and north star Y. Continuing the instance from above, you might discover that customers with the next click-through charge additionally are inclined to have the next retention. But this doesn’t imply {that a} product change that improves the click-through charge may even enhance retention (the truth is, selling clickbait might have the alternative impact). This is as a result of, as any introductory causal inference class will let you know, there are various confounders between S and Y — a lot of which you’ll be able to by no means reliably observe and management for.
- Looking naively at remedy impact correlations between S and Y. Suppose you might be fortunate sufficient to have many historic A/B assessments. Further think about the abnormal least squares (OLS) regression line by means of a scatter plot of Y on S wherein every level represents the (S,Y)-treatment impact from a earlier check. Even if you happen to discover that this line has a optimistic slope, you sadly can’t conclude that product modifications that enhance S may even enhance Y. The purpose for that is correlated measurement error — if S and Y are positively correlated within the inhabitants, then remedy arms that occur to have extra customers with excessive S may even have extra customers with excessive Y.
Between these naive approaches, we discover that the second is the simpler lure to fall into. This is as a result of the hazards of the primary method are well-known, whereas covariances between estimated remedy results can seem misleadingly causal. In actuality, these covariances may be severely biased in comparison with what we really care about: covariances between true remedy results. In the intense — similar to when the unfavorable results of clickbait are substantial however clickiness and retention are extremely correlated on the person degree — the true relationship between S and Y may be unfavorable even when the OLS slope is optimistic. Only extra information per experiment may diminish this bias — utilizing extra experiments as information factors will solely yield extra exact estimates of the badly biased slope. At first look, this would seem to imperil any hope of utilizing current experiments to detect the connection.
To overcome this bias, we suggest higher methods to leverage historic experiments, impressed by strategies from the literature on weak instrumental variables. More particularly, we present that three estimators are constant for the true proxy/north-star relationship below completely different constraints (the paper supplies extra particulars and ought to be useful for practitioners all in favour of selecting the perfect estimator for his or her setting):
- A Total Covariance (TC) estimator permits us to estimate the OLS slope from a scatter plot of true remedy results by subtracting the scaled measurement error covariance from the covariance of estimated remedy results. Under the idea that the correlated measurement error is identical throughout experiments (homogeneous covariances), the bias of this estimator is inversely proportional to the whole variety of items throughout all experiments, versus the variety of members per experiment.
- Jackknife Instrumental Variables Estimation (JIVE) converges to the identical OLS slope because the TC estimator however doesn’t require the idea of homogeneous covariances. JIVE eliminates correlated measurement error by eradicating every remark’s information from the computation of its instrumented surrogate values.
- A Limited Information Maximum Likelihood (LIML) estimator is statistically environment friendly so long as there are not any direct results between the remedy and Y (that’s, S totally mediates all remedy results on Y). We discover that LIML is very delicate to this assumption and advocate TC or JIVE for many functions.
Our strategies yield linear structural fashions of remedy results which are straightforward to interpret. As such, they’re well-suited to the decentralized and rapidly-evolving follow of experimentation at Netflix, which runs 1000’s of experiments per 12 months on many numerous elements of the enterprise. Each space of experimentation is staffed by unbiased Data Science and Engineering groups. While each group in the end cares about the identical north star metrics (e.g., long-term income), it’s extremely impractical for many groups to measure these in short-term A/B assessments. Therefore, every has additionally developed proxies which are extra delicate and immediately related to their work (e.g., person engagement or latency). To complicate issues extra, groups are continually innovating on these secondary metrics to seek out the precise steadiness of sensitivity and long-term impression.
In this decentralized setting, linear fashions of remedy results are a extremely great tool for coordinating efforts round proxy metrics and aligning them in direction of the north star:
- Managing metric tradeoffs. Because experiments in a single space can have an effect on metrics in one other space, there’s a must measure all secondary metrics in all assessments, but in addition to know the relative impression of those metrics on the north star. This is so we are able to inform decision-making when one metric trades off in opposition to one other metric.
- Informing metrics innovation. To decrease wasted effort on metric growth, additionally it is vital to know how metrics correlate with the north star “net of” current metrics.
- Enabling groups to work independently. Lastly, groups want easy instruments so as to iterate on their very own metrics. Teams might provide you with dozens of variations of secondary metrics, and sluggish, difficult instruments for evaluating these variations are unlikely to be adopted. Conversely, our fashions are straightforward and quick to suit, and are actively used to develop proxy metrics at Netflix.
We are thrilled in regards to the analysis and implementation of those strategies at Netflix — whereas additionally persevering with to attempt for nice and at all times higher, per our tradition. For instance, we nonetheless have some strategy to go to develop a extra versatile information structure to streamline the appliance of those strategies inside Netflix. Interested in serving to us? See our open job postings!
For suggestions on this weblog put up and for supporting and making this work higher, we thank Apoorva Lal, Martin Tingley, Patric Glynn, Richard McDowell, Travis Brooks, and Ayal Chen-Zion.