![\"\"](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/2uVa1ChTrVKu3vDAkxkptw\/6c09d4a786c6f66383c0d57e786786ec\/Colette_Kolenda.png?w=60&h=68&q=96&fm=webp)
<\/picture><\/noscript><\/div>\n<\/div>\nMixed Methods Research at Spotify<\/strong><\/h3>\nAt Spotify, our strategy to insights is grounded in our perception in making use of a number of complementary analysis methodologies. Our analysis workforce, Product Insights<\/u><\/a>, is made up of each User Researchers and Data Scientists.<\/p>\nAs we are able to see from visualizing our methodologies on this \u201cWhat-Why Framework,\u201d User Researchers and Data Scientists are pure companions. Data Scientists take a look at the large-scale, overarching traits in person conduct by strategies corresponding to A\/B exams and statistical modeling. User Researchers apply strategies corresponding to interviews and surveys, to discover the self-reported listener expertise to know the psychological fashions and perceptions of Spotify.<\/p>\n<\/div>\n
\n\n\n![\"02](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/5aMDCXXVbq14OHe08noqYI\/fd97e31dca129f6c95c4788db1fef5ae\/02_-_Body01.jpg?w=1000&q=96\")
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nThe \u201cWhat-Why Framework,\u201d highlighting some methodologies in every quadrant. We tailored this framework from an Nielsen Norman Group article by Christian Rohrer. [1]<\/p>\n<\/div>\n
\nTogether, User Research and Data Science present complementary views that mutually improve one another. By combining these disciplines, we are able to acquire a holistic understanding of a number of types of information and mitigate the blind spots of a single analysis methodology alone.<\/p>\n<\/div>\n
\nWhen your information says one factor however your customers say one other<\/strong><\/h3>\nWhat occurs when the totally different strategies you use yield totally different, even contradictory outcomes? Well, this occurred to our workforce once we have been researching a check of skippable advertisements in Australia. In this check, Spotify Free listeners may skip the audio and video advertisements that come between their songs.\n<\/p>\n
As a Data Scientist, Kristie ran an A\/B check the place the experimental group acquired the skippable advertisements function and the management group had a traditional Spotify advert expertise. She recognized three teams of customers with totally different ranges of engagement with this new function: energy skippers, medium skippers, and those that by no means skipped a single advert. To higher perceive why every group used the function as they did, User Research adopted up with interviewing listeners from every group.<\/p>\n
We have been stunned to be taught that the A\/B check information mentioned one factor, however our customers within the interviews mentioned one other. Some listeners we labeled as \u201cpower skippers\u201d from our A\/B check truly had some confusion round which advertisements they may skip. These confused listeners doubtless wouldn’t have labeled themselves as energy skippers, as our information did.\n<\/p>\n
Mental fashions are one thing we may solely be taught from qualitative information from User Research. What we thought was the reality from an A\/B check was truly an incomplete story. This distinction between what our information confirmed and what our listeners mentioned was arduous to reconcile.<\/p>\n<\/div>\n
\n\n\n![\"03](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/N4QpoaqtrKmHa2uxhjqen\/97fd167cb6b551fe20cef9813e210e03\/03_-_Body02.jpg?w=1000&q=96\")
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nDiscrepancies between A\/B check information and person information have been irreconcilable.<\/p>\n<\/div>\n
\nThis discrepancy was found as a result of we have been passing learnings between teammates: from Data Science with the A\/B check to User Research with the person interviews. This is the everyday method of working collectively, however in actuality, the 2 disciplines weren’t really collaborating. Although we used complementary strategies on this mission, they yielded contradictory insights.<\/p>\n
We wanted to take collaboration to the following stage, to concurrently triangulate User Research and Data Science by pointing totally different methodologies, on the similar group of listeners, on the similar time.<\/p>\n
Three steps for mixing strategies successfully: simultaneous triangulation.<\/strong><\/h3>\nWe delineated a course of to extra successfully combine strategies. We name it \u201csimultaneous triangulation.\u201d It generates complete insights with out complicated discrepancies:<\/p>\n
\nStep 1: Hone your analysis questions.<\/strong><\/p>\nClearly defining the analysis targets makes it simpler to determine alternatives to collaborate.<\/p>\n
Step 2: Mix strategies in several quadrants of the \u201cWhat-Why Framework.\u201d<\/strong><\/p>\nFind complementary strategies in several quadrants to counterbalance the strengths and weaknesses of every.<\/p>\n
Step 3: Implement strategies concurrently to yield complete insights.<\/strong>\u00a0<\/p>\nRather than deploying every methodology individually, design your research to have all strategies level on the similar group of customers on the similar time. This will mitigate dangers for unexplainable discrepancies.<\/p>\n<\/div>\n
\nWhat this appears to be like like in observe:<\/strong><\/h3>\nStep 1: Honing our analysis questions.<\/i><\/p>\n
Despite some discrepancies within the first spherical of analysis, we nonetheless acquired helpful learnings about points that have been stopping listeners from totally adopting the function.<\/p>\n
With the second spherical of this analysis, we needed to uncover all the drivers and blockers for function utilization to know the \u2018why\u2019 behind the advert skip conduct. Building off of the earlier learnings, we crafted particular analysis questions for spherical two that may yield a deeper understanding of these focus areas: consciousness, understanding, and value of skippable advertisements.<\/p>\n
Step 2: Mixing strategies in several quadrants of the \u201cWhat-Why Framework.\u201d<\/i><\/p>\n
To holistically perceive these analysis questions, we mixed a diary research with information monitoring. Diary research are within the qualitative-attitudinal quadrant of the \u201cWhat-Why Framework\u201d whereas information monitoring is within the quantitative and behavioral quadrant.<\/p>\n<\/div>\n
\n\n\n![\"04](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/25fRpk7I1OL45GQcSVy6Wz\/ebfe4fee9a9edb2d9d7ef0e8db2effea\/04_-_Body03.jpg?w=1000&q=96\")
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nWe employed two methodologies concurrently: information monitoring and a diary research.<\/p>\n<\/div>\n
\nThese two strategies present complementary views. In the diary research, we recruited members to participate in a three-week research, the place we requested them to inform us about their day by day listening experiences and their reactions to the advertisements. For the information monitoring, we requested for every participant\u2019s consent to have a look at their behavioral log information that was pertinent to the listening periods. We have been in a position to perceive their general expertise from the behavioral information and their perceived expertise by the diary research.<\/p>\n
Step 3: Implementing strategies concurrently to yield complete insights.<\/i><\/p>\n
The wealthy understanding of our customers\u2019 experiences got here from pointing each strategies on the similar group of listeners on the similar time. While members crammed out the diary research entries, we may additionally test the dashboard to evaluate their behavioral information expertise. For every diary research entry, we not solely acquired a way of their self-reported expertise of the place and the way they listened and their response to the advertisements, however we additionally may perceive the behavioral information expertise \u2013 corresponding to how lengthy they listened for, what number of advertisements they acquired, and which advertisements they skipped.<\/p>\n
One methodology alone would have fallen brief. Without the information dashboard, we’d have been blind to their behavioral expertise. And with out the diary research, we’d not have had perception into their perceived expertise.<\/p>\n<\/div>\n
\nThis time, discrepancies have been attention-grabbing follow-ups, not lifeless ends<\/strong><\/h3>\nIn this new research design, discrepancies between the behavioral information and the diary research have been truly actually helpful insights! If we noticed an attention-grabbing pattern in listeners\u2019 information, we may comply with up with them within the diary research to be taught extra concerning the \u2018why\u2019 behind their conduct.<\/p>\n
For instance, Kristie seen that one participant solely ever skipped a most of six advertisements in a day. No matter what number of he acquired, he would by no means skip greater than six. This was attention-grabbing to us, as a result of there was no restrict to the variety of advertisements a person may skip. But his behavioral information would recommend that he was hitting some type of cap. However, he by no means talked about this in his diary research entries. Again, we confronted one other information discrepancy!<\/p>\n
The information mentioned one factor\u2014that he was experiencing a restrict. But the person mentioned one other \u2013 that the function was working wonderful for him. Through simultaneous triangulation, we have been in a position to comply with up with him to know his psychological mannequin and thus the explanation behind this discrepancy. He responded, \u201cI can only skip six ads because I only get six song skips. I guessed ad skips must follow the same rule.\u201d On Spotify Free, a person can solely skip six songs per hour. He knew about this rule and misattributed it to advertisements as effectively. He made his personal psychological mannequin about an advert skip restrict.<\/p>\n<\/div>\n
\n\n\n![\"06](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/1wV6iDsNlFtJf9y6PVrkHb\/5ca955e534f21947fbf48399aad16e68\/06_-_Body05.jpg?w=1000&q=96\")
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nOne participant\u2019s dashboard and corresponding diary research response. We noticed a pattern of their skip conduct and adopted as much as be taught extra within the diary research.<\/p>\n<\/div>\n
\nThis time, discrepancies weren\u2019t a complicated lifeless finish, however somewhat, attention-grabbing new alternatives to dig deeper. Through concurrently combining the behavioral information dashboard and the attitudinal diary research, we discovered nuanced drivers and blockers for advert skipping.<\/p>\n<\/div>\n
\nVerified insights to gas product selections<\/strong><\/h3>\nAs a end result, User Research recognized totally different psychological fashions of consciousness, understanding, and value of this new function. However, the diary research methodology isn’t geared up to quantify how many individuals outdoors of the diary research could be affected by these points.<\/p>\n
So, we collaborated to outline proxies in behavioral information that may measurement the impression of every perception. For instance, we grouped those that had confusion round an \u201cad skip limit\u201d by figuring out customers who at all times plateaued at quite a lot of advert skips per day.<\/p>\n
With product and design stakeholders, we brainstormed options to deal with every subject uncovered within the analysis. We determined to ship instructional messages to teams of listeners who displayed these proxy metrics for consciousness, understanding, and value points. For listeners who have been confused about an \u201cad skip limit,\u201d we may ship them a message informing them that there was not truly a restrict on the variety of advert skips.<\/p>\n<\/div>\n
\n\n\n![\"07](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/6zsAIFgoOi4NH0YpQPBIrF\/17bd3e53906fbe965c1c652c1d46b18e\/07_-_Body06.jpg?w=1000&q=96\")
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\n“Freedom to skip. There’s no restrict to the variety of advertisements you possibly can skip on Spotify. Really.” An instance of an academic message we despatched to listeners who could be confused about an advert skip restrict.<\/p>\n<\/div>\n
\nThese focused instructional messages decreased person confusion and, in consequence, we noticed our function success metrics double. In mixing strategies, we are able to have better confidence in our insights and guarantee product selections are evidence-based.<\/p>\n
Wrapping all of it up<\/strong><\/h3>\nSimultaneous triangulation is an extremely highly effective device to generate complete and verified findings. If you solely use one methodology, you would find yourself with blind spots. If you use strategies sequentially somewhat than concurrently, you would run into unexplainable contradictions…like we did at first.<\/p>\n
The answer is simultaneous triangulation. Next time you may have a posh analysis query, think about using the three-step course of to mitigate blind spots and switch discrepancies in studying alternatives. Choose strategies from totally different quadrants of the \u201cWhat-Why Framework\u201d to offer complementary views. Apply these strategies on the similar time, to the identical group of customers.<\/p>\n
You can apply this technique to your insights observe: simply comply with the steps with any of the analysis strategies you may have entry to. You don\u2019t must be on the Product Insights workforce at Spotify, and also you don\u2019t even want each User Researchers and Data Scientists. You can do simultaneous triangulation with the strategies in your toolkit.<\/p>\n
With many analysis questions forward, we’re excited to proceed using simultaneous triangulation to affect advanced product, design, and tech technique selections. We acquired right here by experimenting with mixing totally different strategies and we encourage you to do the identical. Tell us how this course of works for you, as you apply this in your analysis! You can attain out to us at colettek@spotify.com<\/u><\/a> and ksavage@spotify.com<\/u><\/a>.<\/p>\n\u2014<\/i><\/p>\n
Thank you to George Murphy who led this analysis mission and Peter Gilks for presenting this case research with us on the <\/i>Qualtrics Conference<\/i><\/u><\/a> earlier this 12 months.<\/i><\/p>\n\u2014<\/i><\/p>\n
References:<\/p>\n
As we are able to see from visualizing our methodologies on this \u201cWhat-Why Framework,\u201d User Researchers and Data Scientists are pure companions. Data Scientists take a look at the large-scale, overarching traits in person conduct by strategies corresponding to A\/B exams and statistical modeling. User Researchers apply strategies corresponding to interviews and surveys, to discover the self-reported listener expertise to know the psychological fashions and perceptions of Spotify.<\/p>\n<\/div>\n
![\"02](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/5aMDCXXVbq14OHe08noqYI\/fd97e31dca129f6c95c4788db1fef5ae\/02_-_Body01.jpg?w=250&h=115&q=96&fm=webp)
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nThe \u201cWhat-Why Framework,\u201d highlighting some methodologies in every quadrant. We tailored this framework from an Nielsen Norman Group article by Christian Rohrer. [1]<\/p>\n<\/div>\n
Together, User Research and Data Science present complementary views that mutually improve one another. By combining these disciplines, we are able to acquire a holistic understanding of a number of types of information and mitigate the blind spots of a single analysis methodology alone.<\/p>\n<\/div>\n
When your information says one factor however your customers say one other<\/strong><\/h3>\nWhat occurs when the totally different strategies you use yield totally different, even contradictory outcomes? Well, this occurred to our workforce once we have been researching a check of skippable advertisements in Australia. In this check, Spotify Free listeners may skip the audio and video advertisements that come between their songs.\n<\/p>\n
As a Data Scientist, Kristie ran an A\/B check the place the experimental group acquired the skippable advertisements function and the management group had a traditional Spotify advert expertise. She recognized three teams of customers with totally different ranges of engagement with this new function: energy skippers, medium skippers, and those that by no means skipped a single advert. To higher perceive why every group used the function as they did, User Research adopted up with interviewing listeners from every group.<\/p>\n
We have been stunned to be taught that the A\/B check information mentioned one factor, however our customers within the interviews mentioned one other. Some listeners we labeled as \u201cpower skippers\u201d from our A\/B check truly had some confusion round which advertisements they may skip. These confused listeners doubtless wouldn’t have labeled themselves as energy skippers, as our information did.\n<\/p>\n
Mental fashions are one thing we may solely be taught from qualitative information from User Research. What we thought was the reality from an A\/B check was truly an incomplete story. This distinction between what our information confirmed and what our listeners mentioned was arduous to reconcile.<\/p>\n<\/div>\n
\n\n\nDiscrepancies between A\/B check information and person information have been irreconcilable.<\/p>\n<\/div>\n
\nThis discrepancy was found as a result of we have been passing learnings between teammates: from Data Science with the A\/B check to User Research with the person interviews. This is the everyday method of working collectively, however in actuality, the 2 disciplines weren’t really collaborating. Although we used complementary strategies on this mission, they yielded contradictory insights.<\/p>\n
We wanted to take collaboration to the following stage, to concurrently triangulate User Research and Data Science by pointing totally different methodologies, on the similar group of listeners, on the similar time.<\/p>\n
Three steps for mixing strategies successfully: simultaneous triangulation.<\/strong><\/h3>\nWe delineated a course of to extra successfully combine strategies. We name it \u201csimultaneous triangulation.\u201d It generates complete insights with out complicated discrepancies:<\/p>\n
\nStep 1: Hone your analysis questions.<\/strong><\/p>\nClearly defining the analysis targets makes it simpler to determine alternatives to collaborate.<\/p>\n
Step 2: Mix strategies in several quadrants of the \u201cWhat-Why Framework.\u201d<\/strong><\/p>\nFind complementary strategies in several quadrants to counterbalance the strengths and weaknesses of every.<\/p>\n
Step 3: Implement strategies concurrently to yield complete insights.<\/strong>\u00a0<\/p>\nRather than deploying every methodology individually, design your research to have all strategies level on the similar group of customers on the similar time. This will mitigate dangers for unexplainable discrepancies.<\/p>\n<\/div>\n
\nWhat this appears to be like like in observe:<\/strong><\/h3>\nStep 1: Honing our analysis questions.<\/i><\/p>\n
Despite some discrepancies within the first spherical of analysis, we nonetheless acquired helpful learnings about points that have been stopping listeners from totally adopting the function.<\/p>\n
With the second spherical of this analysis, we needed to uncover all the drivers and blockers for function utilization to know the \u2018why\u2019 behind the advert skip conduct. Building off of the earlier learnings, we crafted particular analysis questions for spherical two that may yield a deeper understanding of these focus areas: consciousness, understanding, and value of skippable advertisements.<\/p>\n
Step 2: Mixing strategies in several quadrants of the \u201cWhat-Why Framework.\u201d<\/i><\/p>\n
To holistically perceive these analysis questions, we mixed a diary research with information monitoring. Diary research are within the qualitative-attitudinal quadrant of the \u201cWhat-Why Framework\u201d whereas information monitoring is within the quantitative and behavioral quadrant.<\/p>\n<\/div>\n
\n\n\nWe employed two methodologies concurrently: information monitoring and a diary research.<\/p>\n<\/div>\n
\nThese two strategies present complementary views. In the diary research, we recruited members to participate in a three-week research, the place we requested them to inform us about their day by day listening experiences and their reactions to the advertisements. For the information monitoring, we requested for every participant\u2019s consent to have a look at their behavioral log information that was pertinent to the listening periods. We have been in a position to perceive their general expertise from the behavioral information and their perceived expertise by the diary research.<\/p>\n
Step 3: Implementing strategies concurrently to yield complete insights.<\/i><\/p>\n
The wealthy understanding of our customers\u2019 experiences got here from pointing each strategies on the similar group of listeners on the similar time. While members crammed out the diary research entries, we may additionally test the dashboard to evaluate their behavioral information expertise. For every diary research entry, we not solely acquired a way of their self-reported expertise of the place and the way they listened and their response to the advertisements, however we additionally may perceive the behavioral information expertise \u2013 corresponding to how lengthy they listened for, what number of advertisements they acquired, and which advertisements they skipped.<\/p>\n
One methodology alone would have fallen brief. Without the information dashboard, we’d have been blind to their behavioral expertise. And with out the diary research, we’d not have had perception into their perceived expertise.<\/p>\n<\/div>\n
\nThis time, discrepancies have been attention-grabbing follow-ups, not lifeless ends<\/strong><\/h3>\nIn this new research design, discrepancies between the behavioral information and the diary research have been truly actually helpful insights! If we noticed an attention-grabbing pattern in listeners\u2019 information, we may comply with up with them within the diary research to be taught extra concerning the \u2018why\u2019 behind their conduct.<\/p>\n
For instance, Kristie seen that one participant solely ever skipped a most of six advertisements in a day. No matter what number of he acquired, he would by no means skip greater than six. This was attention-grabbing to us, as a result of there was no restrict to the variety of advertisements a person may skip. But his behavioral information would recommend that he was hitting some type of cap. However, he by no means talked about this in his diary research entries. Again, we confronted one other information discrepancy!<\/p>\n
The information mentioned one factor\u2014that he was experiencing a restrict. But the person mentioned one other \u2013 that the function was working wonderful for him. Through simultaneous triangulation, we have been in a position to comply with up with him to know his psychological mannequin and thus the explanation behind this discrepancy. He responded, \u201cI can only skip six ads because I only get six song skips. I guessed ad skips must follow the same rule.\u201d On Spotify Free, a person can solely skip six songs per hour. He knew about this rule and misattributed it to advertisements as effectively. He made his personal psychological mannequin about an advert skip restrict.<\/p>\n<\/div>\n
\n\n\nOne participant\u2019s dashboard and corresponding diary research response. We noticed a pattern of their skip conduct and adopted as much as be taught extra within the diary research.<\/p>\n<\/div>\n
\nThis time, discrepancies weren\u2019t a complicated lifeless finish, however somewhat, attention-grabbing new alternatives to dig deeper. Through concurrently combining the behavioral information dashboard and the attitudinal diary research, we discovered nuanced drivers and blockers for advert skipping.<\/p>\n<\/div>\n
\nVerified insights to gas product selections<\/strong><\/h3>\nAs a end result, User Research recognized totally different psychological fashions of consciousness, understanding, and value of this new function. However, the diary research methodology isn’t geared up to quantify how many individuals outdoors of the diary research could be affected by these points.<\/p>\n
So, we collaborated to outline proxies in behavioral information that may measurement the impression of every perception. For instance, we grouped those that had confusion round an \u201cad skip limit\u201d by figuring out customers who at all times plateaued at quite a lot of advert skips per day.<\/p>\n
With product and design stakeholders, we brainstormed options to deal with every subject uncovered within the analysis. We determined to ship instructional messages to teams of listeners who displayed these proxy metrics for consciousness, understanding, and value points. For listeners who have been confused about an \u201cad skip limit,\u201d we may ship them a message informing them that there was not truly a restrict on the variety of advert skips.<\/p>\n<\/div>\n
\n\n\n“Freedom to skip. There’s no restrict to the variety of advertisements you possibly can skip on Spotify. Really.” An instance of an academic message we despatched to listeners who could be confused about an advert skip restrict.<\/p>\n<\/div>\n
\nThese focused instructional messages decreased person confusion and, in consequence, we noticed our function success metrics double. In mixing strategies, we are able to have better confidence in our insights and guarantee product selections are evidence-based.<\/p>\n
Wrapping all of it up<\/strong><\/h3>\nSimultaneous triangulation is an extremely highly effective device to generate complete and verified findings. If you solely use one methodology, you would find yourself with blind spots. If you use strategies sequentially somewhat than concurrently, you would run into unexplainable contradictions…like we did at first.<\/p>\n
The answer is simultaneous triangulation. Next time you may have a posh analysis query, think about using the three-step course of to mitigate blind spots and switch discrepancies in studying alternatives. Choose strategies from totally different quadrants of the \u201cWhat-Why Framework\u201d to offer complementary views. Apply these strategies on the similar time, to the identical group of customers.<\/p>\n
You can apply this technique to your insights observe: simply comply with the steps with any of the analysis strategies you may have entry to. You don\u2019t must be on the Product Insights workforce at Spotify, and also you don\u2019t even want each User Researchers and Data Scientists. You can do simultaneous triangulation with the strategies in your toolkit.<\/p>\n
With many analysis questions forward, we’re excited to proceed using simultaneous triangulation to affect advanced product, design, and tech technique selections. We acquired right here by experimenting with mixing totally different strategies and we encourage you to do the identical. Tell us how this course of works for you, as you apply this in your analysis! You can attain out to us at colettek@spotify.com<\/u><\/a> and ksavage@spotify.com<\/u><\/a>.<\/p>\n\u2014<\/i><\/p>\n
Thank you to George Murphy who led this analysis mission and Peter Gilks for presenting this case research with us on the <\/i>Qualtrics Conference<\/i><\/u><\/a> earlier this 12 months.<\/i><\/p>\n\u2014<\/i><\/p>\n
References:<\/p>\n
![\"03](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/N4QpoaqtrKmHa2uxhjqen\/97fd167cb6b551fe20cef9813e210e03\/03_-_Body02.jpg?w=250&h=115&q=96&fm=webp)
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nDiscrepancies between A\/B check information and person information have been irreconcilable.<\/p>\n<\/div>\n
This discrepancy was found as a result of we have been passing learnings between teammates: from Data Science with the A\/B check to User Research with the person interviews. This is the everyday method of working collectively, however in actuality, the 2 disciplines weren’t really collaborating. Although we used complementary strategies on this mission, they yielded contradictory insights.<\/p>\n
We wanted to take collaboration to the following stage, to concurrently triangulate User Research and Data Science by pointing totally different methodologies, on the similar group of listeners, on the similar time.<\/p>\n
Three steps for mixing strategies successfully: simultaneous triangulation.<\/strong><\/h3>\nWe delineated a course of to extra successfully combine strategies. We name it \u201csimultaneous triangulation.\u201d It generates complete insights with out complicated discrepancies:<\/p>\n
\nStep 1: Hone your analysis questions.<\/strong><\/p>\nClearly defining the analysis targets makes it simpler to determine alternatives to collaborate.<\/p>\n
Step 2: Mix strategies in several quadrants of the \u201cWhat-Why Framework.\u201d<\/strong><\/p>\nFind complementary strategies in several quadrants to counterbalance the strengths and weaknesses of every.<\/p>\n
Step 3: Implement strategies concurrently to yield complete insights.<\/strong>\u00a0<\/p>\nRather than deploying every methodology individually, design your research to have all strategies level on the similar group of customers on the similar time. This will mitigate dangers for unexplainable discrepancies.<\/p>\n<\/div>\n
\nWhat this appears to be like like in observe:<\/strong><\/h3>\nStep 1: Honing our analysis questions.<\/i><\/p>\n
Despite some discrepancies within the first spherical of analysis, we nonetheless acquired helpful learnings about points that have been stopping listeners from totally adopting the function.<\/p>\n
With the second spherical of this analysis, we needed to uncover all the drivers and blockers for function utilization to know the \u2018why\u2019 behind the advert skip conduct. Building off of the earlier learnings, we crafted particular analysis questions for spherical two that may yield a deeper understanding of these focus areas: consciousness, understanding, and value of skippable advertisements.<\/p>\n
Step 2: Mixing strategies in several quadrants of the \u201cWhat-Why Framework.\u201d<\/i><\/p>\n
To holistically perceive these analysis questions, we mixed a diary research with information monitoring. Diary research are within the qualitative-attitudinal quadrant of the \u201cWhat-Why Framework\u201d whereas information monitoring is within the quantitative and behavioral quadrant.<\/p>\n<\/div>\n
\n\n\nWe employed two methodologies concurrently: information monitoring and a diary research.<\/p>\n<\/div>\n
\nThese two strategies present complementary views. In the diary research, we recruited members to participate in a three-week research, the place we requested them to inform us about their day by day listening experiences and their reactions to the advertisements. For the information monitoring, we requested for every participant\u2019s consent to have a look at their behavioral log information that was pertinent to the listening periods. We have been in a position to perceive their general expertise from the behavioral information and their perceived expertise by the diary research.<\/p>\n
Step 3: Implementing strategies concurrently to yield complete insights.<\/i><\/p>\n
The wealthy understanding of our customers\u2019 experiences got here from pointing each strategies on the similar group of listeners on the similar time. While members crammed out the diary research entries, we may additionally test the dashboard to evaluate their behavioral information expertise. For every diary research entry, we not solely acquired a way of their self-reported expertise of the place and the way they listened and their response to the advertisements, however we additionally may perceive the behavioral information expertise \u2013 corresponding to how lengthy they listened for, what number of advertisements they acquired, and which advertisements they skipped.<\/p>\n
One methodology alone would have fallen brief. Without the information dashboard, we’d have been blind to their behavioral expertise. And with out the diary research, we’d not have had perception into their perceived expertise.<\/p>\n<\/div>\n
\nThis time, discrepancies have been attention-grabbing follow-ups, not lifeless ends<\/strong><\/h3>\nIn this new research design, discrepancies between the behavioral information and the diary research have been truly actually helpful insights! If we noticed an attention-grabbing pattern in listeners\u2019 information, we may comply with up with them within the diary research to be taught extra concerning the \u2018why\u2019 behind their conduct.<\/p>\n
For instance, Kristie seen that one participant solely ever skipped a most of six advertisements in a day. No matter what number of he acquired, he would by no means skip greater than six. This was attention-grabbing to us, as a result of there was no restrict to the variety of advertisements a person may skip. But his behavioral information would recommend that he was hitting some type of cap. However, he by no means talked about this in his diary research entries. Again, we confronted one other information discrepancy!<\/p>\n
The information mentioned one factor\u2014that he was experiencing a restrict. But the person mentioned one other \u2013 that the function was working wonderful for him. Through simultaneous triangulation, we have been in a position to comply with up with him to know his psychological mannequin and thus the explanation behind this discrepancy. He responded, \u201cI can only skip six ads because I only get six song skips. I guessed ad skips must follow the same rule.\u201d On Spotify Free, a person can solely skip six songs per hour. He knew about this rule and misattributed it to advertisements as effectively. He made his personal psychological mannequin about an advert skip restrict.<\/p>\n<\/div>\n
\n\n\nOne participant\u2019s dashboard and corresponding diary research response. We noticed a pattern of their skip conduct and adopted as much as be taught extra within the diary research.<\/p>\n<\/div>\n
\nThis time, discrepancies weren\u2019t a complicated lifeless finish, however somewhat, attention-grabbing new alternatives to dig deeper. Through concurrently combining the behavioral information dashboard and the attitudinal diary research, we discovered nuanced drivers and blockers for advert skipping.<\/p>\n<\/div>\n
\nVerified insights to gas product selections<\/strong><\/h3>\nAs a end result, User Research recognized totally different psychological fashions of consciousness, understanding, and value of this new function. However, the diary research methodology isn’t geared up to quantify how many individuals outdoors of the diary research could be affected by these points.<\/p>\n
So, we collaborated to outline proxies in behavioral information that may measurement the impression of every perception. For instance, we grouped those that had confusion round an \u201cad skip limit\u201d by figuring out customers who at all times plateaued at quite a lot of advert skips per day.<\/p>\n
With product and design stakeholders, we brainstormed options to deal with every subject uncovered within the analysis. We determined to ship instructional messages to teams of listeners who displayed these proxy metrics for consciousness, understanding, and value points. For listeners who have been confused about an \u201cad skip limit,\u201d we may ship them a message informing them that there was not truly a restrict on the variety of advert skips.<\/p>\n<\/div>\n
\n\n\n“Freedom to skip. There’s no restrict to the variety of advertisements you possibly can skip on Spotify. Really.” An instance of an academic message we despatched to listeners who could be confused about an advert skip restrict.<\/p>\n<\/div>\n
\nThese focused instructional messages decreased person confusion and, in consequence, we noticed our function success metrics double. In mixing strategies, we are able to have better confidence in our insights and guarantee product selections are evidence-based.<\/p>\n
Wrapping all of it up<\/strong><\/h3>\nSimultaneous triangulation is an extremely highly effective device to generate complete and verified findings. If you solely use one methodology, you would find yourself with blind spots. If you use strategies sequentially somewhat than concurrently, you would run into unexplainable contradictions…like we did at first.<\/p>\n
The answer is simultaneous triangulation. Next time you may have a posh analysis query, think about using the three-step course of to mitigate blind spots and switch discrepancies in studying alternatives. Choose strategies from totally different quadrants of the \u201cWhat-Why Framework\u201d to offer complementary views. Apply these strategies on the similar time, to the identical group of customers.<\/p>\n
You can apply this technique to your insights observe: simply comply with the steps with any of the analysis strategies you may have entry to. You don\u2019t must be on the Product Insights workforce at Spotify, and also you don\u2019t even want each User Researchers and Data Scientists. You can do simultaneous triangulation with the strategies in your toolkit.<\/p>\n
With many analysis questions forward, we’re excited to proceed using simultaneous triangulation to affect advanced product, design, and tech technique selections. We acquired right here by experimenting with mixing totally different strategies and we encourage you to do the identical. Tell us how this course of works for you, as you apply this in your analysis! You can attain out to us at colettek@spotify.com<\/u><\/a> and ksavage@spotify.com<\/u><\/a>.<\/p>\n\u2014<\/i><\/p>\n
Thank you to George Murphy who led this analysis mission and Peter Gilks for presenting this case research with us on the <\/i>Qualtrics Conference<\/i><\/u><\/a> earlier this 12 months.<\/i><\/p>\n\u2014<\/i><\/p>\n
References:<\/p>\n
Clearly defining the analysis targets makes it simpler to determine alternatives to collaborate.<\/p>\n
Step 2: Mix strategies in several quadrants of the \u201cWhat-Why Framework.\u201d<\/strong><\/p>\n Find complementary strategies in several quadrants to counterbalance the strengths and weaknesses of every.<\/p>\n Step 3: Implement strategies concurrently to yield complete insights.<\/strong>\u00a0<\/p>\n Rather than deploying every methodology individually, design your research to have all strategies level on the similar group of customers on the similar time. This will mitigate dangers for unexplainable discrepancies.<\/p>\n<\/div>\n Step 1: Honing our analysis questions.<\/i><\/p>\n Despite some discrepancies within the first spherical of analysis, we nonetheless acquired helpful learnings about points that have been stopping listeners from totally adopting the function.<\/p>\n With the second spherical of this analysis, we needed to uncover all the drivers and blockers for function utilization to know the \u2018why\u2019 behind the advert skip conduct. Building off of the earlier learnings, we crafted particular analysis questions for spherical two that may yield a deeper understanding of these focus areas: consciousness, understanding, and value of skippable advertisements.<\/p>\n Step 2: Mixing strategies in several quadrants of the \u201cWhat-Why Framework.\u201d<\/i><\/p>\n To holistically perceive these analysis questions, we mixed a diary research with information monitoring. Diary research are within the qualitative-attitudinal quadrant of the \u201cWhat-Why Framework\u201d whereas information monitoring is within the quantitative and behavioral quadrant.<\/p>\n<\/div>\n We employed two methodologies concurrently: information monitoring and a diary research.<\/p>\n<\/div>\n These two strategies present complementary views. In the diary research, we recruited members to participate in a three-week research, the place we requested them to inform us about their day by day listening experiences and their reactions to the advertisements. For the information monitoring, we requested for every participant\u2019s consent to have a look at their behavioral log information that was pertinent to the listening periods. We have been in a position to perceive their general expertise from the behavioral information and their perceived expertise by the diary research.<\/p>\n Step 3: Implementing strategies concurrently to yield complete insights.<\/i><\/p>\n The wealthy understanding of our customers\u2019 experiences got here from pointing each strategies on the similar group of listeners on the similar time. While members crammed out the diary research entries, we may additionally test the dashboard to evaluate their behavioral information expertise. For every diary research entry, we not solely acquired a way of their self-reported expertise of the place and the way they listened and their response to the advertisements, however we additionally may perceive the behavioral information expertise \u2013 corresponding to how lengthy they listened for, what number of advertisements they acquired, and which advertisements they skipped.<\/p>\n One methodology alone would have fallen brief. Without the information dashboard, we’d have been blind to their behavioral expertise. And with out the diary research, we’d not have had perception into their perceived expertise.<\/p>\n<\/div>\n In this new research design, discrepancies between the behavioral information and the diary research have been truly actually helpful insights! If we noticed an attention-grabbing pattern in listeners\u2019 information, we may comply with up with them within the diary research to be taught extra concerning the \u2018why\u2019 behind their conduct.<\/p>\n For instance, Kristie seen that one participant solely ever skipped a most of six advertisements in a day. No matter what number of he acquired, he would by no means skip greater than six. This was attention-grabbing to us, as a result of there was no restrict to the variety of advertisements a person may skip. But his behavioral information would recommend that he was hitting some type of cap. However, he by no means talked about this in his diary research entries. Again, we confronted one other information discrepancy!<\/p>\n The information mentioned one factor\u2014that he was experiencing a restrict. But the person mentioned one other \u2013 that the function was working wonderful for him. Through simultaneous triangulation, we have been in a position to comply with up with him to know his psychological mannequin and thus the explanation behind this discrepancy. He responded, \u201cI can only skip six ads because I only get six song skips. I guessed ad skips must follow the same rule.\u201d On Spotify Free, a person can solely skip six songs per hour. He knew about this rule and misattributed it to advertisements as effectively. He made his personal psychological mannequin about an advert skip restrict.<\/p>\n<\/div>\n One participant\u2019s dashboard and corresponding diary research response. We noticed a pattern of their skip conduct and adopted as much as be taught extra within the diary research.<\/p>\n<\/div>\n This time, discrepancies weren\u2019t a complicated lifeless finish, however somewhat, attention-grabbing new alternatives to dig deeper. Through concurrently combining the behavioral information dashboard and the attitudinal diary research, we discovered nuanced drivers and blockers for advert skipping.<\/p>\n<\/div>\n As a end result, User Research recognized totally different psychological fashions of consciousness, understanding, and value of this new function. However, the diary research methodology isn’t geared up to quantify how many individuals outdoors of the diary research could be affected by these points.<\/p>\n So, we collaborated to outline proxies in behavioral information that may measurement the impression of every perception. For instance, we grouped those that had confusion round an \u201cad skip limit\u201d by figuring out customers who at all times plateaued at quite a lot of advert skips per day.<\/p>\n With product and design stakeholders, we brainstormed options to deal with every subject uncovered within the analysis. We determined to ship instructional messages to teams of listeners who displayed these proxy metrics for consciousness, understanding, and value points. For listeners who have been confused about an \u201cad skip limit,\u201d we may ship them a message informing them that there was not truly a restrict on the variety of advert skips.<\/p>\n<\/div>\n “Freedom to skip. There’s no restrict to the variety of advertisements you possibly can skip on Spotify. Really.” An instance of an academic message we despatched to listeners who could be confused about an advert skip restrict.<\/p>\n<\/div>\n These focused instructional messages decreased person confusion and, in consequence, we noticed our function success metrics double. In mixing strategies, we are able to have better confidence in our insights and guarantee product selections are evidence-based.<\/p>\n Simultaneous triangulation is an extremely highly effective device to generate complete and verified findings. If you solely use one methodology, you would find yourself with blind spots. If you use strategies sequentially somewhat than concurrently, you would run into unexplainable contradictions…like we did at first.<\/p>\n The answer is simultaneous triangulation. Next time you may have a posh analysis query, think about using the three-step course of to mitigate blind spots and switch discrepancies in studying alternatives. Choose strategies from totally different quadrants of the \u201cWhat-Why Framework\u201d to offer complementary views. Apply these strategies on the similar time, to the identical group of customers.<\/p>\n You can apply this technique to your insights observe: simply comply with the steps with any of the analysis strategies you may have entry to. You don\u2019t must be on the Product Insights workforce at Spotify, and also you don\u2019t even want each User Researchers and Data Scientists. You can do simultaneous triangulation with the strategies in your toolkit.<\/p>\n With many analysis questions forward, we’re excited to proceed using simultaneous triangulation to affect advanced product, design, and tech technique selections. We acquired right here by experimenting with mixing totally different strategies and we encourage you to do the identical. Tell us how this course of works for you, as you apply this in your analysis! You can attain out to us at colettek@spotify.com<\/u><\/a> and ksavage@spotify.com<\/u><\/a>.<\/p>\n \u2014<\/i><\/p>\n Thank you to George Murphy who led this analysis mission and Peter Gilks for presenting this case research with us on the <\/i>Qualtrics Conference<\/i><\/u><\/a> earlier this 12 months.<\/i><\/p>\n \u2014<\/i><\/p>\n References:<\/p>\nWhat this appears to be like like in observe:<\/strong><\/h3>\n
![\"04](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/25fRpk7I1OL45GQcSVy6Wz\/ebfe4fee9a9edb2d9d7ef0e8db2effea\/04_-_Body03.jpg?w=250&h=115&q=96&fm=webp)
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nThis time, discrepancies have been attention-grabbing follow-ups, not lifeless ends<\/strong><\/h3>\n
![\"06](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/1wV6iDsNlFtJf9y6PVrkHb\/5ca955e534f21947fbf48399aad16e68\/06_-_Body05.jpg?w=250&h=115&q=96&fm=webp)
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nVerified insights to gas product selections<\/strong><\/h3>\n
![\"07](\"https:\/\/images.ctfassets.net\/c1zhnszcah7h\/6zsAIFgoOi4NH0YpQPBIrF\/17bd3e53906fbe965c1c652c1d46b18e\/07_-_Body06.jpg?w=250&h=115&q=96&fm=webp)
<\/picture><\/noscript><\/div>\n<\/div>\n<\/figure>\nWrapping all of it up<\/strong><\/h3>\n