A Behind-the-Scenes Look at How We Release the Spotify App (Part 1)

April 17, 2025

Published by Jacob Vesterlund, Senior Engineer

Developing and releasing cell apps at scale is a giant problem. With every weekly launch of our cell app for iOS and Android, a whole bunch of adjustments exit to greater than 675 million customers everywhere in the world and on all types of cell units. So much can go fallacious, so discovering and mitigating potential and confirmed points is important to making sure a easy listening expertise. Every characteristic might influence the app’s stability and person expertise, so ensuring we roll them out in a coordinated and prioritized manner can also be one thing that needs to be taken care of. 

At Spotify, our Release group has a twin mission: (1) to supervise the discharge of the primary Spotify app and (2) to construct the required instruments to help this course of. Day-to-day coordination is dealt with by the full-time Release Manager with the help of the remainder of the group.

We have a brand, which implies we imply critical enterprise

When it involves releasing the app, the core duties of the Release group are twofold: 

1. Making positive that the time from when the developer merges their code into the primary department to when it’s obtainable to customers is as brief as potential
2. To be sure that high quality meets our requirements

At occasions, friction exists between these two targets, and far of the craft of launch administration is about mitigating this, each with tooling and with knowledgeable coordination and decision-making. Instances when this stability must be struck may embody the next: 

• Prioritizing accordingly — not all bugs are created equal. A crash throughout signup or playback calls for fast consideration, whereas a post-logout crash is perhaps much less pressing.
• Identifying a fallback — if a bug impacts a particular A/B take a look at group, we will quickly route all customers to the working expertise by way of backend changes, addressing the client-side repair within the subsequent launch. This retains the discharge on monitor with out sacrificing person expertise. 
• Acting rapidly when essential — even a minor bug affecting a small however important person group (e.g., crashes in a particular area) may warrant a swift decision.

The launch cycle

To illustrate our launch cycle, let’s comply with the journey of model 8.9.2 from inception to rollout.

Friday, September 20: A brand new model is born

Each launch cycle kicks off on a Friday morning, when the discharge of the earlier model has been reduce. Once this has been achieved, it’s time to start out the work on the upcoming model. 

At Spotify, we observe trunk-based improvement, which means builders merge their code into the primary department as quickly because it’s examined and reviewed.

However, we make an exception for main adjustments: Large-scale or infrastructure updates are merged earlier within the cycle (sometimes on the primary day, Friday of Week 1). This provides us ample time for thorough testing, leveraging each inner groups and exterior alpha customers to determine and repair points early on.

With Spotify 8.9.2, we deliberate to roll out the Audiobooks characteristic in some markets — it had been obtainable behind a characteristic flag within the backend for quite a lot of releases, for inner testing to find any bugs which may impede the deliberate rollout. This was an essential new characteristic for the corporate, and we wished to ensure we bought it proper, notably since advertising actions and occasions had been already scheduled. 

The Release Manager made positive, properly prematurely, that it was the one huge characteristic rolling out with this launch — one other new characteristic that originally had been scheduled to roll out in the identical week was rescheduled for the next week. 

Other groups might nonetheless merge code at any second, however we strongly inspired them to make use of characteristic flags. If that wasn’t potential, we requested them to keep away from merging any high-risk adjustments on that individual week.  

Friday, September 20 – Thursday, September 26

Apart from the extra actions through the first day, every day of the primary week of the discharge cycle mainly seems to be the identical.

• Early every morning, nightly builds of the primary department are despatched out internally and to our alpha customers. 

• Teams develop and merge new code. The builders and their groups ensure that the code is examined and reviewed beforehand.

• Bug studies are filed by inner and exterior alpha customers. When the proprietor of the affected characteristic is unknown to the reporter, the Release Manager makes positive that the bug report will get assigned to the proper group. 

• Crash charges and different metrics are tracked for every construct each robotically and manually. Automatic bug tickets are created when a crash or different problem exceeds our predefined severity threshold; guide tickets are created when one thing is deemed worthy of investigation by the Release Manager or some other worker.

To assist in monitoring the standing of an upcoming launch, we make use of our Release Manager Dashboard, which collects all of the related launch data in the identical place:  

Some examples of the information discovered within the dashboard are as follows:

• Blocking bugs
• Latest construct obtainable, passing assessments, and distributed by the alpha program within the app retailer
• Crashes and ANRs (App Not Responding) per unit of consumption 
• Daily utilization
• Status of the distribution jobs that distribute the app internally and externally 

By this time, the Audiobooks characteristic had been turned on for many staff. So along with the common course of for this specific launch, each the Release Manager and the Audiobooks group regarded by all of the crashes occurring within the consumer to see if something may put the Audiobook rollout in jeopardy. Even a seemingly minor crash affecting a small variety of staff might signify a possible problem impacting a big person base upon rollout. Therefore, it’s essential to research and mitigate any points as quickly as potential. 

Friday, September 27: Fri-yay is department day!

Once per week handed, it was time for the 8.9.2 model to be branched off for releasing,  initiating essentially the most intensive a part of the discharge course of. Once a launch has been branched off, it’s thought to be the present launch and solely important bug fixes are permitted. Our weekly launch cadence permits much less important bugs and new options to be addressed in subsequent releases, and groups typically keep away from last-minute adjustments to attenuate threat.

Once branching is finished, the Release Manager coordinates the work of releasing the model as quickly as potential with all stakeholders. 

To assist us collect extra information on the standard of the discharge department, we have now a public beta program that’s taken from the discharge department — these builds are anticipated to be extra steady than our alpha builds. 

On Fridays throughout Week 2, groups carry out guide regression testing on their owned options and report their outcomes. Teams with excessive confidence of their automated assessments and pre-merge routines can choose out of guide testing.

Throughout the testing course of, groups might uncover bugs in their very own or in different groups’ options and can file tickets for these. Additionally, crash studies and bug studies from inner and exterior customers additionally set off the creation of latest bug tickets.

For 8.9.2, the Audiobooks group was notably vigilant throughout this section, meticulously trying to find any potential points. At this level, it’s not unusual to debate bugs and determine in the event that they warrant a launch blocker. Good launch administration means objectively threat, potential influence, and workarounds.

Friday, September 27 – Monday, September 30: Getting it prepared for submission

During the weekend after branching, our beta customers present extra runtime of the app, which both will increase our confidence in releasing it or helps us to seek out points that had not been discovered earlier, by the point we get again to work on Monday. 

Ideally, we intention to submit the app to the app shops on Monday. However, complicated bugs or unexpected points can prolong this course of by a number of days. To streamline communication and coordination for every launch, the Release Manager, characteristic groups, and different stakeholders share updates, ask questions, and flag potential considerations on a devoted Slack channel.

The Release Manager ensures bugs are assigned to the proper group and prioritized appropriately, guide testing is executed and reported, and any release-blocking bugs (there are typically maybe three to 5 such bugs per launch) are fastened on the discharge department. 

Before we submit a construct to the app shops, we wish to make certain the next standards are met:

• All commits on the discharge department are included within the newest construct and have handed automated assessments.
• No blocking bug tickets stay open.
• All groups have signed off and permitted.
• Crash charges and different key metrics are under our outlined thresholds.
• The app model to be launched has been used to play a ample quantity of content material.

The Release Manager Dashboard gives a transparent overview of those standards, utilizing color-coding (purple/yellow/inexperienced) for fast evaluation.

For the 8.9.2 launch, we offered extra take a look at accounts with the Audiobooks performance enabled, in addition to detailed testing directions to the assorted app shops to make sure they had been conscious of the brand new options and wouldn’t be caught off guard after we began to roll out the performance. 

Tuesday, October 1 – Wednesday, October 2: Rollout

Once the app is permitted for a platform, we roll it out in two phases: first, to a small proportion of customers and, then, the next day, to 100% of customers. 

When a launch has been rolled out to 1%, we count on the dashboard to appear to be this:

The solely remaining objects, indicated in yellow, wanted earlier than full rollout are the ITGC tickets, the place we test to see that reporting from the consumer to the backend is working as essential. It’s commonplace to uncover minor bugs that, whereas acceptable within the present model, can be thought-about blockers for future releases. In extreme instances, we’d quickly halt the characteristic rollout and resume with the following launch. 

Thanks to our massive person base, even a small preliminary rollout proportion permits us to rapidly determine important points that will have slipped by the cracks through the inner and public alpha and beta testing.

If we discover a extreme problem through the first rollout section, we instantly pause the rollout and the group accountable begins to create a repair. Ideally, the repair is prepared earlier than the following model branches, permitting us to submit an up to date construct. However, if the repair isn’t well timed, we might face the troublesome resolution of canceling both the present or the upcoming launch to stop the complexity of managing two energetic launch branches concurrently. Once we have now reached 100%, we proceed monitoring the state of the discharge over the following week.

For model 8.9.2, as soon as a ample person base was utilizing the brand new model, the Audiobooks group initiated their phased rollout. This concerned steadily enabling the Audiobooks characteristic for a small proportion of customers in particular markets utilizing a backend characteristic flag.

Fortunately, for model 8.9.2, the Audiobooks characteristic met our high quality requirements, and the rollout efficiently ramped as much as 100% over the next days.

Using the discharge process described above, we’re in a position to roll out greater than 95% of releases to all our customers. The weekly cadence additionally implies that a canceled launch isn’t the tip of the world for the characteristic groups, since a brand new launch will exit the next week. In the identical manner, customers will be capable to get a brand new model of the app each week so long as it meets our high quality requirements.

Summary

Spotify’s weekly cell app launch course of tries to strike a stability between velocity and high quality. The individual on the helm of the method is the Release Manager, who handles communication and coordination with characteristic groups and different stakeholders all through the discharge cycle with the assistance of the Release Squad. Tools just like the Release Manager Dashboard play an essential function in enabling the Release Manager to make quick and correct choices.

Detailed documentation of the instruments and processes helps information all of the groups concerned. 

This strong system permits Spotify to persistently replace its app, rapidly tackle points, and introduce new options like Audiobooks, all whereas minimizing disruptions to the person expertise.

Stay tuned for half 2, the place we’ll look underneath the hood to see how the techniques (and robots!) that energy the Release Manager Dashboard work.

Tags: developer expertise, engineering tradition, Mobile