A few weeks ago, Festival Queen Anivia brought the spirit of Carnaval to the Rift. It was all costumes and celebrations—until bug reports about the icy bird started flying in. Riot’s QA teams quickly confirmed her ultimate wasn’t properly applying the debuff that doubled Frostbite’s damage.

The bug was game-breaking enough to warrant a disable while we found a fix, and that fix would then have to be QA-tested overnight before implementing to live servers. That means it’d be Friday afternoon, at the earliest, that Anivia would be playable…but we don’t send out hotfixes on Fridays unless the game is severely broken: The risk of unintentionally causing a major problem during peak weekend hours is too high.

Thus, the party was to be cut short as soon as it begun, with Anivia evicted from the Rift until Monday. But wait—Anivia didn’t actually get disabled for the entire weekend.

What allowed us to save The Cryophoenix from that terrible fate? To answer this, we have to dive into the glamorous world of game servers, clients, redeploys, and micropatches.

Let’s Get (a Little) Technical

Game data is held in two locations: the client and the server. Client data is downloaded onto everyone’s computer, and changing it requires that you download a patch. The client contains a lot of visual and audio information, such as the Rift’s appearance, champion models, and voice overs.

The game server is the thing you connect to after champ select; it’s like a little universe that runs your individual 5v5 (or 3v3 or 1v9) game. When we make a change to the game servers, there’s no downtime or downloads. The servers contain numerical information like champion stats as well as scripts that control how abilities function (but not necessarily how they look).

For a long time, the only way to make a change to the client or server was to send out all-new information in the form of patches (deploys) or hotfixes (redeploys). Redeploys most often target server data and are largely how we fixed important live issues (extreme balance problems, champion bugs, etc.) in the past.

Redeploys are risky and require overnight QA-testing, even though most problems are small and require only minor pieces of data to be changed. However, we can’t redeploy just one piece of data—we have to redeploy all of the data. Engineering architect Brian “Riot Penrif” Bossé says, “Doing a large QA-heavy process to push a small change can be wasteful and has a lot more risk than sometimes warranted.”

And even after overnight testing, we still don’t redeploy on Fridays unless there’s an extreme emergency (like ranked queues being busted or only being able to play Teemo). Some of League’s architecture was built a long time ago, before we realized the amount of content we’d place on those foundations, and some these foundations are just not as reliable as we’d like them to be.

Once, after redeploying client data, everybody’s executable file for League was deleted—meaning nobody could open League of Legends. Release Lead Donna “Riot Feithen” Mason says, “Ideally, if we push a button, we expect it to do the same thing every time, but sometimes that’s not the case.”

We’re still working on updating some of League’s dated foundations, but in the meantime, it occasionally makes redeploys unpredictable. We tend to proceed with caution, particularly before the weekend, because redeploys force us to resend all the information regardless of how small the change is.

We knew there had to be a better way to fix minor issues causing major problems.

Micropatches, the Sneaky Superheroes

About a year ago, a new technology called micropatches changed everything. Unlike redeploys, micropatches can make targeted, specific changes to gameplay data. If you think about the game server as a bulletin board, then a micropatch is like a sticky-note posted above the bulletin board. Redeploys are like changing out the whole damn thing, sometimes just to fix one typo. With micropatches, the old game server is still there, but when a new game loads, it looks for sticky-notes and makes only those specific changes.

This makes micropatches way less risky and also way faster than redeploys. For one, micropatches build off of existing server data instead of redeploying all of it, so changes we make are much more contained. And two, micropatches are super easy to implement and revert. We can ship a micropatch globally in less than two minutes—and remove it just as fast.

For awhile, micropatches could only be used to make changes to numerical data—basically, they were there to address balance issues that needed attention, fast. There was still no way to make targeted changes to the scripts that control how champion abilities function, meaning lengthy redeploys were still the only way to fix (most) bugs that would result in disable-dom.

Just a few weeks prior to Anivia’s disabling bug, the champion-saving technology was finished: We could now use micropatches to make targeted changes to ability scripts.

Rebirthing Anivia

The day of Anivia’s Festival Queen debut, we learned her ultimate was bugged across all skins. QA members started piling in, working to pinpoint the source of the problem. Associate game designer Matthew “PhRoXzOn” Leung-Harrison, who worked on the Anivia changes made in Patch 6.23, says, “As soon as I saw the bug, I knew exactly what was going wrong.”

Every champion in League has a logic file containing scripts that control their abilities. This doesn’t necessarily hold information about how the abilities look or how much damage they do. Rather, it controls how the abilities should function—it’s kind of like a motherboard that connects all of the information and synthesizes it into what you see on screen.

Because of a mix-up, we ended up sending a broken version of Anivia’s ult’s logic file to live servers. As a result, her ultimate looked like it worked—it turned on when you pressed “R” and had pretty particle effects—but it wasn’t functioning properly. Glacial Storm should apply a two second debuff for Frostbite’s doubled damage, but it was only applying a .5 second debuff. In the words of QA Lead Brian “Aotus” Brause, “You looked really pretty… while you did no damage.”

After identifying the problem, we updated her logic script and spent some time QA testing. The change fixed the new bug, but a more thorough bug sweep revealed an old exploit that originated in Patch 6.23. Fortunately, the QA process for micropatches is pretty quick, since we’re only checking for the results of one specific change. We made a second modification, and this one worked: Both bugs were fixed and ready to send to players via micropatch.

Forget being disabled for a weekend—within four hours, Anivia was back to (correctly) chilling and fiesta-ing on the Rift.

Micropatches Fix All the Things?

While micropatches have been used in the past to address balance problems, this was the first time we’d modified a logic script and saved a champion from the cruel fate of disable-dom with micropatches. This tech doesn’t work on everything—you won’t see us micropatching a new wing onto Anivia or adding extra-frosty VFX—but they’re a fast, low-risk way of addressing some live issues.

Right now, we’re working on finding a better way to communicate micropatch changes with players. They’re currently added to the top of patch notes, but who the heck goes back to read patch notes long after the patch? We considered broadcasting major changes, like nerfs and buffs, in the in-client ticker, but that’s really only there to notify you if something is broken or unstable. We’re hoping the updated client will provide a better way to share micropatch changes.

In the meantime, Riot Penrif jokes, “If something is unexpectedly working when you thought it was broken, let’s just assume micropatches saved the day.”