Input System Update - Sept 2021: Engine Upgrades, Robots, and More

MCOC Team · September 2021

Greetings Summoners,

It’s been a little while since we’ve communicated on the ongoing Control issues, and today we wanted to give you all a different kind of post, where we’ll explore the cause of the issue, what it entails, and the progress we’ve made on our journey so far in addressing the inconsistencies and changes in control timing.

We also want to take a moment to better outline what the issue actually is, as we know there’s some confusion surrounding the exact symptoms of this issue. We’ve seen players reporting issues such as framerate drops or lag that are not related to this issue, and are being addressed separately with some fixes coming your way soon.

We also want to give you more insight into why this process is taking some time to address, why it’s difficult to address issues based on “feel”, and introduce you to a new member of our team!

If you need a refresher on what we’ve said so far, take a look at this post here.

This is going to be a long post, so let’s get started.

What is the issue?

When we refer to the “Parry and Dex issue”, we’re not actually talking about an issue with Parry or Dex, but a framerate and control input issue exclusive to iOS devices that is most noticeable when trying to Parry or Dex, since these are two mechanics players rely on quite often.

Marvel Contest of Champions is built on a 3rd Party Game Engine built by our partners at Unity, which has an input system built-in. We’ve relied on Unity's input system since the game was built, and have regularly updated the Engine as they have released updates. With the most recent update to the Engine, a bug was fixed where iOS devices had been operating at the device’s own frame rate, instead of being capped to 30fps as it was set to. This meant that iOS devices had a slightly different input window than Android devices, but this was never identified.

Since this is how iOS devices had always worked with our game since launch, we had built game interactions around that input window. Now, because this bug in the engine has been fixed, it changed the input window of inputs for all iOS devices and pushed back the input window by mere milliseconds, but that is enough that it is noticeable when playing the game.

This is the issue that we are working to address, and our fix for it is to integrate an entirely new control input system from our partners at Unity. But this is not a quick or easy task.

Operating on Feel

The most consistent thing we hear from Players, and how this issue was identified, was that Parry and Dex “feel” off. After all, this is how players know the game, through feel. There are no in-game metrics that allow players to see for certain where there is or is not a control issue.

But the problem with “feel” is that it can be misleading, and extremely hard to verify.

You may remember that a couple of months ago, we held a Beta Test for what we believed could help address this issue. Unfortunately, the results of that Beta Test showed that determining a fix from how things feel was not the way to go.

During the Beta Test, we split players into 2 Groups, a control group and a test group that received an update for the fix. We asked players to let us know if the version they downloaded felt better, worse, or the same. Approximately 61% of Summoners that received the version with the update felt that Parry and Dex felt better than before. Comparatively, 63% of our Control group, who did not receive a potential fix, felt that Parry and Dex felt better. Data also showed that many Android users felt that Parry or Dex felt better or worse, but as far as we know, this issue does not affect Android users.

Similarly, we saw an increase in reports of issues with Dexing special attacks during the finale of the Summer of Pain, but a large portion of those reports were coming from devices that were not affected by this issue.

After the Beta Test, we added new timing metrics throughout many systems in-game and were still not able to find anything. This is because the issue exists between the OS/Hardware and the Engine, and not something in-game, which means that software testing was not going to suffice.

This is a major hurdle when trying to address an issue based on how things feel. Many factors can influence the results, and the Beta test did not help us find the issue or solution with any more certainty.

So, how do we operate with more confidence than “feel” when we can’t be sure something is working better or worse (I also was in the control group and thought things felt better)? You build yourself a new coworker.

Meet Kabam Tapsalot

We knew that we needed a consistent way to test inputs, and be sure that we’re taking human error and “feel” out of the picture, and to be absolutely sure that we’re making progress in the right direction.

Since we can’t be 100% certain, we expanded our roster and added a new member to the team. Meet, Kabam Tapsalot!

The name Tapsalot comes from Cat Murdock, and we fell in love with it instantly.

Tapsalot is a robot built by our own engineers to do 2 things, read the screen and tap. This gives us more consistent data to look at, and lets us see the variance between when a user would hit the screen (in this example, for a parry on run in) and when that action is successful vs when you take a punch to the face.

Tapsalot’s crowning achievement so far has been that it gave us an objective measure of the issue. This in turn allowed us to bisect through upwards of 30 engine versions until we were able to find the exact change that was made.

The first 2 Bars in the chart below shows the difference between the input window in v31.1, and 32.0, where the issues first appeared (ignore the 3rd bar for now, we’ll get to that one later). You can see that the window shifted back by approximately 50 milliseconds and 1 frame, making the Parry window open and close earlier than previous versions.

While it’s a great start, we can’t be totally confident in the findings until we have Tapsalot test more devices (both iOS and Android) and ensure that we’re seeing a consistent pattern across those devices as well.

Tapsalot is going to be joined by some friends soon. We’ve started working to build many more Robots just like it, and plan to quickly expand our ability to test multiple devices and scenarios at once, and will be a part of our testing on all builds going forward so we can avoid another situation like this from happening again in the future.

What does a Fix Look like?

Since this issue is the result of a Bug Fix to the engine, reverting back to what it was before is not possible, and our goal is to build a new reliable timing for controls in-game.

We are now rebuilding our entire control and input system to work with a new input system in the Unity engine. This new system will have more stability than our previous input System, which was built on legacy code.

We’ve already started to make progress towards this, and early tests are showing promise.

Let’s take another look at the bar graph above, but this time we want to direct your attention to the third bar that represents the response time in the early stages of development of our new input system. Things are trending the right way, as we get the input window closer to what it was before, but we’re not quite there yet.

Also, remember that this is only showing the results of a single test (Parrying a Medium attack) on one device (iPhone XS Max) and is very preliminary. We still need to get more information from other scenarios and other devices.

As we mentioned before, there is no going back to what was there before, but we feel it’s more important to build a new stable system that you can reliably learn and not have to worry about feeling different again in the future.

While work is steaming ahead on this new Input System refactor, we do not have a definite release date for you yet. This is an extremely fragile and complex system that touches every gameplay aspect of the game, and we need a lot more data (and robots) before we can confidently commit to the new system. We’ll also want to run a large-scale Beta for this new system before pushing it live.

While we want to work as quickly as possible and get this out to you ASAP, this is a crucial part of the game that we can’t risk breaking further, and are going to do our best to get this out to you quickly, but safely. We are currently expecting an early 2022 release for the final iteration, with internal and Beta testing along the way.

What about the Stuttering and Crashing?

As we mentioned earlier, these gameplay issues became sort of a catch-all for other issues occurring in-game that are completely unrelated, including some gameplay stuttering and framerate issues, as well as any other gameplay problems.

We wanted to identify what the actual issue is so that our players know that if they experience something else, they should report it, and also so we can differentiate a timeline for those fixes.

During this time, players have also reported issues with increased game lag and stuttering, as well as some rare crashes (more common with Arena Grinders and players that have long sessions lengths).

We’ve found that an Asset Memory leak is the culprit responsible for these issues, and have a two step fix coming for this issue. The first fix will be coming with v32.3 next month, and another will follow in v33.0 in November.

We know that this issue has been a particular problem for some users, and are happy to have a fix coming your way soon, and want to thank you all for your patience as we worked to address it.

In Conclusion

We know that hearing that these issues are going to persist for a while and that there is, unfortunately, nothing that can be done to speed that up is not the news you likely wanted to hear, nor do we enjoy having to share that with you all, but we wanted to be open with you all about what you can expect and the reason that this happened in the first place.

In the meantime, we will continue our Alliance Quest and Alliance War potion and resource packages and will be delivering a package of Single Player Content Resources next week that we’ll be repeated on an occasional basis.

We want to thank you all for taking the time to read through all of this, and for your patience as we’ve worked on this situation.