Release Process (Dev Diary) Discussion

Latest Dev Diary by @KabamDORK

https://forums.playcontestofchampions.com/en/discussion/392508/dev-diary-mcoc-release-process-and-bug-fixes#latest

I’ll start regarding timing around Submission to Apple/Google.
Says could be up to a few days for Submission to be approved.

So since your example timeframes (2nd pic) seem to indicate you SUBMIT somewhere between 2-3 weeks before RELEASE (were those actually valid examples, that far in advance of Live ?)
Then I gather that is a 2-Step Release process then.

SUBMISSION to Apple/Google, but then you have to manually follow up and do the LIVE RELEASE later on, on the intended day. (as opposed to Apple/Google automatically releasing it as soon as they approve it, which I could imagine would play even more havok with event timings if Apple/Google released it much earlier on their own)

Is that right ? (which would show that releases, even the HotFixes you mention, are no quick process)

Comments

DNA3000

I’ll start regarding timing around Submission to Apple/Google.
Says could be up to a few days for Submission to be approved.

So since your example timeframes (2nd pic) seem to indicate you SUBMIT somewhere between 2-3 weeks before RELEASE (were those actually valid examples, that far in advance of Live ?)
Then I gather that is a 2-Step Release process then.

SUBMISSION to Apple/Google, but then you have to manually follow up and do the LIVE RELEASE later on, on the intended day. (as opposed to Apple/Google automatically releasing it as soon as they approve it, which I could imagine would play even more havok with event timings if Apple/Google released it much earlier on their own)

Is that right ? (which would show that releases, even the HotFixes you mention, are no quick process)

That's how the app store process works on both sides of the fence. Both Apple and Google have a review process where an app (or an app update) must be approved before it can be released through their respective stores. This generally takes a few days, but there is no set timeframe. Apple and Google have been known to take a day, and sometimes a week or more. They make no promises about how long their review process will take, depending on the kind of app it is. They can reject an app for any number of reasons having to do with their app store policies, many of which are sometimes somewhat ambiguous and often change over time. Also, apps can get rejected for *how* they do what they do, separate from what they do, due to the way the two companies (particularly Apple) want people to construct their apps. Use the wrong kind of code to do the wrong thing, and you can get rejected on that basis, even if everything the app appears to do falls within the rules.

Once it is approved, the developer can decide when it actually gets released to their customers or made available for download from the App store. Irrespective of when the developer schedules release, the actual release can get staggered by Apple and Google, so that everyone on Earth doesn't request the download simultaneously.

BringPopcorn

Not gonna lie, this is not the transparency we asked for...
Cool we know the process now, which bugs are getting fixed or being worked?

DNA3000

Not gonna lie, this is not the transparency we asked for...
Cool we know the process now, which bugs are getting fixed or being worked?

I think they are trying to address the questions about why things aren't fixed sooner and quicker, and how anything could possibly take as long as some of these issues take.

Which, to be honest, I'm not sure that development pipeline overview fundamentally addresses. However, I suspect where the whole thing really goes into the weeds is with the merge. Or rather, The Merge. Having not directly seen the brass tacks of their pipelines I can only speculate, but for the MMOs I was more familiar with, The Merge was where things often went kablooey.

But if you haven't seen it, it isn't easy to appreciate.

ViperOfChampion

Not gonna lie, this is not the transparency we asked for...
Cool we know the process now, which bugs are getting fixed or being worked?

Some people on the forums try and say that all they need to do is find code and fixed it, I think his post was just ot humble them

BigBlueOx

Not gonna lie, this is not the transparency we asked for...
Cool we know the process now, which bugs are getting fixed or being worked?

I agree, the corporate mechanics are basically what I would expect. This didn’t tell us anything about which bugs are being addressed, the complexity, etc…

Also makes me question, why it’s so easy to pull from resources to turn off Okoye’s ability to Dex Photon but it takes over a year and a half for Galan to have the right number of incinerates….

The stuff I would have been interested in are as clear as mud still.

Appreciate the attempt, but I feel like this type of thing has been described many times before.

BringPopcorn

Not gonna lie, this is not the transparency we asked for...
Cool we know the process now, which bugs are getting fixed or being worked?
I think they are trying to address the questions about why things aren't fixed sooner and quicker, and how anything could possibly take as long as some of these issues take.

Which, to be honest, I'm not sure that development pipeline overview fundamentally addresses. However, I suspect where the whole thing really goes into the weeds is with the merge. Or rather, The Merge. Having not directly seen the brass tacks of their pipelines I can only speculate, but for the MMOs I was more familiar with, The Merge was where things often went kablooey.

But if you haven't seen it, it isn't easy to appreciate.

All I got from this is the process of fixing something client based or server based. And the timelines required. Which is perfectly fine and transparent. I am not even going to get to the conspiracy **** of some pro player bugs get fixed instantly and other don't. Not even gonna mention the AI one cause I understand is more complex.
For example the VFX one has been awful for months, what is the excuse on that?

unknown

This content has been removed.

BringPopcorn

I feel like I am going to a mechanic asking what is wrong with my car, and they explained the process of making metal...

BringPopcorn

Not gonna lie, this is not the transparency we asked for...
Cool we know the process now, which bugs are getting fixed or being worked?
Some people on the forums try and say that all they need to do is find code and fixed it, I think his post was just ot humble them

If you tell me that post by you know who and the screenshots of coding is what triggered this announcement. I am quitting the game tomorrow....

GroundedWisdom

Not gonna lie, this is not the transparency we asked for...
Cool we know the process now, which bugs are getting fixed or being worked?

They gave an explanation on their process, and why things aren't fixed immediately. They already have a Trello board, and they communicate on the Forum as often as they can. I'm not sure what you're asking for, an itinerary of what they're trying to replicate, or a breakdown of their processes? There's transparency, and then there's an internal review.

GroundedWisdom

As to the subject, I'm happy they made an effort to break things down. This is true, not only for them but many games streamlined like this one. There is a process to prioritization. Sometimes things force their hand because they become much more detrimental to leave. In those cases, hasty actions need to be taken. Otherwise, you need to replicate the issue, then work on the fix, and that fix can either occur server-side or build integrated. Just because things aren't fixed doesn't mean they're ignoring them, or they're not being looked at and worked on.

BringPopcorn

Not gonna lie, this is not the transparency we asked for...
Cool we know the process now, which bugs are getting fixed or being worked?
They gave an explanation on their process, and why things aren't fixed immediately. They already have a Trello board, and they communicate on the Forum as often as they can. I'm not sure what you're asking for, an itinerary of what they're trying to replicate, or a breakdown of their processes? There's transparency, and then there's an internal review.

Good thing you don't need to be sure what I am asking for, the message was not directed for you to act on.
Read the post of BigBlueOx, why are some bugs like Galan or other champs prolonged build after build? Tbh I don't care about your answer, I want theirs.

BigBlueOx

The more I read it… the more it feels like empty words disguised as answers. Too many bugs survive far too long… far longer than this example schedule.

GroundedWisdom

The more I read it… the more it feels like empty words disguised as answers. Too many bugs survive far too long… far longer than this example schedule.

The schedule is an example. Remember, this only applies to issues they can replicate and devise a fix for.

BigBlueOx

The more I read it… the more it feels like empty words disguised as answers. Too many bugs survive far too long… far longer than this example schedule.
The schedule is an example. Remember, this only applies to issues they can replicate and devise a fix for.

Yep I acknowledged it’s an example. But again that’s an obscure metric and timeline on what they consider replication and when they attempt to do it. The approach has never been balanced, and this post offered 0 insights as to why.

BringPopcorn

They took months to replicate Galan not placing the right ammount of incinerates lol

GroundedWisdom

The more I read it… the more it feels like empty words disguised as answers. Too many bugs survive far too long… far longer than this example schedule.
The schedule is an example. Remember, this only applies to issues they can replicate and devise a fix for.
Yep I acknowledged it’s an example. But again that’s an obscure metric and timeline on what they consider replication and when they attempt to do it. The approach has never been balanced, and this post offered 0 insights as to why.

Describe balanced in the sense you're referring to. There are some things that have undoubtedly gone unresolved for a long time. Other things that have taken a long time to be resolved, and others still that are resolved much faster. Part of this is explaining the process.
By your skepticism, it seems like there's nothing they could have said that would have been satisfactory. Perhaps I'm wrong in thinking that. It either sounds like you're asking for them to have their tail between their legs, or a complete list of every issue reported, where they're at in the process, and why things are going wrong. This is the problem when transparency meets expectations. They're rarely in tandem.

PT_99

Just make MCoC-2 broski 🫱

Eakomo

sounds like they're stuck in a prison of thier own making giving very little time to handle already existing bugs and even less time for new bugs that can become higher priority than existing bugs.

now i wouldn't say that i'm an expert at how software development structures, but isn't the goal of having the structure to have the best system possible in aiding the app development and not hinder the process. And such if it is broken might need to be looked at again

Average_Desi

They took months to replicate Galan not placing the right ammount of incinerates lol

Maybe it was just not priority? Like everybody who used Galan might've still been using him in the same frequency and not affected his gameplay at all, so they just kept it down the line. And I don't think it was a case of just changing 7 to a 9. The issue was that harvest was ending too early so they had to change that.

Gentsy12

I appreciate the sentiment and perspective but this post was a swing and miss IMO. If anything, they gave the people that have said to “skip a month’s release” more ammunition.

Based on the process and timeline they outlined it is near impossible for them to release a single update that will not have new bugs and since the the timelines are so compressed and they are working under such limited resources there is no way for they to identify all bugs that will appear, assess the criticality, and fix the critical ones. This is evident in the releases with new bugs and interactions in every release and some not getting fixed for multiple releases. TBH I’m actually shocked some things don’t take even longer because they could be building a fix for something that is going to break in a different way at next release. The code seems to be built on a house of cards and is at risk of falling over multiple times every month.

I don’t know what the right approach is. I know a suggestion such as “change to a bi-monthly release” is not feasible. The other recommendation would be add more dev and testing resources but that is likely above the pay grade of the game and community teams.

With all that said, I have appreciated the additional effort that has gone into communication recently but this one felt like it took a step backwards.

ahmynuts

I appreciate the sentiment and perspective but this post was a swing and miss IMO. If anything, they gave the people that have said to “skip a month’s release” more ammunition.

Based on the process and timeline they outlined it is near impossible for them to release a single update that will not have new bugs and since the the timelines are so compressed and they are working under such limited resources there is no way for they to identify all bugs that will appear, assess the criticality, and fix the critical ones. This is evident in the releases with new bugs and interactions in every release and some not getting fixed for multiple releases. TBH I’m actually shocked some things don’t take even longer because they could be building a fix for something that is going to break in a different way at next release. The code seems to be built on a house of cards and is at risk of falling over multiple times every month.

I don’t know what the right approach is. I know a suggestion such as “change to a bi-monthly release” is not feasible. The other recommendation would be add more dev and testing resources but that is likely above the pay grade of the game and community teams.

With all that said, I have appreciated the additional effort that has gone into communication recently but this one felt like it took a step backwards.

One correction. It would be above what Netmarble allows for kabam. Because netmarble sucks

Bosley

Picture has a typo. It should read "Bug goes live to the public"....

BostonSpidey124

I’m sorry but this post reads like one big excuse. Everyone has processes and limitations at work that allow things to be done at a certain pace. That doesn’t justify the extent of bugs and issues. You can fix Okoye being able to dex when photon turns off dex in less than a week but you can’t fix Peni Parker so she can punish a champion? What are the odds that a problem that make a champion overpowered is always something you can fix immediately but issues that make champions weaker always take months?

This doesn’t address the quality controls that appear to be lacking. You launch a featured six star that doesn’t have the advertised pool? I get that the game is complex and the processes you have to follow make things go slow but this post provides nothing to players other than an excuse on why you are slow fixing the numerous issues in game.

Average_Desi

I’m sorry but this post reads like one big excuse. Everyone has processes and limitations at work that allow things to be done at a certain pace. That doesn’t justify the extent of bugs and issues. You can fix Okoye being able to dex when photon turns off dex in less than a week but you can’t fix Peni Parker so she can punish a champion? What are the odds that a problem that make a champion overpowered is always something you can fix immediately but issues that make champions weaker always take months?

.

This isn't an excuse, but communication. People constantly ask why can't they do this now, why can't you do that then, etc ,etc. this is an answer to *those* questions.

As for the second question, the only reason they do that is because the community thinks they do. There are bugs like MoleMan and kingpin that made them "overpowered" which took years to fix.

GroundedWisdom

I’m sorry but this post reads like one big excuse. Everyone has processes and limitations at work that allow things to be done at a certain pace. That doesn’t justify the extent of bugs and issues. You can fix Okoye being able to dex when photon turns off dex in less than a week but you can’t fix Peni Parker so she can punish a champion? What are the odds that a problem that make a champion overpowered is always something you can fix immediately but issues that make champions weaker always take months?

This doesn’t address the quality controls that appear to be lacking. You launch a featured six star that doesn’t have the advertised pool? I get that the game is complex and the processes you have to follow make things go slow but this post provides nothing to players other than an excuse on why you are slow fixing the numerous issues in game.

Anything can happen. The Crystals are server-side. I can count the amount of times that Champs haven't been in the Crystal on one hand, with fingers to spare. There's no way to test a server bug like that until it comes up when they go live.

Shadowstrike

I appreciate the sentiment and perspective but this post was a swing and miss IMO. If anything, they gave the people that have said to “skip a month’s release” more ammunition.

To be honest, they can have a full nuclear Arsenal but there is one undeniable truth: they will never not take a month off from releasing new characters & content because not only do they have their bosses at Netmarble to answer to, but they also have Marvel Comics to answer to, which means they have two major bosses saying to release content and which characters.

BringPopcorn

They took months to replicate Galan not placing the right ammount of incinerates lol
Maybe it was just not priority? Like everybody who used Galan might've still been using him in the same frequency and not affected his gameplay at all, so they just kept it down the line. And I don't think it was a case of just changing 7 to a 9. The issue was that harvest was ending too early so they had to change that.

You are completely right, it wasn't a priority back then, like a lot of things, and the result is a bunch of non priority things backed up one after another and a bunch of nonsense changes that should not be a priority.
How is a UI change a priority over awful looking VFX that sometimes don't even show SP projectiles. We heard the praise to the art department with their awesome work. The art deparment doesn't miss!!! Well what is the point of them not missing when the end result is champs that look like they were colored with crayons?
Why are they messing around with champ sizes in different game modes when they know the hitbox issue was a thing? It took how many raids to realize "hey maybe they should be normal sized..."
Priorities I guess...

BostonSpidey124

And if this post helps the people asking those questions, I’m happy for them. But to me, it reads as one big excuse on why changing things is hard. I would have more sympathy for it if they didn’t constantly make changes to their benefit in extremely timely manner. Again I will use the Okoye bug as an example. They found the bug (probably because of YouTube videos), figured out the solution, made the fix and pushed it out in less than a week. Another example is the additional rewards to battlegrounds this season. They added those in less than a week when they originally said they couldn’t do it. I understand running a live game is difficult and challenging and the longer it has been running and the more complex it is, the harder it becomes. But the amount and types of bugs is not acceptable. Champions can just whiff their attacks? Your update changed a champions attacks and animations (Juggernaut from a few months ago)? This post doesn’t address any real issues just tries to explain why what they do is hard. I don’t need to know what makes your job hard, everyone’s job has challenges. I need to know what you are doing to fix the issues that exist and how you are going to prevent issues of this extent from happening again.

BigBlueOx

The more I read it… the more it feels like empty words disguised as answers. Too many bugs survive far too long… far longer than this example schedule.
The schedule is an example. Remember, this only applies to issues they can replicate and devise a fix for.
Yep I acknowledged it’s an example. But again that’s an obscure metric and timeline on what they consider replication and when they attempt to do it. The approach has never been balanced, and this post offered 0 insights as to why.
Describe balanced in the sense you're referring to. There are some things that have undoubtedly gone unresolved for a long time. Other things that have taken a long time to be resolved, and others still that are resolved much faster. Part of this is explaining the process.
By your skepticism, it seems like there's nothing they could have said that would have been satisfactory. Perhaps I'm wrong in thinking that. It either sounds like you're asking for them to have their tail between their legs, or a complete list of every issue reported, where they're at in the process, and why things are going wrong. This is the problem when transparency meets expectations. They're rarely in tandem.

Unbalanced, as in there is no discernible rhyme or reason as to what “bug” gets fixed or when. It feels completely reactionary, and anyone who’s played long enough knows that the level of testing is just as unpredictable, as it’s not uncommon to have to deal with unexpected consequences. At this point it’s a meme, the community expects failure and bugs with every release or “fix”.

This is not a “Kabam only fixes pro-player” bugs rant but it’s an observation that as a consumer it’s like a constant spectre that floats over every kit in the game and at the will of a mercurial king will things be addressed. Okoye’s big went from unknown as a bug to fixed from what seems to be a forum comment. Galan’s bug has an 18month old thread begging for acknowledgment, showing proof for it be ignored. There’s little evidence of any prioritization when it comes to character bugs…

And then there’s the Juggernaut debacle where he wasn’t bugged and they decided to “bug fix” him into being bugged… and that took far far longer than it should for one of the most used characters in the game. One they actively sold at a premium price point.

As for the communication, Miike and Jax have effectively communicated things like this before. Many times even, without sounding as dodgy as this one did. So this fell flat. I sincerely appreciate the huge steps forwards and efforts in general by the communication team… I think this is a misfire though. It’s not one that largely detracts from the huge improvement we’ve seen the last month or two, but still a miss.