Arena crystal drop odds analysis

I've been waiting to build up enough crystals (and honestly to see if a summoner bonus would come around) and I finally decided to pull the trigger on this one. I've done an analysis of the drop odds for arena crystals. Pretty much every attempt to measure this in some way has looked at the average rate of gold and units from the crystals. The reason is everyone opens them as fast as possible ten at a time, and there's no way to dig any deeper into the data than to look at overall average results: you can't actually attempt to figure out what the odds of any particular drop are going to be.

Unless you found someone crazy enough to open arena crystals one at a time. But who would be crazy enough to do that?

So anyway, I have opened 1503 crystals one crystal at a time: 900 uncollected crystals and 603 regular crystals (the "3" is a long and boring and probably predictable story). First of all, the raw data (as in, the tabulated data: I have no ability to reliably video record 1500 individual crystal openings).

From this, it is possible to state within reasonable statistical certainty that the drop odds between the two crystals is, over large numbers of openings, basically identical. Of course the rare events like Punisher drops are going to be more variable between the two data sets, but the drop rates of gold and units are identical within the margin of error for this size data set. Because of that, I'm comfortable combining the two data sets to look at the drop rates for gold and units and everything else using a 1500 drop data set. When I combine data and look at the drop rates and the estimated margin for error, and given game developers' propensity to weight drop tables using whole numbers whenever possible, I get this estimate for the weighted drop rates for the arena crystals:

Gold: 84 (out of 100)
Units: 15
Other: 1

Within each type of drop, I get this estimate for the drop weights:

Gold
Large (10k/50k): 15 (out of 100)
Medium (5k/25k): 25
Small (3k/15k): 60

Units
Small (15/75): 70 (out of 100)
Large (45/225): 30

Other
Punisher small: 2 (out of 3)
Energy refill: 1
Punisher large: << 0.1

This then allows me to calculate the average rate of return for the arena crystals. For every one hundred crystals opened, players should expect to get from the Uncollected crystals:

Gold: 1,911,000
Units: 1800
Energy Refills: 1.6
Punisher 4*: 0.67
Punisher 5*: haha

For the normal crystals, it is one fifth the return, so 382,200 gold and 360 units. Also, this does not count the gold that you get from duping Punisher, but that is a small amount overall (it would add about a thousand gold to the overall averages).

There is one way in which the crystals have a different average return, and that is the Punishers that drop. The normal crystal drops 3* Punisher with a rare chance at 4* while the Uncollected arena crystal drops a 4* with rare chance at 5*. However, the 4* Punisher is, in my opinion, more than five times more valuable than the 3*. The 3* when duped generates two 5k ISO bricks while the 4* generates twenty four. The gold from duping actually skews slightly in favor of the normal crystal (it is less than five times as much gold) but the ISO difference is larger and swaps it. Overall, the difference isn't large and the Punisher drop is unucommon anyway, but that small difference is there (the super rare drop doesn't happen often enough to materially change the value of the crystals).

So if you want to know if your own crystal opening was above average or below average, you can now compare to these numbers and see. I cannot say with 100% certainty these are the precise drop odds for the crystals, but I'm confident these numbers are close, and I believe there's a strong chance they are the exact numbers. Of course, if this is the one time that a game developer decided to make the drop odds for a lootbox something like 14.8757 and 84.1243 just to mess with the players, there would be no way for the players to determine that without literally tens of millions of individual crystal openings. It would take less effort to stage a heist of the reward spreadsheets on Kabam's offices.

Comments

Panchulon21

Lotta math that I actually understand and think about when opening crystals. Even though the probabilities reset after every pull, its usually going to fall in the ranges you saw above.

Kudos for the work you put in.

Werewrym

When’re you going to do this with the hero crystals so we can confirm champion drops are random?

Werewrym

Awesome analysis by the way. Unfortunately it will never affect me because arenas are the only part of this game I can’t stand.

DNA3000

When’re you going to do this with the hero crystals so we can confirm champion drops are random?

Back when people used to stream PHC openings, I did a meta analysis of all streamed videos I could find (only live streamed, so no selection bias) and posted my guesses as to the 2*/3*/4* relative drops. It would take too many crystals to look at individual drop rates of individual champions, and since then I don't think anyone seriously believes that Kabam manipulates the relative drop rates of 3* and 4* champions anymore.

I am thinking about other crystals that I get enough of to do a proper analysis, but as you can imagine opening one crystal at a time does take some focused effort. When I'm focused just on that and in a rhythm, it takes me about 30-40 minutes to tabulate a hundred crystals of individual drops. So you're looking at about ten hours of accumulated work for the analysis above.

StevieManWonder

Wow, thanks a bunch for doing all this work!

SummonerNR

Nice work @DNA3000 😀

I do want to point out a couple things...

** Energy Refill... even if merging UC and Reg pulls together, your “Average Rate” should be listed as 0.33 instead of 1.6 (note it should be half of your Punisher drops).

Although since you did not pull ANY refills from Reg crystal (might be within statistical deviation, but just seems odd), I wonder if they are even included in the Regular Crystal at all (and were the UC ones Full Refills or just the Small Refills ?). If they are not even in the Reg crystal, then for UC the Avg Rate for Energy would be about 0.55 since you wouldn’t then count the Reg pulls in the denominator.

** And UNITS... I’m sure you noticed that the spread between the 3 levels of GOLD was consistent no matter if UC or Reg crystal. BUT, the spread between the Low/High for UNITS seemed to be skewed AGAINST the High Units when pulled from UC (versus Low/High from the Regular crystal)

61 vs 30 Low/High from Reg. But 95 vs only 35 for UC (if consistent with Reg it should have been more like 95 Low vs 46 High). Could just be that the true rate between Low/High is indeed the same between Reg and UC, and Reg just statistically fell for you sample size a little over the actual odds, while UC just fell a little under the odds, making the difference appear larger in your data. But seems at least significant enough to maybe take note of (??)

SummonerNR

(answered one of my own question about Refills)..
Both crystal shows 0.35 in their Drop Rates in game for Energy Refills (and pictures Full Refill). So I guess was just unlucky that none came during your Reg Crystal pulls.

ESF

Thanks for crunching those numbers -- you are the man

DNA3000

Nice work @DNA3000 😀

I do want to point out a couple things...

** Energy Refill... even if merging UC and Reg pulls together, your “Average Rate” should be listed as 0.33 instead of 1.6 (note it should be half of your Punisher drops).

1.6 is the expected average number of energy refills. Uncollected crystals drop them in sets of five. If we assume that on average one time in a hundred the drop will be "other" and one time in three that will be a five pack of energy refills, that means the average number of energy refills you should expect to get is 5/3, or 1.67. I truncated that to 1.6, I should have rounded to 1.7.

Also, I probably should have asked myself what I was going to do with twenty five energy refills.

Although since you did not pull ANY refills from Reg crystal (might be within statistical deviation, but just seems odd), I wonder if they are even included in the Regular Crystal at all (and were the UC ones Full Refills or just the Small Refills ?).

I've seen them before, just coincidentally not in these runs.

** And UNITS... I’m sure you noticed that the spread between the 3 levels of GOLD was consistent no matter if UC or Reg crystal. BUT, the spread between the Low/High for UNITS seemed to be skewed AGAINST the High Units when pulled from UC (versus Low/High from the Regular crystal)

61 vs 30 Low/High from Reg. But 95 vs only 35 for UC (if consistent with Reg it should have been more like 95 Low vs 46 High). Could just be that the true rate between Low/High is indeed the same between Reg and UC, and Reg just statistically fell for you sample size a little over the actual odds, while UC just fell a little under the odds, making the difference appear larger in your data. But seems at least significant enough to maybe take note of (??)

That's possible, but it is also within the statistical margin for error. While I'm taking my own reasonable guess - I point out that my analysis is partially based on quantitative statistical analysis, and partially on my "feel" of what simple numbers match the statistical results - I'm also publishing the actual data along with the analysis so that such things are visible.

1500 crystals sounds like a lot, but when that will end up generating about 200 unit drops, and maybe only 60 or 70 of the big unit drops, suddenly the margin for error is like 12%. It could approach 20% for the subsets. That makes the actual real margin for error range for the ratio of the normal crystals by themselves anything between 1.5 and 2.8. Combining the data lets me reduce that to something closer to 2.9 to 2.0. Still pretty big.

It would be fair to say my guestimate for the primary weights is 99% calculation and 1% guess. For the gold weights it is 95% calculation and 5% guess. For the unit weights it is 50% calculation and 50% speculation.

If you stare at the individual pieces of data long enough, you could probably convince yourself that Kabam changed the crystal on the 12th. But since the shift is towards the larger unit drops, no one would believe Kabam did that.

More data will slowly clarify this over time. But as this represents about a year of arena grinding for me, it will take some time to get enough data to add another digit of precision to all the numbers.

DNA3000

(answered one of my own question about Refills)..
Both crystal shows 0.35 in their Drop Rates in game for Energy Refills (and pictures Full Refill). So I guess was just unlucky that none came during your Reg Crystal pulls.

Actually, I didn't bother to look at those rates as I was worried it would bias my analysis. I had actually forgotten that energy refill was specifically called out in there and didn't think to even look at it before you mentioned it just now.

Nice to know that at least to the extent that the numbers are there, my analysis is reasonably close to them.

DNA3000

For completeness sake, the thing my analysis got wrong relative to the published odds is that I assumed the odds of pulling the 5* Punisher were included in the 1% odds of pulling anything but units and gold, but as it turns out those odds were pulled from the odds to get Gold, reducing Gold from 84% chance to 83.98% chance, which is completely impossible to measure.

Also, apparently sometimes game developers design drunk.

SummonerNR

Thx. Guess that’s why you listed Energy Refill x5 for UC. Didn’t know they dropped 5 Refills together when you get one from UC. Although makes sense that you get 5x as much from a UC pull than if you pull a single Refill from the Regular crystal.

Capn_Dante

(actually I really enjoyed this post... Just couldn't help it with the gif)

DNA3000

For those interested, I posted my analysis for (normal) gold crystals here: https://forums.playcontestofchampions.com/en/discussion/173966/gold-crystal-drop-odds-analysis-with-data-speed-up-trick

Turns out that once I thought about it a bit, there was a way to speed this up for gold crystals that unfortunately doesn't work for arena crystals. Or perhaps fortunately, because if it did work for arena crystals I might have cried.

Myradral

When’re you going to do this with the hero crystals so we can confirm champion drops are random?

If you think drop rates for heroes are random in this game, you are nuts.

It is RIGGED RIGGED RIGGED.

TheTalents

I've been waiting to build up enough crystals (and honestly to see if a summoner bonus would come around) and I finally decided to pull the trigger on this one. I've done an analysis of the drop odds for arena crystals. Pretty much every attempt to measure this in some way has looked at the average rate of gold and units from the crystals. The reason is everyone opens them as fast as possible ten at a time, and there's no way to dig any deeper into the data than to look at overall average results: you can't actually attempt to figure out what the odds of any particular drop are going to be.

Unless you found someone crazy enough to open arena crystals one at a time. But who would be crazy enough to do that?

So anyway, I have opened 1503 crystals one crystal at a time: 900 uncollected crystals and 603 regular crystals (the "3" is a long and boring and probably predictable story). First of all, the raw data (as in, the tabulated data: I have no ability to reliably video record 1500 individual crystal openings).

From this, it is possible to state within reasonable statistical certainty that the drop odds between the two crystals is, over large numbers of openings, basically identical. Of course the rare events like Punisher drops are going to be more variable between the two data sets, but the drop rates of gold and units are identical within the margin of error for this size data set. Because of that, I'm comfortable combining the two data sets to look at the drop rates for gold and units and everything else using a 1500 drop data set. When I combine data and look at the drop rates and the estimated margin for error, and given game developers' propensity to weight drop tables using whole numbers whenever possible, I get this estimate for the weighted drop rates for the arena crystals:

Gold: 84 (out of 100)
Units: 15
Other: 1

Within each type of drop, I get this estimate for the drop weights:

Gold
Large (10k/50k): 15 (out of 100)
Medium (5k/25k): 25
Small (3k/15k): 60

Units
Small (15/75): 70 (out of 100)
Large (45/225): 30

Other
Punisher small: 2 (out of 3)
Energy refill: 1
Punisher large:

The smaller area crystals give better results. I only open battlechips 2 million at a time and It only took me two runs to realize the significant different in the amount of units I retrieve from one crystal to another.

DNA3000

The smaller area crystals give better results. I only open battlechips 2 million at a time and It only took me two runs to realize the significant different in the amount of units I retrieve from one crystal to another.

Everyone has their hunches. Here I'm only interested in doing an objective data driven analysis of the drop rates. The current data can't completely rule that possibility out, but it can state with reasonable certainty that any deviation that does exist would take more than that many crystals to prove actually existed.

TheTalents

The smaller area crystals give better results. I only open battlechips 2 million at a time and It only took me two runs to realize the significant different in the amount of units I retrieve from one crystal to another.
Everyone has their hunches. Here I'm only interested in doing an objective data driven analysis of the drop rates. The current data can't completely rule that possibility out, but it can state with reasonable certainty that any deviation that does exist would take more than that many crystals to prove actually existed.

I'm telling players that open no where near the amount of battlechips I open at a time if you're looking for units which would be the most valuable resource you should open the smaller arena crystals. Just by the volume since you open way less for a lot of battlechips you're more than likely to be snubbed opening the larger arena crystals. Plus I already got 5 star punisher so need to proceed with the bigger ones at all. that's all I'm saying. I'm not refuting your data.

unknown

This content has been removed.

DNA3000

The smaller area crystals give better results. I only open battlechips 2 million at a time and It only took me two runs to realize the significant different in the amount of units I retrieve from one crystal to another.
Everyone has their hunches. Here I'm only interested in doing an objective data driven analysis of the drop rates. The current data can't completely rule that possibility out, but it can state with reasonable certainty that any deviation that does exist would take more than that many crystals to prove actually existed.
I'm telling players that open no where near the amount of battlechips I open at a time if you're looking for units which would be the most valuable resource you should open the smaller arena crystals. Just by the volume since you open way less for a lot of battlechips you're more than likely to be snubbed opening the larger arena crystals. Plus I already got 5 star punisher so need to proceed with the bigger ones at all. that's all I'm saying. I'm not refuting your data.

The "snubbing" effect gets averaged out over time, because you get more units with the Uncollected crystal when you get units at all. Assuming the odds for the two crystals are identical, which seems to be true and is consistent with what Kabam says about the crystals (but which technically hasn't been proven yet).

But suppose you didn't care about averaging out over time. Suppose you don't care whether you get more or less units overall, but rather care if you get units at all. Suppose you are primarily concerned with getting more units, without regard for how much more units you get. In other words, suppose we compare two different players, one of which always gets 100 units, and one of which gets 50 units 90% of the time and 1000 units 10% of the time. On average the first player gets 1000 units every ten crystals while the second player gets 1450 every ten crystals. On average, the second player is doing way better. But, when you compare their pulls one batch at a time the first player "wins" 90% of the time. Nine times out of ten he does twice as well as the second player, and only one time out of ten the second player does ten times as well. What if you would rather be the first player than the second player (this is an exaggeration for illustration purposes, I'm not saying anyone wants this dramatically less rewards over time)? What's the best strategy then?

Just for giggles, rather than do the closed form math (lazy) I decided to just simulate this. I took two million virtual players and then handed them each 200000 battle chips. Half bought 20 uncollected arena crystals and the other half bought 100 normal arena crystals. Each pair of players then opened them, and only compared the units they got. What ends up happening?

Winner Normal: 524347
Winner Uncollected: 452020
Tie: 23633
Total Normal: 359933685 100000000
Total Uncollected: 359984100 20000000

So in terms of actual number of units, it is basically a tie: both groups of players collectively got the same number of units. However, when we line them up in pairs the players who opened normal crystals "win" about 52.4% of the time, compared to the uncollected players winning about 45.2% of the time (2.4% of the time they tie).

So if you open normal crystals and compare yourself to someone else that opens uncollected crystals, then for the same number of battlechips you're going to get more units than the other guy more often. But when he gets more than you, he will tend to get much more than you. You win more often, he wins by more units when he wins.

This one is more of a psychological choice than anything else.

unknown

This content has been removed.

DNA3000

Wow this is incredible! I think there are a couple of deeper items that need to be analyzed here statistically so we can just look at what these numbers show us. Is it ok with you if I pull this data just to take a look at it? Anything I find I can DM you. It's your work here, I just want to try and discover a bit more about it.

Thanks for the amazing research!

Unfortunately, as I mentioned, I only have the tabulated data. I did not keep roll-by-roll records of each individual pull. I really, really, really wanted to do that, but when I experimented with doing that it ended up slowing me down by a significant amount. I'm considering figuring out a way to do that in a way that wouldn't slow things down too much, but I'm not quite there yet. I think I know where you might be going here, and I'd like to go there also, but my data collection time is not unlimited.

unknown

This content has been removed.

DNA3000

One thing I'd like to mention that if you just wanted to treat each day like it's own variable, this is something that can be easily added upon to add more data to what you have already. I'd be happy to help out with that if you're interested. I'm not sure if that's something you considered, but it's quite feasible to just add to this data even if you planned on collecting more and structuring it some other way.

If there's any way I can help with data collection or organization, please let me know how and where to send it.

It is extremely likely that the next time I do this, I'll record each individual opening separately, with timestamps. I may also do it less "all at once" than this time, since the 20% gold bonus is interesting, but not particularly useful to me even though I did wait for it this time (between the arena crystals and the gold crystals I'm now over the 100 million mark) and the deficit of doing it all at once is you end up with a ton of energy refills you have to figure out how to make efficient use of.

Wubbie075

When’re you going to do this with the hero crystals so we can confirm champion drops are random?
If you think drop rates for heroes are random in this game, you are nuts.

It is RIGGED RIGGED RIGGED.

DNA posted his methodology, data, and results/analysis. Care to do the same?