|
Post by thegentleman on Nov 30, 2012 18:42:11 GMT -5
Okeydoke, bros.
After hearing some of the discussion as to whether or not SBMM exists in public lobbies (despite Vahn's twitter post that says, "yeah, it does exist to some extent or another"), I decided to apply my rudimentary grasp of the statistics course I took about 7 years ago to trying some ham-fisted measurement of SBMM.
Is this experiment perfect? Hell no. But it's something I could do in 20 minutes and if you'd like to conduct your own number crunching or have a better idea for methodology or measurement, go ahead and do it yourself.
THE METHOD I went into 10 random TDM lobbies and recorded the K/D ratios of the players there, subtracting my own ratio from the data. I imagined that the worldwide K/D ratio should sit at 1.0. We've mentioned that suicides might bring this figure down slightly, but I don't think enough to truly matter. If SBMM wasn't a factor, I hypothesized that the average of all players would hover around 1.0, as was my experience in the first Black Ops and MW3.
My own K/D ratio in BOII is 1.41, with a SPM of 285 and an abysmal W/L ratio of 0.77, given my tendency to back out of lobbies that shaft me connection-wise. Player base -wise, I'm probably in the upper 10 to 20% by my own estimates, but that's nothing I could quantify.
For this experiment, I used the statistical approach of determining a standard error of the population (standard deviation divided by n, the number of player ratios collected). Standard error allows us to determine the chance that the TRUE population mean is somewhere within a certain range of data. By multiplying the standard error by 3 and then adding and subtracting this to the mean of our data, we get a confidence interval of 99.6%.
THE DATA Writing down the data I could get out of 10 lobbies yielded 85 total players. If I joined a game in progress, I backed out. If I didn't have enough time to write down all of the K/D ratios in the lobby, I excluded them from the results.
The mean of all players' K/D was 1.260. The standard deviation of the K/D ratios was 0.314, and our standard error was 0.034. Using the standard error, we can say with 99.6% confidence that the true K/D average of the people I'm being matched with is between 1.158 and 1.362. In other words, and using the assumption that the true K/D ratio of all Black Ops II players is 1.0, there's less than a 4 tenths of one percent chance that the people I'm being matched with would ever even out to a 1.0 K/D or present a representative sample of the total BOII player base.
From that, I conclude that SBMM exists and is a significant factor in online play. Getting into whether this is a good or bad thing has been beaten to death and is beyond the scope of this thread, so save your breath.
ANALYSIS AND LIMITATIONS A few things of note. First, I'm in Los Angeles. With a majorly huge metropolitan center playing at 3:00 on a Friday, there's going to be enough potential matches to put me in lobby after lobby with "best" matchmaking settings. So latency may be "king," as Vahn said, but there's enough people I can get a 4-bar to that the algorithm can probably rely on SBMM. People in less dense metro areas may feel the effects of SBMM far less, as they'll presumably be matched with other players outside of their skill level in order to minimize latency first and foremost.
Second, it's possible that there are simply just more good players online. Bad players who get stomped and end up with a 0.3 K/D ratio might say, "Screw it: I hate this game and I'm never playing it again." From that vantage point, the average K/D ratio of players on at any given time MIGHT certainly be over 1.0. I doubt it would matter to a significant extent, but it's possible.
It would be interesting to repeat this "lobby" test for MW3, WAW, MW2, and Black Ops 1 to determine to what degree this is true in the CoD franchise.
Interestingly enough, only 14 of the 85 players was under 1.0, with the absolute lowest hovering at a 0.67 ratio.The highest ratio I recorded was a 2.23. I'd regularly see people in MW3 and BO lobbies with ratios well under 0.6. The SBMM in public lobbies may work by simply putting a floor and a ceiling over whatever metric you're currently at.
A few final thoughts: SPM may be a metric being used for SBMM. If so, I didn't test it: you guys are welcome to. Also, I don't think W/L is a factor. I say this based on two lines of thinking. First, mine's well below 1.0. If I were being matched with players who lose far more often than they win, I'd probably get a sample of players who are worse, but it's ALSO entirely possible that with so many people aware of the latency issues and backing out of lobbies whenever they don't feel right, everyone's W/L has taken a hit.
Secondly, if W/L was a consideration, it'd be easy to game the SBMM algorithm by just backing out of lobbies constantly rather than by C4 / RPGing yourself over and over. This seems like such an exploitable issue that 3arc would have intentionally made the choice not to factor in W/L as a matchmaking criteria from the get-go. So with that, the wall of text is over. Based on my ghetto-ass statistical test, I'm going to say that SBMM is a verifiable thing in public TDM lobbies and is a significant factor on how well a person plays with respect to previous games. To some of us, that may have already been a conclusion we intuited already, but here's a limited amount of proof that makes that more than wild speculation.
|
|
wittyscorpion
True Brorange
All warfare is based on deception.
Posts: 8,598
|
Post by wittyscorpion on Nov 30, 2012 19:13:57 GMT -5
I can speak with 100% confidence this is well above the range of the players I was matched with. Below are two sample games I mentioned in the other thread (my KDR was around 0.97 - 1.02 when these games were played):
So I am a firm believer on that SBMM exists as well, at least for TDM.
|
|
|
Post by aidsaidsaids on Nov 30, 2012 19:33:37 GMT -5
I honestly think it just sorts by ping and uses k/d or some other measure to break ties.
|
|
asasa
True Bro
fuck
Posts: 4,255
|
Post by asasa on Nov 30, 2012 19:35:36 GMT -5
I was thinking that it separates things rather crudely, such as: K/D / W/L avg < .75 K/D / W/L avg .75 <-> 1.0 K/D / W/L avg 1.0 <-> 1.25 K/D / W/L avg > 1.25 Then it tries to put you into the correct tier, but if it cant, will put you in another. Again, all speculation
|
|
|
Post by RageHulkSmash on Nov 30, 2012 23:54:24 GMT -5
Here's how i think this works (this is just a guess so dont take this as fact) Connection is #1, vahn said it. But people are easily finding games that are within their k/d range. So what i think it does is this Seperates connections into tiers (<50ms, <75 ms, and so on for example) then when it does the "Compatible games" thing it checks for an average k/d and compares them to your own. Its probably using range, average, and median k/d to make some sort of box and whiskers plot to find a good range of players for you. Probably scans a bunch of games where the average k/d fits in the upper and lower quartile of your data. Probably picks the closest one then. Then after each game it updates your little graph. I feel like this could explain why sometimes games stay empty when people leave, since the games still in early life, so every game can fu [b/]ck up the system.
this is all speculation.That's exactly what I was saying earlier. People who are finding many games available in each bracket (<50ms, <75ms, etc) will be put into the closest skill range lobby that is open. The more games available at the time, the more likely you will find other players matching your skill level.
|
|
|
Post by thegentleman on Dec 1, 2012 6:54:52 GMT -5
One interesting thing about the SBMM is that if it uses either K/D or SPM as metrics is that the algorithm is going to be working around the compression effect that we're all noticing. Both the 2.0 and 0.5 k/d ratios were more common in MW3 than they are here.
With that in mind, there might be an entire standard deviation between a 1.2 k/d ratio and a 1.4 k/d ratio even though that might have seemed like a fairly small skill difference in players in any other MW game.
It'll also be interesting to watch the stats once they get all of the crazy, game-breaking lag issues sorted out (hopefully). I'm cautiously optimistic.
|
|
Evan950
True Bro
PSN: xXCrazyBarksXx
Posts: 869
|
Post by Evan950 on Dec 1, 2012 7:33:39 GMT -5
ohkay, first question\ what is SBMM? something* based match making?
|
|
|
Post by corpsecreate on Dec 1, 2012 10:28:33 GMT -5
My guess is its "Skill Based Match Making"
Also, the it might not be a good idea to calc the "average KD" summing all the values and divding by the sample size. The reason I say this is because the "average" is sort of a multiple, not an addition.
2.0 KD 0.5 KD
The "Average" for this should be 1.0, not 1.25.
|
|
sh58
True Bro
Posts: 10,277
|
Post by sh58 on Dec 1, 2012 11:10:18 GMT -5
Skill based matchmaking is so bad when they have league mode. Wtf. I wonder if this is something they can patch.
One of the big reasons I like cod is being able to track my kd and watch it go up as I get better. With skill based matchmaking this probably won't happen to anywhere near the same degree
It also means the higher kill streaks are going to be attained less frequently.
Also, you can't mess around with crappier weapons/set ups if every game you are playing vs similar standard players cos you will get crushed.
If they wanted skill based matchmaking they should have implemented some kind of elo system.
|
|
|
Post by Voice from the Basement on Dec 1, 2012 12:32:24 GMT -5
Well, I decided to analyse recently played games and the players I met there. I played Kill Confirmed game mode.
My K/D is pretty low (1.67 – that's too low for MW2 or MW3, but nearly OK for BOII), but I realized that the highest K/D player I've played with recently was…1.41. Yes, only 1.41. And the average K/D was even lower than 1.00 (lots of 0.8–1.00 players and one player had 0.24…). Where's the SBMM? But it's maybe due to my residence (Eastern Europe) and comparative unpopularity of this mode in BOII. However, there're no problems with searching free lobbies.
|
|
|
Post by funcrusher on Dec 1, 2012 13:09:49 GMT -5
Agreed with mouse. 1.67 is pretty high, even in other cod games. Anyone higher than that is either godly or a fan of the dashboard. In mw3 I got moabs fairly often and was a 1.57. Blops 2 seems as if it was made with the mindset that you aren't supposed to last. My tactical loitering skills are strong and I still haven't played a perfect game.
|
|
|
Post by Voice from the Basement on Dec 1, 2012 13:28:16 GMT -5
Well, in MW3 I had 2.2 K/D ratio (with only 25 M.O.A.B.s so far playing Specialist-only style) and in MW2, after I lost all of the stats 'cause of crashed HDD, I had ~3.00 K/D. And the only game I felt I'm playing not so bad was MW2. I say 1.67 totally sucks 'cause there're lots of tryhаrds good players here with their >2.00 K/D playing Shotgun-only style and so on.
But the subject I was speaking about was not my low K/D – I just mentioned it here for making comparison look clear.
|
|
GoHarvard
True Bro
It's not that serious.
Posts: 390
|
Post by GoHarvard on Dec 1, 2012 14:20:40 GMT -5
Second, it's possible that there are simply just more good players online. Bad players who get stomped and end up with a 0.3 K/D ratio might say, "Screw it: I hate this game and I'm never playing it again." From that vantage point, the average K/D ratio of players on at any given time MIGHT certainly be over 1.0. I doubt it would matter to a significant extent, but it's possible. I don't know how much I believe this. Bad players have always been common and now they are rare? A good test would be on a new account with a crap KDR like 0.4 and go into TDM match making and see if the scrubs can be found. My theory: New and casual players are usually the main culprits when you would see someone going 4-18, 3-20, 0-17 etc. in TDM mode. It still happens, just not as much. They have said it every year that getting more casual/new players hooked into online multiplayer is important. Obviously because it will get them more DLC sales. So the experience for the casual/new/struggling player is important to them. Even though the community has generally hated Death Streaks they never took them out of the game until now. Why would they do that? They don't care about helping the scrubs get a come-back-kill anymore? Or are they helping them in a different way now? I've never been killed so many times in situations where I set myself up for a high % kill because I know an enemy player is coming because he's on the UAV or just killed a teammate near by. It's business as usual to see players at the bottom of the scoreboard (or losing team, read below) pull off these types of miracles on here without having to pre-fire/aim while strafing out. The 4-18 player is still here, he just doesn't go 4-18 anymore though. I'm also noticing a lot more come-from-behind victories or close scores in TDM matches. "It's the skill based match making?" That could be the reason if teams were trading kills more consistently, but that's usually not the case. It revolves around a sudden increase of (artificial) lag. Like the above scenario I mentioned in which a certain group of players or maybe a whole team suddenly have a more difficult time getting kills. I've been on both ends of that. The idea that they would use artificial lag or other means to create a perception of more competitive matches and statistics (KDR's in this case). Is it far fetched? I will write stuff down to keep a log to see if I can come up with something to analyze.
|
|
|
Post by thegentleman on Dec 1, 2012 14:31:50 GMT -5
I don't know how much I believe this. Bad players have always been common and now they are rare? A good test would be on a new account with a crap KDR like 0.4 and go into TDM match making and see if the scrubs can be found. As Asasa's experience has been, this is exactly what's happening. Do terribly, and you're matched with all of the terrible players. A student of mine got the game. I watched him play a round on Standoff. PARTY IN THE STREETS! Everyone was running around in the open without any regard for using cover or staying out of sniper sightlines. I told him just a few tips and he got a dragonfire within 5 minutes of me standing there and watching him. Surreal, to say the least.
|
|
|
Post by randomguy987 on Dec 1, 2012 15:15:53 GMT -5
Well, I decided to analyse recently played games and the players I met there. I played Kill Confirmed game mode. My K/D is pretty low (1.67 – that's too low for MW2 or MW3, but nearly OK for BOII), but I realized that the highest K/D player I've played with recently was…1.41. Yes, only 1.41. And the average K/D was even lower than 1.00 ( lots of 0.8–1.00 players and one player had 0.24…). Where's the SBMM? But it's maybe due to my residence (Eastern Europe) and comparative unpopularity of this mode in BOII. However, there're no problems with searching free lobbies. What the heck?! My KD is barely 1.0 and I am constantly placed in KC lobbies where the average KD is ~1.5 (with the top 1-2 players each lobby having >2.0 and the absolute LOWEST KD I've ever seen in KC being 0.86).
|
|
|
Post by Voice from the Basement on Dec 1, 2012 15:29:56 GMT -5
Well, I decided to analyse recently played games and the players I met there. I played Kill Confirmed game mode. My K/D is pretty low (1.67 – that's too low for MW2 or MW3, but nearly OK for BOII), but I realized that the highest K/D player I've played with recently was…1.41. Yes, only 1.41. And the average K/D was even lower than 1.00 ( lots of 0.8–1.00 players and one player had 0.24…). Where's the SBMM? But it's maybe due to my residence (Eastern Europe) and comparative unpopularity of this mode in BOII. However, there're no problems with searching free lobbies. What the heck?! My KD is barely 1.0 and I am constantly placed in KC lobbies where the average KD is ~1.5 (with the top 1-2 players each lobby having >2.0 and the absolute LOWEST KD I've ever seen in KC being 0.86). Do you live in US and play on PS3/Xbox?
|
|
GoHarvard
True Bro
It's not that serious.
Posts: 390
|
Post by GoHarvard on Dec 1, 2012 15:31:00 GMT -5
I don't know how much I believe this. Bad players have always been common and now they are rare? A good test would be on a new account with a crap KDR like 0.4 and go into TDM match making and see if the scrubs can be found. As Asasa's experience has been, this is exactly what's happening. Do terribly, and you're matched with all of the terrible players. A student of mine got the game. I watched him play a round on Standoff. PARTY IN THE STREETS! Everyone was running around in the open without any regard for using cover or staying out of sniper sightlines. I told him just a few tips and he got a dragonfire within 5 minutes of me standing there and watching him. Surreal, to say the least. Got it. If a bro already did that test then I believe it. The other things I mentioned I'm still curious about though.
|
|
|
Post by anthraxx on Dec 1, 2012 16:15:15 GMT -5
My k/d is >2.2 and I'm still playing against plenty of noobs. Once in a while I'll play against a team of >2 k/d players, but this has always been the case in CoD. I play on PS3. So I"m not really feeling the effects of SBMM.
Is it possible that your connection search settings affect this? I would assume "Best" would yield even less of similar skilled opponents because it would focus on getting a good ping lobby.
|
|
|
Post by vodkanaut on Dec 1, 2012 17:02:18 GMT -5
Disclaimer: I suck at BO2
Anyway, I registered here just to reply to this topic because I think it's interesting and I know you guys have your heads on straight when it comes to actually analyzing these games.
With all the talk of skill-based matchmaking and the anecdotes of being placed in lobbies with players in a certain K/D range, I think people have lost sight of something here. As thegentleman explained, a weighted average of all players' K/Ds should be (slightly under) 1.00. However, a 1.0 for a given player in TDM only means they are "average" if their competition is a representative random sample of the entire playerbase.
In the original post, the average player in those 10 lobbies had 1.26 kills per death. A 1.26 K/D player in TDM has, ipso facto, been consistently matched up against lesser players. What kind of "skill-based" matchmaking is that? Keep in mind that this player isn't an outlier - 1.26 is above average, but nothing spectacular. There are many players in a similar K/D range in the matchmaking pool, so in a highly populated region, a skill-based system should have no problem finding more appropriate lobbies for such a player. And yet, our theoretical 1.26 player continues to be, after hundreds of games, solidly better on average than his competition.
A TDM lobby full of all elite pubstompers wouldn't all put up great scores, and a lobby full of all creampuffs wouldn't all get stomped. Some would go positive, some would go negative, others would break even. If the game consistently put players in lobbies with those of similar skill, everyone - good and bad - would see their K/D drift toward 1.0, and the "noob lobbies" and "pro lobbies" people are reporting wouldn't even exist. So what's really happening here? I'm perfectly open to the notion that skill-based matchmaking exists in some form, but clearly nothing so simple as forming teams of equal K/D ratios.
|
|
|
Post by ovni on Dec 1, 2012 17:27:27 GMT -5
As far as I know, skillbased matchmaking was introduced to have more close games with a few points difference between the two teams. But did any of you guys and gals have a more of such games - which are fun btw - than in any other CoD? I didn't so far. Almost all my games end either a mediocre win or mediocre loss. I must admit, I miss it to have a 10+ KD game every now and then. And why did they add leagueplay when even public matches have SBMM? IMO it fails its purpose and I do not quite understand why they added it to public matches as well.
|
|
|
Post by ovni on Dec 1, 2012 17:36:22 GMT -5
I play pretty much everything except CTF and S&D. Where do you get your close games?
|
|
|
Post by randomguy987 on Dec 1, 2012 17:37:38 GMT -5
Do you live in US and play on PS3/Xbox? Yep: Central time zone & Xbox. I have no idea how I got so "lucky" with my lobbies, but I am almost always at the bottom of the lobby leaderboard (occasionally 2nd from bottom). The highest I've ever seen myself in 2+ weeks of owning the game (I'm on my 2nd prestige now) is 4th from the bottom (15th out of 18 players), and that was a Ground War lobby. Go figure.
|
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 1, 2012 18:19:24 GMT -5
What if you start out with a high KD, then it drops down a bunch after a while?
For example, my first prestige I had a 2.7, then it dropped to 1.8 about halfway through my second prestige.
Will I be matched with lower skilled players?
|
|
asasa
True Bro
fuck
Posts: 4,255
|
Post by asasa on Dec 1, 2012 19:06:38 GMT -5
Yes
|
|
|
Post by Voice from the Basement on Dec 1, 2012 19:27:07 GMT -5
Do you live in US and play on PS3/Xbox? Yep: Central time zone & Xbox. I have no idea how I got so "lucky" with my lobbies, but I am almost always at the bottom of the lobby leaderboard (occasionally 2nd from bottom). The highest I've ever seen myself in 2+ weeks of owning the game (I'm on my 2nd prestige now) is 4th from the bottom (15th out of 18 players), and that was a Ground War lobby. Go figure. Well, then it sounds funny that you, living in a good region for games, connect only to tryhаrds' good players' lobbies and me, living in godforsaken region, connect primarily (not always, of course) to nооbs' sessions. And yeah, in, let me say, 80% of games I've joined from the beginning I finish in one of the first three places (I've played 283 games so far and have already completed "Top Player" challenge requiring 250 times being in one of the first three places, but that sounds ridiculous). But that's maybe just because of playing on usual KC – I haven't visited Ground War since MW2. Remember, that Ground War mode is full of tryhаrds good players.
|
|
|
Post by corpsecreate on Dec 1, 2012 19:32:41 GMT -5
You're nuts if you think 1.67 is bad Really? I would consider it bad. I got a 2.8 KD in Blops II atm, I'm lvl 24 1st prestige.
|
|
|
Post by RageHulkSmash on Dec 1, 2012 20:05:48 GMT -5
Everyone has different perspectives on good and bad stats. Some NBA stars might think 20 points per game is huge, but Kobe would probably be disappointed lol. I think anything less than 2.5 is bad for me, considering how much I've improved over the years without boosting it from dashboarding or playing infected.
Also consider that kdr isn't the only metric used to match people. If spm is included, the game might think high kdr, low spm players are equal to low kdr, high spm players.
|
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 1, 2012 21:19:13 GMT -5
If you go by the average player, 1.67 is really high...
|
|
|
Post by corpsecreate on Dec 2, 2012 9:20:34 GMT -5
Really? I would consider it bad. I got a 2.8 KD in Blops II atm, I'm lvl 24 1st prestige. well sure but i mean its incredibly gametype dependant. Obviously it means worse in TDM than say...demolition Well I only play TDM, I cant play any other game type because there isnt enough players in my area to start a lobby. Hell it can take upwards of 20 mins to find a game in TDM even. My SPM is 340 (ranked 50 on leaderboards).
|
|
|
Post by TheHawkNY on Dec 2, 2012 11:08:11 GMT -5
Two thoughts on this:
You appear to have taken an unweighted average of the ratios, which you would expect to be above 1.00. To make a very simple example, if we 1v1 for our first match and I go 10 and 5, and you go 5 and 10, an unweighted average of our KDRs afterwards would be 1.25 (my 2 + your 0.5 divided by 2). It's only if we weight the average by adding up the kills and deaths and then divide that we get the true average of 1.0.
Second, I would expect the KDRs reflected in the stats to be above 1.00 anyway because of split screeners. The second player usually has a KDR well below 1. The kills of and deaths to that player are recorded in the stats of other players, but those stats are not recorded.
|
|