General Poker Discussion Poker Forums

Hey guys,

A post someone made in a recent thread (paraphrased): "No way is an 80/20 player, even if we only have 16 hands, is going to end up being a 15/13." This was in PLO, and it got me to thinking about the validity of that statement and a lot of the assumptions we make about the utility of statistics, specifically HUD-based statistics.

In the past, I've had a feeling for when a player's stats had normalized, but I'm generally a fan of actually being able to determine how confident we can be in our assumptions. If a player's PFR is 3% over 3000 hands, we can be pretty certain he's a nit, how certain are we? What about over 1500 hands? 250 hands?

I know that HUD stats are something that many players rely on too much, but if we're looking to take in as much information as possible when making decisions, it's probably a good idea to be able to calculate how certain we are in some of that information.

tl;dr - If a player is 80/20 through 16 hands, how likely is it that he could actually be a 15/13 player? How do we calculate this?

What other variables do you need to consider, for other statistics (call vs. 3-bet, for example, or coldcall preflop %)?

If this is covered in one of our many videos, let me know, I'd love to check it out. I'm not looking for a specific formula, but more of a general framework for understanding how we can trust PFR as a stat at (for example) 500 hands, but can't trust 3-bet % until 1500 hands, etc.

thanks,
Rob

I could be wrong and it could be more advanced than this, but a long time ago I calculated these with the assumption that they're a simple binomial distribution. The other assumption is that they're independent events, which is not true (tilt, adjusting to game conditions, etc.) but maybe not far off.

Hopefully someone with a fuller understanding of statistics can chime in. Not home atm but I can show how to calculate the margin of error on Monday, barring someone ripping apart my methodology between now and then.

What other variables do you need to consider, for other statistics (call vs. 3-bet, for example, or coldcall preflop %)?

If this is covered in one of our many videos, let me know, I'd love to check it out. I'm not looking for a specific formula, but more of a general framework for understanding how we can trust PFR as a stat at (for example) 500 hands, but can't trust 3-bet % until 1500 hands, etc.

Been thinking about this also.
Like 3b sb vs bu if someone had a guy that was folding a lot to 3bet in bu and they played a 200 hands at that table. He would have a higher 3b in that spot and it would take a lot of hands before it would be back to normal again.

Been thinking about this also.
Like 3b sb vs bu if someone had a guy that was folding a lot to 3bet in bu and they played a 200 hands at that table. He would have a higher 3b in that spot and it would take a lot of hands before it would be back to normal again.

Great point. Specifically, on my pop up I have "3-bet vs. open vs..." and this type of thing could throw that completely out of whack.

all i can say is that im a 15/12 guy and i have played many times 4/1 over 100+ hands and also 30/25 over 50 hands but no more

it happens a lot though

If a player is 80/20 through 16 hands, how likely is it that he could actually be a 15/13 player? How do we calculate this?

I think to get this accurately is pretty difficult, simplifying this a lot into If a player is 80 vpip through 16 hands, how likely is it that he could actually be a 15vpip player?
This is a lot easier and may lead the way to a better or more complete solution. This, as sthief09 says, is likely to be a simple binomial distribution problem. We know that the player opens 15% so for each trial (ie, a hand) the chance of opening is 0.15, so the chance of scoring 13 (12.8 is 80%) out of 16 is given by : nCk p^n(1-p)n-k
This is really unlikely to happen, the probability is 6.69316*10^-9, or about 1 in 150 mllion.
(This assumes the player is robotically opening 15%, ie, never tilts into being a maniac, or varies in any real way - not like real life)

We can do the same for pfr, scoring 3 out of 16 when we have a probability of 13% for each trial, this is quite likely and is a 20% chance.

edit: this 20% figure is for scoring exactly 3, the result for 3 or more is about 35%, the value for the vpip above is so small it doesn't much matter is you ask for 13 or greater as it's still 1 in 100 mill or so.

In this case I think using the binomial model is fine but this is simplified, we aren't using the extra information of the pfr for the vpip case. The pfr has a fairly complicated relationship to vpip but will have some bearing on the chance of seeing 80% for vpip, and also the vpip will have an influence on the pfr result. I am not sure how to tie these things together in a good manner, I think this is far too complicated to get an algorithmic answer.

I'm not looking for a specific formula, but more of a general framework for understanding how we can trust PFR as a stat at (for example) 500 hands, but can't trust 3-bet % until 1500 hands, etc.

The above answer is really not much use as in that case we somehow knew the player was actually 15/13.
A simple way of getting a feel for how much we can trust hud stats of low samples is to use the "Margin Of Error" approach, also suggested by stief09.
I wrote a script to calculate these and for a vpip of 40% you get:
No. Hands in sample: 20, openedHands: 8, sampledVpip: 40.0%, Approx90%CI: +/-18.544%
No. Hands in sample: 40, openedHands: 16, sampledVpip: 40.0%, Approx90%CI: +/-12.944%
No. Hands in sample: 60, openedHands: 24, sampledVpip: 40.0%, Approx90%CI: +/-10.524%
No. Hands in sample: 80, openedHands: 32, sampledVpip: 40.0%, Approx90%CI: +/-9.094%
No. Hands in sample: 100, openedHands: 40, sampledVpip: 40.0%, Approx90%CI: +/-8.124%
No. Hands in sample: 120, openedHands: 48, sampledVpip: 40.0%, Approx90%CI: +/-7.410%
No. Hands in sample: 140, openedHands: 56, sampledVpip: 40.0%, Approx90%CI: +/-6.856%
No. Hands in sample: 160, openedHands: 64, sampledVpip: 40.0%, Approx90%CI: +/-6.410%
No. Hands in sample: 180, openedHands: 72, sampledVpip: 40.0%, Approx90%CI: +/-6.042%
No. Hands in sample: 200, openedHands: 80, sampledVpip: 40.0%, Approx90%CI: +/-5.730%

Each time you get 4x the number of possible events you half the width of the confidence interval.
(Also this approach breaks down if the sample size is too small as you can get a CI that would push the stat into the -ve)

You always get the chance to pfr each hand but at 6 max you are only likely to be able to 3-bet on 20% of hands so this would imply that in a 1500 sample of hands the 3bet CI accuracy is equivalent to having 1500/5 = 300 hands of pfr or vpip.

To get better accuracy I think you can use some Bayesian type analysis on these stats. We know the players stat will come from a distribution (probably Normal) for the player popualtion and we can use this to influence our CI. I think this may be to do with joint distributions or covariance stuff but my knowledge fails around here.

I had also best add my usual disclaimer that I might have calculated wrong and I am just reading up on stats, I am not an expert.

I'll take 1 in 150 million as "never" (I'm the one who made the statement in Rob's OP).

FWIW I think my particular classification - players being loose and passive based on a small sample of hands - is about the only sub-50 hand metric you can use with a HUD, and it's the combination that's important. Someone who's 60/50 over 10 hands could just have seen a rush of cards. Someone who's normally 25/20, however, just isn't calling 11 hands and raising 2 in a 16 hand stretch. So 80/15 gives you enough information, while 60/50 doesn't. I'm sure there's relevant thresholds for lots other stats, but I haven't given them much thought.

I'll also say that 75% of the posts I see in the PLO forum have people making decisions based on incredibly small sample sizes - as a whole we need to do a better job of discounting that sort of stuff.

If this is covered in one of our many videos, let me know, I'd love to check it out. I'm not looking for a specific formula, but more of a general framework for understanding how we can trust PFR as a stat at (for example) 500 hands, but can't trust 3-bet % until 1500 hands, etc.

I think about it in terms of opportunity. If you start with x# of total hands, everything beyond vpip (and even that, somewhat), is a subset of total hands. In other words, for every other stat in your HUD, _something else_ has to happen for it to be possible. 3-bet? Someone needs to raise in front of you. Fold to turn c-bet? You have to call pf and on the flop. So as you add these extra layers, there's just fewer opportunities for that specific set of events to occur. If you've played 250 hands, it's entirely possible you've only had 50 chances to 3-bet and 3 chances to fold to a turn c-bet. I'd say that if you have 500 3-bet opportunities for someone - you can see in a HUD pop-up what the sample is for any stat - you likely have enough information. But that may take you 2500 hands to get.

eta: you may be looking for something way more complicated than that

One option is to use bayesian inference.

You might also want to read the post:

http://www.deucescracked.com/forums/34-Small-Stakes-Shorthanded-NL/topics/537241-Creative-Grinders-Episode?#post_4729441

I'll also say that 75% of the posts I see in the PLO forum have people making decisions based on incredibly small sample sizes - as a whole we need to do a better job of discounting that sort of stuff.

I would agree with this, it is fine to use any sample size if you do know how to interpret it but many people do under-estimate the range these stats have in small sample sizes. In fact with hud stats many people don't even know how to interpret them with large samples. How many know if at 6max say and at their buy-in level a 3-bet of 8% is medium, high or very high? And this is for a quite commonly used one.
My advice on hud stats would be to start off with just the main stats vpip/pfr and get used to what these mean then add an extra one doing a bit of work to get used to this, and keep slowly adding more (or removing any not found useful) until you are happy with them.
I play mostly tournaments and in these there isn't much point in trying to go too far with them as the sample sizes are typically very small and also the context these samples have come from varies massively. Sometimes you are playing with an average of 100+bb sometimes just 5 or 6bbs. Similarly the context changes for table size and bubble effects. It is possible to filter but it all just makes it harder to get much from them.

At the moment my best guess for analytically viewing these stats is to use something like a
Creditable Interval or some of the techniques/maths from machine learning, eg, classifiers. I guess eventually these things will get built into trackers - I am not sure if they are already available as extra apps to Hem or PT. Something to look forward too , on the bright side lots of people still won't know how to interpret even these well.

One thing I've always wanted to do, was create a custom stat with PT for the credible intervals of certain stats, that'd be sweet. If anyone knows the jist of the math for let's say, the credible interval of VP$IP, post it, and I'll show it to the PT forums and see if it can be done.

You mean like you have x hands on the villain and his VPIP is showing at 25, your stat would show something like
"VPIP [20...30] with a 95% confidence"?

I think it could be cooked up. Not sure if you'd really want that showing up in your hud though?

Anyone with a statistics class from this millennium please verify, but I think the standard deviation can be calculated in PT4 like
sqrt(#VPIP#*(100-#VPIP#) / (10000)*#Hands#)
If that's the case, then it's trivial to put this in HUD.

Ok, here we go (and PLEASE keep in mind my statistics studies aren't too recent):
create a variable called 'std_vpip' in PT4 with the expression
100 * sqrt(((cnt_vpip / (cnt_hands - cnt_walks))) * (1 - ((cnt_vpip / (cnt_hands - cnt_walks)))) / cnt_hands)

create a stat called 'VPIP conf' (or what ever you like) with the value expression of
(cnt_vpip / (cnt_hands - cnt_walks)) * 100
and formatting expression of
format("VPIP between {1} and {2} with a 95% confidence" , format_number((cnt_vpip / (cnt_hands - cnt_walks)) * 100 - 2 * std_vpip,0,false,false),format_number((cnt_vpip / (cnt_hands - cnt_walks)) * 100 + 2 * std_vpip,0,false,false) )

(You can modify the format expression to your liking, of course... it's the {1} and {2} where the values show up.)

Put your newly created stat in the hud or a report.

This would seem to create somewhat reasonable numbers imo, but when someone has a vpip close to 100, multiplying the std by 2 and adding to his vpip seems to produce numbers above 100 (likely the same on the lower end of the scale). Not sure if this implies a problem with the stat or should I just format so that nothing outside of [0...100] gets displayed.

You mean like you have x hands on the villain and his VPIP is showing at 25, your stat would show something like
"VPIP [20...30] with a 95% confidence"?

I think it could be cooked up. Not sure if you'd really want that showing up in your hud though?

Anyone with a statistics class from this millennium please verify, but I think the standard deviation can be calculated in PT4 like
sqrt(#VPIP#*(100-#VPIP#) / (10000)*#Hands#)
If that's the case, then it's trivial to put this in HUD.

In the above we have been analysing stats like vpip and pfr as a binomial distribution (**) and if so the std dev is:
stdev = sqrt( np(1-p)) where in this case n number of hands, p = prob of a single true event.

(**) I think this seems a pretty reasonable model to me

No. Hands in sample: 200, openedHands: 80, sampledVpip: 40.0%, Approx90%CI: +/-5.730%

In plain English, what does this translate to? We are 90% confident that our opponent has a VPIP of somewhere between 34.3 and 45.7?

One thing I've always wanted to do, was create a custom stat with PT for the credible intervals of certain stats, that'd be sweet. If anyone knows the jist of the math for let's say, the credible interval of VP$IP, post it, and I'll show it to the PT forums and see if it can be done.

Yeah this has always seemed like a good idea to me. The gist of what I was getting at in this post (which others seem to be thinking about as well) is that we have these "guidelines" we all have mentally set for when we can trust a stat - opponent is 100/100 through 40 hands, ok, we know he's a maniac. 40/20 through 500 hands, we're pretty sure he's a LAG. but 40/20 through 40 hands, we "don't have enough info... he could be a TAG or a LAG."

'll also say that 75% of the posts I see in the PLO forum have people making decisions based on incredibly small sample sizes - as a whole we need to do a better job of discounting that sort of stuff.

Yep. My overall goal with this topic is to come up with a way of weighting specific commonly-accessed statistics using something other than gut intuition. Most of the information I pass on to students currently is based on experience - "you've only played 40 hands with him, forget his fold BB to steal % until you've played a lot more." I'm looking to solidify that experience a bit, so if someone asks me "when can I start to trust someone's fold BB%" I can give them an better understanding.

Rob