Quote:
Originally Posted by Snaggles
[You must be logged in to view images. Log in or Register.]
This is a sample of a lot of hits but there still is a chance of the relatively small sample pool providing scatter. Someone like bcbrown could probably use the right terms for what I’m trying to say.
|
I don't have a snappy name for the principle you're trying to describe, but I can probably illustrate it. Imagine you have two coins, and you're trying to figure out which one is more likely to come up heads - one or both of them might not be exactly fair. Imagine that although you do not know this, one has a 50% chance of heads and one has a 60% chance. Lets say you decide to flip them each ten times, and then say that the coin with more heads is the one that's more likely to come up heads than the other one.
If you flip a 50% coin 10 times, there's a 37.7% chance you get at least 6 heads. If you flip a 60% coin 10 times, there's a 36.7% chance it comes up no more than 5 times. Since these probabilities are independent, there's a 0.377 * 0.367 = 13.8% chance that this scenario happens.
If you instead flip each coin 100 times, there's a 2.8% chance that the 50% coin has at least 60 heads. There's a 2.7% chance that the 60% coin has no more than 50 heads. There's a .08% chance that this scenario occurs.
You can plug your own numbers in here to run any variation on these calculations:
https://www.wolframalpha.com/input?i...2C+X+%3E%3D+60
Neither of these calculations is exactly answering the question "what's the likelihood that the 50% coin has more heads than the 60% coin with n flips", but they illustrate the principle that the likelihood that you pick the wrong coin drops as you increase the number of flips.
This page discusses the exact answer to this question, but the answers are too filled with jargon to be readily understandable.
Perhaps this is a succinct description of what Snaggles was trying to say: "Are we confident that the sample size for each toon is large enough that the possibility that the one that appears to do better was just exceptionally lucky, and/or the one that appears to do worse was just exceptionally unlucky, is small enough that we can conclude that the one that appears better actually is better." Maybe that's not very succinct. It's hard to put into words.
I cannot help myself but to note that in the past on this forum I've had interactions with people who were aggressively uninterested in this question when an experiment with a small sample size resulted in an outcome that supported their argument.