Biased Sampling is a fallacy committed when some members of a group are less likely to be included in a selection than others biasing an outcome. This fallacy takes two general forms and I’ll discuss both below.

1) True statistical bias due to poor sampling like using a phone poll to determine average pay in the 1950s. This sampling method is unreasonable because a minimum income was required to have a phone so a group from the population was ignored. Wikipedia has more examples of this.

2) Biased Sampling caused by ignoring a group that can’t contribute to a result. This is sometimes refered to as the exclusion fallacy but I haven’t been able to find what I’ll call a “canon” source to call it this.

Example- This scenario occurred when Mykie Noble and Jeremy Bush were discussing the effects of changing the legal cards in a particular format for Magic.

Jeremy Bush: My local play group has largely approved of the recent changes
Mykie Noble: I’m still trying to rebuild my community after the last set of changes.
Jeremy Bush: Well, all the people that I play with have no problem with them.

Players that disapprove of the changes would have left the format so it’s much more likely that Jeremy who deals with local players through the format would largely run into players that support the changes. People from partisan groups largely fall for this fallacy when discussing social, political and economic issues as they’re unlikely to encounter people of opposing view points making statements that “Americans support this position” easier as they’ve never encountered opposition face-to-face.

Nasim Talib discusses this fallacy in his book The Black Swan and provides a wonderful example about a ship wreck. A historian is talking with a priest about the power of prayer and is shown a stele created by Poseidon worshipers who survived a ship wreck. The historian asks for the stele made by the worshipers that died in the storm and the priest rightfully can’t produce it as those worshipers are obviously dead. Power of prayer arguments commonly depend on biased sampling as those not saved by prayer are largely dead.

Overcoming and Avoiding This Fallacy: Always try to think of exceptions and determine if these are genuine exceptions or a large class that’s being excluded. If it’s the latter, biased sampling is likely and the argument should be dropped, attacked or changed to incorporate the exception. For instance, in the Jeremy/Mykie argument, Jeremy could have avoided biased sampling by saying that the size of his play community hadn’t changed much thus showing that the numbers lost due to the changes were insignificant. In the priest/historian example, evidence that those who died were non-believers would lend evidence although this would be difficult to prove.

Power of Prayer, more information: eMJA has a nice summary of Australian research

The Skeptical Inquirer chimes in a little less politely

Comments: This is my first post on a logical fallacy and arguing. I’d like to know what everyone thinks.