Comments on: It always seems worse than you think
http://mindhacks.com/2009/09/24/it-always-seems-worse-than-you-think/
Neuroscience and psychology news and views.Thu, 17 Apr 2014 12:37:41 +0000hourly1http://wordpress.com/By: Jeff Beck
http://mindhacks.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5315
Wed, 07 Oct 2009 14:20:44 +0000http://mindhacksblog.wordpress.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5315here’s a hasty blog post explaining why this ‘bias’ is rational from a Bayesian statistical perspective: http://justaddnoise.blogspot.com/2009/10/here-is-probability-that-true.html
but the short answer is just that the median estimate is greater than the mean and maximum a posterori estimates of small quantities and so it’s actually likely that small quantities are underestimated.
]]>By: Jeff Beck
http://mindhacks.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5314
Wed, 07 Oct 2009 13:52:24 +0000http://mindhacksblog.wordpress.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5314It’s actually good statistical reasoning to assume that rare events are underestimated in their frequency. A Bayesian statistical analysis can demonstrate this. In short, if you have N observations of a Poisson distributed (rare) random variable, then the maximum likelihood estimate of the mean of the underlying process (lambda) is N.
However, one could take a proper Bayesian approach and ask, what is the probability that the true mean, lambda, is greater than this estimate? It turns out that the smaller N is the more likely it is that the true rate is greater.
To see this, place the conjugate Gamma prior on lambda so that the posterior distribution on lambda is also Gamma distributed to obtain,
p(lambda|n)
Now we can ask, what is the probability that lambda is actually greater than the mean of this distribution or even greater than the maximum aposteriori estimate of lambda? It turns out the answer is more or less independent of the parameters of the prior in this case. And its quite high. See figure here: http://4.bp.blogspot.com/_QrGZ_bdq8kE/SsyMyTgcrwI/AAAAAAAAAmw/b7dykNNKUZk/s1600-h/underestimation.jpg
This result also holds for binomial distributions with uniform prior. x=lambda in the figure. For the Poisson case, the observed rate has been normalized by 1000 to fit on the same plot.
]]>By: Rhodora Online
http://mindhacks.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5313
Sat, 26 Sep 2009 15:13:39 +0000http://mindhacksblog.wordpress.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5313To me it seems like that same bias for hypothesis-proved rather than hypothesis-disproved results. The latter ones are more often filed away and neglected and the former ones have a better chance of being sent and accepted for publication.
]]>By: raestyr
http://mindhacks.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5312
Sat, 26 Sep 2009 00:10:23 +0000http://mindhacksblog.wordpress.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5312Yet Google also says:
Results 1 – 10 of about 284 for “the true number may be higher” -”the true number may be higher or lower”. (0.19 seconds)
Ah, search engines. And english.
]]>By: Neuroskeptic
http://mindhacks.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5311
Fri, 25 Sep 2009 22:57:04 +0000http://mindhacksblog.wordpress.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5311This may just be my cynicism showing but I think it’s because generally the people responsible for estimating the prevalence of a given disease are the same people with an interest in making that prevalence seem high.
Say you’re an academic who researches GID (or anything else). If you can say “GID affects maybe 1 in 100 people” that looks a lot better when you’re asking for grant money than if you say “GID affects 1 in 100,000 people”.
Or say you’re an activist for the interests of people with GID, you will naturally want to make it seem as common as possible because this is, unfortunately, equated with “normality” in most people’s minds.
]]>By: Maia Szalavitz
http://mindhacks.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5310
Fri, 25 Sep 2009 22:06:56 +0000http://mindhacksblog.wordpress.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5310It’s simply because if it’s a bigger number, it’s a bigger story.
]]>By: WhiskerBiscuit
http://mindhacks.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5309
Fri, 25 Sep 2009 21:43:32 +0000http://mindhacksblog.wordpress.com/2009/09/24/it-always-seems-worse-than-you-think/#comment-5309> “the true number may be higher” 20,300 hits
Yes, but the true number may be higher.
]]>