Friday, September 11, 2009

Conficence Interval vs. Credible Interval

I recently participated in a project to compare two different ways to do the meta analysis: the traditional way to pool the database directly (sort of the integrated analysis) and the Bayesian approach (prior distribution + likelihood function -> posterior distribution). When we try to compare the results from two different approaches, we run into the issue of comparing 'confidence interval' and 'credible interval'. While these two terms have some similarities, the interpretations are quite different.

The "confidence interval" is a term used by frequentist - I am a frequentist. If we say an estimate has its 90% confidence interval of 35-45, it means that with a large number of repeated samples, 90% of times, the true value of the parameter will fall within the range of 35-45.

The term 'credible interval' is used by Bayesian statisticians and it may also be called 'Bayesian Posterior Interval'. In Bayesian statistics, a credible interval is a posterior probability interval, used for purposes similar to those of confidence intervals in frequentist statistics. Bayesian inference is statistical inference in which probabilities are interpreted not as frequencies or proportions or the like, but rather as degrees of belief. ...

The posterior probability can be calculated by Bayes theorem from the prior probability and the likelihood function. ... In statistics, a confidence interval (CI) is an interval between two numbers, where there is a certain specified level of confidence that a population parameter lies. ... Statistical regularity has motivated the development of the relative frequency concept of probability. ...

For example, a statement such as "following the experiment, a 95% credible interval for the parameter t is 35-45" means that the posterior probability that t lies in the interval from 35 to 45 is 0.9.

A Bayesian credible interval incorporates information from the prior distribution into the estimate, while confidence intervals are based solely on the data.

Like 'confidence interval' vs 'credible interval, there is also 'confidence region' vs 'credible region'.

Here are some links for further reading:

at

5 comments:

  1. Anonymous10:55 AM

    I just stumbled on your blog - it's very nice! However, I had to comment on this:

    "The 'confidence interval' is a term used by frequentist - I am a frequentist. If we say an estimate has its 90% confidence interval of 35-45, it means that with a large number of repeated samples, 90% of times, the true value of the parameter will fall within the range of 35-45."

    This is not the correct interpretation of a confidence interval. A 90% confidence interval of 35 - 45 doesn't mean that 90% of the time the true parameter will be between 35 and 45. If you think about it, that doesn't even make any sense from a frequentist perspective. The true value is fixed, it's the confidence interval that has sampling variability.

    A 90% confidence interval is an interval constructed in such a way that 90% of such intervals will contain the true value.

    ReplyDelete
  2. Web blog from Dr. Deng11:31 AM

    I agree that the true value is fixed, but unknown. What I mean is that if you sample many times, for 90% of times, the point estimate from these samples will fall under the range of 35 - 45.

    ReplyDelete
  3. CAB5:42 PM

    It is still incorrect to say that 90% of the time the true value will fall between 35 and 40. A frequentist confidence interval is a random variable.

    If you sample the population 1000 times you will get 1000 sample confidence interval derived from the sample means. 900 of these sample confidence intervals will contain the true value of the mean.

    The diagram in the following illustrates the concept.
    http://www.statisticalengineering.com/frequentists_and_bayesians.htm

    Great blog btw. Helps a lot in my work.

    ReplyDelete
  4. Anonymous3:52 PM

    how about saying we are 90% confident that the true value lies within the interval of 35-45 - this is a consequence of the fact that 9 out of 10 times our estimated interval constructed this way should include the true value?? sounds a lot like the bayesian interpretation!

    ReplyDelete
  5. McCult12:24 AM

    To quote: "What I mean is that if you sample many times, for 90% of times, the point estimate from these samples will fall under the range of 35 - 45."

    That is incorrect. You have one sample from which you calculated a confidence interval. If your sample happens to be an outlier and your interval does not include the true value then the point estimate of other samples will lie between 35-45 less than 50% of the time (not 90% of the time).

    Essentially, you are trying interpret a confidence interval as if it was a Bayesian credible interval. The confidence interval cannot be interpreted that way. The 90% confidence interval can be thought of as a range of hypotheses for each of which the conditional probability of the data is ≥90%. It is not valid to assume the inverse probability is also 90%. This is the same mistake as thinking the p-value gives the probability of the null hypothesis being true.

    ReplyDelete