A technique I have seen used for statistical confidence testing of non-Gaussian distributions is to generate 1000 random examples of the distribution. If the you want to be 95% confident that answer to be checked comes from the same distribution then it should be "like" 950 of the 1000 examples.
Eg if the distribution is reasonably well behaved then if the answer to be checked lies outside the range of the 25th to 975th example we can say we confidently reject the null hypothesis and say our answer is not from the distribution used to generate the 1000 examples. We do not need Z-scores, t-tests etc.
This non-parametric test should be ok with any distribution. We are effectively burning CPU cyles rather than spending brain cycles on devising and validating a statistical technique specifically for our new distribution.
GECCO 2011: Call for Papers
4 years ago