Inferential Statistics

Inferential Statistics#

Exercise 1a: P-values#

The p-value indicates the likelihood of obtaining the observed results if the null hypothesis is correct. Small p-values indicate that the Null hypothesis is unlikely to have generated the data - a strong evidence against the null hypothesis.

Let’s explore how the p-value behaves if we know that the null hypothesis is true.

Generate 1000 datasets with a Normal distribution using np.random.randn, each with 10000 samples, with mean 0 and standard deviation 1. We know that the samples are normally distributed!

  1. For each of the 1000 datasets (each of them with 10000 samples), run a test for normality, scipyt.stats.normaltest, and store the obtained p-value in array. You should have a set of 1000 p-values.

  2. Plot the distribution of p-values. Does it look as you expected?

  3. What fraction of p-values fall below 0.05? In other words, what is the likelihood of the normaltest producing a wrong result (p<0.05 indicates that the data is not normally distributed)?

# your solution here

Exercise 1b: P-values#

Now do the same as above, but instead of generating from a Gaussian distribution, for which we cannot reject the null hypothesis of normality, let’s use samples from a uniform distribution.

Do the same as above with these data:

Generate 1000 datasets with a Uniform distribution using np.random.rand, each with 10000 samples. We know that these samples are not normally distributed!

  1. For each of the 1000 datasets (each of them with 10000 samples), run a test for normality, scipyt.stats.normaltest, and store the obtained p-value in array. You should have a set of 1000 p-values.

  2. Plot the distribution of p-values. Does it look as you expected?

  3. What fraction of p-values fall below 0.05? In other words, what is the likelihood of the normaltest producing a wrong result (p<0.05 indicates that the data is not normally distributed)? Run the normaltest repeatedly time for 1000 different datasets, and plot the distribution of the resulting p values.

Does the distribution look as expected?

# your solution here

Exercise 2: Statistical tests#

Download the experimental data file 5.03_inferential_stats_exercise.npy from studip.

  1. plot the distribution

  2. test for normality

  3. test whether the values differ significantly from

  • a=3.1

  • b=3.0

  • b with signficance level 0.001

  • smaller than b

  • larger than b

Inspect the p-values for the different tests and try to understand their relative values (why do the tests for difference vs. smaller vs. larger produce different p-values?).

Hint You can use the scipy.stats.ttest_1samp test, to test whether a dataset’s mean differes significantly from a specific value.

# your solution here

Exercise 3 (bonus): Multiple groups#

Download the data file 5.03_inferential_stats_exercise_multi.npy. The file contains a 2D array with 50 rows (samples) and 3 columns (several different treatments). We want to know whether the treatments have different effects.

  • Visualize the distribution of the data

  • Test the data for normality

  • Run an ANOVA to check whether there is any treatment effect (scipy.stats.f_oneway).

  • Run a post-hoc test to determine which treatments differ from each other (scipy.stats.tukey_hsd).

# your solution here
Contents