I'm working with 400+ variables (mostly 2D gel data) across 2 conditions for which I've calculated p values with a students t-test. Some of the variables are highly correlated. I'm wanting to adjust the p values to allow for multiple comparisons and I've used p.adjust in R with the "BH" method but I understand that it supposed to be for independent variables. I also tried the "BY" method as I thought being "under dependency" meant that it allowed for some variables to be non-independent (ie correlated), and so I assumed the adjusted p values would be lower than with the "BH" method, but they're not. (When I use the p.adjust BY method on my data the adjusted p values are all 1). Is there a method of adjusting p values that allows for some correlated variables? Thanks for any help.
> pvals$adjusted_BY <- p.adjust(pvals$p_value, method = "BY") > pvals$adjusted_BH <- p.adjust(pvals$p_value, method = "BH") > head(pvals) ID p_value adjusted_BY adjusted_BH 1 2366 0.001361787 1 0.5828448 2 746 0.008377390 1 0.9516818 3 1635 0.017267163 1 0.9516818 4 1577 0.025868990 1 0.9516818 5 1588 0.029862316 1 0.9516818 6 891 0.036119770 1 0.9516818
You could build a permutation test. The basic idea is find the null distribution of the minimum p-values (equivalently, the maximum statistic of the test) when you do 400 tests, one for variable. This is a multivariate permutation approach, taking into account the correlation structure of your data
First, calculate the unadjusted p-value with the observed response (you already have this)
The process of permutations is the next:
- Permute the response variable (the 2 conditions in your case)
- Do the 400 tests with the permuted response
- Save the minimun p-value of the tests from these 400 p-values
- Repeat the process 1000 or 10000 times
Finally, you calculate the adjusted p-value. For each unadjusted p-value, the adjusted p-value is the number of p-values obtained from the permutations smaller than that p-value, divided by the number of permutations.
- Solved – Many p-values are equal to 1 after Bonferroni correction; is it normal
- Solved – Why is the combined p-value, obtained using the Fisher’s method, so low
- Solved – Why are (almost) all of the corrected (Benjamini-Hochberg) p-values equal
- Solved – FDR calculation for each p value
- Solved – Multiple hypothesis testing with FDR in R – FDRtool and p.adjust