Links : Home Index (Subjects) Contact StatsToDo

Explanations and References This page provides 3 statistical tests for paired differences in scalar measurements. These are the Paired t Test, the Wilcoson, Paired Signed Rank Test, and the Permutation Test for paired differences. Paired t Test Twin 1 Twin 2 Difference 3163 3124 39 3245 2807 438 3391 3014 376 2547 2727 -180 3042 3254 -211 3200 3826 -626 3115 2596 519 3294 2952 343 3019 3279 -260 3222 3325 -103 2831 2984 -153 3043 2765 277 2646 3109 -463 3327 2757 570 3182 3781 -599 2984 3061 -77 2878 3658 -780 2770 3665 -895 3092 2739 353 2735 3324 -589 The following example is used to demonstrate the Paired t Test for parametric data. Please note: the data is computer generated and does not represent any research information. We purports to conduct a study to test whether the order of birth resulted in differences in the birth weight of twins. We obtained the birth weights (grams) from 20 sets of twins, which are shown in the first two columns in the table to the right, and the differences (Twin1 - Twin 2), as shown in the third columns. The calculations are as follows Number of Paires = 20 Mean Difference = -101 Standard Deviation of Difference = 456 Standard Error of Mean = 105 t Test : t = -0.9659 df = 19 p(α) = 0.17 (1 tail) and 0.35 (2 tail) 95% Confidence Interval of Paired Difference (1 tail) = >-282 or <80 95% Confidence Interval of Paired Difference (2 tail) = -320 to 118 Our conclusion is therefore that birth weights between first and second born twins are not significantly different Wilcoxon Paired Signed Rank Test Before After Diff 3 1 2 2 1 1 2 0 2 3 0 3 1 1 0 1 0 1 2 2 0 1 2 -1 1 1 0 3 1 2 1 2 -1 1 1 0 1 1 0 3 0 3 3 1 2 1 2 -1 1 1 0 1 3 -2 1 1 0 3 1 2 3 2 1 1 1 0 The following example demonstrates the Wilcoson Paired Signed Rank Test for non-parametric data. Please note: the data is computer generated and does not represent any research information. We purports to study whether a new analgesic is effective in relieving headaches. We ask the subjects to describe their headache as none (0), some (1), moderate (2), and severe (3), a 4 point scale (0 to 3), before and after administering the analgesics, and use the paired differences to evaluate the analgesics. We collected data from 22 subjects. The headache scales are as shown in the first 2 columns, and the paired difference (Before - After) in column 3 of the table to the left. Paired diff - + 1 3 3 2 1 5 3 0 2 The differences are ranked and counted as in the table to the right. The - ranks are in column 1 and the positive ranks in column 2 There were 8 subjects whose headache scores did not change (0), and these are not included in the table of counts. On the negative side, there were:   - 3 subjects that got worse (-1)   - 1 subjects that got a lot worse (-2) On the positive side, there were   - 3 subjects that got better (1)   - 5 subjects that got a lot better (2)   - 2 subject that got completely better(3) The sums of negative ranks (T-) = 20, and sums of positive ranks (T+) = 85. As the numbers included in the count is 14, less than 16, Segal's Table H is used to determine statistical significance, and this is p<0.05. We can therefore conclude that headaches decreased significantly after receiving the analgesic. Permutation Test for paired differences The general principle underlying the permutation of paired differences is that, in a randomly allocated study, the data obtained could have been in either of the paired measurements. In ther words, each paired difference can either be + or - of that difference. The test therefore consists of calculating every possible permutation of the paired differences, and examine the results. If the results from the original data is near the extremes (e.g. less than 5 percentile or more than 95 percentile in a one tail model), then a decision can be made that it is unlikely to be null and therefore statistically significant. The advantages of using the Permutation test are : Exhaustive permutation allows the calculation of the precise probability that the data presented is null, so the tests calculate the Type I Error (α), with the power (1-β) of 100%. The test is not dependent on any assumption of data distribution, so they can be used in any regular interval data (where 10-9 is the same as 4-3). The test can therefore be used on parametric measurements, ratios, variances, and time. Because of the above two characteristics, the test can be used with a very small sample size The disadvantages of using the test are related to the computation resources (time and memory) required. The number of permutation is 2n, were n is the number of pairs. Computation time therefore increases exponentially with increasing sample size, and large dataset may either crash the program when available RAM is exhausted, or the computation becomes unacceptably too long. In theory, the Permutation Test can cope with any number of pairs. However, α<0.05 cannot be computed with less than 6 pairs unless the differences are uniformly in one direction, and computation will take an unacceptably long time with 22 pairs or more. The Permutation Test is therefore ideal for handling small sets of interval data with uncertain distributions. With larger sample size, Wilcoxon PSRT for non-parametric data or Paired t test for parametric data should be preferred. The mathematical argument of the Permutation Test is as follows In a pair of measurements, the null hypothesis is that there is no difference between the pair. In other words, that the values observed can be in either of the pair. The Permutation Test therefore consists of examining the sum of paired differences, in all permutations where the values in each pair are in either groups. The total permutation is therefore 2n, were n is the number of pairs. The sum of differences in the original data is then compared with all possible outcomes, so that its position (thus probability) can be estimated. The following example uses the same data as that presented in the Paired t Test. There are 22 pairs of birth weights, so the number of possible permutations are 222 = 1048576 The sum of paired differences in the input data = -2021 There are 173010 permutated sums that are less than that from the input data (<-2021) There are 262 permutated sums that are the same as that from the input data (=-2021) There are 875304 permutated sums that are more than that from the input data (>-2021) Therefore, there are 262 + 173010 = 173272 permutated sums that are equal or less than from the input data (<=-2021) Therefore, the probability of having a permuted sum equal or less than that of the input data is 173272 / 1048576 = 0.1652 Putting this another way The input data is at the 16.5th percentile of all possible permutationed values Statistical significant (α) = 0.1652 (1 tail) and 2 * 0.1652 = 0.3305 (2 tail) We can conclude that paired differences is not significant differenct to null (0) Comparison of the 3 methods Using the same twin data by the 3 tests provides the following results Paired t Test: α = 0.1731 (1 tail) Wilcoxon PSRT: α = 0.1753 Permutation Test: α = 0.1652 (1 tail) Although the differences between the α values in this example are trivial, they do show the relatively greater power of the Permutation Test (lower value of α) and less power of the Wicoxon Test (higher value of α), compared with the Paired t Test References Paired t Test : Armitage P. Statistical Methods in Medical Research (1971). Blackwell Scientific Publications. Oxford. P.189-207 Wilcoxon Paired Signed Rank Test : Siegel S and Castellan Jr. NJ (2000) Nonparametric Statistics for the Behavioral Sciences. Second Edition. McGraw Hill, Sydney. ISBN0-07-100326-6 p. 95. Table H Critical values p.332-334 Permutation Test : Siegel S and Castellan Jr. NJ (2000) Nonparametric Statistics for the Behavioral Sciences. Second Edition. McGraw Hill, Sydney. ISBN0-07-100326-6 p. 95-101. Javascript Program The data is for a single study, a 2 column table of numbers.   - Each row represents data from a pair   - The two columns are paired values from that pair Paired t Test (parametric data)    Wilcoxon's Paired Signed Rank Test (non-parametric data)    Permutation Test for Paired Differences (small sample size)    R Codes Program 1: Paired t test # Paired t test # data entry dat = (" V1 V2 3163 3124 3245 2807 3391 3014 2547 2727 3042 3254 3200 3826 3115 2596 3294 2952 3019 3279 3222 3325 2831 2984 3043 2765 2646 3109 3327 2757 3182 3781 2984 3061 2878 3658 2770 3665 3092 2739 2735 3324 ") df <- read.table(textConnection(dat),header=TRUE) # conversion to data frame df$Diff <- df$V1 - df$V2 #df # optional display of input data and paired differences # Clculation: paired t n = nrow(df) # sample size degFm = n - 1 # degrees of freedom mean = mean(df$Diff) # mean of paired difference sd = sd(df$Diff) # Standard Deviation of paired differences se = sd / sqrt(degFm) # Standard Error of mean t = mean / se # paired t p = 1 - pt(abs(t),degFm) # probability of Type I Error α # 95% confidence interval t = qt(1 - 0.05, df=degFm) # t for 95% CI 1 tail ll1 = mean - t * se # lower limit 1 tail ul1 = mean + t * se # upper limit 1 tail t = qt(1 - 0.025, df=degFm) # t for 95% CI 2 tail ll2 = mean - t * se # lower limit 2 tail ul2 = mean + t * se # upper limit 2 tail # result output c(n, mean, sd) # sample size, mean difference and SD difference c(degFm, se, t, p, p*2) # deg freedom, Standard Error t, p(α) 1 and 2 tail c(ll1, ul1) # 95% CI 1 tail c(ll2, ul2) # 95% CI 2 tail The results are as follows > # result output > c(n, mean, sd) # sample size, mean difference and SD difference [1] 20.0000 -101.0500 455.9936 > c(degFm, se, t, p, p*2) # deg freedom, Standard Error t, alpha (1 and 2 tail) [1] 19.0000000 104.6121044 2.0930241 0.1731023 0.3462046 > c(ll1, ul1) # 95% CI 1 tail [1] -281.93822 79.83822 > c(ll2, ul2) # 95% CI 2 tail [1] -320.0057 117.9057 Program 2 Wilcoxon Paired Signed Rank Test # Data input dat = (" V1 V2 3 1 2 1 2 0 3 0 1 1 1 0 2 2 1 2 1 1 3 1 1 2 1 1 1 1 3 0 3 1 1 2 1 1 1 3 1 1 3 1 3 2 1 1 ") df <- read.table(textConnection(dat),header=TRUE) # conversion to data frame df$Diff <- df$V1 - df$V2 # paired difference df # optional display of input data and paired difference # Make Wilcoxon table with non-equal cases x <- vector() # rank of non-zero differences in abs values y <- vector() # neg or pos ranks for(i in 1:nrow(df)) { v = df$Diff[i] # v = diff if(v !=0) # non-zero v { x <- append(x, abs(v)) # abs diff # abs(diff) y <- append(y, v / abs(v)) # sign + or - 1 # -1 or +1 } } n = length(x) # number of non-zero differences x <- rank(x) # abs diff ranked #x mx <- xtabs(~ x + y) # matrix +/- ranks # calculate Wicoson W ranks <- rownames(mx) # ranks as value array #ranks # optional display during debug tp = 0; # T+ sum of positive ranks tm = 0; # T- sum of negative ranks for(i in 1 : nrow(mx)) { r = as.numeric(ranks[i]) # rank of ith row tm = tm + mx[i,1] * r # T+ sum of positive ranks tp = tp + mx[i,2] * r # T- sum of negative ranks } t = tp if(tm>t) t = tm # use the larger of the 2 sums for calculation # significance testing if(n>15) # large sample size use z and its probability { z = (t - n * (n + 1) / 4) / sqrt(n * (n + 1) * (2 * n + 1) / 24) p = 1-pnorm(abs(z)) res = sprintf("z=%.4f p=%.4f", z, p) } else # small sample size use coefficients { Sig <- c(c(0,0,0),c(0,0,0),c(0,0,0),c(0,0,0),c(0,0,0),c(15,0,0), # 1-6 c(19,0,0),c(24,28,0),c(31,45,0),c(37,42,0),c(45,50,55), # 7-11 c(53,59,65),c(61,69,76),c(70,79,87),c(80,90,99),c(90,100,111)) # Table H. Siegal p.332-334 mxSig <- matrix(data = Sig, nrow=3,ncol=16) # converted to matrix for reference nn = n + 1 if(n<5) { res = "p = n.s." } else if(mxSig[3,nn]>0 && t>=mxSig[3,nn]) { res = "p = <0.001"; } else if(mxSig[2,nn]>0 && t>=mxSig[2,nn]) { res = "p = <0.01"; } else if(mxSig[1,nn]>0 && t>=mxSig[1,nn]) { res = "p = <0.05"; } else { res = "p = n.s."; } } # result output mx # matrix of +/- ranks c(nrow(df), n, tm, tp) # sample size, pairs different, Sum Rank + and - res # statistical significance The results are as follows Table of neg and pos ranks. x = absolute rank, y= - or + initial sample size (nrow), number of non-zero differences (n), sums of - (tm) and + (yp) ranks Statistical significance (p, α) > # result output > mx # matrix of +/- ranks y x -1 1 3.5 3 3 9.5 1 5 13.5 0 2 > c(nrow(df), n, tm, tp) # sample size, pairs different, Sum Rank + and - [1] 22 14 20 85 > res # statistical significance [1] "p = <0.05" > Program 3: Permutation of paired differences Section 1: declare global variabes # Global variables rows <- 0 # number of rows (pairs) sumDiff <- 0 # sum of paired difference from input data arV <- vector() # array of abs(paired diff) from all pairs nLess <- 0 # number of pairs with values less than sumDiff nSame <- 0 # number of pairs with values same as sumDiff nMore <- 0 # number of pairs with values more than sumDiff Section 2: subroutine for recursive calculation of paired differences All variables except col, sum, sumPlus and sumMinus are globally assigned and read Recurse <- function(col,sum) # col = pair number, sum = sum of paired diff { sumPlus = sum + arV[col] # sum with +ve [aired diff sumMinus = sum - arV[col] # sum with -ve [aired diff if(col==(rows)) # reaching last row { if(abs(sumPlus-sumDiff)<1e-10) # sum == sumDiff (allowing minor differences by binary math) { assign("nSame", nSame+1, envir = .GlobalEnv) # add to nSame } else { if(sumPlus<sumDiff) { assign("nLess", nLess+1, envir = .GlobalEnv) # add to nLess } else { assign("nMore", nMore+1, envir = .GlobalEnv) # add to nMore } } if(abs(sumMinus-sumDiff)<1e-10) { assign("nSame", nSame+1, envir = .GlobalEnv) # add to nSame } else { if(sumMinus<sumDiff) { assign("nLess", nLess+1, envir = .GlobalEnv) # add to nLess } else { assign("nMore", nMore+1, envir = .GlobalEnv) } } return() } arCount <- Recurse(col+1,sumPlus) # recursive call if not end row arCount <- Recurse(col+1,sumMinus) # recursive call if not end row return () } Main program # input data dat = (" V1 V2 3163 3124 3245 2807 3391 3014 2547 2727 3042 3254 3200 3826 3115 2596 3294 2952 3019 3279 3222 3325 2831 2984 3043 2765 2646 3109 3327 2757 3182 3781 2984 3061 2878 3658 2770 3665 3092 2739 2735 3324 ") df <- read.table(textConnection(dat),header=TRUE) # conversion to data frame rows <- nrow(df) # number of pairs df$Diff <- df$V1 - df$V2 # paired differences #df # optional check t debug sumDiff <- sum(df$Diff) # sumDiff, sum of paired diff from input data arV <- abs(df$Diff) # vector of abs(paired differences for use in permutation col = 1 # start permutation with first row sum = 0 # start permutation with sum of diff = 0 arCount <- Recurse(col, sum) # permutate calculations subroutine # Assemble results nTot = nLess + nSame + nMore # total number of possible permutations pLess = nLess / nTot # proportion with smaller effect size pSame = nSame / nTot # proportion with the same effect size pMore = nMore / nTot # proportion with larger effect size # probability of obtaining reference effect size or more extreme p = min(pLess,pMore) + pSame # significance (probability, α) # result output c(rows, sumDiff) # number of pairs and sum of paired diff from input data c(nTot, nLess, nSame, nMore) # result numbers c(pLess, pSame, pMore) # result proportions p # probability of sum(x) or more extreme (p, α) The results are as follows > # result output > c(rows, sumDiff) # number of pairs and sum of paired diff from input data [1] 20 -2021 > c(nTot, nLess, nSame, nMore) # result numbers [1] 1048576 173010 262 875304 > c(pLess, pSame, pMore) # result proportions [1] 0.1649951935 0.0002498627 0.8347549438 > p # probability of sum(x) or more extreme (p, α) [1] 0.1652451