 | Content Disclaimer Copyright @2020. All Rights Reserved. |
StatsToDo: Discriminant Analysis
Links : Home Index (Subjects) Contact StatsToDo |
|
Explanations
This page provides explanations, clarifications, and supports to Linear Discriminant Analysis, as shown in the Javascript program, and the R codes
Javascript Program
Discriminant Analysis is clearly and succinctly described in Wikipedia, and only a brief description will be provided here, to help the user to understand the data input required and interpreting the results. Different approaches to calculationsDiscriminant Analysis was introduced by Fisher more than a century ago, as a variant of the multiple regression model, but with multinomial groups as dependent variable and normally distributed measurements as independent variable. It enjoys continued usage, but with time many modifications and additions are made, producing options for users to produce results that are conceptually similar but numerically different. This confusion is further aggravated by some statistical packages keeping to the original algorithm, some including useful additions and and modifications, and some providing menus from which users can choose different options. The following comments attempts to clarify some of these issues
ReferencesGenral references on Discriminant analysisWikipedia on Discriminant Analysis.George D and Mallery P (1999) SPSS for Windows Step by Step. A Simple Guide and Reference. Allyn and Bacon, Sydney. ISBN 0-205-28395-0 Chapter 26. The Discriminant Procedure p.313-328. Resources used to developed the web bases Javascript programThese are very old books, that still present actual formulae and algortithms for all step in the calculations. Most newer references do not provide detailed algorithms, but advise users to access available packages such as SAS, SPSS, R, and PythonOverall JE and Klett CJ (1972) Applied Multivariate Analysis. McGraw Hill Series in Psychology. McGraw Hill Book Company New York. Library of Congress No. 73-14716407-047935-6
Press WH, Flannery VP, Teukolsky SA, Vetterling WT (1989). Numerical Recipes in Pascal. Cambridge University Press IBSN 0-521-37516-9 p.395-396 and p.402-404. Jacobi method for finding Eigen values and Eigen vectors Norusis MJ (1979) SPSS Statistical Algorithms Release 8. SPSS Inc Chicago Chapterr 23 : Discriminant p. 69-83. Formulae for algorithm provided by SPSS. Useful references and advice on Discriminant analysis using Rhttps://www.geeksforgeeks.org/linear-discriminant-analysis-in-r-programming/https://www.statmethods.net/advstats/discriminant.html Venables, W. N. and Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth edition. Springer.
Help & Hints
R Code
This panel provides supports for data entry and interpretation of results from the Javascript program.
Calculations
There are two programs.
Default Example
Data EntryThe data are entered as a table
Result OutputGroups are organized in alphabetical order, and in this example DR=1, DW=2, SR=3, SW=4Predictor variables are in the same order as that of the columns in data entry Discriminate functions are ordered in order of magnitude (statistical significance). The most significant being f1, then f2, f3, etc in decreasing significance. The mark # is used to designate functions that are not statistically significant and can be ignored in subsequent calculations Step 1: Data DefinitionThe entered data are summarised. In this example, number of cases (rows) = 16, number of variables = 4 and number of groups = 4.Table 1a. shows group names in alphabetical order and number of cases in each group Group 1b shows tha means and Standard Deviations of the variables in the same order as the columns Step 2: Create Discriminant functions, coefficients and parametersThe values of the predictor variables are converted to z values, where z = (value-mean) / SD, and used in all subsequent calculations. In other words, all values are transformed to have a mean of 0 and SD of 1An analysis of variance and covariance is carried out to obtained the within group covariance matrix, which is used to produce the Eigen values and vectors in orders of magnitude.
Using the eigen values and chi square, the statistical significance of the functions, in decreasing order of chi square and significance is calculated, as shown in the table to the left. Given that there are 4 predictor variables nv=4), the maximum number of functions is nv-1 = 3. However, only the first 2 functions are statistically significant (p<0.05)
The Eigen vectors are then used to establish the function coefficients, which are shown in the table to the right. Each function value is the sum of the z value of the variable and the coefficient of the function/variable. Please note that all functional coefficients are presented, the third function is not statistically significant in this example and can be ignored as trivial
Using the function coefficients and the mean z values of each variable in each group, the centroid value of each function (equivlent to the mean) in each group is estimated, as shown in the table to the left. Again values for all 3 functions from this example are calculated, but the third one, being not statistically significant, can be ignored as trivial. At this point, the initial analysis is completed. The group names, mean and Standard Deviation for rach predictor variables, the function coefficients, and the centroid values, collectively represents the parameters and coefficients of the Discriminant model, and can be used to calculate discriminant functions and allocate to groups any similar data. Step 3. Validating AnalysisThe parameters and coefficients developed are now used to evaluate the modelling data, and check whether the Discriminant calculation predicts the same group as that designated in the modelling data.
Each row of the modelling data is analsed in turn. The last column is the designated group, and other columns are the predictor variables. All functions (3 in this example) are used in calculations, although the third function is non-significant and, if ignored, will only make trivial differences to the results
The results show that the Discriminate functions correctly assign all the cases. This is not surprising as the data was artificially created to discriminate the groups well. When real data are used, particularly when large number of cases are involved, the random variations involved means that a proportion of erroneous allocation often occurs. Step 4: Transfer Parameters and Coefficients as Input Data for Program 2The tables for means and Standard Deviation, the function coefficents, and the centrod values, reprsents the Discriminant model developed from the modelling data. These are copied to the appropriate text areas of program 2, so they can be used to analyse additional and new data in the future.Should the user intend to use the coefficients developed in the future, it would be appropriate to archive these parameters and coefficients, so they can be use in future analysis. Should the user wishes to use only the significant functions, then the last column of the function coefficients and centroid values can be deleted, resulting in the table as follows. Furthermore, the results can be further altered by Bayesean probability, if the apriori probabilities of the groups are defined. The results of calculations, using all functions, and having the same apriori probability for all groups, produces the same results as when the data is modelled.
Step 5: Analyse the Modelling DataThe predictor variables of the reference data are copied to the data input box of program two and analysed. The same table of results as the validating exercise is produced, as shown in the table to the right. The same input data can be used on clicking the Esample button. However this is merely to demonstrate how program 2 can be used. The intention is that users may wish to enter new data here for analysis. To demonstrate that the third and not significant discriminant function is trivial and unnecessary, the caculations are repeated after the third column from the functions and centroid matrices are deleted, and the results are shown in the table to the right. Comparing to the validating results, the probability coefficients are very similar, and group designations unchanged.
This is essentially a repeat of the validating exercise, and carried out to demonstrate how new data can be analysed.
Step 6Plotting data The user can designate any two functions to be plotted in a x/y scatter plot. The program will firstly mark out in color the areas occupied by each group in the plot, then plot the function scores from the two functions in each case as in an x/y scatter plot. The intention is to allow a visual display of the data in any two function, and how they are related to each group.Ten default background colors are available (assuming that no more than 10 functions will be required in any single Discriminant Analysis). The colors are, in alphabetical order of group names, Grp 1 Grp 2 Grp 3 Grp 4 Grp 5 Grp 6 Grp 7 Grp 8 Grp 9 Grp 10 The plot using the example data with x=function 1 and y=function 2 are as shown above and to the left. The 4 cases from each group can be seen to fit clearly within the areas for each group, with DR (group 1) in green, DW (group 2) in red, SR (group 3) in blue, and SW (group 4) in yellow It can be seen that function 1 can clearly separate DR (grp 1 green), SR (grp 3 blue) and SW (grp 4 yellow). Data points from DW(grp 2 red) however overaps other groups, and require the additional function 2 to separately identify them Program 1. Produce Discriminant Coefficients from Reference DataData Input for Discriminant Analysis Using Reference Data The data is for a single analysis It is a table of multiple columns Each row contains data from a case or a record All columns except the last are predictors and must be numerical The last column (on the right) is outcome group name, single character or text with no gaps
Program 2. Using Discriminant Coefficients for Calssification
The Linear Discriminant analysis using R was carried out to check the accuracy of the Javascript program. Only the minimum amount of coding is used. User can search R for the numerous version of calculation and graphic support for Discriminant analysis.
Please note that R codes are in maroon, and results in blue Step 1. Data entry
myDat = ("
v1 v2 v3 v4 Grp
1.2 45 3.16 72.7 SR
1.3 67 3.38 102.4 SR
1.1 48 3.61 33.7 SR
1.6 36 3.51 58.2 SR
1.5 47 3.20 44.2 DR
1.5 74 3.21 91.8 DR
1.7 47 3.39 53.1 DR
1.6 56 3.36 88.5 DR
1.1 27 3.30 36.3 SW
1.0 53 3.55 74.7 SW
0.9 37 3.23 94.2 SW
1.2 23 3.07 53.8 SW
1.4 44 3.34 20.7 DW
1.3 34 3.24 9.5 DW
1.1 37 3.24 17.8 DW
1.4 55 3.35 35.9 DW
")
myDataFrame <- read.table(textConnection(myDat),header=TRUE)
Please note that the headers are included in R as they are required to call the algorithm
Step 2. Calculate the Standard Deviate z=(v-mean/SD) myDataFrame$z1<-(myDataFrame$v1-mean(myDataFrame$v1)) / sd(myDataFrame$v1) myDataFrame$z2<-(myDataFrame$v2-mean(myDataFrame$v2)) / sd(myDataFrame$v2) myDataFrame$z3<-(myDataFrame$v3-mean(myDataFrame$v3)) / sd(myDataFrame$v3) myDataFrame$z4<-(myDataFrame$v4-mean(myDataFrame$v4)) / sd(myDataFrame$v4)Step 3. Display the data object, including the calculated z values summary(myDataFrame)The results are
v1 v2 v3 v4 Grp z1 z2 z3 z4
1 1.2 45 3.16 72.7 SR -0.45185501 -0.0460777 -1.1033570 0.58784387
2 1.3 67 3.38 102.4 SR -0.02657971 1.5758573 0.4019983 1.60105898
3 1.1 48 3.61 33.7 SR -0.87713031 0.1750953 1.9757788 -0.74264062
4 1.6 36 3.51 58.2 SR 1.24924619 -0.7095966 1.2915264 0.09317656
5 1.5 47 3.20 44.2 DR 0.82397089 0.1013709 -0.8296560 -0.38443326
6 1.5 74 3.21 91.8 DR 0.82397089 2.0919275 -0.7612308 1.23944012
7 1.7 47 3.39 53.1 DR 1.67452149 0.1013709 0.4704235 -0.08080988
8 1.6 56 3.36 88.5 DR 1.24924619 0.7648898 0.2651478 1.12686066
9 1.1 27 3.30 36.3 SW -0.87713031 -1.3731154 -0.1454036 -0.65394166
10 1.0 53 3.55 74.7 SW -1.30240561 0.5437168 1.5652273 0.65607384
11 0.9 37 3.23 94.2 SW -1.72768091 -0.6358722 -0.6243803 1.32131609
12 1.2 23 3.07 53.8 SW -0.45185501 -1.6680127 -1.7191841 -0.05692938
13 1.4 44 3.34 20.7 DW 0.39869559 -0.1198020 0.1282973 -1.18613545
14 1.3 34 3.24 9.5 DW -0.02657971 -0.8570452 -0.5559551 -1.56822331
15 1.1 37 3.24 17.8 DW -0.87713031 -0.6358722 -0.5559551 -1.28506892
16 1.4 55 3.35 35.9 DW 0.39869559 0.6911655 0.1967226 -0.66758765
Please note: z1, z2, z3, and z4 are the z values for v1, v2, v3, and v4
Step 4. Perform Linear Discriminant analysis and display results
#install.packages("MASS") # if not already installed
library(MASS)
fit <- lda(Grp ~ z1 + z2 + z3 + z4, data=myDataFrame)
fit
Please note: the calculations are based on the z values and not the original measurements
Prior probabilities of groups:
DR DW SR SW
0.25 0.25 0.25 0.25
Group means:
z1 z2 z3 z4
DR 1.14292737 0.7648898 -0.2138289 0.4752644
DW -0.02657971 -0.2303885 -0.1967226 -1.1767538
SR -0.02657971 0.2488196 0.6414866 0.3848597
SW -1.08976796 -0.7833209 -0.2309352 0.3166297
Coefficients of linear discriminants:
LD1 LD2 LD3
z1 1.3710438 -0.7462890 0.1364612
z2 1.7208986 0.4156835 -0.2198221
z3 -0.6709095 -0.2961205 -0.8885895
z4 -1.5797134 -1.4082862 0.2128178
Proportion of trace:
LD1 LD2 LD3
0.7899 0.1899 0.0202
The prior probabilities are calculated from the sample sizes of the groups
The LD1, 2, and 3 are the 3 Linear Discriminant functions The proportion of trace represents the proportion of discriminating power of each function, and can be used to test for statistical significance Step 5. Calculate function scores predict(fit,newdata=myDataFrame,prior=c(1,1,1,1)/4)$x #calculate function scoresplease note that the prior term is actually unnecesary, as it is not used when function scores are calculated When the data object is called as newdata, a separate sets of data can be used, providing the the appropriately labelled independent variables are present (in this example z1, z2, z3, and z4)
LD1 LD2 LD3
1 -0.8871802 -0.1830651 1.054003243
2 -0.1234701 -1.6988951 -0.366512945
3 -1.0536723 1.1881588 -2.071887461
4 -0.5220622 -1.7409330 -0.801348480
5 2.4680678 0.2142880 0.745565901
6 3.2824521 -1.2654109 0.592784805
7 2.2823362 -1.2330374 -0.228987474
8 1.0710619 -2.2798046 0.006542448
9 -2.4349834 1.0478052 0.172180524
10 -2.9365081 -0.1894505 -1.548469220
11 -5.1313958 -0.6508717 0.740034619
12 -2.2466446 0.2331076 1.820538675
13 2.1281402 1.2850848 -0.285692745
14 1.3390091 2.0367135 0.345040379
15 0.1061805 2.3646456 0.240614788
16 2.6586689 0.8716648 -0.414407058
Step 6. Calculate the posterior (Bayesean) Probability of belonging to each group
predict(fit,newdata=myDataFrame,prior=c(1,1,1,1)/4)$posterior #calculate posteriori (Bayesean) probabilitiesPlease note that the prior term for apriori probability is used here. If this is left out, the program assumes the prior probabilities are the same as that in the reference data, depending on the sample sizes of the groups there.
DR DW SR SW
1 1.027719e-02 1.738244e-02 0.8056435151 1.666969e-01
2 7.584478e-02 1.695596e-03 0.9196816594 2.777961e-03
3 2.543569e-04 5.741198e-02 0.8883330540 5.400061e-02
4 1.793459e-02 5.428730e-04 0.9760628920 5.459645e-03
5 6.615467e-01 3.338972e-01 0.0045559379 1.919581e-07
6 9.948814e-01 4.791425e-03 0.0003272035 5.284128e-10
7 9.744939e-01 1.355788e-02 0.0119480765 1.254452e-07
8 8.326310e-01 1.399772e-03 0.1659475333 2.173213e-05
9 2.634318e-06 5.441063e-04 0.0758854570 9.235678e-01
10 7.201969e-07 1.160473e-05 0.1924352689 8.075524e-01
11 9.464596e-12 1.011071e-10 0.0001827173 9.998173e-01
12 2.029302e-05 2.288982e-04 0.0560473803 9.437034e-01
13 5.420201e-02 9.416221e-01 0.0041755258 3.779071e-07
14 4.865341e-03 9.917932e-01 0.0033348804 6.545613e-06
15 7.663001e-04 9.729023e-01 0.0250258317 1.305537e-03
16 2.029808e-01 7.940588e-01 0.0029603250 4.639190e-08
when translated to normal numerical format with 2 decimal point precisions
DR DW SR SW 1 0.01 0.02 0.81 0.17 2 0.08 0.00 0.92 0.00 3 0.00 0.06 0.89 0.05 4 0.02 0.00 0.98 0.01 5 0.66 0.33 0.00 0.00 6 0.99 0.00 0.00 0.00 7 0.97 0.01 0.01 0.00 8 0.83 0.00 0.17 0.00 9 0.00 0.00 0.08 0.92 10 0.00 0.00 0.19 0.81 11 0.00 0.00 0.00 1.00 12 0.00 0.00 0.06 0.94 13 0.05 0.94 0.00 0.00 14 0.00 0.99 0.00 0.00 15 0.00 0.97 0.03 0.00 16 0.20 0.79 0.00 0.00The function coefficients, scores, and probabilities are the same as that produced by the Javascrip program, other than minor discrepancies caused by different rounding errors. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
