Discussion and Conclusions
This study analyzed and evaluated the identification and selection systems used in state supported residential schools of mathematics and science. In order to evaluate these systems from different perspectives, quantitative and qualitative methods were used. The following questions were addressed.
Question One: What are the common policies and procedures used for identifying students in state supported residential schools of mathematics and science?
The typical identification/selection system found in residential schools can best be described as flexible, uses multiple criteria, and follows a multiple stage format. Five selection criteria and five stages were involved in most schools. The selection criteria included standardized tests (verbal and mathematics sections), the home school (the high school in which the student was enrolled before attending the residential school) GPA, ratings of all the information gathered on a student by a selection committee, and interviews. The identification and selection stages included: a recruitment campaign, application file development, file review, interviews, and selection decisions. Except for one school, there were no cutoffs on the selection measures, a practice that allows administrators to make adjustments for representation by region and race/ethnicity. While Stanley (1986), for example, takes a strong position in favor of setting minimal standards without exceptions, other researchers recommend using practices similar to those used in the residential schools (Baska, 1989; Maker, 1989).
Different committees were used for different stages of the selection process. Subjective judgments were used either as components of the overall composite index for selection (i.e., file review, interview rating) or as a final index for selection based on the file review. In the latter case, all transcripts and standardized test scores were included in the applicants' files. One big school did not inform the reviewers about scores. Zero correlations between the SAT-M and SAT-V scores and file ratings were found in the analysis of data from this big school. In contrast, significant correlations were found between these two variables and file ratings in most other schools. These findings support a policy of not including standardized test scores in the applicant's file for the review process. Reviewers can be influenced by extreme scores while rating student files. Uncorrelated components of a selection system, when combined properly, can generate a better validity coefficient with criterion variables.
Grades, ratings, and standardized scores were combined in order to generate a composite index for selection. All schools except two used weighted raw form grades and different methods for developing composite scores. There were no empirical data concerning the validity of any of these methods. At least two problems are involved in these methods. First, adding variables together in raw score form will weight them in unknown ways and produce undesirable statistical artifacts (Lauer and Asher, 1988). Second, the reliability of composite scores is a product of the reliabilities of their components, meaning the unreliabilities of the components will be reflected in the reliability of the composite. Some of the components used in the selection process probably lack reliability. A composite score with low reliability, when correlated with a criterion with low reliability, results in a low correlation coefficient since the upper limit of the correlation coefficient between two variables is restricted by their internal consistency reliabilities. The higher the reliabilities, the higher the correlation. In two schools there was no mechanical addition of different data sources; rather, a holistic score was assigned to each student based on personal judgments of the credentials and accomplishments by members of the selection committee.
Following this discussion and based on the results of this study, two questions can be raised: Why use a multiple criterion system if the final product of the system is invalid, as is the case with composite scores? Why use personal judgments to qualify what has already been quantified as standardized test scores and HS-GPA? Knowing that the reliabilities of the subjective portions of the composite score (i.e., interviews, file ratings) are questionable and costly relative to other objective components, the value of the whole system is therefore questionable.
Question Two: Are the identification and selection systems as used in residential schools valid for predicting success as measured by students' grades?
The correlational analysis of data for most schools indicate that the final index for selection, the composite score, is invalid for predicting success as measured by first year adjusted grade point average (GPA1). The correlation of the composite score with first year GPA was lower than the correlations of home school GPA, SAT, ACT, and interviews with the first year GPA. The stepwise multiple regression analysis excluded the composite score as one of the variables selected for an optimum prediction equation for the criterion GPA1, given the set of variables used for all schools. Composite scores, therefore, function poorly as a predictor for first year adjusted GPA earned at residential schools. The correlations of composite scores with other criterion variables were also lower than the correlations of most predictors.
Another aspect of the analysis carried out in this research was to check the validity of statistical versus clinical judgment since two schools used the latter method. The correlational and regression analyses of data indicate that the use of statistical prediction is far superior to professional judgments for predicting the criterion variables. This finding is in agreement with previous research (Sawyer, 1966). The use of regression equations to combine data and for selection can actually support the strategy of using multiple criteria. Once a regression equation has been cross validated, it can be used for selection of future classes. Mainframe or personal computer statistical programs can do the job in a very short time. The process simply requires entering data for the variables in the equation, calculating the predicted criterion scores, ranking students based on their predicted scores, and using the rankings as selection indices.
A correlation coefficient in the range of .30 to .40 is commonly considered meaningful for educational selection (Kaplan and Saccuzzo, 1989). The HS-GPA was the only variable to meet this criterion consistently with GPAs earned at the end of the first and second years in the residential schools, and for both males and females. This finding was based on data from all schools, from two big schools, and from the only school offering a three year program. Accordingly, the home school GPA proved to be a valid predictor of success as measured by residential school grades. Also, both the SAT-M and SAT-V (or ACT) increased the prediction power of the HS-GPA for the criterion variables. These three variables were selected as the best linear combination for predicting GPA1 in most cases. This result is consistent with previous research done at two of the largest residential schools. An evaluation study conducted at the North Carolina School of Mathematics and Science in 1987 concluded that "high school grade point average is the best predictor of first year grade point average (r=.49)" (p. 24). Another, research study, conducted in 1988 at the Louisiana School for Math, Science, and the Arts, concluded that the highest relationship was between the grade point average earned at the student's home school and the grade point average earned at the Louisiana residential school. What was missing in both studies was the inclusion of composite scores in the analyses.
As for interview ratings, the results of this study suggest a great deal of fluctuation and inconsistency in their correlations with criterion variables. The file ratings correlated significantly and positively with only one (GPAO1) of four criterion variables. They were not consistent in their correlations with criterion variables across schools. File ratings in most schools were based mainly on references and biographical data. As reported by many researchers, interviews, letters of recommendation, and biographical data, in addition to their cost, are poor predictors of future academic success (Hills, 1971).
Question Three: Are teachers trained for and involved in selection?
This study found the involvement of teachers in the selection process to be minimal, as is their training for identification and selection. Inadequate training and involvement of teachers in the selection processes may lead to unrealistic expectations of students and may be related to both lower student grades and the high attrition rates found in most schools. Also, the range of student abilities, as reflected in SAT scores, is close to what is expected in the general population even though the mean would be much higher. Additionally, it was found that identification data are used only for placement in mathematics and language courses rather than for general instructional planning and counseling. Yet, several researchers have urged that such planning and counseling should be guided by info