Introduction
In the article, The effects of band Director leadership style and student leadership ability on band festival ratings, Davison (2007) outline presented a quantitative research to examine different relationships between leadership styles of band directors and student leaderships on band festival ratings. The present paper is a critique on specific aspects of the article such as the design method, validity and reliability.
Design
The research design of the article was not explicitly stated by the author. In that sense, it can be said that the design is non-experimental correlational design. In correlation designs the researcher does not control or alter the research setting. The design can be called correlation when obtained results only imply to which degree the dependant and independent variables “tend to co-occur or are related to each other.” (Price, 2000)
Applying the aforementioned to present study the research examines to which degree the independent variables, which sometimes called predictors, i.e. reported director leadership styles or reported strength of student leadership, can be related to the dependant variable, which sometimes called criterion variables, i.e. concert festival ratings. The selection of such design can be understood as non-experimental design is chosen basically due to the inability to manipulate the independent variable, which in the present case is specifically true. However, the main weakness of choosing such design is the inability to establish the causal assessment, i.e. to prove that changes in independent variable caused the changes in dependant variable. The study could be strengthened by collecting longitudinal data, i.e. data during several points in time, and developing a causal model in which “he relation among the variable could be tested simultaneously, so that indirect effects can be investigated without having to run separate regression analysis.” (Green, Camilli, & Elmore, 2006, p. 218)
Validity
Validity is the assurance that the researcher is measuring the exact relation that the researcher is intended to measure, or “the degree to which a test measures what it is supposed to measure.”
The different aspects of validity can be stated as following:
- Content validity, which is checking “whether or not the content of the manifest variables (e.g. items of a test or questions of a questionnaire) is right to measure the latent concept that we are trying to measure.” (Muijs, 2004, p. 66) In regards of the article, the content of the manifest variables is the instrument designed by the author, called “The Leadership Measurement Instrument” (LMI), which is basically, a survey addressing student leadership strength and director leadership style. The content validity was checked by a panel of four experts who reviewed this survey to ensure its appropriateness for measurement. (DAVISON, 2007) This check for validity required revising the research survey scale implementing answers with percentages (i.e. 1-20%, 21-40%, 41-60%, 61-80%, 81-100%), rather than with five point Likert scale.
- Criterion Validity, which is the prediction of certain outcome of the measurement, where this cannot be applied to the present research as the designed instrument(LMI) was intended to describe and establish a score of the leadership characteristics. In that sense, the variance of the scores cannot be predicted as they resemble individual characteristics, rather than a defined path of behavior. The usage of Likert-scale and the resulting score is rather a range of the outcomes, thus stating that the research does not have criterion validity.
- Construct Validity, which is the approach concerning “the degree to which the test measures the construct it was designed to measure.” The check for construct validity was not performed in that research.
Analyzing internal and external validity, the threats to both must be defined. Internal validity threats are “experimental procedures, treatments, or experiences of the participants that threatens the researcher’s ability to draw correct inferences from the data in an experiment” (Creswell, 2003, p. 171) External validity threats, on the other hand, are the possibility of drawing “incorrect inferences from the sample data to other persons, other settings, and past or future situations.” (Creswell, 2003, p. 171)
From the research it can be seen that the results were externally valid in terms of the question of the relationship between director leadership styles and the strength of student leadership, where the two-tailed correlation between student leadership strength and director leadership style produced a statistically significant positive relationship (r =.39; p=.01), where p is the level of significance or the probability of type I error; the p was set to.01. The other two questions revealed no significant interaction between the variables, thus the results were externally invalid. Threats to internal validity could include the effect of other factors such as individual characteristics unrelated to leadership.
Reliability
Reliability is “the extent to which test scores are free of measurement error.” (Muijs, 2004 71) One form of reliability which is repeated measurements was conducted by the researcher in the form of pre-pilot test which was examined on a group of eight graduate music education students and additionally on a group of Arkansas band directors.
The researcher also conducted internal consistency reliability tests, which is the reference to “how homogenous the items of a test are”, where the results of several analysis tests resulted of an estimated alpha of.87. It can be seen that based on several tests made by the author, it can be said that the results of the research are reliable.
Generalizability
Generalizability can be referred to the ability of the research results to be generalized to the whole population. The size of the sample and the method of selecting subjects for the experimental groups make the research externally invalid and thus greatly reducing generalizability of the results. In addition, the statistically significant difference only in part of the questions put doubt on the external validity of the research results.
In that sense, using a larger sample and the consideration of other variables might result in an improvement in the accuracy of the results. Other external factors might include experience, age and etc, which assumingly might influence the outcome of the analysis. Implementing the approach of a cross-case analysis, which is “collecting and analyzing data from multiple examples” (Green, et al., 2006, p. 123), could be found more accurate in establishing the relation between the examined variables.