|
Article Excerpt
The authors examined the extent to which the Realistic, Investigative, Artistic, Social, Enterprising, and Conventional scales and 3-point codes of the Self-Directed Search may be considered statistically and practically equivalent across 3 different modes of administration: paper-and-pencil, personal computer, and Internet. Student preferences for the respective versions were also examined.
**********
As the number of computer and Internet-based versions of career assessment instruments grows, there is an increased need for research to ascertain the equivalency of these instruments, especially as equivalency relates to reliability and validity issues. The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999) specify "a clear rationale and supporting evidence should be provided for any claim that scores on different forms of a test may be used interchangeably" (Standard 4.10, p. 57). In order to be confident in the results of new versions or administration formats of career assessment instruments, equivalency must be established.
We located more than 11 studies that demonstrated the equivalence of score results obtained from paper-and-pencil and computer-assisted administrations of assessment instruments (Beaumont & French, 1987; Booth-Kewley, Edwards, & Rosenfeld, 1992; Davis, Hoffmann, Morse, & Luehr, 1992; Davis & Morse, 1991; Hansen, 1987; Kapes & Vansickle, 1992; Reardon & Loughead, 1988; Roper, Ben-Porath, & Butcher, 1991; Sawyer, Sarris, & Baghurst, 1991; Simola & Holden, 1992; Vansickle & Kapes, 1993; Wilson, Thompson, & Wylie, 1982). However, a few published studies have shown that different administration modalities affect the equivalence of the score results (Allred & Harris, 1984; Beaumont & French, 1987; Watson et al., 1990; Watson, Thomas, & Anderson, 1992).
Although the existing literature seems to suggest that traditional paper-and-pencil administration and computer administration tend to be equivalent more often than not (Hofer & Green, 1985; Moreland, 1987; Roid, 1984), some data suggest these alternate forms are not necessarily equivalent. This would suggest that it is important for each instrument to be examined individually to clearly establish equivalency of scores and to ascertain if assumptions of equivalency are warranted (French, 1986). With the Internet being the newest modality for many forms of testing, there is a need to have research data on the equivalence of the Internet version to paper-and-pencil and computer versions (Gati & Saka, 2001).
The Self-Directed Search Form R (SDS; Holland, 1994) is one of the most widely used interest inventories (Spokane & Holland, 1995) and one of only a few instruments available in multiple formats, for example, paper-and pencil, personal computer, Internet, mail-in scoring (Professional Report Service), and kit (Vocational Exploration and Insight Kit; Reardon & Lenz, 1998). Our study examined only the first three versions of the SDS (i.e., paper-and-pencil, personal computer, and Internet). The SDS personal computer version (Reardon, PAR Staff, & Holland, 1996) was first published in 1985, and in a study, Reardon and Loughead (1988) looked at equivalence of administration procedures and score results for the paper-and-pencil and personal computer versions. The researchers examined scale scores, Iachan agreement index scores, and differentiation scores and found no significant differences between the two formats. However, they did find differences in administration time and student preferences. The personal computer version was completed in about 25% less time than the paper-and-pencil version, and students strongly preferred the personal computer version to the paper-and-pencil version. Such differences in administration times and user preferences regarding the SDS in its various formats raise issues for program administrators about the use of the SDS in practice.
The SDS Internet (Holland et al., 1999) is the most recent version of the SDS, and no prior research has been reported on the equivalence of this format to the paper-and-pencil or personal computer versions. The purpose of this study was to examine the extent to which score results from the administration of three different formats of the SDS Form R (i.e., the paper-and-pencil version, the personal computer version, and the Internet version) may be considered equivalent. We also wanted to assess the reactions of students who used the three administration formats of the SDS. Results of the study would enable us to ascertain the possible impact of the three modes of administration on SDS scores.
METHOD
Design
This study used a 2 X 2 correlational within-subjects design that was counterbalanced for each of the three groups (paper-and-pencil vs. personal computer, personal computer vs. Internet, Internet vs. paper-and-pencil). Counterbalancing the order of the administrations of the SDS was intended to distribute any potential ordering effect. The independent variable in each group was the method of the administration of the SDS--personal computer versus paper-and-pencil versus Internet format. The dependent variable was the equivalence between the SDS summary codes for the two formats that were completed by each participant.
Participants
Ninety-three undergraduate students who were enrolled in an introductory psychology course at a large southeastern university volunteered to participate in a study of career interest inventories. Their participation also fulfilled a requirement for participation in research for the course. Numerous other research options existed for students to fulfill this course...
|
