3. Method

Selection and assignment of subjects

The Human Resources Department selected 66 employees as learners to receive the instruction. These were new hires who had not received the instruction previously. These employees were potential subjects of the study. Three cycles of instruction were planned, in October 1999, January and April 2000; to coincide with the company's existing quarterly training frequency. The Human Resources department specified the order in which the learners were to receive the instruction. The learners were divided in three groups, one for each training cycle, using the order specified. The first group comprised 24 learners.

Each learner was interviewed to introduce the researcher and describe the study. Learners were informed that while the instruction was a mandatory condition of employment, their participation in the study was part of an independent academic research project and was voluntary. Consent forms were distributed and reviewed. After a few days, the learners were again contacted and asked if they chose to become a subject in the study. Completed consent forms were collected from 23 of the learners and they became subjects of the study for the first cycle. Subjects were given photocopies of their signed consent forms. The learner who declined to participate was scheduled for the classroom session and did not become a subject.

The subjects then were divided between the two instruction formats. Twenty-four assignment slips of paper were prepared, half indicating "computer session", half indicating "classroom session". The assignment slips were mixed and placed in a container. Subjects drew a slip from the container to determine which format instruction they would receive. This random selection was used to provide an equitable distribution of computer skill levels between the two formats. Subjects were not allowed to select their choice of format, as it was thought the computer-literate would tend to choose the computer format and vice-versa, potentially biasing the results of the study.

Curriculum development

The curriculum chosen for the study was electrical theory. This included six separate topics: (a) terms and concepts, (b) electrical components, (c) interpretation of wiring diagrams, (d) safety, (e) instruments and measurements, and (f) electro-static discharge awareness. Each topic would require approximately 30 minutes of classroom time. The classroom format curriculum was scheduled in three 60-minute sessions on consecutive days, with two topics presented per session. The CAI format curriculum was scheduled in three similar one-hour time blocks on consecutive days, but the learners could use less than 60 minutes per session if they finished early, or use additional 60-minute blocks (or portions) as required. Time was allowed to be a variable for the CAI format.

The CAI format instruction was developed first. Existing classroom lecture notes and illustrations used previous to the study were converted into a format compatible for computer delivery. The curriculum was written in Hypertext Markup Language (HTML), with the illustrations converted to Graphics Interchange Format (GIF). Navigation buttons and page counters were included at consistent positions on each page. Each page had a note at the bottom instructing the subject to call the Help Desk by telephone at any time if there were questions.

The CAI format curriculum was proofread by an Electrical Engineer for correctness of technical content. Several revisions were made until the engineer was satisfied with the content's accuracy. The curriculum was then usability tested by three persons: a graphic designer, a technical writer, and a web developer. These persons evaluated the curriculum for appearance, typography, spelling, grammar, operation and logic of hyperlinks, readability, and also for clarity from the viewpoint of persons unfamiliar with the content. These persons also compared the assessment tool with the curriculum, to ensure all answers could be found within the curriculum.

In order to keep the content of the CAI and classroom formats as consistent as practical, the classroom format was developed by printing each individual screen of the CAI format. The pages were bound to be used as the lecture script. Illustrations from the CAI format were enlarged, converted to transparencies and displayed using an overhead projector whenever a page appeared in the script containing that illustration.

Delivery of Instruction

A quiet individual office was obtained for the CAI format instruction. This office contained a desk, chair, personal computer, paper, clock, and a telephone. The door could be closed for privacy if the subject wished.

The CAI format lessons were installed on the hard drive of the personal computer and navigated using a web browser. Other applications on the personal computer, such as internet access and games, were removed or disabled. The CAI was set up so that only the present and past content was available to the subject. Future class content was not. This was an attempt to simulate the classroom environment: In the classroom, students were always able to ask questions about the present or previous material, but not of future material.

Each subject was introduced to the CAI format, and used this format under supervision long enough to ensure they were familiar with hypertext, the navigation buttons, the mouse, and were able to navigate the lessons independently. They were encouraged to take notes that could be used as reference material during the posttest. This was allowed as the intent of the instruction was for the learners to be able to retrieve the instruction as needed, not necessarily to memorize it. They were shown that each CAI screen contained a footnote instructing them to phone the Help Desk at any time if any questions arose about curriculum or computer operation. The phone number was also posted on the wall in the event computer failure prevented its retrieval. The subjects were instructed to call the Help Desk and report when they finished each session.

Each subject was monitored at the beginning of each session to ensure the room and computer were ready, the subject was able to begin the day's instruction without trouble, and to note the actual starting time for each session. Sometimes the learners would use that opportunity to ask questions about material covered previously.

A dedicated classroom was used for the classroom format sessions. The room contained a white board, overhead projector, paper, seating and writing tables for all subjects. The instructor delivered the material using the binder of CAI screen printouts as a script and the related transparencies. The white board was used as necessary to clarify points. Subjects were encouraged to take notes and advised the notes could be used as reference material during the test. The start and end times of each session were recorded. Subjects could ask questions as necessary, and review current and prior content as desired.

To ensure the instructor was available at the Help Desk while learners were using the CAI format curriculum, the classroom and CAI sessions were never scheduled concurrently.

Assessment

A 40 question paper multiple choice test was compiled from the curriculum, with a representative number of questions from each of the six topics. This test was given as a pretest to all subjects prior to beginning instruction. Scrap paper was provided. Each subject could use as much time as necessary, and their elapsed time was noted. Upon completion, the tests were scored using 2.5 points for each correct answer. The subjects were informed of their score, but not which questions were missed. The intent of withholding that information was to prevent the explanation of the incorrect answers from becoming inadvertent undocumented instruction. Subjects were told the same test would be used as a posttest.

A posttest was given to all subjects following completion of all instruction. The subjects were reminded they could refer to their notes during the test. Scrap paper was provided. Each subject could use as much time as necessary, and was timed. The tests were scored using 2.5 points for each correct answer. Upon completion, the subjects were informed of their score; and the incorrect answers were reviewed if requested. At this time, the pretest was also offered for review if requested.

All learners used the same paper tests, regardless of which format instruction they received. Although computerized testing is available and common in CAI instruction, it was believed that introducing a variable of different testing methods could potentially bias the results of the study, which was not intended to include differences resulting from test method.

Satisfaction Survey

After the posttest was reviewed, subjects were given the satisfaction survey instrument. The same survey instrument was used for both classroom and CAI subjects. The questions were grouped to apply to one group or the other. The inappropriate group of questions was stricken with an X so the subjects would not inadvertently complete the wrong section.

The surveys identified the instructional format used, but not the individual subject. Subjects were instructed not to indicate their names, so that they would not be inhibited in providing honest answers.

Time log

A time log was kept to document time spent in developing the curriculum, presenting the curriculum, and other related tasks. This time log would be used to figure costs of the two methods, using a per-hour figure supplied by the company. The company provided a figure of $50 per hour as their estimate of the actual cost of a person developing the curriculum.

Subsequent cycles

The selection and assignment of subjects, instruction, and testing were repeated in the second and third cycles. The only deviation in method was to adjust the relative number of assignment slips in the container prior to the last cycle, to ensure an equal distribution of subjects between the two formats. This adjustment was required as three subjects did not complete the instruction and were removed from study, unbalancing the classroom/CAI mix. The very last subject was presented a forced choice rather than a random choice, as only one assignment slip remained in the container by that point. Table 1 indicates the numbers of learners who became subjects of the study, and in which formats and instruction cycles they participated.

Table 1
Distribution of Learners and Subjects by Training Cycles and Instruction Format

	Learners	Subjects	Classroom	CAI
Cycle 1	24	23	11	12
Cycle 2	20	18	9	9
Cycle 3	22	19	10	8
Total	66	60	30	29

Six learners did not become subjects. Of these, three declined to participate in the study. The remainder initially consented but did not complete the instruction, and were removed from the study.