Use of Expertiza in Classes, 2005-2006: Impact on Student Learning

During the period from October 2005 to July 2006, the Expertiza system was used in five classes. In three of these classes, the system was used more than once. This report provides an evaluation of those uses.

CSC/ECE 517, Fall 2005. Expertiza was used to help improve a textbook, Dale Skrien's An Introduction to Object-Oriented Design and Design Patterns using Java. The class used a prepublication copy of this textbook and engaged in three peer-reviewed assignments during the semester:

• Write a better explanation of a concept covered in the text.
• Devise another example of a concept covered in the text.
• Generate an exercise for the back of one of the chapters in the text.

Our experience is described in a paper to be presented in October at the OOPSLA Educators' Symposium, the premier conference for work related to teaching object-oriented programming. Seventeen students' contributions have been selected for inclusion in the book. Nearly two thirds of the students felt they learned a lot from these exercises (n = 74, 63% response rate).

The majority of them enjoyed doing the assignments. While they felt they learned more from the assignments than they enjoyed them, homework is not something that students typically enjoy.

The students were less enthusiastic--almost neutral--about the feedback from their peers.

However, they thought that the material they reviewed was correct.

One of the benefits touted for Expertiza is its ability to be used by distance-education students on the same basis as on-campus students. And 70% of the DE students responding (n = 30) did report that it made them feel more a part of the class.

ECE 463/521, Spring 2006, and CSC/ECE 506, Summer 2006. ECE 463/521 and CSC/ECE 506 are both computer-architecture classes, focusing respectively on processor architecture and parallel architecture. Students in the former class did three peer-reviewed excercises, and students in 506 did four. These consisted of reviews of papers from the computer-architecture literature (since these did not produce resources for future classes, these were not really Expertiza experiments), madeup problems over the course-material, machine-scorable questions on a particular lecture, and animations (usually PowerPoint animations) of algorithms covered in class. All students were not required to do all these types of assignments, and ECE 521 had different requirements than ECE 463. The same questions were asked to students from each class.

In ECE 463/521, fourteen of the 30 student responded (47%); in CSC/ECE 506, 10 of 16 students responded (63%). By about a 5-to-4 margin, they said that writing for their peers helped them to better understand the concepts in the course.

Like the CSC/ECE 517 students, they believed their peer reviewers offered correct feedback:

Most of them also believed they received more feedback than they would have from a TA.

Only half, however, thought the feedback was more useful than that from a TA. That is not quite as bad as it sounds, since most of the others thought that the feedback was as useful as from a TA. Only about 1/5 thought it was less useful.

Finally, three-quarters of the students agreed that peer feedback helped them improve their work.

In-class exercise. In addition to the peer-reviewed homework assignments, in CSC/ECE 506, two class periods were spent on "synchronous" peer review. These exercises related to memory consistency, one of the most challenging topics covered in a parallel-architecture class. Three exercises were performed. In the first exercise, students were given a diagrams showing how different consistency models worked (i.e., what value was in each memory cell at each time), and asked to explain them. In the second exercise, they were asked to devise diagrams illustrating certain phenomena (e.g., "Create your own series of reads/writes that is legal under both processor and sequential consistency, but will have different results due to the differences in the consistency models"). In the third set, they were asked to create a step-by-step simulation of a consistency model. Distance-education students were allowed to phone in for the live class session, but only one did. Consequently, only eight students participated in the exercises. All eight responded to the survey

To begin, note that the students considered the material fairly difficult.

The students felt that they learned a lot by doing the in-class exercises ...

more than they would have learned by listening to a lecture.

They found it easier to pay attention than in a normal class.

Summary. In almost all cases, the students reacted favorably to peer review, the exception being the quality of feedback in CSC/ECE 517. The most positive results were obtained when comparing peer review with a lecture (all eight students said they learned more from peer review). Though the results were positive, there is still much room for improvement. One of the chief complaints was that the quality of reviewing was uneven--some students did a cursory job and provided little guidance on how to improve. This can be addressed by better review rubrics and better training for authors before the assignments are due.

The experiments we have done this year only scratch the surface. First, though we can measure how well the exercises themselves helped students learn, we cannot measure the impact of having the resuable learning objects that were produced until the next time the course is taught. In addition, there are many other paradigms in which Expertiza could be used.