Marcus Specht, Alfred Kobsa
GMD FIT – German National Research Center for Information Technology
D-53754 Sankt Augustin
email: {marcus.specht, alfred.kobsa}@gmd.de
http://fit.gmd.de/hci/pages/marcus.specht.html,
http://fit.gmd.de/hci/pages/alfred.kobsa.html
Abstract: This paper presents results from a post-analysis of three previously reported experimental studies on adaptive educational hypermedia. In this new analysis, interaction effects of the post-hoc variable "previous knowledge of learners" and the adaptive treatments were found. The interaction effects have an impact on learners’ scores in knowledge tests, the time learners needed to browse adaptive hypertexts, the number of their page requests, and the type of information requested by them. While learners with higher previous knowledge seem to prefer non-restricting adaptive methods, learners with low previous knowledge can profit from the guidance of more restrictive adaptive methods.
Keywords: Adaptive educational hypermedia, navigation recommendations, student modeling, previous student knowledge, interface design
Empirical evaluations of learning with hypertext have yielded contradictory results. On the one hand, adaptive annotations in educational hypermedia were shown to increase the effectiveness of learning and the learning speed (Eklund & Brusilovsky, 1998; Specht, 1998a; Weber & Specht, 1997). On the other hand, some results suggested an impact of learners’ previous knowledge on the supporting effects of adaptive annotation. Therefore, the following study tries to clarify the interactional effects between previous knowledge of learners and different variants of adaptive annotation in educational hypertexts. The data for the analysis were obtained in two laboratory experiments and one field study with the learning environment AST.
In experiment 1, learning with three different forms of adaptive hypertext and learning with a static hypertext were compared. The four experimental treatments were realized by a combination of the two adaptive methods of adaptive annotation and incremental linking. A combination of both adaptive methods was expected to support students best.
Four groups of students had to work with different versions of a tutorial hypertext (http://hippie.gmd.de:8080/ExpInc) in the area of prionic diseases (a group of infectious diseases). In treatment Text (see Fig. 1), they were administered a regular non-adaptive hypertext. Treatment Anno (adaptive annotation, see Fig. 2) used red and green bullets for marking those links that were (not) recommended to students based on the system’s current learner model. In treatment Inc (incremental linking, see Fig. 3), disrecommended links were removed, but were incrementally added as soon as the student had learned the necessary prerequisites. In treatment IncAnno, the adaptive methods of Inc and Anno were combined.
Figure 1: The main page of the hypertext in the experimental treatment Text
Figure 2: The main page of the hypertext in the experimental treatment Anno (adaptive annotation)
Figure 3: The main page of the hypertext in the experimental treatment Inc (incremental linking)
Figure 4: The main page of the hypertext in the experimental treatment IncAnno
At the beginning of the experiment, all subjects had to answer a demographic questionnaire and a knowledge test (pre-test). The pre-test included 12 questions about central concepts of a curriculum in the area of prionic diseases with varying difficulty. Then, a short introduction to using hypertext and information about the specific experimental treatment was given. Subjects were then asked to carefully study all material that was available in the hypertext system, because they would be quizzed again with the same knowledge test at the end of experiment. After the subjects had visited all hypernodes, the system automatically presented the final questionnaire (post-test). The post-test included all questions from the pre-test and additional questions about the usability and helpfulness of the adaptive methods. The time to read all hypernodes and the number of correctly answered questions were measured as the main dependent variables.
85 subjects completed the experiment. In the demographic questionnaire there were no differences in the experience with computers and the WWW experience between the four groups. In all experimental conditions there was a significant improvement of correctly answered questions from the pre-test to the post-test (t(84)=18.41 ; pē0.01). In the pre-test, the treatment group Text had the best results, while in the post-test the group IncAnno showed the best results. For browsing the hypertext, the group IncAnno needed less time than all other groups. Subjects in the condition Text needed more time than all others. The means of the two knowledge tests and the time to browse the hypertext are shown in Table 1, for all four treatments.
Table 1: The mean scores for the knowledge tests and the time to browse the hypertext, for the four groups (significant figures are in bold).
|
|
Treatment |
|||
|
|
Anno |
Inc |
IncAnno |
Text |
|
Questionnaire before Learning (pre-test) |
4,67 |
5,22 |
4,88 |
5,53 |
|
Questionnaire after Learning (post-test) |
10,01 |
10,3 |
11,33 |
9,94 |
|
Mean time of browsing (in sec.) |
692 |
765 |
618 |
893 |
To ascertain that the experimental groups did not differ in their previous knowledge about the domain, a variance analysis was computed on the data of the pre-test, which showed no significant differences (F(3,81)=0.37 ; pē0.05). An ANOVA comparing the differences between the pre-test and the post-test scores showed no significant main effect for the factor adaptive annotation of hyperlinks Anno (F(1,81)=3.91 ; p=0.052) nor for incremental linking Inc (F(1,81)=2.410 ; pē0.05). With regard to time needed to browse the hypertext, significant effects for both adaptive annotation (F(1,81)=13.17 ; pē0.05) and incremental linking (F(1,81)=4,49 ; pē0.05) could be shown. Comparing the experimental group which used both adaptive methods combined (IncAnno) with the group that had no adaptivity (Text) showed a significant effect with respect to both the number of correctly answered questions (t(39)=2.38; pē0.05) and the time needed to browse the hypertext (t(39)=-4.23; pē0.05). More detailed results and a discussion thereof can be found in (Specht, 1998b).
However, a post-hoc split of learners into three groups (mean +/- 1 standard deviation) depending on the score in the pre-test revealed that the significant knowledge gain in treatment IncAnno only holds true for students with low previous knowledge (F(2,82)=46.9 ; pē0.01). Learners with average or high previous knowledge did not learn significantly more. They profit from the shorter browsing time in the adaptive treatments, though.
We also found that in the adaptive treatments IncAnno and Anno, all learners worked faster through the whole hypertext than in the adaptive treatment Inc. (the difference between IncAnno and Inc was even significant (t(42)=2.6 ; pē0.05)). ). One possible explanation is that in the Inc treatment, learners had to search for new hyperlinks that possibly appeared on some pages after learners had viewed their prerequisite pages. In the annotated treatments, in contrast, all potential hyperlinks were visible from the beginning. This interpretation is supported by the additional finding that the learners in the Inc group had a significantly higher number of visits to already seen hypernodes than all other groups (F(3,81)=16.32 ; pē0.01), i.e. even the Text group. These findings allow for the hypothesis that the unpredictability of the interface in the Inc treatment has a very negative impact on the overall navigation support. The addition of annotations in IncAnno reconciles the violation of the predictability requirement of HCI caused by incrementally appearing hyperlinks, by virtue of the fact that they mark locations where hyperlinks may appear sometimes in the future (their adaptive colors are thereby irrelevant). We regard this finding as a confirmation of the old wisdom in HCI that adaptive methods can never be a remedy for bad interface design.
The second experiment investigated the effects of introductory lessons that are dynamically computed based on the results of a preceding knowledge test. An adaptive system component generated a questionnaire containing necessary prerequisites for a curriculum. It then collected those prerequisites that a student had not yet mastered into an introductory lesson and presented this lesson to the student.
The experiment was a classical pre-post design where the students had to answer a knowledge test about a given curriculum at the beginning of the experiment and at the end of the curriculum. The post-test was automatically administered after students had seen all units of the curriculum. The experimental variation was that students either got a static introductory lesson (All) that contained the main prerequisites for the curriculum, got no introductory lesson (No), or that the system computed a special introductory lesson depending on the introductory knowledge test of the learners (Filter). All subjects worked in an annotation treatment comparable to the Anno treatment in the first experiment.
46 subjects took part in the experiment. No significant differences could be reported for the experimental treatments in the introductory knowledge test (for details see Specht, 1998a). In the All treatment, learners showed a significant improvement from the pre-test to the post knowledge test (t(8)=-5.7; pē0.01).
A post-hoc split of learners into two groups was then performed, where the split point was the mean of the pre-test results (this bifurcation rather than a tripartite split as in experiment 1 was chosen because of the smaller number of subjects). We found that it was mostly the students with low previous knowledge who got better in the Filter treatment while students with high previous knowledge got better in the All treatment (ceiling effects are unlikely). The mean scores are shown in table 2.
Table 2: The means of the two post-hoc groups for the knowledge tests across all conditions.
|
|
Low previous knowledge |
High previous knowledge |
|
Questionaire before Learning |
7.79 |
12.28 |
|
Questionaire after Learning |
12 |
12.9 |
These results support the findings of experiment 1 in that there seems to be an interaction between previous knowledge and the effectiveness of different adaptive treatments in terms of knowledge gain.
In the field study, the courseware Adaptive Statistics Tutor (AST, http://hippie.gmd.de:8080/ACE) was accessible to students of the University of Trier, Germany. AST is based on a knowledge-based architecture for delivering adaptive and adaptable courseware on the WWW. Adaptability in the system allows learners to specify preferences about learning materials, and gives teachers the possibility to adapt a curriculum and specify criteria and pedagogical strategies for the learning process. Adaptivity in AST uses methods like adaptive annotation, adaptive testing, and incremental linking. Learners can start working on the curriculum wherever they want. The system checks whether a learner lacks any prerequisite knowledge to work on a selected section, and presents tests for lacking prerequisites. If a learner is not able to solve the given tasks, AST recommends to work on the prerequisites first. Learners can explore new concepts and get immediate feedback about his/her actions by using JAVA-based playgrounds and interactive HTML forms for tests (for details see Specht, 1998a).
Before working with AST, students had to fill out a demographic questionnaire and work on a pre-test about the statistics curriculum. The curriculum contained 23 concepts of descriptive statistics in 8 learning units (sections). With each section and concept, 5 to 15 tests were associated. When learners mastered a certain amount of tests, the system assumed that they had learned the respective concept. Students were allowed to work with AST as long and as much as they wanted, and the system was able to preserve the learner model over multiple sessions. Students were randomly assigned to the following three adaptive treatments:
1. Annotation of Hyperlinks (Annotation): A colored bullet was presented with each hyperlink, which gave some information about the concept behind the hyperlink. The color of the bullets was adapted to the knowledge state of the student. Green balls marked the corresponding link as a recommendation, orange balls were presented when all prerequisites to this concept had been learned, and red balls meant that the hyperlink leads to a hypernode whose prerequisites were not yet fully learned by the student.
2. Annotation of Hyperlinks and hiding of "red" hyperlinks (Hide): In this treatment, adaptive annotation of hyperlinks was realized in the same way as in treatment 1, except that those hyperlinks were hidden that lead to hypernodes that were "not ready to be learned". When a student had mastered all prerequisites of a concept, the hyperlink to this concept was made visible and presented with an orange ball. The annotation of hyperlinks with green balls was computed by the system taking into account the knowledge state of a student, the learning material that he or she had already viewed, and a didactic model for sequencing concepts and learning materials.
3. Annotation of learned and not-learned concepts (Static): In the third treatment, all annotations had the form of white balls and check marks, so learners only got information about what concepts they had already learned (check mark) and what concepts they still needed to work on (white ball).
During a period of three months, 180 subjects worked with AST. In the following study only 67 subjects are taken into account that had issued more than 20 page requests to the system. One result of the study was that the number of requests and the requested type of learning material were dependent on the adaptive treatment. Summarized over all 22 units of the curriculum, subjects in the Annotation group requested significantly more text material (F(2,63)=6.11; pē0.05) than the other groups, while the subjects in the Hide condition requested more tests (F(2,63)=5.77; pē0.05) than the other groups. The number of requests was not confounded with the preferences for different materials specified in the introductory questionnaire. In the pre-test there were no differences between the experimental groups. A post-hoc split into three groups (mean +/- 1 standard deviation) depending on the results in the pre-test showed a significant interaction effect between the previous knowledge and the adaptive treatment on the number of information requests (F(4,71)=3.35;pē0.05). Students with the best results in a preliminary knowledge test worked more intensively with the system when they were in the Annotate group. Vice versa, students with medium results in the introductory test worked better (more requests) in the Hide group. There were too less subjects in the low previous knowledge group.
In all three post-hoc studies, interaction effects between the previous knowledge of learners and the adaptive treatments could be shown. First, learners with high previous knowledge seem to prefer working in less restrictive adaptive environments, and work more intensely and have more profit if they have full access to all information. They can benefit from non-restrictive adaptive methods like the adaptive annotation of hyperlinks, however only as far as browsing time is concerned (see experiment 1 and field study). Learners with low previous knowledge seem to profit from more guidance by adaptive methods and the adaptation of the available information to their current knowledge. Guidance by incremental linking must however be combined with indicators for the locations where links will appear, in order to save users from having to search for new links and request pages multiple times (experiment 1). When integrating adaptive methods in learning environments one should keep in mind that certain adaptive treatments can enforce certain learning strategies and the preferred usage of certain learning materials (see study 3).The results and trends presented in these experiments should be validated in follow-up studies with experimental designs where previous knowledge is a controlled experimental variation.
Eklund, J., & Brusilovsky, P. (1998). The Value of Adaptivity in Hypermedia Learning Environments: A Short Review of Empirical Evidence. Proceedings of the the Ninth ACM Conference on Hypertext and Hypermedia (pp. 11-17). Pittsburgh.
Specht, M. (1998a). Adaptive Methoden in computerbasierten Lehr/Lernsystemen (Vol. 1). St. Augustin, GMD Research Series
Specht, M. (1998b). Empirical Evaluation of Adaptive Annotation in Hypermedia. Proceedings of the ED-MEDIA98 (pp. 1327-1332). Freiburg, Germany.
Weber, G., & Specht, M. (1997). User Modeling and Adaptive Navigation Support in WWW-Based Tutoring Systems. In: A. Jameson, C. Paris and C. Tasso, eds.: User Modeling: Proceedings of the Sixth International Conference. (pp. 289-300). Springer Verlag, Wien New York.
