plb@cs.cmu.edu
http://www.contrib.andrew.cmu.edu/~plb/home.html
Abstract: Adaptivity is a particular functionality of hypermedia that may be applied through a variety of methods in computer-based learning environments used in educational settings, such as the flexible delivery of courses through Web-based instruction. Adaptive link annotation is a specific adaptive hypermedia technology whose aim is to help users find an appropriate path in a learning and information space by adapting link presentation to the goals, knowledge, and other characteristics of an individual user. To date, empirical studies in this area are limited but generally recognised by the Adaptive Hypertext and Hypermedia research community as critically important to validate existing approaches. The purpose of this paper is two fold: to briefly report the results of an experiment to determine the effectiveness of adaptive link annotation in educational hypermedia (fully described in Brusilovsky & Eklund, 1998), and to situate the study within a summarised survey of the literature of adaptive educational hypermedia systems and the empirical studies that have been undertaken to evaluate them.Keywords: Adaptive, evaluation, hypermedia, navigation, user model, WWW
An important technique for navigation support in educational hypermedia is adaptive annotation technology. This annotation augments links with a comment which provides users with information about the current state of the nodes behind the links (Brusilovsky, Pesin & Zyryanov, 1993; de La Passardiere & Dufresne, 1992; Hohl, Böcker & Gunzenhäuser, 1996; Schwarz, Brusilovsky & Weber, 1996). It has been convincingly argued (Brusilovsky & Pesin, 1995; Zeiliger, Reggers & Peeters, 1996; Brusilovsky & Eklund, 1998) that this method is desirable for educational hypermedia, and this is the particular technology used in the adaptive learning environments which feature in the review of empirical work covered in this paper. Link annotations can be provided in textual form or in the form of visual cues, for example, using different icons, colours, font sizes, font types etc. This technique can be considered a "soft form" of hiding (Brusilovsky, 1996), but it is considerably more cognitively coherent in that it consistently presents to the user a full view of the hyperspace.
In a comprehensive review of adaptive hypermedia which emphasised adaptive navigation support, Brusilovsky (1996) identified thirteen adaptive hypermedia systems in the educational domain: ANATOM-TUTOR (Beaumont, 1994), C-BOOK (Kay & Kummerfield, 1994), <CLIBBON> (Clibbon, 1995) , ELM-ART (Schwarz et al, 1996), ISIS-TUTOR (Brusilovsky & Pesin, 1994), ITEM/PG (Brusilovsky, Pesin & Zyranov, 1993), HYPERTUTOR (Perez et al, 1995) , LAND USE TUTOR (Kushniruk & Wang, 1994) MANUEL EXCEL (de La Passardiere & Dufresne, 1992), SHIVA (Zeiliger, 1993), SYPROS (Gonschorel & Herzog, 1995), HYPADAPTER (Hohl et al, 1996) and HYPERCASE (Miccarelli & Sciarrone, 1996). Of course we must add to this list InterBook (Brusilovsky, Schwarz & Weber, 1996) which features in this paper. In addition, recent work in adaptive hypermedia systems development also includes 2L670 (De Bra, 1996; Calvi & De Bra, 1997), 2M350 (De Bra, 1998 submitted), WEST-KBNS (Eklund & Sawers, 1996), AST (Specht et al, 1997), ADI (Schöch, Specht & Weber, 1988 in press), ELM-ART-II (Weber & Specht , 1997), and AHM (da Silva et al, 1998 submitted). Of the adaptive education systems listed above, a number use adaptive annotation, either alone or in combination with other adaptive technologies. These are HYPADAPTER, ELM-ART (as well as the ELM-ART derivatives), ISIS-TUTOR, HYPERCASE, AST, ADI, 2M350 and MANUEL EXCEL.
A survey of the literature on adaptive navigation support in educational hypermedia reveals relatively few empirical studies, or at least papers with a non-trivial evaluation component, that serve to set the context for the study reported in this paper. In the first of these, de La Passardiere & Dufresne (1992) conducted experiments with MANUEL EXCEL, the first working system that used adaptive link annotation. Their work focussed on the value of history-based mechanisms such as a three-stage footprint (unseen, partially seen and completed) and the use of adaptive advice. The advice was calculated from an error diagnosis of the student's answer to tests and the number of trials. These two user-model based components of the learning environment were found through the use of audit trails to significantly increase "exploration of explanations". In other words, students were responding to feedback and investigating the responses to their answers. de La Passardiere & Dufresne's (1992) experiment used a "with and without" adaptivity approach, comparing the paths of a group of 20 students using the environment without the adaptive components to a group of 30 students using it with the adaptive advisement and cognitively coherent three stage footprints. It appears that in this experiment the two environments are largely different. One offers feedback while the other does not, and this is a significant difference in the basic functionality of the systems, regardless of adaptive navigation support. The experiment was valuable in the sense that it gave the researchers an indication of how link annotation might work in an educational environment, and an intuition that it could be valuable but it is hardly an empirical result in favour of adaptive link annotation. This work stresses link annotation as being more than just a physical coherence of the information: that footprints should reflect the cognitive structure of the hyperspace. The importance of offering an individualised historical progression, and a diversified dialogue with the learner, were as relevant as being able to let the user "affect" the information space in which they find themselves.
Brusilovsky & Pesin (1995) conducted one of the earliest and most significant studies with the adaptive link annotation and hiding mechanisms in the ISIS-Tutor. ISIS-Tutor is a reasonably old system (first version completed in 1992) which used colours and symbols on a non-graphic DOS screen to adaptively mark the links on a page to related pages. The environment used annotations of "not ready to be learned", "ready to be learned", and "in work", very similar to ELM-ART and InterBook for which Brusilovsky was partially responsible. The first version of the system was reported in (Brusilovsky, Pesin & Zyryanov, 1993). The more advanced version used in the experiment was reported in (Brusilovsky & Pesin, 1994a; Brusilovsky & Pesin, 1994b). The first empirical work with ISIS-Tutor was completed in 1994. The paper (Brusilovsky & Pesin, 1995) written immediately after the experiment reported the very first processed results: "...the overall number of navigation steps, the number of repetitions of previously studied concepts, the number of transitions from concept to concept and from index to concept are seriously less for [adaptive hypermedia]." Later in 1995 these results were analysed in more detail and finally published in (Brusilovsky, 1997). A third, more comprehensive analysis was completed in 1996 and belatedly reported in (Brusilovsky & Pesin, 1998) which presented the carefully processed results from the same experiment. This work showed that due to a great variety of navigation styles within each group very few dependent variables show significant difference between the with/without adaptivity groups. At the same time, the work demonstrated by analysis of variance that the overall number of navigation steps, as well as the number of unforced repetitions of concept and task pages were significantly fewer with the adaptive version of the tutor. No difference was found between annotation and hiding techniques. No difference was also found between all three groups for the quality of mental maps of the hyperspace developed by students.
In 1996 ISIS-Tutor was used again with almost one hundred subjects but at the time of writing this paper (1998) the publication of the results is forthcoming.
In a less formal context, Eklund conducted trials with a group of 30 university students studying HTML using a system called WEST-KBNS (Eklund & Sawers, 1996) which produced a annotated global overview map of the nodes in the courseware. Using questionnaires and focus groups it was found that students thought that the navigable and annotated global map was a useful aid to moving through the hyperspace. Using a scale of 1=strongly disagree to 5=strongly agree a mean of 3.9 and a standard deviation of 1.3 was obtained for students' impressions of the usefulness of this map. Focus groups confirmed that students generally found the feature worthwhile, and the annotations on it which gave them an individual suggestion of where to proceed after each test valuable. Again this work offers no firm empirical evidence in favour of adaptive link annotation but is of value in the sense that it used a range of data obtained from sources that were not generally used in other trials. The use of these sources acknowledges that a qualitative approach can provide very useful and positive feedback about a particular interface feature, in this case link annotation. Audit trails provide a detailed picture of user paths but often the interpretation of them fails to account for a wide range of individual differences in users, and hence they are best used in conjunction with other information sources.
In a recent study using ELM-ART-II, Weber & Specht (1997) compared the mean number of pages and tests that visitors to the ELM-ART-II Web site made (those who made a non-trivial visit), both with and without adaptive link annotation and adaptive curriculum sequencing. It was found that there was a statistically significant effect of adaptive curriculum sequencing at p<0.05 but no significant difference in link annotation. Here, it is the ability of these adaptive interface tools to motivate the users to proceed that is being evaluated. Further, Weber & Specht (1997) report that adaptive link annotation had no significant effect on the number of navigation steps. There are two difficulties with this experiment - the small number of participants (a total of twenty-four across four groups), and the fact that users were totally unscreened except for the fact that they visited more than the first few pages. Users' knowledge of the content (LISP programming), the time spent on the computer before the session, the speed of their connection, their motivation, and so forth, are all almost completely uncontrolled variables. They qualify their results as not necessarily supporting the case against adaptive link annotation on the basis of these obvious experimental limitations.
Weber, Brusilovsky & Specht (1998 submitted) used the ELM-ART system to test the effect of both adaptive curriculum sequencing, as mentioned, and link annotation. No significant effect on the number of navigation steps was found through the use of adaptive link annotation. There was some evidence that the adaptive component was useful for beginners, but the small effect faded as the users became more familiar with the system. This is not to say that as a group, the subjects did not benefit from using the system, as they clearly did in this case. They concluded that link annotation appears useful to novice users as it provides an initial support for them in the use of an unfamiliar interface.
Brusilovsky & Pesin's (1995; 1998) study of the efficiency of adaptive link annotation in the ISIS Tutor, as described above, remains the only convincing piece of empirical work with a favourable outcome for the adaptive navigation support technique of link annotation in educational hypermedia. The overall result is that learning using link annotation is faster, more goal-oriented, and significantly reduces the number of steps to cover the hypermedia. It many ways it points to the value of link annotation in the information retrieval domain. Is the fact that students used a "more efficient" path with adaptive navigation support particularly relevant in an educational setting? If it is, could a more efficient path be achieved through a careful layout in a static hyperspace, or through the inclusion of other more traditional non-adaptive strategies such as the use of case studies, interactive examples, or problem solving?
InterBook is a system for authoring and delivering adaptive electronic textbooks on WWW (Brusilovsky, Schwarz & Weber, 1996). InterBook is a well-published environment and it is the intention here to provide just the briefest overview. InterBook-served electronic textbooks are based on a domain model represented in a form of concept network. All sections of an electronic textbook are indexed with domain concepts. ie., for each section a list of related concepts with their roles is provided (each concept can be either an outcome concept or a background concept). For each domain model concept, an individual student's knowledge model stores some value which is an estimation of the student knowledge level of this concept. InterBook traces student actions and keep the student model updated. Using the student model, InterBook distinguishes several educational states for each unit of ET: the content of the unit can be known to the student (all outcome concepts are learned or well-learned), ready to be learned (all prerequisites are learned or well-learned), or not ready to be learned (some prerequisite concepts are not yet learned). According to the idea or adaptive annotation, the icon and the font of each link presented to the student are computed dynamically from the individual student model. They always inform the student about the type and the educational state of the unit behind the link. In InterBook, a red ball and italic font means "not ready to be learned", a green ball and bold font suggests "ready and recommended", while a white ball means "learned, no new information". A checkmark is added for already visited units. The same way can be used to distinguish and show several levels of students knowledge of the concepts shown on the concept bar. In InterBook, no annotation means "unknown", small checkmark means "known" (learning started), medium checkmark means "learned" and big checkmark means "well-learned".
In a study involving 25 undergraduate teacher education students in an educational computing elective at the University of Technology, Sydney, students were exposed to two chapters of a textbook about the database and spreadsheet mdoules of the integrated package ClarisWorks, and used the InterBook system both with and without adaptive link annotation. The experiment was created to be in a real-world teaching and learning context, with the use of InterBook as an integral part of a university subject. The goal of this experiment was to assess what impact, if any, user-model based link annotation would have on student's learning and on their paths through the learning space, in this realistic situation. Tests of knowledge were carried out, audit trails and questionnaires were gathered and the results analysed. The experiment took place over a four-week period. In the first two hour session, students were introduced to InterBook and its features explained to them. They used the system for an hour, and answered a questionnaire about its features. This questionnaire showed that almost all students were familiar with what each of the buttons and annotations meant. They were then free to use the system at any time during the following week. In the second session, students were randomly divided into two groups of equal size, one group receiving the link annotation (12 subjects), while the other group did not (13 subjects). They were allowed access to the chapter of the textbook on databases which had been authored into InterBook, and they completed a questionnaire. Students had access to the database chapter for the following week. In the third session, students took a multiple choice test on the database section of the textbook. For the purpose of brevity we omit the description of the following sessions which are outside the goal of this paper.
All student transactions with InterBook were recorded by audit trails which were used to examine how particpants navigated through InterBook with and without ANS. For each user these trails showed the number of times they selected a green ball, red ball, as well as their use of all the other features of the interface. This information was collated with the test results and the questionnaires. We initially found using a two-sided t-test that students with adaptive link annotation performed significantly worse in the database section test and that there was no difference in the spreadsheet section. This unexpected result suggested that further investigation was needed. First, students who did not spend a reasonable time with the system were excluded, and once this was done we found no significant difference in the test results for the group with adaptivity and the group without. A clear correlation (R=0.670) was found between the agreement rate and score in the database tests: the more students agree with system's suggestion, the better is the score - for the group receiving link annotation.
For the database section, four separate groups were identified on the basis of their agreement rate: high positive, low positive, low negative and high negative, and all the users were categorised into one of these groups. We found that for the ANS group a better agreement rate results in generally better test results.
Using sequential navigation (i.e., continue-back) versus non-sequential navigation is known behaviour exposed by novices in hyperspaces. However, we were able to show that ANS encourages the novices to use annotated non-sequential tools more often. This was achieved using the count of hits on annotated links such as table of contents links versus non-annotated links such as Continue button.
This implies that ANS provides the learner with a non-linear guide through the learning space, and learners are more likely to use non-sequential paths with adaptive link annotation. It again reflects the student's trust in the annotations - ANS provides some security for those users who would like to follow non-linear paths but might be afraid of becoming lost.
When the number of hits on pages of various 'states' (suggested, etc) is examined, it is clear that students prefer to visit ready-to-be-learned pages than those which are annotated as "no information". Students spent approximately twice as much time reading ready-to-be-learned pages than reading all other pages combined. So a green and unchecked page is one that students read most. This is naturally due to the fact that the Continue button was used most of the time, bringing the user into a page with a "ready to be learned" status.
What we observed was a mixture of two effects: an effect of page state and an effect of annotation. This means that students of the ANS group noticed the annotations, and may have decided apriori to spend less time on "nothing new" pages and more on those annotated as "not ready". The effect of annotation clearly dominates in the case of "not-ready" pages. Those rare users who selected a page with full understanding that this page is not ready are willing to allocate significantly more time (ANOVA, p=0.012) for reading this page. Again, they understood how the system worked and trusted the integrity of the annotations.
These results are far more fully described in Brusilovsky & Eklund (1998) and Eklund, Brusilovsky & Schwarz (1998, in press). The trial had some clear limitations: Students had varying levels of access to the learning materials in printed form, and some obviously relied on this for their learning. Also, since the non-annotated continue link (which takes the user to the next page) was used in over 90% of transactions, so the effect of link annotation on student paths was relatively small. Further trials with a modified experimental design are currently underway at The University of Technology, Sydney.
The literature on the evaluation of adaptive link annotation in educational hypermedia is limited and sometimes characterised by anecdotal evidence from a variety of less formal experimental designs. Further, the evaluation of adaptive hypermedia systems has generally has been noted by writers in the field as a much under-investigated area. In this paper we have sought to provide a brief survey of adaptive educational systems which offer link annotation, and reviewed the work that has been undertaken to evaluate their adaptive component. We also offer some definite but qualified empirical evidence from a recent study that adaptive link annotation is advantageous to learners who choose to accept the navigation advice. We found that they will exhibit more exploratory paths in the closed corpus educational context, as well as achieve a greater understanding of the content. There is clearly a need for continued studies in this area.
We wish to express our thanks to Elmar Schwarz, University of Karlsruhe, Germany, the primary developer of InterBook, for his advice about this research and for making available InterBook as an experimental vehicle. Also Associate Professor Ken Sinclair and Dr Mike Bailey of The University of Sydney for their advice about experimental procedure. Part of this work is supported by grants from Alexander von Humboldt Foundation and MacDonnell Foundation.
Brusilovsky P (1996) Methods and techniques of adaptive hypermedia. User Modeling and User-Adapted Interaction 6, 2-3, 87-129.
Brusilovsky P (1997) Efficient techniques for adaptive hypermedia. In: C. Nicholas and J. Mayfield (eds.): Intelligent hypertext: Advanced techniques for the World Wide Web. Lecture Notes in Computer Science, Vol. 1326, Berlin: Springer-Verlag, pp. 12-30.
Brusilovsky P & Eklund (1998) A Study of User Model Based Link Annotation in Educational Hypermedia. Journal of Universal Computer Science. 4, 4. p. 429-448. Springer Science Online. http://www.iicm.edu/jucs_4_4/a_study_of_user/paper.html
Brusilovsky P & Pesin L (1994a) An intelligent learning environment for CDS/ISIS users. In: J. J. Levonen and M. T. Tukianinen (eds.) Proceedings of The interdisciplinary workshop on complex learning in computer environments (CLCE94), Joensuu, Finland, May 16-19, 1994, EIC, pp.29-33. http://cs.joensuu.fi/~mtuki/www_clce.270296/Brusilov.html
Brusilovsky P & Pesin L (1994b) ISIS-Tutor: An adaptive hypertext learning environment. In: H. Ueno and V. Stefanuk (eds.) Proceedings of JCKBSE'94, Japanese-CIS Symposium on knowledge-based software engineering, Pereslavl-Zalesski, Russia, May 10-13, 1994, EIC, pp. 83-87.
Brusilovsky P & Pesin L (1995) Visual annotation of links in adaptive hypermedia. CHI'95 Conference Companion (Denver CO, May 1995), ACM Press, 222-223. http://www.acm.org/sigchi/chi95/Electronic/documnts/shortppr/plb_bdy.htm
Brusilovsky P & Pesin L (1998) Adaptive navigation support in educational hypermedia: An evaluation of the ISIS-Tutor. Journal of Computing and Information Technology.
Brusilovsky P, Pesin L & Zyryanov M (1993) Towards an adaptive hypermedia component for an intelligent learning environment. In Human-Computer Interaction, Lecture Notes in Computer Science, Vol. 753, L. J. Bass, J. Gornostaev and C. Unger (eds), Springer-Verlag, Berlin. pp. 348-358.
Brusilovsky P, Schwarz E & Weber G (1996) A tool for developing adaptive electronic textbooks on WWW, in Proceedings of WebNet'96 (San Francisco CA, October 1996), AACE, 64-69. http://www.contrib.andrew.cmu.edu/~plb/WebNet96.html
Calvi L & De Bra P (1997) Using dynamic hypertext to create multi-purpose textbooks. In: T. Müldner and T. C. Reeves (eds.) Proceedings of ED-MEDIA/ED-TELECOM'97 - World Conference on Educational Multimedia/Hypermedia and World Conference on Educational Telecommunications, Calgary, Canada, June 14-19, 1997, AACE, pp. 130-135.
Clibbon K (1995) Conceptually adapted hypertext for learning. In I.Katz, R. Mack, & L. Marks (Eds.), Proceedings of CHI'95 (pp.224-225). Denver: ACM. http://www.acm.org/sigchi/chi95/Electronic/documnts/shortppr/kc_bdy.htm
da Silva D, Van Durm R, Duval E & Olivie H (1998 submitted) Adaptive Navigation Facilities in Educational Hypermedia. Paper to be presented at WebNet98.
De Bra P (1996) Teaching Hypertext and Hypermedia through the Web. Proceedings of WebNet'96, World Conference of the Web Society. San Francisco, CA, October 15-19, 1996.
de La Passardiere B & Dufresne A (1992) Adaptive navigational tools for educational hypermedia. In I. Tomek (Ed.), ICCAL'92, 4-thInternational Conference on Computers and Learning (pp. 555-567). Berlin: Springer-Verlag. http://mistral.ere.umontreal.ca/~dufresne/Publications/ical91.htm
Eklund J, Brusilovsky P & Schwarz E (1988, in press) A study of adaptive link annotation in educational hypermedia. Paper to be presented at EDMEDIA'98 Freiburg, Germany. June 1998.
Eklund J & Sawers J (1996) Customising Web-based course delivery in WEST® with navigation support. Proceedings of WebNet'96, World Conference of the Web Society. San Francisco, CA, October 15-19, 1996, pp. 534-535.
Gonschorek M & Herzog C (1995) Using hypertext for an adaptive help system in an intelligent tutoring system. In J. Greer (Eds.), Proceedings of AI-ED'95, 7th World Conference on Artificial Intelligence in Education (pp. 274-281). Washington, DC: AACE.
Hohl H, Böcker H, Gunzenhäuser R (1996) Hypadapter: An adaptive hypertext system for exploratory learning and programming. User Modeling and user adapted Interaction 6, 2-3, 131-156.
Kay J & Kummerfeld R (1994b) An Individualised Course for the C Programming Language. Proceedings of Second International WWW Conference "Mosaic and the Web". Chicago, IL.http://www.ncsa.uiuc.edu/SDG/IT94/Proceedings/Educ/kummerfeld/kummerfeld.html
Kushniruk A & Wang H (1994) A hypermedia-based educational system with knowledge-based guidance. In T. Ottoman & I. Tomek (Eds.), Proceedings of ED-MEDIA'94 - World conference on educational multimedia and hypermedia (pp. 335-340). Vancouver, Canada: AACE.
Micarelli A & Sciarrone F (1996) A case-based toolbox for guided hypermedia navigation. Proceedings of Fifth International Conference on User Modelling, UM-96. Kailua-Kona, Hawaii: User Modelling, Inc. (pp. 129-136)
Pérez T, Gutiérrez J & Lopistéguy P (1995) An adaptive hypermedia system. Proceedings of AI-ED'95, 7th World Conference on Artificial Intelligence in Education. Edited by J. Greer. Washington, DC, 16-19 August 1995, pp. 351-358.
Schöch V, Specht M & Weber G (1998, in press) ADI - an evaluation of a tutorial agent. Proceedings of ED-MEDIA/ED-TELECOM'98 - World Conference on Educational Multimedia/Hypermedia and World Conference on Educational Telecommunications, Freiburg, Germany, June, 1998, AACE.
Schwarz E, Brusilovsky P & Weber G (1996) World-wide intelligent textbooks. In: Proceedings of ED-TELECOM'96 - World Conference on Educational Telecommunications, Boston, MA, June 17-22, 1996, AACE, pp. 302-307, http://www.contrib.andrew.cmu.edu/~plb/ED-MEDIA-96.html.
Specht M, Weber G, Heitmeyer S & Schöch V (1997) AST: Adaptive WWW-Courseware for Statistics. In: P. Brusilovsky, J. Fink and J. Kay (eds.) Proceedings of Workshop "Adaptive Systems and User Modeling on the World Wide Web" at 6th International Conference on User Modeling, UM97, Chia Laguna, Sardinia, Italy, June 2, 1997, Carnegie Mellon Online, pp. 91-95, http://www.contrib.andrew.cmu.edu/~plb/UM97_workshop/Specht.html.
Weber G & Specht M (1997) User modeling and adaptive navigation support in WWW-based tutoring systems. In: A. Jameson, C. Paris and C. Tasso (eds.) User Modeling. (Proceedings of 6th International Conference on User Modeling, Chia Laguna, Sardinia, Italy, June 2-5, 1997) Wien: Springer-Verlag, pp. 289-300.
Weber G, Brusilovsky P & Specht M (in press) ELM-ART: An adaptive educational system on the WWW. In User Modeling and User-Adapted Interaction .
Zeiliger R (1993) Adaptive testing: contribution of the SHIVA model. In Item banking: Interactive testing and self-assessment, NATO ASI Serie F, Vol. 112, D. Leclercq and J. Bruno (eds), Springer-Verlag, Berlin. pp. 54-65.
Zeiliger R, Reggers T & Peeters R
(1996) Concept-map based navigation in educational hypermedia : a case
study. Proceedings of ED-MEDIA'96 - World conference on educational
multimedia and hypermedia. Boston, MA, June 17-22,
1996.