Abstract: Adaptive hypermedia has the potential to break through traditional educational barriers by allowing the tailoring of applications to specific user needs and requirements, to do this effectively the application must hold a model of the user. In this paper we present an empirical evaluation which was carried out to investigate various techniques in adapted hypermedia in education by comparing; adapted presentation, adapted navigation with a non adapted control hypermedia application. This experiment was also used to verify a static model (derived from the SaD static and dynamic) user model which was the result of initial analysis from a research project undertaken at Southampton University. Initial results from the evaluation look promising, with a shift in attitude of subjects towards the acceptance of adaptation in hypermedia applications with a high percentage of users preferring the adapted presentation application.
Keywords: Adaptive Hypermedia, User Modelling, Educational Hypermedia, Static User Modelling, Empirical Evaluation
In general not all personality variables were found to be
relevant to the interaction process however, traits such as shy,
pessimistic and quiet did reveal some significance. Again the most
relevant 'computing experience' attributes were quite clear i.e. users
computing skill, windows, application experience proved to be extremely
relevant for users when interacting with the application. More
confident users required less help from the system. It was interesting
to find that most users irrespective of their skill level wanted some
level of information when errors were made. When analysing the results
it became evident that a division in the type of relevant user model
variables fell into two distinct categories which required the
construction of two user models; a static user model which contained
static user information and a dynamic user model which contained data
which could be used by the system to dynamically adapt to the
individual user. Hence, SaD (Static and Dynamic User Modelling), the
basic SaD model can be seen in Figure 1.
Varying windows management experience between subjects presented a problem, therefore a software tool called SHEP (screen handler enabling process) was used to keep the interface consistent, whereby issues such as the overlapping or multiple opening of windows etc. could be addressed by manipulating settings in the SHEP files called Shepherds (Hall et al, 1997).
In all, four basic (static) stereotypical user models were devised based on system and content knowledge. Each of these models corresponded to a particular linkbase which held the links relevant to that particular user model group. Each subject was required to complete a pre-test questionnaire which was used to allocate that subject a user model. A matrix describing the codes used for each user model can be seen in Figure 2.
For Application A, four linkbases (database of links in Microcosm) were
used each corresponding to one of the four different user models so
that the application displayed different links for the four types of
For Application B only two linkbases (CNSN and CNSE) were applied i.e. only the subject's system skill level was taken into account in the user model, because for this application we were only interested in the users' subjective view of the adapted presentation. In this application the entire information base and highlighted link structure was available to all users whatever their content skill level. However, more complicated and dense material which was included solely for the experts, was shaded out so that users could choose whether to read it or not, thus giving them a form of self adaptation.
Application C was used as a non-adapted control hypermedia application, which the subjects could use to compare the methods of adaptation. This application also utilised only one linkbase where only one user model was allocated to all types of users, however, No form of adaptation was used and all the subjects were presented with the same links, documents and hypermedia structure.
The evaluation was carried out over three separate sessions using a laboratory set up, the subjects came from Computer Science and Archaeology backgrounds. 7 females and 9 males with an age range from 18 to 35 took part in the study with experience ranging from 1st year undergraduates, PhD students to lecturers.
A correlation analysis carried out on the data proved to be very useful. It clearly showed the variables which were significant to the way the subjects worked. In this experiment personality variables only showed some positive correlation. Having said this it is important to point out that the results from the first survey from which SaD was proposed did find certain personality variables extremely relevant in how well a user accepted a system. There is also a general acceptance within the user modelling community (Beck, 1997) that they are relevant to how a user interacts with a system. Therefore the lack of significance from the current analysis can be put down to the small population sample who took part in the study as opposed to the larger sample which took part in the first survey.
It was interesting to find that the number of hours a subject watched television and the number of hours a subject used a computer during a week was very significant in how a user accepted and worked with a system. The results showed that in general a high correlation between the subjects who spent more than 15 hours a week watching television and subjects with a low level of expertise in computing however, subjects who spent less then 15 hours a week watching television were more likely to be expert computer users. Also, the more windows a user claimed to work with at one time the more experienced they were on most of the packages listed in the questionnaire.
From the analysis it could be observed that some of the
variables which were studied could be used to dynamically determine
users' expertise. The time spent per document is a good indicator as
the analysis showed, the CESE group spent less time on a document
compared with the other two groups. Total session completion time per
group was also significantly varied as the CESE expert group had a
faster completion time and the CNSE group had the slowest session
completion time. There was not much difference found in the number of
nodes the different groups visited, however, the CESN group did visit
more nodes within each application compared with the other two groups.
The time spent on relevant documents proved to be more significant than
the average time spent on each document, as the CNSE group spent more
time on relevant documents than the CESN group and the CESE group
overall spent the least time per relevant document.
During the analysis it was obvious that members of each user model worked in a similar manner as the number of links they followed, the time they spent on each document and the total session times etc were alike. This emphasised the relevance of allocating users stereotypical user models at the start of an interactive session.
Figure 3 The Change of Preference Towards Adaptive Hypermedia (full size)
Some subjects even proposed the use of a combination of adaptation
methods within one application. Again the reasons stated for this was
that users would still have access to all the material but could avoid
wasting time reading material which would be too complex or irrelevant
for their particular task and that the information would be there in
case they wanted to know more about a particular topic, other
statements given included;
These views were reflected in the answers given in the
questionnaire where 78% of the subjects felt that all the material
within an application should be made available to all types of users,
just like a conventional book and 63% agreed that shading out chunks of
irrelevant text reduced information overload.
Figure 4 clearly shows that most of subjects spent the highest amount
of time visiting nodes while using Application A. The subjects also
spent on average a significantly greater amount of time visiting each
node in Application C as opposed to Application B where the average
time spent visiting each node was significantly less than with the
other two applications.
The results from this graph are significant in that although the
subjects spent less time per node on average within Application B the
percentage of correct answers was higher for this application in
comparison to correct answers for the other two applications. This can
be seen in Figure 5 below.
Figure 5 Correct Answers For Each Application (full size)
It is important to note here that the users cognitive model is one
division of the user model and due to it's complexity this area was
deliberately left out from this work.
Brusilovsky, P. (1997), Efficient techniques for adaptive hypermedia in Intelligent hypertext: Advanced techniques for the World Wide Web, Lecture Notes in Computer Science, vol.1326 eds. C. Nicholas, J. Mayfield, pp. 12-30. Springer-Verlag, Berlin.
Davis, H., Hall, W., Heath, I. and Hill, G., (1992), Towards an Integrated Information Environment with Open Hypermedia Systems in Proceedings of the ACM conference on Hypertext eds. D. Lucarella, J. Namard, M. Namard and P. Paolini, ACM Press.
Gutkaut, B., Thies, S. and Domik, G., (1997), A User-Adaptive Chart Editing System Based On User Modelling and Critiquing in User Modelling; Proceedings of the 6th International Conference eds. A. Jameson, C. Paris and C. Tasso, Springer Wien.
Hall, W., Weal, M., Heath, I., Wills, G.B., & Crowder, R.M., (1997), Flexible Interfaces in the Industrial Environment in International Conference Managing Enterprises- Stakeholders, Engineering, Logistics and Achievement (ME-SELA'97), pp453-460, Loughborough, UK.
Murphy, M. and McTear, M., (1997), Learner Modelling for Intelligent CALL in User Modelling; Proceedings of the 6th International Conference eds. A. Jameson, C. Paris and C. Tasso, Springer Wien.
Pohl, W., Kobsa, A. and Knutter, O., (1995), User Model Aquisition Heuristics Based on Dialogue Acts in International Workshop on the Design of Cooperative Systems, 471-486, France.
Totterdell, P. and Cooper, P., (1987), Design and Evaluation of the AID Adaptive Front End to Telecom Gold in BCS Workshop Series Proceedings of the 3rd conference of BCS HCI Specialist People and Computers eds. D. Diaper and R. Winder, Cambridge University Press.