Abstract: Electronic commerce has recently shown enormous potential to take over the sales market. There is a need to provide services that can reach individual computer users with different information profiles and levels of expertise. In this paper we introduce the novel concept of web assistants, human assistants working in an electronic web shop. This human-computer cooperation provides intelligent and personal services via an integrated communication media. A prototype of a web assistant system has been implemented. While browsing through the system the user can call for human assistance should the need arise. We present the results of a usability study performed on our prototype system. The results are encouraging especially when it comes to the attitude aspects of usability. The subjects were extremely enthusiastic about the concept of web assistants and its implications.
Keywords: Adaptivity, electronic commerce, usability, www, data collection.
In spite of the apparently warm reception of electronic commerce, most people are not willing to base serious decisions on information or recommendations provided by a computer program. For example, in bank services or insurances customers need highly qualified help to decide what to buy. In such cases, when customers are about to make risky decisions the opinion of a human assistant may be very important as guidance [Friberg, 1998]. One reason for this is that automated services are not intelligent in the sense that they are very limited and do not allow the customer to have a dialogue and ask follow-up questions or ask for explanations [Mertens and Schumann, 1996]. People also tend to trust humans more than machines, at least when it comes to taking advice. Some experiments have shown that users have problems in placing the right level of trust in computer systems (e.g. [Bonsall and Joint, 1991]). If a user takes advice from a human assistant, the user at least has a name of someone to contact if anything goes wrong. Taking advice from a machine means that the responsibility situation is unclear.
Sales assistants in ordinary shops have the ability to adapt to a customer's personal information needs and requirements. An example of this is when a sales person recognises a customer's decision style [Driver et al., 1993], and adapts his or her assistance accordingly [Perrault and Brousseau, 1989]. Another example is a sales person in a local convenience store who knows the customers well, and can anticipate their needs and serve them in a personal manner.
A general problem with web shops is that many people do not like computers or electronics in general. They are not good at using the equipment since they do not understand how it works and they are afraid of the consequences of their actions. Therefore the interface aspects of web shops are important, and the interface needs to be flexible. This kind of personal and intelligent service with a flexible user interface is lacking in the web shops of today, and is what we are aiming at creating.
To summarise, the objective of the work presented in this paper is to design, implement and evaluate a system with the following properties as motivated by the previous discussion.
To evaluate the first part of our objectives we have performed a usability study on an implemented prototype system. The remaining parts of this article are structured as follows. In section 2 we describe the design of a flexible system that allows personal and intelligent service. In section 3 we present a usability evaluation of a prototype implementation of our proposed system. In section 4 we give an overview of related work and finally in section 5 we conclude and give directions for future work.
In Figure 1 we illustrate the structure of our proposed system. The web shop is a web system for electronic commerce (a typical example is Amazon.com). The customer system contains knowledge about customers. The assistant's role is to support the customers of the web shop. Many assistants can work at the same time, helping different customers.
The communication between the customer and the system takes place via a single user interface. The customer can choose to use the system as a traditional web shop, and ask for assistance if he or she wishes.
The assistant has access to the web shop as an information source to deal with enquiries. The assistant can also follow the customer's actions in the web shop. This gives interesting possibilities for collaboration between a customer and an assistant. Even more importantly, to also be able to give a personal service, an assistant can use knowledge from the customer system. The customer system's knowledge about customers is collected by assistants and the web shop. The assistants collect knowledge about a customer from their conversations, while the web shop can use explicit or implicit feedback from the customer (e.g. [Nichols, 1997]). Note that the customer system can also be used for different kinds of functionalities in the web shop (e.g. personal recommendation of shop items).
In Figure 1 we illustrate the primary interface functions of our system. The customer can interact via the browser, a chat window or via a phone or any combination of these. Chatting with an assistant while browsing is possible and also very useful for the collaboration between a customer and an assistant. The phone in combination with the browser can be a particularly useful option for a beginner to the system. Then an assistant can guide the customer to use the system as well as to look for the items in the shop. Note that other interface functions can also be incorporated, for example by adding functionality for voice chatting or video.
As an example of how the system works consider the following scenario. Mary is a regular customer at a web shop. She logs in and begins to browse among the information looking for some things to buy. After a while she needs help to decide between two items of similar kind. She also wants to know if there are any good alternatives to the items that she simply cannot find in the shop. The information and the help functionality available in the web shop is simply not enough to enable her to make up her mind. She calls for assistance by pressing the corresponding button. A chat window pops up and an assistant greets her by name. The assistant asks her if she is satisfied with her latest purchase. She explains what she thinks about it and then goes on to ask some questions regarding her current problem. While Mary is chatting with the assistant she continues to browse in the shop, checking up on things that the assistant recommends. The conversation continues until she makes up her mind, and quit the chat connection to the assistant. She then goes on to purchase the items she chose.
Note that Mary's preference for how to do shopping (e.g. lots of chatting, or just concise information exchange) can influence how the conversation proceeds (depending on the assistant's ability to adapt to Mary's communication profile). She also has the possibility to choose the communication media (e.g. chat window or voice connection).
The same scenario from the assistant's perspective could be as follows. John works as a web assistant for a company. He gets a request from the assistant router that a customer needs assistance. He quickly checks up on the customer (evidently called Mary) and reviews the latest purchases and some other data from the customer system. He then greets her by name and asks if she was satisfied with her latest purchase. Mary answers and asks some questions on a few of the products and he answers the best he can, sometimes using information from the web shop to check up on details, and sometimes using the customer system to get ideas of what Mary potentially likes. When Mary is satisfied and ends the conversation, he updates the customer system with the new knowledge he has gained about Mary from the conversation. He then tells the assistant router that he is ready to help another customer.
The customer system reduces the risks for John to make misstakes in his communication with Mary (e.g. suggesting items not useful), it also saves time by allowing the conversations to be more efficient. Note that John must be observant of how the conversation proceeds to be able to adapt to Mary's communication profile. While this kind of sensitivity and adaptability is no match for a trained assistant like John, it is completely impossible for a computer program (at least with the current technology).
The field trial consists of three parts. In the first part the subject has to log in to the system and submit answers to personal questions such as age group, web experience and taste in movies. This information is sent to the customer system which is accessible to the assistant as a help when answering questions from the subject.
Then in the second part the subject has to perform two exercises in the system in order to get a decent experience with web assistants. The exercises are formulated in such a way that they could potentially be solved using the functionality in the web shop, but using an assistant would probably make the exercises easier. In the first exercise the subject is asked to name three of his or her favourite movies and find three movies with similar plot and three movies with similar actors. In the second exercise the scenario is that the subject would rent three videos together with two friends (the favourite movies of the friends were provided). The requirement on the rented movies was that the subject and the two friends would all like the movies. The subject also had to decide when he or she had found movies that were good enough. The conversations between subjects and the assistant were performed through a chat interface. No voice connection was used. All conversations were logged.
In the third part of the trial the subjects had to answer a set of evaluation questions. The questions and the answers are presented in the following section. The questions were formulated to evaluate the different aspects of usability. The subject had to indicate his or her disagreement or agreement with each question according to a 1 to 10 scale. We had labelled the extreme value 1 with "No, not at all", and 10 with "Yes, definitely".
The assistant only had one subject to deal with at a time. The subjects spent an average of an hour and a half on the exercises. We had nine subjects geographically distributed throughout southern Sweden. There were five female and four male subjects. They had different backgrounds, e.g. different kinds of previous experience with computers and the web (from beginners to professionals), different amount and type of education (junior high school, senior high school, master of science in different areas, up to PhD in computer science) and different age groups (from 16-25 up to 46-55).
| Questions regarding relevance | Mean | Min | Max |
|---|---|---|---|
| Do you think that web assistants can make it easier for customers in a web shop? | 8.9 | 8 | 10 |
| Do you think that web assistants is a good idea? | 9.1 | 5 | 10 |
| Questions regarding efficiency | Mean | Min | Max |
| Do you think you can get flexible and good help from a web assistant? | 7.7 | 3 | 10 |
| Questions regarding attitude | Mean | Min | Max |
| Is it fun to be able to have a dialog with a web assistant? | 9 | 5 | 10 |
| Does the atmosphere become more personal in a web shop with web assistants? | 9.1 | 7 | 10 |
| Does your trust for a web shop increase if you can get help from a web assistant? | 9.3 | 5 | 10 |
| Would you want web assistants in other web shops too? | 9.2 | 6 | 10 |
| Questions regarding learnability | Mean | Min | Max |
| Do you think it was easy to get started and get help by a web assistant when you needed it? | 8.2 | 3 | 10 |
The subjects asked the assistant a range of different kinds of questions. It was common to ask questions of a simple nature like how to find information about some item in the shop, and how some particular functionality in the shop worked. It would definitely be possible to provide computer support for this kind of questions. Such extensions are especially important when one considers the problems of feasibility discussed in Section 5. Note though that even if computer support were introduced to deal with these kinds of problems, some customers would still prefer to use human assistants. This was evident from the information we got in our evaluation. One reason for this seems to be that the special human touch of a conversation with a web assistant is very important.
Subjects also asked questions of a more complex and subjective nature, such as "Has the actor X done any movie similar to the movie Y", or "Do you think that my friend and I would like the movie X?". Finding an answer to questions like this requires more effort by the assistant and having a dialogue with follow-up questions is essential. This kind of support is difficult to automate. Observe that the web shop we used in our field trial has advanced search functionality. Still, it was not enough for the subjects to solve their exercises. In some cases the search functionality allowed them to find potential solutions, but they still wanted to verify it with the assistant. The conversations that follow this kind of complex questions usually takes several turns and can go on for a long time. We hypothesise that using an advanced customer system could make these conversations shorter and more efficient. In our experiment the assistant had very limited information about the customers and basically had to start from scratch with each customer.
It is interesting to study the way the subjects tried to explain their taste in movies to the assistant. They used quite general statements of a different nature than the information typically gained by the more traditional data collection methods (i.e. explicit feedback using for example questionnaires, and implicit data collection [Nichols, 1997]). For example, one subject said "I like movies by John Travolta because he is so good looking". Another said "I generally like movies with scifi theme and UFOs, I do not mind if they are really bad, that is just cool."
There was a rather wide variety on how much the subjects used the assistants. Some of the more experienced computer users were confident in their abilities to solve the exercises on their own. However, they confessed that they would make more use of the assistants if they had to solve the exercises again, because they thought it would be more efficient.
As mentioned previously, only nine subjects participated in the evaluation. While the subjects had different backgrounds it is still a small number, and part of future work is definitely to continue the field trial with more subjects. However, since every single subject was very positive and gave generally high scores to most questions it is a strong indication to the usability of the concept of web assistants.
Kristina Höök and colleagues argue for the usefulness of combining human and machine intelligence to achieve filtered information. In [Höök et al., 1997] they describe an approach which they call "edited adaptive hypermedia". The idea is to have a human editor who collects and structures information for the benefit of other information users. User profiles are suggested to handle the individual user interests and preferences. The editor is supposed to be an expert in the special domain and also an expert computer user and have various search tools at his or her disposal. The advantages with having a human editor in combination with machine intelligence in the form of advanced search tools are, they argue, that users find it easier to place the right level of trust in a human compared to a machine, and that humans usually have a greater flexibility and domain knowledge than machines.
The collaboration of humans and machines is a central issue in the related work outlined above. This can be seen as support for our approach of integrating human assistants in web shops.
Several researchers have compared the service provided in today's state of the art electronic web shops with the service given by assistants in ordinary shops (e.g. [Schumann et al., 1998, Jörding, 1997]). It is argued that the electronic counterpart is lacking in the kind of service that can be provided. The idea to include human assistants in web shops is never raised though.
Based on the results of the evaluation we can conclude that we have clearly satisfied our first goal. The subjects were extremely enthusiastic and indicated that they got truly adaptive support and enjoyed the special human touch of the system.
When it comes to the second goal we cannot claim to have fulfilled it since we have not tested an advanced customer system. However, based on the collected data from conversations between the subjects and the assistant, we conclude that conversations is a data collection method that could be valuable for fulfilling the goal of personal support and should be of interest to the user modelling community. We also have indications from the person who played the role of the assistant that an assistant could be of help when it comes to separating the important information from the noise in the conversation data.
As for the third goal we have not evaluated the interface aspects of our proposed system (we only used the chat interface to assistants in the evaluation). However, some subjects suggested improvements of the system by introducing a voice interface to the assistants, which is actually already part of our proposed system. We interpret this as an indication that we are on the right track towards fulfilling the third goal.
We believe that the concept of web assistants has a large potential when it comes to dealing with the global problem of urbanisation. Technically, web assistants could easily work from home. The only additional requirement is a fast connection to internet.
An important issue with web assistants is whether it is technically and economically feasible. The chatting in itself is not much of a problem. What could be a difficulty though is if there are a massive amount of simultaneous users wanting assistance at the same time. Then some kind of routing functionality would be needed to queue up users for the assistants. Many large sales companies have call centers for customer support open twenty four hours a day. Then web assistants would be a natural extension to the electronic commerce market. Currently many sales companies are training their employees in the technique of servicing customers. This training will come in handy for web assistants. For small companies the concept of web assistants may be too expensive to realise.
The most important part of our future work is to design and test an advanced customer system. The agent framework used for the prototype implementation has good extensibility [Kindborg et al., 1999], allowing for a simple inclusion of an advanced customer system. We hypothesise that using conversations as a data collection method for user models could play an important role in this task. We are currently studying machine learning algorithms that can take user information collected by assistants into consideration for generalisation. Testing our system with a more flexible interface such as the suggested voice connection to assistants is also an interesting part of the future work.
The problems with feasibility mentioned above need to be dealt with in real world applications. In many cases it may not be feasible to let customers have human assistance for every possible reason, for example for trivial questions. A compromise between the quality of service and the waiting time for the customers may be necessary. Therefore we would like to investigate if assistants can help in identifying problems common to many customers. An attempt could then be made to provide automatic help with these problems (when possible), perhaps by introducing new help functionality in the web shop or by extending the knowledge in the customer system. The goal of these extensions is to improve the quality of information provided to the customers and thereby reduce the number of redundant questions.
Two of our subjects commented that it was important to feel the presence of the assistant. It was suggested that an animated character of some kind could be used to indicate what the assistant was doing. This would be interesting to evaluate. Another idea was to use characters somehow indicating the personality of the assistant, so that the user could select an assistant to his or her liking.