End-User Quality of Experience Oriented Adaptive E-learning System: Muntean and McManis: JoDI

End-User Quality of Experience Oriented Adaptive E-learning System

Cristina Hava Muntean and Dr. Jennifer McManis
Performance Engineering Laboratory, Dublin City University, Glasnevin, Dublin 9, Ireland.
Tel: (+353) 1 700 7645 194 Fax: (+353) 1 700 5508
Email: {havac, mcmanisj}@eeng.dcu.ie

Abstract

In the context of new devices and with a variety of network technologies that allow access to the Internet, the providers of e-learning materials have to ensure that the users have a positive experience using their e-learning systems and they are happy to re-use them. Adaptive Hypermedia research aims to provide personalised educational material that ensures a positive learning experience for the end-users. However, user experience is dependent not only on the content served to them, but also on the user perceived performance of the e-learning system. This leads to a new dimension of individual differences between Web users: the end-user Quality of Experience (QoE). We have proposed a solution for Adaptive Hypermedia Systems (AHS) that provides satisfactory end-user QoE through the use of a new QoE layer. This layer attempts to take into account multiple factors affecting QoE in relation to the delivery of a wide range of Web components such as text, images, video, audio.

The effectiveness of our QoE layer has been tested in comparison to a standard educational AHS and the results of these tests are presented in this paper. Different educational-based evaluation techniques such as learner achievement analysis, learning performance assessment, usability survey and correlation analysis between individual student performance and judgment on system usability were applied in order to fully assess the performance of the proposed QoE layer. Results of the tests showed that the use of the QoE layer brought significant improvements in terms of user learning performance, system usability and user satisfaction with the personalised e-learning system while not affecting the user learning achievement.

Keywords

end-user QoE, adaptive hypermedia, e-learning, end-user perceived performance, learning performance

1 Introduction

Extensive research in the area of Web-based adaptive hypermedia has demonstrated the benefit of providing personalized content and navigation support for specific users or user categories. A comprehensive review on the developed adaptive hypermedia systems, the techniques used during the adaptation process and the applicability areas of these systems can be found in Brusilovsky. 2001 and Brusilovsky 1996.

Web users differ in skills, aptitudes, goals and preferences for processing accessed information. They may have different perceptions of the same content and performance factors. Finally, they may have special needs due to disabilities. Therefore, the Web-based Adaptive Hypermedia Systems (AHS) try to capture and analyse these user-related features in order to optimise the user experience with the Web site. A variety of AHS have been applied in the educational area, providing e-learning services. This research area has attracted huge interest due to its capability for facilitating personalized e-learning, its distributed nature and its simplicity of interaction. Several good examples exist in the academic community including ELM-ART II, AHA! and JointZone. These systems build a model of the goals, knowledge and preferences of each individual person and use this model throughout the interaction with the user in order to propose content and link adaptations, which would best suit e-learners. Lately, researchers started to integrate learning styles in the design of a AHS along with the classic learner's features. Several systems providing adaptation to users' learning styles have been created such as INSPIRE and AES-CS.

With the advance in computer and communication technology a variety of Internet access devices (e.g. laptop, pocketPC, PDA, mobile phone) have been launched on the market. The type and capacity of the access device, the network the device operates on, the available bandwidth, the state of the network that may very dynamically over course of session and the complexity of the Web pages delivered, all affect the quality of experience for the end-user. Thus, end-users of educational and training services expect not only high-quality and efficient educational material but also a perfect integration of this material with the day-to-day operational environment and network framework. In this context it is significant to highlight a new problem faced by network-based education over the Internet: providing a good level of end-user perceived Quality of Service (QoS), also called Quality of Experience (QoE).

Currently Adaptive Hypermedia Systems for Education (AHSE) place very little emphasis on QoE and its effect on the learning process. This QoE-unaware approach is perhaps unsuited to a general learning environment (Figure 1) where one can imagine a student with a laptop moving from a low bandwidth home connection, to a higher bandwidth school connection, and potentially to public transport with a mobile connection with a widely varying bandwidth connection. It should be noted that some adaptive hypermedia systems have taken into consideration some performance features (e.g. device capabilities, the type of the access, state of the network, etc.) in order to improve the end-user QoE. For example GUIDE system considers hand-held units as tools for navigation and display of an adaptive tourist guide. INTRIGUE, a tourist information system that assists the user in the organization of a tour, provides personalized information that can be displayed on WAP phones. Merida et al. have considered HTTP protocol, type of the access and the server load in the design of the SHAAD. However, these account for only a limited range of factors affecting performance and do not fully address QoE.

figure 1

Figure 1. A New E-learning Environment

Therefore, adaptive hypermedia systems should also take into consideration QoE characteristics when the user profile is built and regularly monitor in real-time any change in the system that might indicate variations of QoE. These include changes in the user's operational environment and also modifications of user behaviour, which might possibly indicate dissatisfaction with service (such as an abort action). This would allow for better Web content adaptation that suites varying delivery conditions.

This paper presents an approach that introduces a new QoE-based content adaptation strategy that enhances the functionality of a classic adaptive hypermedia system and aim to improve the end-user QoE. The QoE-based enhancement (QoE layer) measures and analyses various factors that may affect QoE. QoE layer consists of different components (Figure 2) The Performance Monitor measures a variety of performance metrics in order to learn about the Web user's operational environment characteristics, changes in network connectivity between the user's computer and Web server, and assesses the consequences of these changes on the user's QoE. This information is synthesized in the Perceived Performance Model, which proposes strategies for tailoring Web content in order to optimise QoE.

In order to demonstrate the benefits of the proposed QoE layer we have deployed it in the open-source AHA! system creating Quality of Experience-aware AHA!(QoEAHA). In this paper we present results from subjective evaluation in the educational area. The goal of this evaluation was to assess the learning outcome, learning performance, system usability and user QoE when the original AHA! and the QoEAHA systems are used in a low bit rate home-like environment. The results indicated that QoEAHA significantly improves performance and user satisfaction with their experience. The usage of the QoE-layer did not affect the user-learning outcome.

2 Quality of Experience

The term Quality of Experience (QoE) relates to end-user expectations for QoS. QoE is defined by Empirix as the collection of all the perception elements of the network and performance relative to expectations of the users. The QoE concept applies to any kind of network interaction such as Web navigation, multimedia streaming, voice over IP, etc. Depending on the type of application the user interacts with, different QoE metrics that assess the user's experience with the system in term of responsiveness and availability have been proposed. QoE metrics include subjective elements and can be influenced by any sub-system between the service provider and the end-user. ITU-T Recommendation G.1010 provides guidance on the key factors that influence QoS from the perspective of the end-user (i.e. QoE) for a range of applications that involves voice, video, images and text.

In the area of World Wide Web applications, QoE has been also referred as end-to-end QoS or end-user perceived QoS. Measuring end-to-end service performance, as it is perceived by end-users is a challenging task. Previous research (Bhatti et al. 2000, Krishnamurthy et al. 2000, Bouch et al. 2000) shows that many QoS parameters such as download time, perceived speed of download, successful download completion probability, user's tolerance for delay, and frequency of aborted connections factor into user perception of provided quality. Measurement of these parameters may be used to assess the level of user satisfaction with performance. The interpretation of these values is complex, varying from user to user and also according to the context of the user task.

End-user perceived QoS has also been addressed in the area of multimedia streaming. Research such as (Blakowski 1996, Ghinea 1998, Watson 1997) assesses the effect of different network-centric parameters (i.e. loss, jitter, delay), the continuous aspect of multimedia components that require synchronization, or the effect of multimedia clip properties (i.e. frame size, encoding rate) on end-user perceived quality when streaming different type content.

In this paper QoE is addressed only in the area of Web-based AHS with applicability in education. Typical e-learning systems may involve a combination of text, images, audio and video, and their quality of service is based on the combination of all of these rather than any individual component. The educational context also has its own set of requirements and user expectations in terms of learning outcome and it is against these that user perceptual quality will be evaluated.

3 QoE-aware Adaptive Hypermedia System for Education

Starting from a generic architecture of an AHS that consists of a Domain Model (DM), a User Model (UM), an Adaptation Model (AM), and an AHS engine (Wu 2001) we have enhanced the system with a QoE layer that was presented in Muntean 2004a and Muntean 2004b. The QoE layer includes the following new components (see Figure 2): the Perceived Performance Model (PPM), the Performance Monitor (PM), the Adaptation Algorithm (AA) and the Perceived Performance Database (PP DB).

figure 2

Figure 2. QoE-aware AHSE Architecture

3.1 Performance Monitor

The PM is in charge of monitoring and measuring in real time performance metrics which are then used to infer information regarding user QoE. the performance metrics include download time, round-trip time, throughput and user behaviour-related actions (e.g abort requests). The utility of a session (Bouch et al. 2000) is also calculated and reflects the fact that users become less tolerant to delay as time passes. Tests over high speed connections showed that a 10 sec download time was considered acceptable to 95 % of the participants during the first four Web page accesses, still acceptable for 80 % of the participants during the access of an extra 6 pages, but only for 60 % of accesses over the 11th page were still acceptable (Bouch et al. 2000). Similar tests were performed for download time values between 16 sec and 6 sec and for different type of connections. The conclusion of these results was that the download time should improve over the duration of a session in order to keep acceptable the client's experience with navigation on a web site positive.

The information gathered by the PM during the user access sessions is delivered to the PPM. The mechanism used to measure the performance metrics is based on filtering TCP packets that carry information and monitoring the signals exchanged by the HTTP protocol.

3.2 Perceived Performance Model and Perceived Performance Database

The PPM has the important function of providing a dynamical representation of the user perceived QoE. It models the performance related information in order to learn about the user operational environment characteristics, about changes in network connection and the consequences of these changes on the user's quality of experience. PPM also considers the user's subjective opinion about his/her QoE explicitly expressed by the user. This introduces a degree of subjective assessment, which is specific to each user. The user related information is modelled using stereotype-based technique that makes use of probability and distribution theory (Muntean 2004a) and saved in the PP database.

Finally, the PPM suggests the optimal Web content characteristics (e.g. the number of embedded objects in the Web page, the dimension of the based-Web page without components and the total dimension of the embedded components) that would best meet the end-user expectation related to QoE. PPM aims to ensure that the access time per delivered page, as perceived by the user, respects the user tolerance for delay and it does not exceed the satisfaction zone.

Based on a survey of the current research into user tolerance for delay, three zones of duration that represent how users feel were proposed in (Sevcik 2002): zone of satisfaction, zone of tolerance and zone of frustration. According to a number of studies (Bhatti et al. 2000, Bouch et al. 2000, Servidge 1999, Ramsay et al. 1998) on the effects of download time on users' subjective evaluation of the Web site performance it was indicated that users have some thresholds (user tolerance) for what they consider adequate or reasonable delay. A user is "satisfied" if a page is loaded in less then 10-12 sec, but higher values cause disruption and users are distracted. Any delay higher then 30 sec causes frustration. At the same time it is significant to mention that when the user is aware of the existence of a slow connection, he/she is willing to tolerate a delay that averages 15 sec but does nor exceed 25 sec (Chiu 2001).

3.3 Adaptation Algorithm

The objective of the Adaptation Algorithm (AA) is to determine and apply the correct transformations on the personalised Web page (according to the User Model) in order to match the PPM suggestions on the Web page characteristics. Two types of transformations are considered: modifications in the properties of the embedded components (presented as concepts in the DM) and/or elimination of some of the components. These actions are applied to those components the user is the least interested in as recorded by the UM. The work presented in this paper considers that the Web pages consist of text and images. Since images contribute with the largest quantity of information to the total size of a web page, in this work they were the only ones taken into consideration by this algorithm.

In order to match the PPM suggestion related to the total size of the embedded images, image compression is first applied and, if further reduction is necessary, image elimination is applied. Different compression rates (expressed as percentage) are applied to each image depending on: the total reduction suggested on the total size of embedded images, the image size and user interest in the image as specified in the UM. Thus, if a user is more interested in image A than image B, image A will be reduced less than image B. If one of the computed compression rates cannot be applied to an image (e.g. due to the fact that the quality will be lower than acceptable for the end-users) an image elimination strategy is applied. In the case when an image has to be eliminated, a link to the image is introduced. In this way, if a user does really want to see the image, the link will offer this possibility.The algorithm used for image compression/elimination for the tests in this paper is described in Muntean 2004c. Naturally, the quality of the image relative to its size will depend on the sophistication of the compression technique, as is the decision regarding user perception of the image quality. This is a subject of ongoing research.

Further extension of the algorithm may consider multimedia clips (audio and/or video) that could be embedded in a Web page. For this situation, techniques that involve size and quality adjustments for audio and video can be applied (e.g. for video compression techniques involving frame rate, resolution and colour depth modifications and respectively for audio silence detection and removal technique). These adaptation techniques are studied by the multimedia networking area and they are not addressed in this paper. In addition, a component elimination strategy may be replaced by one of substituting a less bandwidth intensive equivalent for the information eliminated. For example, if video clips could not be supported, images or a sequence of images could be sent instead.

4 Assessing the Benefits of the QoE Layer

For illustration and testing purposes the proposed QoE Layer was deployed on the open-source AHA! system, creating QoEAHA. The AHA! system was developed at the Eindhoven University of Technology, in the Database and Hypermedia group. The system is used in educational area as an adaptive hypermedia courseware application that supports the "Hypermedia Structures and Systems" course (TU/e course, DeBra 1997). AHA! was used in order to demonstrate the benefits brought by the QoE Layer.

Among the advantages of the AHA! system are the following:

  • AHA! has been extensively tested and accepted by the research community.
  • AHA! is a simple general-purpose hypermedia adaptive system.
  • The AHA! architecture respects the general reference AHAM model (DeBra 1998).
  • AHA! is open source and therefore allows for extensibility.
AHA! system also provides an adaptive tutorial as testing material. The content of this tutorial was used as educational material for the students in the experimental tests performed during our research. As the material was already designed prior to the proposal of the QoE Layer, it provides independent testing material for the subjective evaluation.

4.1 Objectives of the Evaluation Experiment

The QoE evaluation investigates the feasibility and usability of applying the QoE Layer in order to support performance-based adaptation. This adaptation is performed based on the end-user perceived performance and their experience with the adaptive system when interacting with the system in a low bit rate operational environment (connection bandwidth up to 128 kbps).

The objectives of the experiment were the following:

  • To investigate the impact of QoE extension on student performance
  • To assess the usability and effectiveness of the QoEAHA system in comparison to the original AHA! system
  • To determine the improvements brought by the QoEAHA in terms of end-user QoE

The impact of the QoE Layer on student performance was investigated by comparing the performance of the students when the two systems AHA and QoEAHA were used. Students' performance was assessed in terms of the two most important metrics: learner achievement and learning performance. The number of revisited web pages was also investigated.

Usability evaluation was performed through an on-line usability questionnaire filled-out by the students after they completed a study task.

The analysis of students QoE was performed through an on-line questionnaire that assessed the user opinion in relation to performance issues and user satisfaction on the perceived quality.

4.2 Setup Conditions

The experiments took place in the Performance Engineering Laboratory, School of Electronic Engineering, Dublin City University. A task-based scenario involving an interactive study session was developed and carried out in laboratory settings. The test environment was designed to be uniformed for all participants. The tests took place in closed medium-size laboratory room where no other people were allowed in and no other activities were performed. The room had no windows and the level of artificial light was the same for all participants.

The laboratory-network setup used for testing involved four desktops PC Fujitsu Siemens with Pentium III (800MHz) processors and 128 MB memory, a Web server IBM NetFinity 6600 with two processors Pentium III (800 MHz) and 1GB memory and one router Fujitsu Siemens with Pentium III (800MHz) processor and 512 MB RAM that has a NISTNET network emulator installed on it. The NISTNET instance that allows for the emulation of various network conditions characterized by certain bandwidth, delay and loss rate and pattern was used to create a low bit rate modem-like operational environment with a 56 kbps connection speed (Figure 3). This setup offers similar connectivity to that experienced by residential users. The emulated network conditions determined performance-related adaptations when the QoEAHA was used.

figure 3

Figure 3. Laboratory-Network Configuration for the Subjective Testing

The subjects involved in this study are comprised of forty-two postgraduate students from the Faculty of Engineering and Computing at Dublin City University. They were randomly divided into two groups. One group used the original AHA! system, whereas the second one used QoEAHA. The subjects were not aware of what system version they were using during the experiment. No time limitation was imposed on the execution of the required tasks. None of the students had previously used any of the two versions of the AHA! system and none of them has accessed the test material prior taking the tests. Therefore no previous practice with the environments was assumed for any of them. The material on which the students performed the task consisted of the original adaptive tutorial delivered with the AHA! system version 2.0.

Interactive Study Session

The students were asked to complete a learning task that involved the study of the AHA! installation chapter from the AHA! tutorial over a 56 kbps connection speed. At the start of the study session the subjects were asked to read a short explanation concerning the use of the system and the required duties. Their duties were as follows:

  • Complete an online Pre-Test evaluation consisting of a questionnaire with six questions related to the learning topic. The test is used to determine subject's prior knowledge in this domain.
  • Log onto the system and proceed to browse and study the material.
  • Complete an online Post-Test evaluation at the end of the study period. The Post-Test consists of a questionnaire with fifteen questions that test recollection of facts, terms and concepts from the supplied material, as suggested in Bloom's taxonomy.
  • Answer a Usability Questionnaire consisting of ten questions related to navigation, accessibility, presentation, perceived performance and subjective feedback.

In order to fully assess the subjects learning achievement, both Pre-Test and Post-Test questionnaires (Muntean 2005) were devised from the four different types of test-items most commonly used in the educational area: "Yes-No", "Forced-Choice", "Multi-Choice" and "Gap-Filling" test items. These test items have different degrees of difficulty and their corresponding answers were assigned weights in the final score accordingly. The maximum score for Pre-Test is 10 points and the maximum score for Post-Test is 30 points. The final scores were normalized in the range of 0 to 10.

4.3 Learner Achievement

Learner achievement is defined as the degree of knowledge accumulation by a person after studying certain material. It continues to be a widely used barometer for determining the utility and value of distance learning technologies.

During the study-based scenario learner achievement was assessed by comparing Pre-Test and Post-Test scores achieved by the subjects using the QoEAHA and AHA! systems respectively. The results of the Pre-Test and Post-Test are shown in Table 1 and Table 2.

Table 1. Pre-Test Results
Mean Score
Min Score
Max Score
SDEV
AHA!
0.35
0.0
2.0
0.552
QoEAHA
0.30
0.0
2.0
0.530

Table 2. Post-Test Results
Mean Score Min Score Max Score
SDEV
AHA!
6.70
4.30
9.30
1.401
QoEAHA
7.05
4.60
9.0
1.395

A two-sample T-Test analysis, with equal variance assumed, performed on the Pre-Test scores shows that statistically both groups of students had the same prior knowledge of the studied subject (significance level alpha=0.01, t=0.21, t_critical= 2.42, p(t)=0.41). This result means that the learner achievement can be assessed by processing only the Post-Test score.

Following the Post-Test results evaluation, the mean score of the subjects that used QoEAHA was 7.05 and the mean score of those that used AHA! was 6.70. A two-sample T-Test analysis on these mean values does not indicate a significant difference in the final marks of the two groups of users. (alpha=0.05, t=-0.79, t_critical=1.68, P (t)=0.21). Therefore it can be stated that there is no significant difference in the learning outcome between the users of the QoEAHA and AHA! systems.

Since the answers for three questions from the Post-Test questionnaire required the subjects to study the images embedded in the Web pages affected by performance-based adaptations, an analysis of the students learning outcome on these questions was performed. After the scores related to these three questions were normalized in the range 0 to 10, the mean value of the students' scores was 6.30 for the QoSAHA group and 6.40 for AHA! group. A two-sample T-Test analysis, with equal variance assumed again indicates with 99% confidence level that there is no significant difference in the student learning achievements (t=-0.08, t-critical=2.71, p(t)=0.93, confidence level alpha=0.01). This result is very important as an adaptive degradation up to 34 % in the image quality was performed by the QoEAHA.

In summary, these test results indicate that the QoEAHA system did not affect the learning outcome and offered similar learning capabilities to the classic AHA! system.

4.4 Learning Performance

The term learning performance refers to how fast a study task (learning process) takes place. The completion time for a learning session (Study Session Time) is measured from the start of the session, when the subject logs into the system and starts to study until the student starts answering the Post-Test questionnaire.

The distribution of the Study Session Time measured for the students involved in performing the required learning task using the AHA! and QoEAHA systems respectively is presented in Figure 4.

figure 4

Figure 4. Distribution of the Study Session Time Measured for the Students Involved in the Learning Task

On average, students that made use of the QoEAHA system (Average Study Time = 17.77 min) performed better than the ones that used the AHA! (Average Study Time = 21.23 min). This fact was confirmed with 99% confidence level by the T-Test analysis. The very large majority of the students that used QoEAHA (71.43 %) performed the task in less than 20 minutes, with a large number of students (42.87 %) requiring between 15 and 20 minutes. In comparison, when the AHA! system was used, only 42.85 % of the students succeeded to finish the learning task in 20 min. The majority of them (71.42 %) completed in less than 25 minutes, with the largest number of students (28.57 %) completed in the interval 20-25 minutes. (Table 3).

Table 3. Percentage of Students that Have Succeeded to Learn over Different Periods of Time
Study Time Interval (mins)
QoEAHA

Number of Students (%)

AHA

Number of Students (%)

0-10
9.50
0.0
0-20
71.43
42.85
0-25
85.71
71.42
20-25
14.28
28.57
25-35
14.29
28.57

Apart of the Study Session Time, Number of Accesses to a page performed by a person was also measured in order to investigate the students learning performance. This metric can provide an indication on the quality of learning. Any re-visit to a page may indicate that the student was not able to recall the information provided in the page and thus the learning process was of poor quality. On average the students from the QoEAHA group performed a smaller number of re-visits (avg. =1.40) to a page than those from the AHA! group (avg. =1.73). An unpaired two-tail T-Test with unequal variance assumed, confirmed with 92% confidence that there is a significant difference in the number of visits performed by a student when the two versions of AHA! systems were used.

Another important metric for assessing the quality of the learning process is Information Processing Time per page (IPT/page). IPT represents the time taken by a student to read and assimilate the information displayed on a Web page. It was measured from the moment when the web page was delivered and displayed on the computer screen until the user sends a request for another page. The web page was not loaded in a progressive way. The test results indicated that on average a lower time per page (IPT=4.31 min) was spent by a student to process the information when QoE-aware version was used, in comparison to the case when AHA! system was used (IPT=4.95 min).

Summarising these results, the students that used the QoEAHA system had shorter Study Session Times than those that used the AHA! system. This was due to the fact that the material was delivered faster. Since the download time per page did not exceed the user tolerance for delay threshold, the students were constantly focused on their task, resulting in shorter Information Processing Time per page as well. Results showed that an improvement of 16.27 % in the Study Session Time for the whole learning session was obtained when the QoE-aware version was used. On average, an improvement of 26.5% on the access time/page was obtained when the QoE-aware version was used. An access time per page no higher than 12 sec provided by the QoE-aware system has ensured a smooth learning process. This observation is confirmed when assessing the number of re-visits to a page (on average19% decrease with QoEAHA) and information procession time per page (on average13% decrease provided with QoEAHA).

4.5 Usability Assessment

The main goal of the usability evaluation strategy is to measure the usability and effectiveness of the QoE-aware AHA system in comparison to the original AHA! system. The methodology of study involved the usage of the online questionnaire technique. This is one of the most widely used techniques in the education area.

At the end of the interactive study session both group of subjects were asked to complete an online usability evaluation questionnaire consisting of ten questions with answers on a five point scale (1-poor - 5-excellent). The questions were devised to respect the widely used guidelines suggested by Preece for evaluating Web sites. They relate to navigation, presentation, subjective feedback, accessibility and user perceived performance. The accessibility and user perceived performance questions assess the end-user QoE. Four questions of the survey relate to these two categories. These four questions assess user opinion in relation to the overall delivery speed of the system (Q6), the download time of the accessed information in the context of Web browsing experience (Q7), the user satisfaction in relation to the perceived QoS (Q9) and whether the slow access to the content has inhibited them or not (Q5). The results of the QoE related questions for both AHA! and QoEAHA systems are graphically presented in Figure 5.

figure 5

Figure 5. Usability Evaluation Results on Questions that Assessed the End-User QoE

As seen from the chart the QoEAHA system has provided a better QoE for the end users, improving the users' satisfaction, which was above the good level for all questions. The AHA! system scored just above the average level, significantly lower than QoEAHA. This good performance was obtained in spite of the subjects using a slow connection (56 kbps) during the study session and not being explicitly informed about this. Overall, the mean value of QoE usability assessment was 4.22 for QoEAHA and 3.58 for AHA!. This lead to an improvement of 17.8 % brought by the QoEAHA system. A two-sample T-Test analysis on the results of these four questions confirmed with a confidence level above 99 %, (p<0.01) that users' opinion about their QoE is significantly better for QoEAHA than for AHA!.

The usability assessment on the other questions related to the navigation and presentation features achieved an average score of 3.83 for AHA! and 3.89 for QoEAHA, demonstrating that these features were not affected by the addition of the QoE enhancements.

Finally, an overall assessment when all ten questions were considered of equal importance shows that the students considered the QoEAHA system (mean value=4.01) significantly more usable then the AHA! one (mean value=3.73). These results were also confirmed by the unpaired two-tailed T-Test (t=2.44, p<0.03) with a 97 % degree of confidence. This increase of 7.5 % in the overall QoEAHA usability was mainly achieved due to the higher scores obtained in the questions related to end-user QoE.

4.6 Correlation Analysis

One aspect worth examining is to determine whether or not there is any correlation between the performance of individual students and their perception of system usability. Therefore, the goal of this analysis that computes the Spearman coefficient is to examine if students that performed well in the Post-Test evaluation, thought that the system was more usable, while students with much lower scores expressed bad opinions. A strong correlation between the two set of results (Post-Test and Usability) would discredit to a certain extent the results of the usability evaluation experiment.

The correlation analysis has been performed for both QoEAHA and AHA! systems. For the QoEAHA the value of the Spearman coefficient was rs=0.23 while for the AHA! this value was rs=0.03. Values lower than 0.33 indicate a weak correlation, while values higher than 0.67 indicate a strong correlation. As both computed coefficients have values lower than 0.67 there is no strong correlation between the two data sets.

Summarising the results, no correlation has been found between the students learning outcome and their judgment on the system usability. The opinions expressed by the students in the usability questionnaire were not influenced by their final scores in the Post-Test evaluation.

5 Conclusions

This paper has proposed the Quality of Experience (QoE) as another dimension of user characterisation that should be taken into consideration by the personalization process provided by adaptive hypermedia applications. QoE is directly influenced by the operational environment through which the user interacts with the AHS (bandwidth, delay, loss, device capabilities, etc) and by the user subjective assessment of their perceived performance. The goal of any AHS should be not only to provide the content that would best suit the user's goals, knowledge or interest but also to provide the best content that would fit the user's operational environment. In this context we have proposed a QoE-Layer enhancement for AHS that analyses some key factors that influence QoE and makes a correlation between their values and Web page characteristics that provide the best QoE for the end-user.

The QoE layer was implemented as an independent module providing an extra layer of adaptation (performance-based content adaptation) on a personalised content generated by an AHS. It can be easily integrated with a classic AHS that respects the following conditions: the application domain is defined as a collection of concepts and concept relationships (Domain Model) and the system builds a user profile and maintain a sorted list (e.g. percentage) of the user interest in the concepts defined in the Domain Model.

For evaluation purposes QoE Layer was deployed on the open-source AHA! system, creating the QoEAHA. QoEAHA was tested in the educational area. This paper presents a study on the impact of the usage of the QoE Layer as part of an adaptive e-learning system when the students access the educational material using a low bit rate home-like operational environment. Different educational-based evaluation techniques such as learner achievement analysis, learning performance assessment, usability survey and correlation analysis between individual student performance and judgment on system usability were applied in order to fully assess the performance of the QoEAHA.

The most significant conclusions drawn from the subjective testing presented in this paper are the following:

  • The usage of QoE Layer brought significant learning performance improvements. These improvements were achieved as the educational material is delivered faster and the students are constantly focused on performing their tasks. Long periods of waiting for a required page annoy people and disturb their concentration on the task.
  • 16.27 % improvement in the execution time of a learning task was obtained when the QoEAHA was used. Most of the QoEAHA group students (71.43 %) finished the task in less than 20 minutes, while only 42.85 % of the AHA! group students finished in the same period of time.
  • The students from QoEAHA group performed on average a smaller number of re-visits to a Web page than those from the AHA! group.
  • An overall assessment of system usability has shown that students considered QoEAHA system significantly more usable than AHA!. QoEAHA achieved a 7.5 % increase in the usability survey results due to the higher marks awarded on the QoE-related questions. Focusing only on the end-user QoE-related usability an improvement of 17.8 % was obtained.
  • QoE Layer does not affect the learning outcome of the students. Both groups of students received similar marks on the final evaluation test. Therefore, the QoEAHA offers similar learning capabilities to the classic AHA! system.

6 Further Work

The proposed QoE layer has been shown to bring improvements to learning performance for material consisting of text and images delivered in a low bitrate environment. Further work is necessary to explore its effectiveness in a wider range of situations. Two possible directions are the extension to multimedia content where performance problems may arise even in higher bandwidth environments and the application to Adaptive Hypermedia Systems in areas other than education.

The delivery of multimedia content to end-users over heterogeneous networks with variable delivery conditions presents significant challenges. We intend to broaden the use of the QoE layer to applications that deliver personalised multimedia content to e-learners. The extended QoE layer will monitor and analyse in real-time the values of multimedia-streaming-related parameters (e.g. delay, loss, jitter, end-user perceived multimedia quality estimation metrics) and will make suggestions about the optimal type and characteristics of multimedia stream (bit rate, frame rate, resolution) delivered to the user in order to provide a good level of QoE. The adaptation algorithm will consider new adaptation strategies for the multimedia clips that may involve size and quality adjustments techniques for audio and video, and in the worst case, substitution of alternative forms of material such as sequences of images.

Another direction to explore is to combine the QoE layer with adaptive hypermedia systems applied in other areas such as on-line information systems and to investigate usability and benefits brought by the new system. These systems differ from the educational ones by providing a bigger navigational space, a higher flexibility to the users to navigate in the hyperspace and the users of the system have different objectives.

References

AHA! Project http://aha.win.tue.nl/

Ardissono, L., Goy, A., Petrone, G., Segnan, M., Torasso, P., (2002) "Ubiquitous User Assistance in a Tourist Information", 2nd International Conference on Adaptive Hypermedia and Adaptive Web Based Systems (AH2002), Malaga, Spain, pp. 14-23

Bhatti, N., Bouch, A. and Kuchinsky, A. (2000), "Integrating User Perceived Quality Into Web Server Design". Computer Networks Journal, Vol. 33, No. 1-6, pp.1-16

Blakowski, G. and Steinmetz, R. (1996), "A Media Synchronisation Survey: Reference Model, Specification, and Case Studies". IEEE Journal on Selected Areas in Communications, Vol. 14, No.1, pp. 5-35

Bloom, B. S., Mesia, B. B. and Krathwohl, D. R. (1964), "Taxonomy of Educational Objectives (two vols: The Affective Domain & The Cognitive Domain)", New York. David McKay Inc.

Brusilovsky, P., (2001), "Adaptive Hypermedia", User Modeling and User-Adapted Interaction Journal, Vol. 11 No. 1-2, pp. 87-110

Brusilovsky, P., (1996), "Methods and Techniques of Adaptive Hypermedia", User Modeling and User-Adapted Interaction Journal, Special Issue on Adaptive Hypertext and Hypermedia, Vol. 6, No. 2-3, pp. 87-129

Bouch, A., Kuchinsky, A. and Bhatti, N. (2000), "Quality is in the Eye of the Beholder: Meeting Users' Requirements for Internet Quality of Service". In Proceedings of the ACM CHI 2000 Conference on Human Factors in Computing Systems, Hague, Netherlands

K. Cheverst, K., Mitchell, K., Davies, N., (2002) "The Role of Adaptive Hypermedia in a Context-Aware Tourist Guide", Communications of the ACM Journal, Vol. 45, No. 5, pp. 47-51

Chiu, W. (2001), "Best Practices for High Volume Web Sites", IBM RedBooks

De Bra, P. (1997), "Teaching Through Adaptive Hypertext on the WWW". Journal of Educational Tele-communications, Vol. 3, No. 2/3, pp.163-180

De Bra, P. and Calvi, L. (1998), "AHA! An Open Adaptive Hypermedia Architecture". The New Review of Hypermedia and Multimedia, Vol. 4, pp. 115-139

Empririx, "Assuring QoE on Next Generation Networks", White paper

Ghinea, G. and Thomas, J. P. (1998), "QoS Impact on User Perception and Understanding of multimedia Video Clips". In Proceedings of ACM Multimedia Journal, Bristol, UK, pp. 49-54.

Krishnamurthy, B. and Wills, C.W. (2000) "Analysing Factors that Influence End-to-End Web Performance". Computer Networks Journal, Vol. 33, No.1-6, pp. 17-32

ITU-T Recommendation G.1010 (2001) "End-User Multimedia QoS Categories"

Merida, M., Fabregat, R., Matzo, J.L, (2002), "SHAAD: Adaptible, Adaptive and Dynamic Hypermedia System for Content Delivery", 2nd International Conference on Adaptive Hypermedia and Adaptive Web Based Systems (AH2002), Workshop on Adaptive Systems for Web-based Education Málaga, Spain

Muntean, C., H. and McManis, J. (2004a) "A QoS-Aware Adaptive Web-based System", In Proceedings of the IEEE International Conference on Communications (ICC04), Paris, France

Muntean, C., H. and McManis, J. (2004b) "QoSAHA: A Performance Oriented Learning System". In Proceedings of AACE ED-MEDIA '04 Conference, Lugano, Switzerland

Muntean, C., H. and McManis, J. (2004c) "End-User Quality of Experience Layer for Adaptive Hypermedia Systems", In Proceedings of 3rd International Conference on Adaptive Hypermedia and Adaptive Web-based systems, Workshop on Individual Differences in Adaptive Hypermedia, Eindhoven, The Netherlands, pp. 87-96

Muntean, C. (2005) Pre-Test and Post Test Questionnaires for QoEAHA Evaluation http://www.eeng.dcu.ie/havac/QoEAHAEvalForms/

Ng, M., H., Hall, W., Maier, P., Armstrong, R., (2002), "The Application and Evaluation of Adaptive Hypermedia Techniques in Web-based Medical Education", Association for Learning Technology Journal, Vol.10, No. 3, pp. 19-40

Papanikolaou, K., A., Grigoriadou, M., Kornilakis, H., Magoulas, G. D. (2003), "Personalizing the Interaction in a Web-based Educational Hypermedia System: the Case of INSPIRE", User Modeling and User-Adapted Interaction Journal, Vol. 13, No. 3, pp. 213-267

Preece, J. (2000) "Online Communities: Designing Usability, Supporting Sociability". In John Willey & Sons (eds), Chichester UK

J. Ramsay,J., Barbasi, A., Preece, J. (1998), "A Psychological Investigation of Long Retrival Times on the World Wide Web", Interacting with Computers Jurnal, Elsevier Ed., March.

Servidge, P. (1999), "How Long is Too Long to Wait for a Web Site to Load?", Usability News

Sevcik, P. J. (2002) "Understanding How Users View Application Performance", Business Communications Review, Vol. 32, No. 7, pp. 8-9

Triantafillou, E., Pomportsis. A, and Georgiadou, E. (2002), "AESCS: Adaptive Educational System Base on Cognitive Styles", 2nd International Conference on Adaptive Hypermedia and Adaptive Web Based Systems (AH'2002), Workshop on Adaptive Systems for Web-Based Education, Malaga, Spain, pp.10-20.

TU/e course 2L690 Hypermedia Structures and Systems, http://wwwis.win.tue.nl/2L690/

Watson, A., Sasse, M. A. (1997), "Multimedia Conferencing via Multicasting: Determining the Quality of Service Required by the End User". In Proceedings of AVSPN '97, Aberdeen, UK, pp. 189-194.

Weber G., Brusilovsky, P., (2001), "ELM-ART: An Adaptive Versatile System for Web based Instruction", Artificial Intelligence in Education, Special Issue on Adaptive and Intelligent Web-based Educational Systems, Vol. 12, No. 4, pp. 351-384

Wu, H., De Kort E., De Bra, P. (2001), "Design Issues for General-Propose Adaptive Hypermedia Systems". In Proceedings of the 12th ACM Conference on Hypertext and Hypermedia, Aarhus, Denmark, pp.141-150