Testing and Evaluation of initial eRaUI Prototype - WP4

On Monday the 28th November 2011 we undertook the first formal testing of an eRaUI protoype using MSc students at the London Metropolitan University Faculty of Computing. This was a very small initial test and involved 4 students, none of whom had any prior experience of eRaUI or the concepts related to this project.

The testing process consisted of assigning a set of 8 questions to each of the 4 participants. The participants answered a subset of these questions using a proxy version of the NaCTeM website on which the eRaUI widget was embeded, and another subset consisting of a proxy of the website without the widget. eRaUI recording and analysis tools were deployed on both proxy websites. The questions were divided up so that each participant answered 4 questions using the website with the widget and 4 questions on the version without the widget. Users were given a maximum of 2 minutes to complete each question, and the results and completeness of users' answers were recorded so as to allow comparison of the efficacy of embeding the eRaUI widget on the NaCTeM interface as compared to using the interface without the widget.

Upon completion of the 8 questions, participants were asked to complete a quick questionnaire asking questions as to which version of the website facilitated a better perception of user experience. The participants were then invited to give feedback by means of a brief informal spoken interview.

This intitial testing process has yielded some very useful feedback and data which can be analysised so as to best know how to go forward with the research and development of the eRaUI widget tool. Results of this analysis, which will include the recordings produced by the eRaUI analysis tools, are pending and will be disseminated in a forthcoming blog post.

WP2 Meeting - 24th Nov 2011

On the 24th November 2011 Eamonn and Farhi met to discuss work package 2 deliverables and progress. We discussed the implementation of Nearest Neighbour algorithms within eRaUI to match content to particular users based on automatic user categorisations. We also showed how text mining has been implemented in such a way that we are able to extract information from user sessions and structure the information so as to inform automatic categorisation of users. The process of extracting and structuring information is undertaken in realtime as users navigate the interface on which eRaUI has been deployed.

We also discussed testing plans for the prototype eRaUI widget. We shall be testing the widget next week.

WP2 Meeting 16th November

Farhi and Eamonn met up on the 16th November to discuss Work Package 2 deliverables and progress. First of all we discussed the proposed eRaUI logo which is currently under development. We have decided to adopt an arrow design using some elements from the London Metropolitan University logo, including the colours. This logo will be unveiled shortly once final considerations have been addressed.

The next point of discussion centred around the design and implementation of Nearest Neighbour algorithms within eRaUI. We discussed a current model which is now operational, and also considered a new model which will involve attributing a weighting to each link within a web application. These weights will then be summed wherever there is a 1 to 1 match between a user's click history and each previous existing use case. This will form the kernel of a sophisticated algorithm which should enable us to classify users most effectively.

Work Packages 1 and 2 - Meeting

Today (10th November) Farhi and Eamonn met to discuss Work Package 1 and 2 deliverables and progress. Primarily our conversation centered on the following two areas:

• Configuration of the university server to host the NaCTeM codebase. Difficulties have been encountered in enabling PHP short tags so as to allow the current codebase to execute correctly. This problem has now been resolved and the code is running according to expectations.
• Implementation of nearest neighbour algorithms within eRaUI in order to classify users into the correct level. Work has been done on this already, and we decided that we should store in the database a historical record of users matched to url clicks and categories. This can be used to generate a case-base which will then be used to categorise users as novice, student, expert etc.

We have also proposed a test plan for the eRaUI widget (see previous blog post) which forms a component of our WP4 deliverables.

eRaUI Testing and Prototype Evaluation Plan - Work Package 4

As part of our Work Package 4 deliverables we require a means of testing and evaluating eRaUI to try to quantify the effectiveness of the prototype which has been developed so far. Although we cannot yet gain interactive user feedback in regard to the finished software, it would be useful to test the efficacy of the current model in regard to enhancing user experience and productivity on the NaCTeM website. To this end, we propose to do the following:

1) Assemble a test group of subjects, perhaps only 4 persons for the initial test. These could be staff, students or Ph.D. students, although it would be preferable not to involve persons who have direct experience with eRaUI already.

2) Decide upon a set of three tasks which could be performed on the NaCTeM website. These would vary in complexity and could for instance involve:

• Sending some feedback via the feedback form.
• Looking for a paper using the text mining tools on the NaCTeM website.
• Downloading U-Compare and learning how to use it to complete a simple text-mining task.

3) Divide the test group into two halves. One half would be asked to complete the three tasks on a NaCTeM clone website without the widget enabled, and the other would be asked to complete the task on a version of the site with the widget.

4) Using the recordings produced from the test user interaction along with times taken and verbal feedback, we could assess to what extend the eRaUI widget improves the experience of interaction with the NaCTeM website, and to what extent it improves the times taken and the correctness of results obtained using the site alone as opposed to using the widget-enabled version.

This simple method of testing should hopefully enable us to glean some insights into the initial reactions to eRaUI and will form a vital part of our Work Package 4 deliverables. Later as the software develops we can expand this testing methodology so as to incorporate a larger test group and a wider range of tasks.

Meeting 3rd November - WP1 and WP2

Farhi and Eamonn met up today (3rd November 2011) to discuss WP1 and WP2 deliverables and progress. We discussed the various softwares which might be required for extending the functionality of eRaUI. We are in need of a solution which can programatically replicate the action of a web browser as a background task so as to assist in the server-side generation of heat maps. We also discussed the need to port the current code base from its existing server to the universities own equipment. This task shall hopefully be completed within the next several days.

eRaUI FoC Seminar - 2nd Nov 2011

On the 2nd of November we gave our first seminar demonstrating the concept and software of eRaUI for an audience of around 30 professors, students and staff from London Metropolitan University. This gave us an opportunity to gain useful feedback as to the current concept of eRaUI and to unveil some of the features we have been working on for the past weeks.

First Farhi Marir gave a slide presentation outlining the concept of eRaUI along with screenshots of the widget and administrative interface. After this we proceeded to a technical demonstration of most of the currently working aspects of eRaUI, including the analytics (recordings, heatmaps etc.) and the widget. We also included a live demonstration of eRaUI inviting members of the audience to connect to the NaCTeM website using their smartphones so as to monitor their activity. We saw a demonstration of the 'live chat' aspect of eRaUI in this way, with members of the audience sending and receiving messages via the widget.

The seminar gave us the opportunity to gauge overall interest in the project which has been on the whole overwhelmingly positive. We were also able to field some interesting questions and suggestions from members of the audience. A particularly interesting question considered the feasibility of making eRaUI accessible to users via any website. This is indeed technically feasible using a browser extension or toolbar although it raises some obvious privacy concerns because of the nature of the data collected.

There have been no weekly meetings for two weeks due to preparations for this seminar and work to complete the initial eRaUI prototype. After the seminar the eRaUI team met to discuss the feedback received during the seminar and to consider ways in which we shall progress. It has been established that we shall attempt to consolidate our WP2 goals by the introduction of Nearest Neighbour algorithms and data mining techniques to match users to the the content they require.