Download Notess_Swan-CHI_03.pdf

Contents : Predicting User Satisfaction from Subject Satisfaction Mark Notess University Information Technology Services Cook Music Library Indiana University Bloomington IN 47405 USA mnotess@indiana.edu Margaret B. Swan School of Library and Information Science 1320 E. 10th Street Indiana University Bloomington IN 47405 USA mbswan@indiana.edu ABSTRACT Variations User Satisfaction In this paper we describe work-in-progress in comparing user satisfaction ratings after user tests with ratings obtained following actual use of a digital music library software. We identify some of the variables that hamper prediction and we reflect on the utility of surveys for predicting user/subject gaps in satisfaction. Students in Cook Music Library were recruited as they entered the computer lab. 30 students who were planning to use Variations during their visit filled out QUIS-style 1 surveys after completing their planned work. They were asked to base their responses on the work they accomplished during that session. Most users reported using Variations 1-5 times each week and half of the users had first used Variations over two years previous. Keywords User testing satisfaction questionnaire digital library. INTRODUCTION Questionnaires measuring user satisfaction are a frequent accompaniment to user testing. A recent article by Lewis 3 reports on five years use of the PSSUQ instrument during user tests at IBM. This and other articles focus on the reliability and generalizability of the various instruments. Our own work has a different interest: Can these satisfaction data gathered during user testing predict satisfaction in real use Measuring satisfaction during lab tests can help identify interface issues or guide selection between alternate designs but if it cannot predict realworld usage satisfaction one might well question the value of such measurements and of user testing generally. A secondary research interest is exploring the value of satisfaction questionnaire findings generally. Our current work compares satisfaction of user test subjects (subject satisfaction) with ratings gathered from people using software for their normal tasks in a normal context (user satisfaction). The software in question is successive versions of digital music library software Variations 2 and Variations2 4 used by music students in the Indiana University Simon Music Library. THE STUDIES To date we have completed three studies: a baseline of user satisfaction with the present Variations software subject satisfaction during a usability test of Variations2 and user satisfaction with Variations2 during pilot usage in a real course assignment. COPYRIGHT IS HELD BY THE AUTHOR/OWNER(S). CHI 2003 APRIL 5 10 2003 FT. LAUDERDALE FLORIDA USA. ACM 1-58113-630-7/03/0004. Variations2 Subject Satisfaction Ten subjects were recruited from three music classes to participate in a Variations2 user test. Subjects were given two representative tasks to complete during the test which lasted approximately one hour. None of the subjects had previously used Variations2 but all were frequent users of Variations. After completing the test subjects filled out a satisfaction questionnaire. Variations2 User Satisfaction Students in a graduate vocal literature class were asked to complete a listening assignment using Variations2. Some improvements to Variations2 were made part-way through the study based on lab test results. Students were asked to fill out upon completing the assignment an online satisfaction survey. Twelve of the approximately 30 students in the class filled out the survey. RESULTS Table 1 shows the Likert scale data for the three studies. Two comparisons are of interest. First we compare the usage data from Variations (V1 Use) and Variations2 (V2 Use). The mean rating is lower for the new software although only one of the item differences straightforwardness of doing tasks approaches significance t(21.5) 2.048 p . 05 two-tailed. The second comparison of interest is the difference between the subject and user satisfaction for Variations2. Test subjects (V2 Test) rated the software lower than did realworld users (V2 Use) but no differences approached significance at the .05 level. Of primary interest is whether the V2 Test data correlate well with the V2 Use data can test satisfaction data predict satisfaction in real use A Pearson correlation revealed a low to moderate positive correlation (r .51) between the two data sets as shown in Figure 1. Table 1 - Mean Item responses Questionnaire Item (1-7 Likert scale) 1. overall: terrible wonderful 2. overall: difficult easy 3. overall: frustrating satisfying 4. overall: dull stimulating 5. overall: slow fast 6. component navigation difficult easy 7. tasks can be performed in a straightforward manner never always 8. my location within system never/always apparent 9. characters hard/easy to read 10. organization of information confusin

INFO HASH : cc01dbae5666c5c65ad31863256877d8c2e0ce80

Download now

File Size: 193 kb