Presenters: Ted Sienknecht, Marcia Kerchner

Concept: We're overlooking a wealth of usability-related data we have on hand!

Where to gather information?

. Lessons learned: Study system usage statistics and user feedback from current/prior releases for possible improvements and functionality
. Software evaluations: Study other software users currently use for implementations they expect or understand
. Field observations: Watch users while they perform relevant tasks and note process, actions, systems, problems, needs, etc.
. Interviews & Focus groups: Use structured inquiry with users about their opinions and experiences
. Task analysis: Investigate typical tasks users perform on the system
. User profiles: Create representative identities for user subgroups (personas)
. Help desk logs: Read help requests for areas for improvement or new functionality

How to evaluate?

. Cognitive walkthrough: Construct scenarios based on task analysis and user profiles and then “walk through” the interface as a typical user
. Heuristic evaluation: Inspect elements of the interface based on established usability principles
. Group walkthrough: Step through task scenarios together (users + developers), discussing and evaluating interaction
. Scripted observation (Formal usability testing): Present users with tasks and observe their interactions with the system, minimizing intervention

Approach for Mining Usability

1. Survey organization for sources of feedback:
. Call/email/chat logs from customer contact centers/help desk
. Search engine logs (e.g., top terms, terms with no results)
. Web quality assurance reports
. Web statistics and site usage data
. Satisfaction surveys
. External studies, market surveys
. Emails to webmaster/points of contact
. Bug reports and feature requests
. Previous usability studies

2. Obtain access to sources, collect data
. Usually available in some sort of user-readable report (though access to raw data may be desirable)
. In larger organizations, data sharing may require extra work to address logistical, technical, security/privacy concerns
. Identify who will approve, and who will actually provide data
. Define what data delivery format(s) are usable, but may initially need to just ingest whatever is given/available

3. Clean up, aggregate, and synthesize data
. Ask data “owner” about accuracy, reliability of data and any known quirks or trends
. Combine data in a spreadsheet, database, or tool of choice
. Elbow grease and/or automated tools to clean up data and put in user-readable format (search, sort, filter, categorize)

4. Identify trends/issues
. Scan/read through data (or a representative sample for very large data sets) to get familiar with nature of feedback
. Note items of interest, begin formulating hypotheses and identifying trends
. Use manual analysis, tools (e.g., frequency counts, sorting/filtering, search) to evaluate hypotheses and trends

5. Define priority for identified issues
. For usability mining, issues include not only the expected usability problems, but also those items/trends that merit further investigation or study (i.e., to fully define them)
. Use your preferred usability severity rating scale for issues that are sufficiently defined as to be evaluated that way
. Remember: Because these issues were derived from customers, the frequency count for a given issue is a good indicator of the magnitude of the issue (in terms of exposure, or number of customers affected)

6. Prepare report of prioritized issues with supporting data
. Use the preferred reporting method/template
. Where appropriate, indicate the frequency of occurrence for issues
. Overarching recommendations:
  . Incorporate usability engineering efforts for issues requiring further study in next cycle/iteration
  . Formalize the data mining process, including incorporating lessons learned
  . Consider tweaking/adding to feedback data that is captured (may include coding to log/collect more usability-related data)

7. Develop remediation plan
. Use report to prioritize activities
. Draw on your usability toolkit to address code/chrome/content issues
. Because users are increasingly using search (beyond the engine on your site), consider:
  . Content engineering (e.g., descriptive titles/headings, correct spelling, consistent nomenclature and naming conventions, minimized jargon, thoughtful structure metadata use)
  . Search engine enhancements (e.g., adjusting weightings, prioritizing results, enabling semantic capabilities such as synonyms and related terms, categorizing results)