Skip to Content
SBMI Horizontal Logo

Turf Quick Help

" Please click on +/- sign to expand/collapse the sections "

  • 1-TURF

    • What is Turf?

      Turf is an EHR usability toolkit software developed by SHARPC at the University of Texas Health Science Center at Houston. It is an all-in-one integrated software system and the first software program designed specifically for safety-enhanced design validation for EHRs. Turf hosts a variety of features for usability testing of an EHR by streamlining User Testing and Heuristic Evaluation using descriptive analysis, inferential statistics, etc. The methodology of Turf is based upon the TURF framework of four analyses (Task, User, Representation, and Function) that should be conducted prior to and during development of an EHR system. This framework created by Drs. Zhang and Walji (2011) is intended to aid researchers define, evaluate, measure, and design EHR systems. The purpose of this guide is to explain and demonstrate the many features of the Turf software.

      References

      Zhang J, Walji MF. TURF: Toward a unified framework of EHR usability. Journal of Biomedical Informatics. 2011; 44(6):1056-67. Doi:10.1016/j.jbi.2011.08.005.

  • 2- Installation Guide

    • System Requirement
      •  

        Administrator privileges are required to install Turf. Turf 2.0.1 or later will not interfere with previous installation of Turf and multiple versions can be used simultaneously. Turf can be installed in any drive. However, the files generated when using Turf will be stored in the user’s My Documents folder.

        List of System Requirements

        Operating System: Windows® 8.1, and 10
        Prerequisites: .NET framework version 4.5.3 or higher
        Processor: 1 GHz or faster
        RAM: 4GB or above
        Free disk space: 250 MB (for application only)
        Screen Resolution: Up to 1920 x 1080
        User Access Control Privileges: Local system administrator or higher

        Note:
        -Turf 4.0 has been tested in Window 8.1 and 10.

        -Storage requirements depend on usage. A typical recording with audio, video, and key stroke events will generate approximately 5-10MB data per minute. For seamless recording and playback, a dedicated video memory (GPU) is preferred.

         

    • System requirements to install Turf for Android in your smartphone or tablet
      •  

        Operating System: Android® 5.0 or higher
        Free disk space: 2 GB (to store the recorded video)
        System Privileges:

        Enable developer mode, USB debugging and install of third party apps.

        ..0+

    • Installing Turf: First Steps
        •  

          Download the Turf software (available as a zip archive) form the Turf website. Go to the extracted folder and click the Turf 3.0 Setup.msi file.

        •  
        • Turf Quick Help

           

          Click Next to proceed.

        •  
        • Turf Quick Help

           

          Read the Research and Evaluation License Agreement and accept the terms to proceed by marking the checkbox and clicking the Next button.

        •  
      • Turf Quick Help

         

      •  
      • Turf Quick Help

         

        Choose if you want to install Turf for all users or just for you. Click the Install button to begin installation.

      •  
      • Turf Quick Help
         

        Installation will begin and the process is indicated in the progress bar.

      • Turf Quick Help
    • Installing Turf: Final Step
        •  

          Turf installation is completed. Click Finish to close the setup window.

      • Turf Quick Help

  • 3- Settings

    • Use Cases
      • Creating a new Use Case
        •  

          Create a new use case in the Settings tab. Click on Use Cases button in the System Configuration Management sidebar on the left. Click New Use Case in the Use Case mini tab. A popup box will appear, add a name and a description then click the Save button.

           

          Turf Quick Help

           

      • Upload Files to Use cases
            •  

              To upload/import files to a Use case: Once the new use case has been created, click on the new use case then click on the Add Files button. A popup box will appear which prompt you to define a path and write a description. When complete, click Save.

               

              To organize your uploaded documents: you can create a new folder by clicking on your use case then clicking on the New Folder button. A popup box will appear and enter your new folder name and description and click Save when complete.

               

              Turf Quick Help

         

    • Descriptive Templates
      •  

        Descriptive Templates are forms used for data entry. They are used in Heuristic Evaluations to describe problems and in User Testing to record demographics, moderator notes, performance evaluation and system usability surveys.

        .
      • Turf Quick Help

         

        To create a new descriptive template:
        - Click Descriptive Templates in the Settings tab
        - Click the New button
        - Enter a name, specify dimensions of width and height in pixel, and write a brief description
        - Click the Add button to add elements of the form
        - Enter a name for the element. This name will become the variable name or column header label in the analytics functionality of Turf as well as in exported files
        - Choose Type from the dropdown menu and select from a list of editable templates.

         

        Turf Quick Help

      • - Click on Content to enter information and values of the field
        - Once the Fields tab has been completed, click on the Design Tab to view the template in a visual layout. You can move the elements using the cursor and arrange them by specifying the top and left coordinates.

        *NOTE: If the dimensions are small, some of the elements will not be visible in the Design tab view. Adjust the dimensions until all the elements can been seen.*
        *NOTE: Templates cannot validate input.

        You can export the template in XML format using the Export button and can later be imported in another system using the Load button.

        - Click the Preview tab to preview the template form
        - Click Save to save the template form

         

        Descriptions of Design Type

        Text Box:

        A text box accepts only one line of input. Enter a name to label the text box and select Text Box under the Type column. You can adjust the width of the textbook, but you cannot set the minimum or maximum characters allowed.

        Text Area:
        A text area accepts multi-line input. Enter a name and select TextArea under the Type column. Width is adjustable.
        Date Picker:
        Date picker can be used to input a date. The default format is: mm/dd/yyyy. The format cannot be changed and input ranges cannot be set.
        Combo Box:

        Combo Box provides a drop down selection. To enter the options, click the Content button. Enter the options for each line. The Combo Box only allows one option to be selected.

        Turf Quick Help
        Radio Box:

        Radio box lets you set radio button options. The options are entered for each line and only one option is selectable. 

         Turf Quick Help
        Check Box:

        Check box lets you set check box options. This allows multiple options to be selected. The options are entered for in each line.

         Turf Quick Help
        Caption:

        A simple text display that can display text, but does not accept input.

        Turf Quick Help

    • Color Sets
      •  

        Turf comes with a small set of default color sets for marking windows events and images. These colors can be edited in the appropriate categories.


        Edit color set by clicking on Color Set in the Settings tab.
        Choose Window event color or Mark property color from the dropdown Color Category box.
        Click Add Field and enter a name for color, choose the correct color and enter a description. Check the Usable checkbox to make the color active.
        Then click Save to complete Color edits.

         

        Turf Quick Help

         

    • Widget Definitions
      •  

        Users can assign different icons, such as, Button, TextBox, List Box, Check Box or ComboBox to problems found in the images captured from different resources, such as, a HTML page, an Android device, an IOS device, or a windows screenshots.


        To add a new widget:

        - Select your OS (Windows/Android)
        - Click on the "Add Widgets" button at the top of the right panel to open up the "Add Widget" window
        - Select your desired icon, enter a name and description, and click on the Save button


        To delete a widget:

        - Select your desired widget
        - Click on the X button that is assigned to the selected widget

         

        Turf Quick Help

         

    • Hotkeys
        •  

          Hotkeys can be set and modified in Turf. 
          To modify and set hotkeys in turf:
          - Click on Hotkeys in the Settings tab
          - Change the hotkeys for each tab in Turf by clicking on the dropdown menu of the Hotkey Category
          - Click on the hotkey to modify and click on the Modify Hotkey Value button. A popup box called Set Hotkey will allow modifications
          - Click OK to save the modifications

          - Click the Active checkbox on different hotkeys to activate or deactivate them.

           

          Turf Quick Help

           

      • Video Settings
        •  

          The Video settings allow users to specify the maximum number of video tabs that can be opened at the same time.

           

      • OneDrive Settings
        •  

          OneDrive settings allow you to customize the way that you want Turf interact with your OneDrive account.

           

      • Install Turf for Android
        •  

          Install Turf for Android settings allow you to install Turf on your android smartphone or tablet device.

           

  • 4- Introduction to Heuristic Evaluation

    • Introduction
      •  

        Introduction to Heuristic Evaluation

        Heuristic evaluation or expert review is a usability inspection method in which usability experts use usability heuristics (i.e., "rules of thumb") to identify user interface challenges. This technique is often described as a " discount" method for finding usability problems as it can identify potential usability issues in a timely and economical fashion. To conduct heuristic evaluation, usability experts inspect a system for violations against established usability guidelines. Numerous guidelines of good user interface design exist (Nielson and Molich, 1990; Zhang, 2011). The National Institute of Standards and Technology (NIST; 2012) published the following design principles:

         

        NIST Design Principles
        Visibility of system status
        Match between system and the real world
        User control and freedom
        Consistency and standards
        Help users recognize, diagnose and recover from Error
        Error Prevention
        Recognition rather than recall
        Flexibility an deficiency of use
        Aesthetic and minimalist design
        Help and documentation
        Pleasurable and respectful interaction with the user
        Privacy

        After usability problems have been identified, they are then merged into a single master list. Evaluators then independently rate the severity of each problem. Average severity scores are then calculated. For more information please see NIST Guideline "Technical evaluation, Testing, and Validation of the Usability of Electronic Health Records" (2012).

        Sources:

        Nielsen, J., Molich, R. (1990). Heuristic evaluation of user interfaces. CHI '90 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 249-256.

        Lowry, S. Z., Quinn, M. T., Ramaiah, M., Schumacher, R. M., Patterson, E. S., North, R., Zhang, J., Gibbons, M.C., Abbott, P. (2012). Technical evaluation, Testing, and Validation of the Usability of Electronic Health Records. National Institute of Standards and Technology

        Zhang, J., Walji, M.F. (2011). TURF: Toward a unified framework of EHR usability. J Biomed Inform. 44(6): 1056-1067.

         

    • Launch and Set Up a New Heuristic Evaluation Project
      • User launch Turf – Step 1
        •  

          Step 1: Launch Turf. After loading the software on the computer, a user will launch Turf by double clicking on the Turf desktop icon.


          Turf Quick Help

           

          Then the user will see the File interface where user can create a new product for the heuristic evaluation.

           

      • Create a new product evaluation folder – Step 2, 3
        •  

          Step 2:

          Click the Add New Product

           

        • Turf Quick Help


          Step 3
          : Fill in product information in the Product dialog box.

        • - Enter the product name (e.g., SharpC-EHR)

        • - Enter the product version (e.g., Alpha)

        • - Enter the vendor name (e.g., SharpC)

        • - Click the Save button

        • Turf Quick Help

           

      • User sets up a new heuristic evaluation project – Step 4, 5
        •  

          After you set up a new product, choose the type of usability method (e.g., Heuristic Evaluation or User Testing) to conduct. In this case, choose Heuristic Evaluation.

           

          Step 4: Select Heuristic Evaluation

          Click Heuristic Evaluation icon in the Home tab.


          Step 5: Fill in Heuristic Evaluation project information in the pop-out window

          The popup box collects information about the evaluation you will conduct. For example, the Project Name is the EHR name and Description is the type of evaluation. Filling in these boxes completely help to recognize distinct projects in the panel tree.

           

          - Enter project Name (e.g., SharpC EHR Heuristic Evaluation)

          - Select predefined use case (e.g., check Electronic Prescribing checkbox).

          - Click the Next button

          Turf Quick Help


          - Select predefined heuristic evaluation template (e.g., select Expert Review). The default Expert Review templates are provided by Turf according to NIST expert review test procedures (2012). Your administrator can also create custom forms for you to select.

          Turf Quick Help


          - Click the Finish button.

        • There is an option to set naming conventions for the image, video and problem files when the Advanced Settings button is clicked.

           

          Note:You cannot currently add these templates to a project later, so choose any you think you will use now.

           

    • Evaluate Heuristic Problems
      • Capture a screenshot – Step 6, 7
        •  

          Screenshots can be captured to inspect interface design violations.

          Step 6:
          -Select the HE Project
          -Click on New Folder in the Heuristic Evaluation tab
          -Rename folder to reflect the project and product (e.g., HE SharpC EHR)

           

          Step 7: Capture a Screenshot

          Configure Screen settings before capturing a screenshot:
          - Click Settings under the Screen checkbox
          - Click Identify Monitors button, then select the monitor that displays the image to be evaluated
          - Select a region (e.g., click Full Screen button)
          - Click OK button
          - Click the Capture Image (or press F5 on the keyboard), (e.g., Click the Capture Image (F5) button to capture the Medication Search screen)

           

          Turf Quick Help

           

      • Physically mark and identify problems on the screenshot– Step 8, 9

        •  

          Step 8: Mark a problematic widget

        • - Identify the problem by marking the widget, (e.g., the medication search textbox).

        • - To mark the problem, drag the mouse over the widget (e.g., mark the medication search textbox)

        • - Click Save to save the mark

        •  
        • Turf Quick Help

           

          Step 9: Describe the identified problem above

          First, open a problem description form, which is default by Turf. Administrator can create new forms or change forms in the administration setting. There are two ways to open a problem description form.

          Method One: Open a problem description form, right Click a mark, then left click Add a Problem about This Mark option

           

          Turf Quick Help

           

          Method TwoOpen a problem description form:

        • - Left click New problem button

        • - Left click Select Marks button

           

        • Turf Quick Help
        •  
        • - Select Mark “1” in the Scope Select pop-out window where you select the marks which you want to document problems.

          - Click OK button

        •  Turf Quick Help
        •  
        • Once the default template form provided by Turf is open, please complete the steps below.

        • 1- Enter the Problem Location (e.g., Medication Search Box).

          2- Fill Problem Description (e.g., Truncated drugs’ name and dosage).

          3- Select an appropriate Design Violation (e.g., Select "08-Recognition rather than recall")

          4-Click Save button to save successfully

          5- Click X to close the problem description window

          1. You repeat steps 1-5 until all problems have been identified and documented.

             

        • Audit and Rate Heuristic Problems

          • Audit heuristic problems- Step10

            •  

              Step10: After you take screenshots and mark potential problems, you proceed to audit the identified problems by selecting the appropriate option. For each problem the evaluator can either:

               

              - Archive a problem, saving it for discussion with another assessor

              - Agree with the problem and the description

              - Discard the problem as not being problematic


              During the auditing, Turf also indicates how many violations remain unclassified.

               

          • Rate Severity for each heuristic problem- Step11

              •  

                Once the agree/discard decision is made, the “agreed” violations can then be scored for severity.

                Step 11: Rate Severity for each heuristic problem:

                 

                - Click Rate Problems button

                -Select the identified problems you want to perform the Rate function on

                -Click OK button

                 

                Turf Quick Help
              •  

                -Rate the severity from “1 (Minor) to 4 (Catastrophic)” for the problem (e.g., Select “2”)

                -Click Save button

                Repeat the last two steps until all problems have been rated for severity


              • Turf Quick Help

                 

        • Generate Heuristic Evaluation Report- Step12

          •  

            All the results can be summarized into a report. Preview the heuristic evaluation report and save it in a preferred file format.

             

            Step 12: Generate heuristic evaluation report:

             

          • -Click the Report button

          • -Check/uncheck the checkboxes and radio buttons based on your preferences

          • -Click the print button at the bottom of the page

          • -Print the report or export it in your preferred format by clicking on the correspond buttons
          •  

            Turf Quick Help

             

            -Preview the report and export the file in a format based on users’ preferences, like PDF, HTML, Word, CSV, Text, Image, etc.

            Turf Quick Help

             

        • Heuristic Evaluation Findings Analyzing Heuristic Evaluation Findings

          • Descriptive Analysis

            •  

              After you have the heuristic evaluation findings, you can analyze these findings by using Turf analytical functions.
              The descriptive analyses within Turf include of calculating the sum, mean, variance, standard deviation, range, minimum and maximum data, kurtosis and skewness. You choose the analytic method(s) that you want to apply, and then generate analysis results. These descriptive analysis results can help to assess the overall usability level of a product and identify the major usability problems.

               

              Step 13: Calculate the mean, range, minimum and maximum of severity scores:

              For the severity scores assigned to problems, we are interested in mean severity scores, the ranges, as well as the minimum and maximum scores.

               

              Query Files:

            • -Go to Analyze tab

              - Click Query Files

              -Select the project – Click rEHR, then click rEHR-Prescribing, click Electronic prescribing folder

              -Select the images where usability problems have been identified in previous steps

              -Click OK


            • Turf Quick Help
            •  

              - Drag the Image Name to the lower panel

              - Select the column of Average Score

              - Uncheck the Sum, Variance, Std. deviation, Kurtosis and Skewness checkboxes

              - Click the Run button

               

            • Turf Quick Help

              Here, Turf displays the descriptive analysis results based on your selections.

               

              Turf Quick Help
          • Count the Numbers

            •  

              In addition, you can count the numbers of each documented error type and each design violation by using Turf Count Function. The count function helps evaluators understand which are the most frequent errors and violations in one project.

               

              Query Files:

            • - Go to Analyze tab

              - Click Query Files

              -Select the project – Click rEHR, then click rEHR e-prescribing, click Electronic prescribing folder.

              -Select the problems (e.g., select “Prob_1”, “Prob_2”, “Prob_3”, “Prob_4”).

              -Click OK

               

              Turf Quick Help

              -Select the column of Design Violation

              Turf Quick Help


              -Click the Count button

              You will get the count of design violation.

               

              Turf Quick Help

      • 5- User Testing

        • What is User Testing?

          • Introduction

            •  

              User testing is a technique used to evaluate an EHR product by testing it with representative users. In the test, these users will try to complete typical tasks while moderators watch, listen and takes notes. The goal is to identify usability problems, collect quantitative data on participants' performance (e.g., time on task, error rates), and determine participant's satisfaction with the product. There are 3 unique types of user testing that can be performed on a product: formative testing, summative testing and validation testing. These different types will be further detailed for clarification.

               

          • Formative Testing

            •  

              User testing can be part of the formative testing of a product that is conducted early in the development lifecycle. In this regard, it is used to assess the usability of preliminary designs and prototypes in order to facilitate rapid improvements during the design process. Consequently, formative user testing does not require a fully developed system, but needs enough functionality to answer the goals of the usability study. Formative user testing focuses on qualitative data collection in which the user walks through processes or explores an interface rather than performing a well-defined scenario.

               

          • Summative Testing

            •  

              User testing is also conducted after high-level design has been specified or in summative testing. In this case, it is used to determine how well a user performs various tasks. Summative testing involves collection of both quantitative and qualitative measures. Summative tests can be designed to measure system effectiveness and efficiency as well as user satisfaction with a system.

               

          • Validation Testing

            •  

              Finally, user testing is conducted for validation testing of a product before it is released to the general public. Validation testing measures the usability against specific predetermined standards or benchmarks such as “Assess if 80% of users can successfully complete task.” This involves the collection of mainly quantitative data.

               

        • Launch Turf and Set Up a New User Testing Project

          •  

            Step 1: Launch Turf

            After loading the software on the computer, a user will launch Turf by double clicking the desktop icon. Then the user will see the Home tab where the user can create a new product for the user testing.

             

          • Step 2: Create a new product evaluation folder

          • You will create a folder to hold all of the user testing projects for one EHR product. This allows you to test multiple tasks (e.g., e-prescribing, medication reconciliation) for a single product. You will create one folder for the product and separate folders within it for each task.

            - Go to Home tab

            - Click the Add New Product button

             

            Step 3: Fill in product information in the Product dialog box

            - Enter the Product name (e.g., Demo EHR)

            - Enter the product Version, (e.g., 1.0; any alphanumeric characters accepted)

            - Enter the Vendor name, (e.g., SHARPC)

            - Click the Save button

          •  Turf Quick Help

            Step 4: Create a user testing project folder

            - Click the User Testing button. Make sure the product folder you created is highlighted.
            - Enter a project name (e.g., User Testing 1). If you like, you can add a description, such as: "Users: Doctors".
            - Select the predefined use cases, e.g., check Electronic Prescribing and Clinical Information Reconciliation. You cannot add these to a project later, so choose all you think you will use now. We have included the NIST Use Case documents for the eight required functions for Meaningful Use Stage 2. Your administrator may add other reference materials for you).
            - Click the Next button

             Turf Quick Help

          • Step 5: Associate data forms with your project

            There are four templates for recording user-testing data preloaded with Turf, which include:

            Demographics:

            Collect demographics and experience data about your users. Completed by the user.

            System Usability Scale:

            Assesses satisfaction with the system. Completed by the user.

            Moderator Notes:

            Free text for notations about the user testing experience. Completed by moderator.

            Performance Evaluation:

            Structured form to evaluate the user performance. Completed by moderator.

            In this step you will associate the correct form with the button from which you will access it. You cannot add these to a project later, so it is best to choose any you think you will use now.

            - Using the drop down menu, select the data templates you want to use.

            - The Advanced Settings button allows you to customize the naming conventions for your image, video and problem files. For now, we will use the default settings.

            - Click Finish

             Turf Quick Help

          • Step 6: Create folders to hold each user’s data

            - Click on your desired project

            - Click New Folder

            Rename the folder to reflect the project and product.

          •  
        • Test Users

          • Complete Administrative Documents – Step 7

            •  

              There are three generic forms for administrative purposes associated with each Use Case document. These forms can be edited within Turf to customize them for your organization and purposes. They can then be printed and signed by your users.

              The available generic administrative forms include:

              - Non-Disclosure Agreement (NDA)

              - Acknowledgement of Receipt Form

              - Informed Consent Form

              Step 7: Prepare Administrative Documents

              - Select a use case folder (e.g., Click the §170.314b(3) Electronic Prescribing icon under User Testing 1 project

            • - Click on the use case to open its content in the Details Panel which is at the bottom of the screen

              - Double click on the form you want to edit

              - Edit the form as you would in a standard word processing program

              - Print the form

              Turf Quick Help

          • Capture Data from a User – Steps 8, 9, 10, 11, 12, 13

            •  

              Step 8: Collect data on the Demographics Form

              - Select the folder for the user you are going to test

              - Click the Demographics icon on the Forms ribbon

              - Have your user complete the form

              - Click Save

               Turf Quick Help


              In Step 9, we will set up electronic data capture. For this example, we are not using the webcam. But we demonstrate accessing settings for capturing Audio and Keyboard. In your testing session, you can accept default settings without clicking on Settings for these.

              Step 9: Set up electronic data capture

              Be sure the EHR product you want to test is available to your users at the point you want them to begin the test. Also, uncheck the checkbox for any electronic data capture method you will not be using.



            • Audio Settings:
              - Click Settings under the Audio check box to change audio settings

            • - Click Ok

              Turf Quick Help

            • Keystroke Settings:
              - Click Settings under the Keyboard check box to change keyboard and mouse settings
              - Click Ok

              Turf Quick Help


            • Video Settings:
              - Click Settings under the Video check box to change video screen capture settings
              - Set Delay to 3 seconds so that video capture begins when the user is working on the EHR

            • Screen Settings:
              If the user is on a second monitor, choose the appropriate monitor
              - Click Settings under the Screen check box to change screen settings
              - Click Ok on the Screen Settings dialog box

              Turf Quick Help

            •  

              Step 10: Capture video and audio of the user performing a task in the EHR being tested

              - Click Record
              Turf will be minimized and the view will switch to the EHR being tested. Let the user perform the tasks per your instructions
              - When you are ready to stop the test, click on the Turf icon in the taskbar to maximize it then click on the Stop button or use the hotkey F9
              - The system will display the data after a few minutes

               

              Step 11: Administer the System Usability Survey

              - Click the Usability Survey button on the user testing ribbon to open the SUS
              - After the user complete the survey, click Save
              - Name the saved file (e.g., “SUS_DocA”)
              - Close the SUS form

              Turf Quick Help

              Step 12: Complete the Moderator Notes form

              - Click the Moderator Notes button to open the form
              - Make notes about the test, click Save
              - Name the saved file (e.g., “Note_DocA”)
              - Exit out of the Moderator Notes form

              Turf Quick Help

              Step 13: Export User data

              - Select the file or files you want to export
              - Click Export Files
              - Select a folder in which to save the exported files
              - Click OK

               Turf Quick Help



        • Set up an Autoflow Session

          • Autoflow Configuration

            •  

              You can use Autoflow to predefine some actions within Turf. The program will then run these actions automatically when the moderator starts the user testing. Before using Autoflow, you first must configure its settings.

              Autoflow Configuration:

              In Autoflow configuration, you can add a sequence of actions. The actions may include:

              -Read PDF File
              -Read RTF File
              -Play Video
              -Capture Image
              -Media Record
              -Fill in demographic
              -Fill in SUS
              -Fill in Moderator Notes
              -Fill in Performance Note
              -Display Text Screen

              You can specify the actions and the orders of the actions to complete a user testing.

               

              Step 1: Add a new Autoflow configuration

            • - Select a product, for example:Select Demo EHR

              - Select a project, (e.g., User Testing 1)
              - Select a folder, (e.g., Electronic prescribing)
              - Click Set Up Autoflow button
              - Click New button in the popup box that appears
              - Enter the configuration Name (e.g., e-prescribing)
              - Enter the configuration Description (e.g., "Product: Demo EHR")
              - Set this configuration as the default Autoflow configuration by checking the Default checkbox

               Turf Quick Help

            • Step 2: Read PDF File

            • - Click the Add button

            • - Enter "Informed Consent" in the Name column

            • - Select Read PDF File in the Action column

            • - Click the

              - Click the button in the Content column to choose which PDF file will be read and click Open

               Turf Quick Help

            • Step 3: Fill in Demographic Form

            • - Click the Add button

            • - Enter "Demographic Form" in the Name column

            • - Select Fill in Demographic in the Action column

            • - Click the button in the Folder column to choose a folder to store your demographic forms. The demographic form filled by the user will be saved automatically into this folder at the end of the Autoflow session. You can Add Folder, Rename or Delete an existing folder

            • - Click OK button

              Turf Quick Help

            • Step 4: Media Record

            • - Click the Add button

            • - Enter Recording Task 1 in the Name column

            • - Select Record Selected Audio/Video/Keystroke/Screen in the Action column

            • - Choose a folder to store your media files. The media files recorded will be saved automatically into this folder at the end of the Autoflow session

            • - You can Add Folder, Rename or Delete an existing folder

            • - Click OK button

               Turf Quick Help

               Step 5: Display Text Screen

            • - Click the Add button

            • - Enter "Debrief" in the Name column

            • - Select Display Text Screen in the Action column

            • - Enter "Thank you for your time!" in the Content column

               Turf Quick Help

            • Step 6: Save Autoflow Configuration

            • - Click Save button

            • - Close Auto Flow dialog box

          • Run an Autoflow Session

            •  

              The steps listed here are based on the configurations we performed on the "Auto Flow Configuration" section.

              Step 1
              : Run a default Autoflow session

            • - Click Run button

            • - A PDF file of Informed Consent will be displayed

               

              Step 2: Read PDF File

            • - Click Next button or Skip button to go to the next action

            • - Click End Session button to end this Autoflow session

            • We click Next button here, a demographic form will be displayed

               Turf Quick Help

            • Step 3: Fill in Demographic Form

            • - Fill in this User Demographics form

            • - Go to the next action

            • - If you click Next button, Turf will go to the next action and save this form automatically

              - If you click Skip button, Turf will go to the next action and drop this form

            • - If you click End Session button, Turf will end this Autoflow session

            • We click Next button here and Turf will minimize and start recording user’s operations.

               Turf Quick Help

            • Step 4: Media Record

            • - Press F9to stop recording

              You can pause or restart the recording by clicking Pause/Restart Record button or by pressing F8. Or you can stop the recording by clicking Stop Record button or by pressing F9. A video or a windows event data will be displayed on the screen

            • - Go to the next action

              If you click Next button , Turf will go to the next action and save the media file(s) automatically.

              If you click Skip button , Turf will go to the next action and drop the media file(s)

              If you click End Session button , Turf will end this Autoflow session

              We Click Next button here and a Text Screen will be displayed here.

               Turf Quick Help

            • <

              Step 5: Display Text Screen

            • - Click End Session button to exit this Autoflow session

      • 6- OneDrive

        • Upload to OneDrive
          •  

            Follow the following steps to upload to OneDrive

             

            - Go to the File Tab

             

            - Click OneDrive button on the left navigation bar

             

            - Click on the login button at the bottom right of the screen

             

            - Enter your username and password to log in to your OneDrive account. If you do not have an account, you will need to create one

             OneDrive


            - An alert window will pop up warning that Turf will access your account info- Click on the Yes button to proceed to the next step

             OneDrive OneDrive


            - Go to Setting Tab

             

            - Click OneDrive Settings button

             

            - Change the value of the first field, “Automatically upload files to OneDrive” to True, if it is set to False. Setting the value to True allows you to synch your local files with the uploaded files. This means that every changes that you make to your local files will be automatically applied to their counterparts in OneDrive.

             OneDrive


            - Go to the Projects panel tab- (Click on the Home tab)

             

            - Right click on your desired folder. Go to “OneDrive” menu item, next, click on “Include in Sync”

             

            - If the value for ”Automatically upload files to OneDrive” field is not set to True, you must click “Sync Now” from the context menu.

             OneDrive


            - An icon on the right lower corner of the screen will appear indicating that the data are being uploaded

             OneDrive


            - Upon uploading a project / folder, you will notice that TurfData4.0 folder is also created in OneDrive. This folder will include data from Turf. The organization of sub-folders is as follows: Product >> Project >> Folders >> Files.

             

          • Sharing Turf data on OneDrive with other team members
                •  

                  To share data with others, you need to access to your OneDrive folder:
                •  

                  -Use your browser or any OneDrive applications to go to your OneDrive account where you will be able to share your files with others

                •  

                  -Locate the folder “TurfData4.0”.
                •  Turf Quick Help

                   

                   

                • -Select the folder and edit its share options.
                •  

                  -Share the folder with Edit privileges.
                •  Turf Quick Help

                   

                • -The shared user will get an email notification.
                •  

                   

                  Accepting shared data from others:

                   

                  If someone has shared their Turf data with you, you will receive an email notification about the share. Follow the instructions in the email to accept the share. You do not need to add the shared folder to your OneDrive. Simply accepting the share will be enough. You should be able to access the folder under Shared Files section in OneDrive.

                   

          • Importing Turf data from OneDrive
            •  

              In general, the file (s) /folder (s) that are uploaded to OneDrive can be imported into a Turf project, or product. They come from two different sources: (1) The ones that you have already uploaded to OneDrive, or (2) the ones that your colleagues have shared with you on OneDrive.

               

              To import a project from OneDrive to a Turf product:

               

              Right click on a product >> Go to OneDrive >> Select import a project from the OneDrive menu item.

               

              Similarly, to import a folder to a Turf project:

               

              Right click on a project >> Go to OneDrive >> Select import a folder from the OneDrive menu item.

              The files/folders that you have uploaded to OneDrive will be displayed in “My Files” tab, whereas, the files that your teammates have shared with you will be displayed on “Shared Files” tab. When importing a Project or Folder to “My Files” tab, you will only see the folders and files that are synced with that particular project and folder. This means that you will not be able to import folders or files from a different project or product using OneDrive function. However, if you use “Shared Files” tab, you will be able to import any projects / folders / files to your local projects or folders .

               

              The following steps demonstrate how you can import your uploaded data files to a project folder:

               

              -Go to the Projects panel. Right click on the project that you want to import the files into.

               

              -From the OneDrive menu item, choose “Import a folder”. This will pop up a new window.

               

            • Turf Quick Help 

              - The new window contains two tabs, “My Files”, and “Shared File”. Go to “My Files” tab and select the files that you want to import

               Turf Quick Help

               

              -Click OK button

               

      • 7-Turf for Android

        • How to setup your device
          •  

            The number of steps or the suggested configuration may vary depending on the devices. The following are suggested steps for any devices with Android 5 or greater. Here, we will present the instructions for Nexus 9 tablet.

             

            Step 1: Check your android version

             

            Setting >> System >> About tablet >> Android Version The version of your Android device OS must be at least 5.0. If the OS version is less than 5, please upgrade it to Android 5.0 or later.

             

            Step 2: Enable Developers mode

             

            -Go to Settings
            -Slide down to the “System” panel
            -Click on “About tablet”`
            -Press the “Build Number” item seven times

             Turf Quick Help

            Turf Quick Help

             

          • Step 3: Configure Developer settings

             

            -Go to Settings
            -Slide down to the “System” panel
            -Click on “Developer options”
            -Check the checkbox to select “USB debugging”

             Turf Quick Help


          • Step 4: Enable side loading apps

             

            Open Settings >> Personal >> Security >> Device Administration
            -Set Unknown Sources to ON
        • Device Driver Setup
            •  

              Connect your device to the computer using USB. If you are connecting the device for the first time, windows should automatically detect and install it and you will see a “Installing device….” window. This will appear only the first time you connect your device to your computer.

               Turf Quick Help


              After successfully installing the device, you will see a notification, such as the one below.

               

               Turf Quick Help

               

            • Go to “My Places / This PC / My Computer", and check if you can see your device name.
            •  Turf Quick Help

               

            • If Windows does not recognize your device, you won’t see your device name. Contact your device vendor to obtain appropriate drivers. Now, you can launch Turf and start installing the Turf for Android.

               

              Note: If Turf does not detect your device, you need to check if you have 1) Activated USB debugging and permitted the installation of 3rd party applications on your device.

               

              2) Installed the appropriate device drivers for ADB interface. Please follow the following steps to install drivers for ADB interface (Only for Google Nexus Devices. For others, refer to vendor sites) :

               

              -Go to“Device Manager”
              -Click on the small arrow next to “Android Device” to expand it.
              If “Android Composite ADB interface” is present, drivers are correctly installed.
              a. If you do not see this, your device may appear as “Other device”. Click on the small arrow next to the “Other devices” to expand it.
              -Right click on the “ADB interface”, and select “Update Driver Software”. Before doing this, make sure you have already downloaded the appropriate driver software.
              -Browse for Driver software on your computer and wait for the device to be installed and enabled
              -After installation unplug the device

        • Install USB driver in Windows
          •  

            Step1: Download

             

            -Go to the following link to download the Google USB Driver on your desktop computer https://developer.android.com/studio/run/win-usb.

          •  

            Google USB Driver 

             

            Step2: Instructions for installing the driver

             

            -Unzip the downloaded folder by extracting its components - Connect your Android device to your computer using a USB
            -Right click on the Computer icon and click on the “Manage” item
            -Click on “Device Manager” from the left panel
            -Click on the small arrow next to the “Other devices” to expand it
            -Right click on the “ADB interface”, and select “Update Driver Software”
            -Browse for Driver software on your computer and wait for the device to be installed and enabled
            -After installation unplug the device

             

        • Configuration and Installation
          •  

            Follow the following steps to utilize the Turf android app. The number of steps or the suggested configuration may vary depending on the devices. The following are suggested steps for any devices with Android 4.4 or greater.

             

            Step1: Install Android in Turf

            -Connect the device to your computer
            -Open Turf
            -Go to Settings
            -Select “Install android”
            -Click on the “Install” button

             

            Step2: Activating Turf Keyboard and Turf Launcher

             

            -Open the Android device
            -Go to Settings
            -Go to “Personal” panel
            -Click on “Language and input”
            -On the “Language & input” screen, click on “Current keyboard”
            -Click on “CHOOSE KEYBOARDS”
            -Turn on the “turf Keyboard” item
            -Click on the OK button on the “Attention” window
            -Get back to the “language & input” window and click on “Current Keyboard”
            -Click on the “turf Keyboard” radio button to select it
            -Go back to Settings
            -Go to the “Device” panel
            -Click on the “Home” item
            -Click on “Turf launcher” radio button to select it
            After successfully installing Turf on your Android device, you will be able to use Turf Android features.

          •  Turf Android


            Turf Android


            Turf Android


            Turf Android


            Turf Android


            Turf Android

            Turf Android


          • Note: After you finish working with Turf, make sure to turn off the Turf keyboard and launcher in your android device. This is to ensure that your device activities are not being logged.

        • Capture events using Turf Android
          •  

            Follow the following steps to use Turf Android features

             

            -Go to User Testing tab, select the folder where you want to store your data
            -Check the checkbox for Android in the Capture Media ribbon, under the User Testing tab at the top of the screen, to select it
            -Click on Android Settings to select the parameters that you would like to capture
            -Press Record (F7) button to record data
            -Once you are done with recording, click Stop Record (F9) button to stop the recording.

            Turf Android

      • 8- User Testing Analytics

        • Analyze User Testing Data
          • Introduction

            •  

              Currently Supported Statistics in Turf

              Commonly used measures of descriptive statistics have been included in Turf. These include:

              <<Statistic Type>>

              Summative

              Inferential

              Sum

              One-way ANOVA

              Minimum & Maximum

              Mann-Whitney U Test

              Range

              Two-sample t Test

              Mean

               Linear Regression

              Variance

               Pearson Correlation coefficient

              Standard Deviation

               

              Kurtosis

               

              Skewness

               

               

          • Summarize, Describe or Present Data

            •  

              After collecting data from the user testing, Turf analytics can help you analyze the data and present the results.

               

              Descriptive Analysis of Demographic Data

              Query demographic data:

              - Go to Analyze tab
              - Click the Query Files button
              - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
              - Select demographic form (e.g., Demo_1 and Demo_2). The name and path of the selected files will be displayed at the bottom of the Query files dialog

              - Click OK

              Turf Quick Help

              Select descriptive measures:

              Choose the descriptive measure(s) you want to calculate by checking the checkbox of the measure(s), e.g., Mean and standard deviation.

              Choose the descriptive measure(s) you want to calculate by checking the checkbox of the measure(s), e.g., Mean and standard deviation.

              Select data and calculate:

              - Select the data you want to analyze (e.g., Age)
              - Click Run button

               Turf Quick Help


              Calculate Average SUS Score


              The System Usability Survey (SUS) can be used to quickly test the usability of a product. After collecting the surveys with 10 items from all participants, you can convert the raw scores of each survey into an overall SUS score that ranges from 1 to 100. Then you can use the descriptive analytics in Turf to calculate the average SUS score.

              Query SUS:

              - Click Query Files button
              - Select a Product
              - Demo EHR
              - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
              - Select SUS files, e.g., SUS_1, SUS_2, and SUS_3
              - Click OK

              Turf Quick Help

              Calculate overall SUS score:
              - Select the three rows
              - Click SUS Score button

              Turf Quick Help

              Calculate average SUS score:

              - Select SUS Score column

              - Select Mean by checking the checkbox of mean

              - Click Run button

               Turf Quick Help

          • Comparison between Different Groups

              <
            •  

              Turf provides statistical methods to compare the usability of different products. Also available within Turf are comparative tests including: two-sample ttest or Mann-Whitney U test to compare two groups of samples and one-way ANOVA to compare three or more groups of samples.

               

              Compare Task Time with Pivot Table/Chart

               

              Step1: Query task time
              - Go to Analyze tab
              - Click Query Files button

              - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
              - Select windows event files, e.g., Win_1 to Win_10
              - Select Task time
              - Click OK

               Turf Quick Help


              Step2: Add categorical columns


              Step2: Add categorical columns

              -Right click on Task time column header and click Insert Column
              -Enter Column Name, e.g., Product Name
              -Enter EHR 1 for the first five rows of Product Name Column to indicate it’s Product 1, enter EHR 2 for the last five rows of Product Name Column to indicate it’s Product 2
              -Click OK

              Use the same steps to add another column named User Group.

               Turf Quick Help


              Step3: Switch to pivot table and chart

              - Click Switch to Pivot Table/Chart button
              - Put your mouse over Data Headers
              - Drag Product Name button to Drop Row Field Here, drag User Group button to Drop Column Fields Here and drag Name button to Drop Filter Fields Here

               

              Compare Task Time with One-way ANOVA

               

              Step1: Query task time

              - Go to Analyze tab
              - Click Query Files button

              -Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
              - Select windows event files, e.g., Win_1 to Win_10
              - Select Task time
              - Click OK

               Turf Quick Help


              Step2: Add categorical columns

              - Right click on Task time and click Insert Column
              - Right click on Task time and click Insert Column
              - Enter

              - Right click on Task time and click Insert Column
              - Enter Column Name, e.g., Product Name
              - Click OK
              - Enter EHR 1 for the first three rows of Product Name Column to indicate it’s Product 1, enter EHR 2 from the fourth to sixth rows of Product Name Column to indicate it’s Product 2, enter EHR 3 for the last four rows of Product to indicate it’s Product 3

              Turf Quick Help

              Step3: Choose variables and calculate
              - Select Product Name and Task time Column
              - Click One-way ANOVA button
              - Add Product Name as Factor
              - Add Task time to the Dependent list
              - Click OK

               Turf Quick Help


              Compare Task Time with Mann-Whitney UTest

               

              Step1: Query task time

              - Go to Analyze tab
              - Click Query Files button

              - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
              - Select windows event files, e.g., Win_1 to Win_10
              - Select Task time
              - Click OK

               Turf Quick Help


              Step2: Add categorical columns

              - Right click on Task time and click Insert Column
              - Enter Column Name, e.g., Product Name
              - Click OK
              - Enter EHR 1 for the first five rows of Product Name Column to indicate it’s Product 1, enter EHR 2 for the last five rows of Product Name Column to indicate it’s Product 2

               Turf Quick Help

              <


              Step3: Choose variables and calculate

              - Select Product Name and Task time Column
              - Click Mann-Whitney U button
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list
              - Click OK

               Turf Quick Help


              Compare Task Time with Two-Sample t Test

               

              Step 1: Query task time

              - Go to Analyze tab
              - Click “Query Files” button

              - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
              - Select windows event files, e.g., Win_1 to Win_10
              - Select Task time
              - Click OK

               Turf Quick Help


              Step2: Add categorical columns

              - Right click on Task time and click Insert Column
              - Enter Column Name, e.g., Product Name
              - Click OK
              - Enter EHR 1 for the first five rows of Product Name column to indicate its Product 1, enter EHR 2 for the last five rows of Product Name Column to indicate it’s Product 2

               Turf Quick Help


              Step3: Choose variables and calculate

              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list.
              - Select Unpaired and None
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list.
              - Select Unpaired and None
              - Click OK
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list.
              - Select Unpaired and None
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list.
              - Select Unpaired and None
              - Click OK- Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list.
              - Select Unpaired and None
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list.
              - Select Unpaired and None
              - Click OK
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list.
              - Select Unpaired and None
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Select Product Name and Task time Column
              - Click Two-Sample t button.
              - Add Product Name as Group Variable
              - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
              - Add Task time to the Test Variable(s) list.
              - Select Unpaired and None
              - Click OK

               Turf Quick Help

      • 9- Reporting Bugs and Feedback

        • To submit the bugs or your feedbacks, Email us at [email protected]. We will try our best to respond to you within one working day.
      • 10- Technical Support

        • Technical support is available by appointment only. Email us at [email protected] for technical support.
      • 11- Glossary

        • Errors

          Errors may include clicking on an incorrect menu item, incorrect link, or interacting incorrectly with an on-screen control.

          Formative Testing 
          User testing can be part of formative testing of a product that is conducted early in the development lifecycle. In this regard it is used to assess usability of preliminary designs and prototypes in order to facilitate rapid improvements during the design process. Consequently, formative user testing does not require a fully developed system, but needs enough functionality to answer the goals of the usability study. Formative user testing focuses on qualitative data collection in which the user walks through process or explores an interface rather than performing a well-defined scenario.
          Heuristic Evaluation

          A heuristic evaluation is a usability inspection method for computer software that helps to identify usability problems in the user interface (UI) design. It specifically involves evaluators examining the interface and judging its compliance with recognized usability principles (the "heuristics"). These evaluation methods are now widely taught and practiced in the New Media sector, where UIs are often designed in a short space of time on a budget that may restrict the amount of money available to provide for other types of interface testing.

          Online surveys

          Online surveys are structured interviews with users, where you display a list of questions online and record users' responses.

          Persona 
          A persona is a fictional person who represents a major user group for your product. Personas help you identify major user groups of your product. You select the characteristics that are most representative of those groups and turn them into a persona.
          Prototype 
          A prototype is a draft version of a product. Prototypes allow you to explore your ideas before investing time and money into development. A prototype can be anything from a paper drawing (low-fidelity), click-through of a few images or pages, or a fully functioning product (high-fidelity).
          Scenario

          A scenario is a short story about a specific user with a specific goal at your site. Scenarios are the questions, tasks, and stories that users bring to your Web site and that the Web site must satisfy. Scenarios are critical both for designing Web sites and for doing usability testing.

          Summative Testing

          User testing is also conducted after high-level design has been specified or in summative testing. In this case, it is used to determine how well a user performs various tasks. Summative testing involves collection of both quantitative and qualitative measures. Summative tests can be designed to measure system effectiveness and efficiency as well as user satisfaction with a system.

          SUS

          In systems engineering, the system usability scale (SUS) is a simple, ten-item attitude Likert scale giving a global view of subjective assessments of usability. It was developed by John Brooke at Digital Equipment Corporation in the UK in 1986 as a tool to be used in usability engineering of electronic office systems.

          Task Analysis

          Task analysis involves learning about your users' goals, what they want to do on your Web site and how they work. It can also mean learning about specific task users must do to meet those goals and what steps they take to accomplish those tasks. A task analysis complements a user analysis.

          Task Failure

          User abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion.

          Task success

          User can complete the task in the allotted time without assistance from the administrator.

          Time on task

          The length of time in seconds required for user to complete the task.

          Turf
          Is a software program based on the TURF Unified Framework for Defining, Evaluating, Measuring, and Designing EHR Usability. The acronym TURF comes from the four usability inspection methods that are utilized in the process: Task analysis, User analysis, Representational analysis, and Functional analysis. For details, please check: NIST, or NCBI.
          Usability
          Usability is defined, in the TURF framework developed by the SHARP-C project funded by ONC, as "how useful, usable, and satisfying a system is for the users to accomplish goals in the work domain by performing certain sequences of tasks."
          Usability Testing

          Usability testing is a technique used to evaluate a product by testing it with representative users.

          Use case

          A use case is a description of how users will perform tasks on your Web site. A use case includes two main parts:

          1-the steps a user will take to accomplish a particular task on your site.

          2-the way the Web site should respond to a user's actions.

          A use case begins with a user's goal and ends when that goal is fulfilled.
          User testing

          User testing is a technique used to evaluate a product by testing it with representative users. In the test, these users will try to complete typical tasks while observers watch, listen and takes notes. The goal is to identify any usability problems, collect quantitative data on participants' performance (e.g., time on task, error rates), and determine participant's satisfaction with the product.

          Validation Testing

          Finally, user testing is conducted for validation testing of a product before it is released to the general public. Validation testing measures usability against specific predetermined standards or benchmarks such as “Assess if 80% of users can successfully complete task.” This involves the collection of mainly quantitative data.

      • 12-Statistics Refresher and Guide to Selecting the Proper Statistical Test

        • Purpose

          Many members of the Human-computer Interaction, Human Factors, Ergonomics, or Usability fields will likely have had some training in basic statistics; we realize that this is not a universal truth. We have many colleagues with backgrounds in Graphic Design or other fields that have never been exposed to this material. It is far outside of the scope of this document to teach basic statistics, but we can at least try to focus on the most common statistical methods used in the usability testing of electronic health records (EHR). This document is meant to support those using the Turf software by providing information about the various methods available in the software, how to choose the most appropriate statistical method, and how to properly interpret the results.

          The tone of this document is intended to be easy-going because the intention is to provide a reader-friendly guide to statistical analyses, not to turn readers into statisticians. In order to achieve this intent, this document tries to explain things in simpler terms than what might be found in textbooks or scientific journal articles. However, in doing so, there is a chance that parts of this document will make professional statisticians cringe because of how lightly we have treated, described, or defined some of the topics in this document. With that in mind, this text should not be considered an authoritative text on statistics, but it should still be useful to provide readers with sufficient knowledge to be able to select and utilize the most appropriate statistical methods for usability tests and how to properly report results.

          Chapter Overview

          This chapter covers both the summative and inferential statistics available in Turf. These statistics are found in the following table.

          Statistics used by Turf
          Statistic Type
          SummativeSummativeInferential
          Sum One-way ANOVA
          Minimum & Maximum Mann-Whitney U Test
          Range Two-sample t Test
          Mean  
          Variance  
          Standard Deviation  
          Kurtosis  
          Skewness  

          Guide to Selecting the Appropriate Statistical Test

          Normally a section like this might come at the end of a statistics text. In that this document is intended as a refresher and quick reference, it is placed here to prevent unnecessary searching. Those that need additional help will find additional information in later sections.

          Turf currently supports three inferential statistical tests. These are the One-way analysis of variance (ANOVA), the Mann-Whitney U test, and the Two-sample t test. The proper test depends upon the usability experimental design (i.e., within subjects, between subjects, or mixed design), the type of data gathered (i.e. qualitative or quantitative), the measurement type (i.e., nominal, ordinal, interval, and ratio), and the number of variables (e.g., number of comparison groups) to be evaluated. Turf currently only supports statistical analysis capabilities for quantitative data from between subjects experimental designs. This means that in order to utilize the statistical analysis features of Turf, your study will need to use a different group of people for each condition (e.g., old design vs. new design; design A vs. design B).

          Using the same people in both groups is a legitimate study design choice, but is known as a within-subjects or repeated-measures design. When comparing performance from only two groups, a dependent-samples t test can typically be used and is relatively simple to calculate. However, comparing more than two conditions (e.g., design A vs. design B vs design C), requires a repeated measures ANOVA, which is considerably more sophisticated than the one-way within-subjects ANOVA provided by Turf. We highly recommend that your research requires a repeated measures experimental design, and having someone in your organization trained in statistics.

          If you want to design a usability study around the current statistical functionality of Turf, consider the following: Your study must use a different set of participants for each EHR design being evaluated. You cannot ask participants to be evaluated in both EHR conditions.

          If you want to design a usability study around the current statistical functionality of Turf, consider the following: Your study must use a different set of participants for each EHR design being evaluated. You cannot ask participants to be evaluated in both EHR conditions.

          Most Statistics Are Fancy Ratios

          It is likely that most statistics that you will calculate and report for usability testing will be based on a ratio of some sort; most likely a “mean.” Actually, this is pretty true for a lot of research in general. The field of statistics often has a tendency to turn many people off by requiring them to learn a lot of new, seemingly arbitrary formulas and symbols. However, most statistics that you are likely to encounter and use for usability testing are really just a fancy ratio or the mean of some values. As long as you remember this in addition to the fact that you probably learned all the necessary mathematics required to calculate them in elementary school, you should be able to follow the logic behind the statistics a lot more closely and prevent yourself from letting the formulas and symbols scare you off track.

          Descriptive Statistics
          Let us start with the concept of a ratio. What is a ratio? mean? A mean, not to be confused with a median or mode, is the arithmetic average of a set of numbers. In very simple terms, it is the sum of a bunch of something (e.g., time in seconds to complete a task) divided by the number of some other unit (e.g., number of participants measured), which results in the number of something per unit. Another way to think about it is to imagine that you collect something from a bunch of contributors and then redistribute the values evenly among all of the contributors. Although this seems like a simple enough concept, many people fail to realize that means are just an equal distribution of some “resource” among all contributors. Understanding this principle will go a long way in helping you see many of the formulas for what they are – just fancy arithmetic means.

           

          Basic Experimental Designs
          There are three common experimental designs. They consist of using two different groups of participants to evaluate a single system/design; using the same group of participants to evaluate two different systems/designs; and using three or more groups.