Turf Quick Help

" Please click on +/- sign to expand/collapse the sections "

1-TURF
- What is Turf?
  
  Turf is an EHR usability toolkit software developed by SHARPC at the University of Texas Health Science Center at Houston. It is an all-in-one integrated software system and the first software program designed specifically for safety-enhanced design validation for EHRs. Turf hosts a variety of features for usability testing of an EHR by streamlining User Testing and Heuristic Evaluation using descriptive analysis, inferential statistics, etc. The methodology of Turf is based upon the TURF framework of four analyses (Task, User, Representation, and Function) that should be conducted prior to and during development of an EHR system. This framework created by Drs. Zhang and Walji (2011) is intended to aid researchers define, evaluate, measure, and design EHR systems. The purpose of this guide is to explain and demonstrate the many features of the Turf software.
  
  References
  
  Zhang J, Walji MF. TURF: Toward a unified framework of EHR usability. Journal of Biomedical Informatics. 2011; 44(6):1056-67. Doi:10.1016/j.jbi.2011.08.005.

2- Installation Guide

System Requirement

Administrator privileges are required to install Turf. Turf 2.0.1 or later will not interfere with previous installation of Turf and multiple versions can be used simultaneously. Turf can be installed in any drive. However, the files generated when using Turf will be stored in the user’s My Documents folder.

List of System Requirements

Operating System:	Windows® 8.1, and 10
Prerequisites:	.NET framework version 4.5.3 or higher
Processor:	1 GHz or faster
RAM:	4GB or above
Free disk space:	250 MB (for application only)
Screen Resolution:	Up to 1920 x 1080
User Access Control Privileges:	Local system administrator or higher

Note:
-Turf 4.0 has been tested in Window 8.1 and 10.

-Storage requirements depend on usage. A typical recording with audio, video, and key stroke events will generate approximately 5-10MB data per minute. For seamless recording and playback, a dedicated video memory (GPU) is preferred.

System requirements to install Turf for Android in your smartphone or tablet
- Operating System: Android® 5.0 or higher
  
  Free disk space: 2 GB (to store the recorded video)
  
  System Privileges:
  Enable developer mode, USB debugging and install of third party apps.
  
  ..0+

Installing Turf: First Steps
- Choose if you want to install Turf for all users or just for you. Click the Install button to begin installation.
- Installation will begin and the process is indicated in the progress bar.

Installing Turf: Final Step

3- Settings
- Use Cases
  - Creating a new Use Case
    - Create a new use case in the Settings tab. Click on Use Cases button in the System Configuration Management sidebar on the left. Click New Use Case in the Use Case mini tab. A popup box will appear, add a name and a description then click the Save button.
  - Upload Files to Use cases
- Descriptive Templates
  - Descriptive Templates are forms used for data entry. They are used in Heuristic Evaluations to describe problems and in User Testing to record demographics, moderator notes, performance evaluation and system usability surveys.
    .
  - To create a new descriptive template:
    - Click Descriptive Templates in the Settings tab
    - Click the New button
    - Enter a name, specify dimensions of width and height in pixel, and write a brief description
    - Click the Add button to add elements of the form
    - Enter a name for the element. This name will become the variable name or column header label in the analytics functionality of Turf as well as in exported files
    - Choose Type from the dropdown menu and select from a list of editable templates.
  - - Click on Content to enter information and values of the field
    - Once the Fields tab has been completed, click on the Design Tab to view the template in a visual layout. You can move the elements using the cursor and arrange them by specifying the top and left coordinates.
    
    *NOTE: If the dimensions are small, some of the elements will not be visible in the Design tab view. Adjust the dimensions until all the elements can been seen.*
    *NOTE: Templates cannot validate input.
    
    You can export the template in XML format using the Export button and can later be imported in another system using the Load button.
    
    - Click the Preview tab to preview the template form
    - Click Save to save the template form
    
    Descriptions of Design Type
    
    Text Box:
    
    A text box accepts only one line of input. Enter a name to label the text box and select Text Box under the Type column. You can adjust the width of the textbook, but you cannot set the minimum or maximum characters allowed.
    
    Text Area:
    
    A text area accepts multi-line input. Enter a name and select TextArea under the Type column. Width is adjustable.
    
    Date Picker:
    
    Date picker can be used to input a date. The default format is: mm/dd/yyyy. The format cannot be changed and input ranges cannot be set.
    
    Combo Box:
    
    Combo Box provides a drop down selection. To enter the options, click the Content button. Enter the options for each line. The Combo Box only allows one option to be selected.
    
    Radio Box:
    
    Radio box lets you set radio button options. The options are entered for each line and only one option is selectable.
    
    Check Box:
    
    Check box lets you set check box options. This allows multiple options to be selected. The options are entered for in each line.
    
    Caption:
    
    A simple text display that can display text, but does not accept input.
- Color Sets
  - Turf comes with a small set of default color sets for marking windows events and images. These colors can be edited in the appropriate categories.
    
    Edit color set by clicking on Color Set in the Settings tab.
    Choose Window event color or Mark property color from the dropdown Color Category box.
    Click Add Field and enter a name for color, choose the correct color and enter a description. Check the Usable checkbox to make the color active.
    Then click Save to complete Color edits.
- Widget Definitions
  - Users can assign different icons, such as, Button, TextBox, List Box, Check Box or ComboBox to problems found in the images captured from different resources, such as, a HTML page, an Android device, an IOS device, or a windows screenshots.
    
    To add a new widget:
    
    - Select your OS (Windows/Android)
    - Click on the "Add Widgets" button at the top of the right panel to open up the "Add Widget" window
    - Select your desired icon, enter a name and description, and click on the Save button
    
    To delete a widget:
    
    - Select your desired widget
    - Click on the X button that is assigned to the selected widget
- Hotkeys

4- Introduction to Heuristic Evaluation

Introduction

Introduction to Heuristic Evaluation

Heuristic evaluation or expert review is a usability inspection method in which usability experts use usability heuristics (i.e., "rules of thumb") to identify user interface challenges. This technique is often described as a " discount" method for finding usability problems as it can identify potential usability issues in a timely and economical fashion. To conduct heuristic evaluation, usability experts inspect a system for violations against established usability guidelines. Numerous guidelines of good user interface design exist (Nielson and Molich, 1990; Zhang, 2011). The National Institute of Standards and Technology (NIST; 2012) published the following design principles:

**NIST Design Principles**
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Help users recognize, diagnose and recover from Error
Error Prevention
Recognition rather than recall
Flexibility an deficiency of use
Aesthetic and minimalist design
Help and documentation
Pleasurable and respectful interaction with the user
Privacy

After usability problems have been identified, they are then merged into a single master list. Evaluators then independently rate the severity of each problem. Average severity scores are then calculated. For more information please see NIST Guideline "Technical evaluation, Testing, and Validation of the Usability of Electronic Health Records" (2012).

Sources:

Nielsen, J., Molich, R. (1990). Heuristic evaluation of user interfaces. CHI '90 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 249-256.

Lowry, S. Z., Quinn, M. T., Ramaiah, M., Schumacher, R. M., Patterson, E. S., North, R., Zhang, J., Gibbons, M.C., Abbott, P. (2012). Technical evaluation, Testing, and Validation of the Usability of Electronic Health Records. National Institute of Standards and Technology

Zhang, J., Walji, M.F. (2011). TURF: Toward a unified framework of EHR usability. J Biomed Inform. 44(6): 1056-1067.

Launch and Set Up a New Heuristic Evaluation Project
- User launch Turf – Step 1
  - Step 1: Launch Turf. After loading the software on the computer, a user will launch Turf by double clicking on the Turf desktop icon.
    
    Then the user will see the File interface where user can create a new product for the heuristic evaluation.
- Create a new product evaluation folder – Step 2, 3
  - Step 2:
    
    Click the Add New Product
  - Step 3: Fill in product information in the Product dialog box.
  - - Enter the product name (e.g., SharpC-EHR)
  - - Enter the product version (e.g., Alpha)
  - - Enter the vendor name (e.g., SharpC)
  - - Click the Save button
- User sets up a new heuristic evaluation project – Step 4, 5
  - After you set up a new product, choose the type of usability method (e.g., Heuristic Evaluation or User Testing) to conduct. In this case, choose Heuristic Evaluation.
    
    Step 4: Select Heuristic Evaluation
    Click Heuristic Evaluation icon in the Home tab.
    
    Step 5: Fill in Heuristic Evaluation project information in the pop-out window
    The popup box collects information about the evaluation you will conduct. For example, the Project Name is the EHR name and Description is the type of evaluation. Filling in these boxes completely help to recognize distinct projects in the panel tree.
    
    - Enter project Name (e.g., SharpC EHR Heuristic Evaluation)
    
    - Select predefined use case (e.g., check Electronic Prescribing checkbox).
    
    - Click the Next button
    
    - Select predefined heuristic evaluation template (e.g., select Expert Review). The default Expert Review templates are provided by Turf according to NIST expert review test procedures (2012). Your administrator can also create custom forms for you to select.
    
    - Click the Finish button.
  - There is an option to set naming conventions for the image, video and problem files when the Advanced Settings button is clicked.
    
    Note:You cannot currently add these templates to a project later, so choose any you think you will use now.
Evaluate Heuristic Problems
- Capture a screenshot – Step 6, 7
  - Screenshots can be captured to inspect interface design violations.
    
    Step 6:
    -Select the HE Project
    -Click on New Folder in the Heuristic Evaluation tab
    -Rename folder to reflect the project and product (e.g., HE SharpC EHR)
    
    Step 7: Capture a Screenshot
    
    Configure Screen settings before capturing a screenshot:
    - Click Settings under the Screen checkbox
    - Click Identify Monitors button, then select the monitor that displays the image to be evaluated
    - Select a region (e.g., click Full Screen button)
    - Click OK button
    - Click the Capture Image (or press F5 on the keyboard), (e.g., Click the Capture Image (F5) button to capture the Medication Search screen)
- Physically mark and identify problems on the screenshot– Step 8, 9
  - Step 8: Mark a problematic widget
  - - Identify the problem by marking the widget, (e.g., the medication search textbox).
  - - To mark the problem, drag the mouse over the widget (e.g., mark the medication search textbox)
  - - Click Save to save the mark
  - Step 9: Describe the identified problem above
    
    First, open a problem description form, which is default by Turf. Administrator can create new forms or change forms in the administration setting. There are two ways to open a problem description form.
    
    Method One: Open a problem description form, right Click a mark, then left click Add a Problem about This Mark option
    
    Method Two: Open a problem description form:
  - - Left click New problem button
  - - Left click Select Marks button
  - - Select Mark “1” in the Scope Select pop-out window where you select the marks which you want to document problems.
    
    - Click OK button
  - Once the default template form provided by Turf is open, please complete the steps below.
  - 1- Enter the Problem Location (e.g., Medication Search Box).
    
    2- Fill Problem Description (e.g., Truncated drugs’ name and dosage).
    
    3- Select an appropriate Design Violation (e.g., Select "08-Recognition rather than recall")
    
    4-Click Save button to save successfully
    
    5- Click X to close the problem description window
  - You repeat steps 1-5 until all problems have been identified and documented.
Audit and Rate Heuristic Problems
- Audit heuristic problems- Step10
  - Step10: After you take screenshots and mark potential problems, you proceed to audit the identified problems by selecting the appropriate option. For each problem the evaluator can either:
    
    - Archive a problem, saving it for discussion with another assessor
    
    - Agree with the problem and the description
    
    - Discard the problem as not being problematic
    
    During the auditing, Turf also indicates how many violations remain unclassified.
- Rate Severity for each heuristic problem- Step11
Generate Heuristic Evaluation Report- Step12
- All the results can be summarized into a report. Preview the heuristic evaluation report and save it in a preferred file format.
  
  Step 12: Generate heuristic evaluation report:
- -Click the Report button
- -Check/uncheck the checkboxes and radio buttons based on your preferences
- -Click the print button at the bottom of the page
- -Print the report or export it in your preferred format by clicking on the correspond buttons
- -Preview the report and export the file in a format based on users’ preferences, like PDF, HTML, Word, CSV, Text, Image, etc.
Heuristic Evaluation Findings Analyzing Heuristic Evaluation Findings
- Descriptive Analysis
  - After you have the heuristic evaluation findings, you can analyze these findings by using Turf analytical functions.
    The descriptive analyses within Turf include of calculating the sum, mean, variance, standard deviation, range, minimum and maximum data, kurtosis and skewness. You choose the analytic method(s) that you want to apply, and then generate analysis results. These descriptive analysis results can help to assess the overall usability level of a product and identify the major usability problems.
    
    Step 13: Calculate the mean, range, minimum and maximum of severity scores:
    
    For the severity scores assigned to problems, we are interested in mean severity scores, the ranges, as well as the minimum and maximum scores.
    
    Query Files:
  - -Go to Analyze tab
    
    - Click Query Files
    
    -Select the project – Click rEHR, then click rEHR-Prescribing, click Electronic prescribing folder
    
    -Select the images where usability problems have been identified in previous steps
    
    -Click OK
  - - Drag the Image Name to the lower panel
    
    - Select the column of Average Score
    
    - Uncheck the Sum, Variance, Std. deviation, Kurtosis and Skewness checkboxes
    
    - Click the Run button
  - Here, Turf displays the descriptive analysis results based on your selections.
- Count the Numbers
  - In addition, you can count the numbers of each documented error type and each design violation by using Turf Count Function. The count function helps evaluators understand which are the most frequent errors and violations in one project.
    
    Query Files:
  - - Go to Analyze tab
    
    - Click Query Files
    
    -Select the project – Click rEHR, then click rEHR e-prescribing, click Electronic prescribing folder.
    
    -Select the problems (e.g., select “Prob_1”, “Prob_2”, “Prob_3”, “Prob_4”).
    
    -Click OK
    
    -Select the column of Design Violation
    
    -Click the Count button
    
    You will get the count of design violation.

5- User Testing
- What is User Testing?
  - Introduction
    - User testing is a technique used to evaluate an EHR product by testing it with representative users. In the test, these users will try to complete typical tasks while moderators watch, listen and takes notes. The goal is to identify usability problems, collect quantitative data on participants' performance (e.g., time on task, error rates), and determine participant's satisfaction with the product. There are 3 unique types of user testing that can be performed on a product: formative testing, summative testing and validation testing. These different types will be further detailed for clarification.
  - Formative Testing
    - User testing can be part of the formative testing of a product that is conducted early in the development lifecycle. In this regard, it is used to assess the usability of preliminary designs and prototypes in order to facilitate rapid improvements during the design process. Consequently, formative user testing does not require a fully developed system, but needs enough functionality to answer the goals of the usability study. Formative user testing focuses on qualitative data collection in which the user walks through processes or explores an interface rather than performing a well-defined scenario.
  - Summative Testing
    - User testing is also conducted after high-level design has been specified or in summative testing. In this case, it is used to determine how well a user performs various tasks. Summative testing involves collection of both quantitative and qualitative measures. Summative tests can be designed to measure system effectiveness and efficiency as well as user satisfaction with a system.
  - Validation Testing
    - Finally, user testing is conducted for validation testing of a product before it is released to the general public. Validation testing measures the usability against specific predetermined standards or benchmarks such as “Assess if 80% of users can successfully complete task.” This involves the collection of mainly quantitative data.
- Launch Turf and Set Up a New User Testing Project
  - Step 1: Launch Turf
    
    After loading the software on the computer, a user will launch Turf by double clicking the desktop icon. Then the user will see the Home tab where the user can create a new product for the user testing.
  - Step 2: Create a new product evaluation folder
  - You will create a folder to hold all of the user testing projects for one EHR product. This allows you to test multiple tasks (e.g., e-prescribing, medication reconciliation) for a single product. You will create one folder for the product and separate folders within it for each task.
    
    - Go to Home tab
    
    - Click the Add New Product button
    
    Step 3: Fill in product information in the Product dialog box
    
    - Enter the Product name (e.g., Demo EHR)
    
    - Enter the product Version, (e.g., 1.0; any alphanumeric characters accepted)
    
    - Enter the Vendor name, (e.g., SHARPC)
    
    - Click the Save button
  - Step 4: Create a user testing project folder
    
    - Click the User Testing button. Make sure the product folder you created is highlighted.
    - Enter a project name (e.g., User Testing 1). If you like, you can add a description, such as: "Users: Doctors".
    - Select the predefined use cases, e.g., check Electronic Prescribing and Clinical Information Reconciliation. You cannot add these to a project later, so choose all you think you will use now. We have included the NIST Use Case documents for the eight required functions for Meaningful Use Stage 2. Your administrator may add other reference materials for you).
    - Click the Next button
  - Step 5: Associate data forms with your project
    
    There are four templates for recording user-testing data preloaded with Turf, which include:
    
    Demographics:
    
    Collect demographics and experience data about your users. Completed by the user.
    
    System Usability Scale:
    
    Assesses satisfaction with the system. Completed by the user.
    
    Moderator Notes:
    
    Free text for notations about the user testing experience. Completed by moderator.
    
    Performance Evaluation:
    
    Structured form to evaluate the user performance. Completed by moderator.
    
    In this step you will associate the correct form with the button from which you will access it. You cannot add these to a project later, so it is best to choose any you think you will use now.
    
    - Using the drop down menu, select the data templates you want to use.
    
    - The Advanced Settings button allows you to customize the naming conventions for your image, video and problem files. For now, we will use the default settings.
    
    - Click Finish
  - Step 6: Create folders to hold each user’s data
    
    - Click on your desired project
    
    - Click New Folder
    
    Rename the folder to reflect the project and product.
- Test Users
  - Complete Administrative Documents – Step 7
    - There are three generic forms for administrative purposes associated with each Use Case document. These forms can be edited within Turf to customize them for your organization and purposes. They can then be printed and signed by your users.
      
      The available generic administrative forms include:
      
      - Non-Disclosure Agreement (NDA)
      
      - Acknowledgement of Receipt Form
      
      - Informed Consent Form
      
      Step 7: Prepare Administrative Documents
      
      - Select a use case folder (e.g., Click the §170.314b(3) Electronic Prescribing icon under User Testing 1 project
    - - Click on the use case to open its content in the Details Panel which is at the bottom of the screen
      
      - Double click on the form you want to edit
      
      - Edit the form as you would in a standard word processing program
      
      - Print the form
  - Capture Data from a User – Steps 8, 9, 10, 11, 12, 13
    - Step 8: Collect data on the Demographics Form
      
      - Select the folder for the user you are going to test
      
      - Click the Demographics icon on the Forms ribbon
      
      - Have your user complete the form
      
      - Click Save
      
      In Step 9, we will set up electronic data capture. For this example, we are not using the webcam. But we demonstrate accessing settings for capturing Audio and Keyboard. In your testing session, you can accept default settings without clicking on Settings for these.
      
      Step 9: Set up electronic data capture
      
      Be sure the EHR product you want to test is available to your users at the point you want them to begin the test. Also, uncheck the checkbox for any electronic data capture method you will not be using.
    - Audio Settings:
      - Click Settings under the Audio check box to change audio settings
    - - Click Ok
    - Keystroke Settings:
      - Click Settings under the Keyboard check box to change keyboard and mouse settings
      - Click Ok
    - Video Settings:
      - Click Settings under the Video check box to change video screen capture settings
      - Set Delay to 3 seconds so that video capture begins when the user is working on the EHR
    - Screen Settings:
      If the user is on a second monitor, choose the appropriate monitor
      - Click Settings under the Screen check box to change screen settings
      - Click Ok on the Screen Settings dialog box
    - Step 10: Capture video and audio of the user performing a task in the EHR being tested
      
      - Click Record
      Turf will be minimized and the view will switch to the EHR being tested. Let the user perform the tasks per your instructions
      - When you are ready to stop the test, click on the Turf icon in the taskbar to maximize it then click on the Stop button or use the hotkey F9
      - The system will display the data after a few minutes
      
      Step 11: Administer the System Usability Survey
      
      - Click the Usability Survey button on the user testing ribbon to open the SUS
      - After the user complete the survey, click Save
      - Name the saved file (e.g., “SUS_DocA”)
      - Close the SUS form
      
      Step 12: Complete the Moderator Notes form
      
      - Click the Moderator Notes button to open the form
      - Make notes about the test, click Save
      - Name the saved file (e.g., “Note_DocA”)
      - Exit out of the Moderator Notes form
      
      Step 13: Export User data
      
      - Select the file or files you want to export
      - Click Export Files
      - Select a folder in which to save the exported files
      - Click OK
- Set up an Autoflow Session
  - Autoflow Configuration
    - You can use Autoflow to predefine some actions within Turf. The program will then run these actions automatically when the moderator starts the user testing. Before using Autoflow, you first must configure its settings.
      
      Autoflow Configuration:
      
      In Autoflow configuration, you can add a sequence of actions. The actions may include:
      
      -Read PDF File
      -Read RTF File
      -Play Video
      -Capture Image
      -Media Record
      -Fill in demographic
      -Fill in SUS
      -Fill in Moderator Notes
      -Fill in Performance Note
      -Display Text Screen
      
      You can specify the actions and the orders of the actions to complete a user testing.
      
      Step 1: Add a new Autoflow configuration
    - - Select a product, for example:Select Demo EHR
      
      - Select a project, (e.g., User Testing 1)
      - Select a folder, (e.g., Electronic prescribing)
      - Click Set Up Autoflow button
      - Click New button in the popup box that appears
      - Enter the configuration Name (e.g., e-prescribing)
      - Enter the configuration Description (e.g., "Product: Demo EHR")
      - Set this configuration as the default Autoflow configuration by checking the Default checkbox
    - Step 2: Read PDF File
    - - Click the Add button
    - - Enter "Informed Consent" in the Name column
    - - Select Read PDF File in the Action column
    - - Click the
      
      - Click the button in the Content column to choose which PDF file will be read and click Open
    - Step 3: Fill in Demographic Form
    - - Click the Add button
    - - Enter "Demographic Form" in the Name column
    - - Select Fill in Demographic in the Action column
    - - Click the button in the Folder column to choose a folder to store your demographic forms. The demographic form filled by the user will be saved automatically into this folder at the end of the Autoflow session. You can Add Folder, Rename or Delete an existing folder
    - - Click OK button
    - Step 4: Media Record
    - - Click the Add button
    - - Enter Recording Task 1 in the Name column
    - - Select Record Selected Audio/Video/Keystroke/Screen in the Action column
    - - Choose a folder to store your media files. The media files recorded will be saved automatically into this folder at the end of the Autoflow session
    - - You can Add Folder, Rename or Delete an existing folder
    - - Click OK button
      
      Step 5: Display Text Screen
    - - Click the Add button
    - - Enter "Debrief" in the Name column
    - - Select Display Text Screen in the Action column
    - - Enter "Thank you for your time!" in the Content column
    - Step 6: Save Autoflow Configuration
    - - Click Save button
    - - Close Auto Flow dialog box
  - Run an Autoflow Session
    - The steps listed here are based on the configurations we performed on the "Auto Flow Configuration" section.
      
      Step 1: Run a default Autoflow session
    - - Click Run button
    - - A PDF file of Informed Consent will be displayed
      
      Step 2: Read PDF File
    - - Click Next button or Skip button to go to the next action
    - - Click End Session button to end this Autoflow session
    - We click Next button here, a demographic form will be displayed
    - Step 3: Fill in Demographic Form
    - - Fill in this User Demographics form
    - - Go to the next action
    - - If you click Next button, Turf will go to the next action and save this form automatically
      
      - If you click Skip button, Turf will go to the next action and drop this form
    - - If you click End Session button, Turf will end this Autoflow session
    - We click Next button here and Turf will minimize and start recording user’s operations.
    - Step 4: Media Record
    - - Press F9to stop recording
      
      You can pause or restart the recording by clicking Pause/Restart Record button or by pressing F8. Or you can stop the recording by clicking Stop Record button or by pressing F9. A video or a windows event data will be displayed on the screen
    - - Go to the next action
      
      If you click Next button , Turf will go to the next action and save the media file(s) automatically.
      
      If you click Skip button , Turf will go to the next action and drop the media file(s)
      
      If you click End Session button , Turf will end this Autoflow session
      
      We Click Next button here and a Text Screen will be displayed here.
    - <
      Step 5: Display Text Screen
    - - Click End Session button to exit this Autoflow session
6- OneDrive
- Upload to OneDrive
  - Follow the following steps to upload to OneDrive
    
    - Go to the File Tab
    
    - Click OneDrive button on the left navigation bar
    
    - Click on the login button at the bottom right of the screen
    
    - Enter your username and password to log in to your OneDrive account. If you do not have an account, you will need to create one
    
    - An alert window will pop up warning that Turf will access your account info- Click on the Yes button to proceed to the next step
    
    - Go to Setting Tab
    
    - Click OneDrive Settings button
    
    - Change the value of the first field, “Automatically upload files to OneDrive” to True, if it is set to False. Setting the value to True allows you to synch your local files with the uploaded files. This means that every changes that you make to your local files will be automatically applied to their counterparts in OneDrive.
    
    - Go to the Projects panel tab- (Click on the Home tab)
    
    - Right click on your desired folder. Go to “OneDrive” menu item, next, click on “Include in Sync”
    
    - If the value for ”Automatically upload files to OneDrive” field is not set to True, you must click “Sync Now” from the context menu.
    
    - An icon on the right lower corner of the screen will appear indicating that the data are being uploaded
    
    - Upon uploading a project / folder, you will notice that TurfData4.0 folder is also created in OneDrive. This folder will include data from Turf. The organization of sub-folders is as follows: Product >> Project >> Folders >> Files.
7-Turf for Android
- How to setup your device
  - The number of steps or the suggested configuration may vary depending on the devices. The following are suggested steps for any devices with Android 5 or greater. Here, we will present the instructions for Nexus 9 tablet.
    
    Step 1: Check your android version
    
    Setting >> System >> About tablet >> Android Version The version of your Android device OS must be at least 5.0. If the OS version is less than 5, please upgrade it to Android 5.0 or later.
    
    Step 2: Enable Developers mode
    
    -Go to Settings
    -Slide down to the “System” panel
    -Click on “About tablet”`
    -Press the “Build Number” item seven times
  - Step 3: Configure Developer settings
    
    -Go to Settings
    -Slide down to the “System” panel
    -Click on “Developer options”
    -Check the checkbox to select “USB debugging”
  - Step 4: Enable side loading apps
    
    Open Settings >> Personal >> Security >> Device Administration
    -Set Unknown Sources to ON
- Device Driver Setup
- Install USB driver in Windows
  - Step1: Download
    
    -Go to the following link to download the Google USB Driver on your desktop computer https://developer.android.com/studio/run/win-usb.
  - Step2: Instructions for installing the driver
    
    -Unzip the downloaded folder by extracting its components - Connect your Android device to your computer using a USB
    -Right click on the Computer icon and click on the “Manage” item
    -Click on “Device Manager” from the left panel
    -Click on the small arrow next to the “Other devices” to expand it
    -Right click on the “ADB interface”, and select “Update Driver Software”
    -Browse for Driver software on your computer and wait for the device to be installed and enabled
    -After installation unplug the device
- Configuration and Installation
  - Follow the following steps to utilize the Turf android app. The number of steps or the suggested configuration may vary depending on the devices. The following are suggested steps for any devices with Android 4.4 or greater.
    
    Step1: Install Android in Turf
    -Connect the device to your computer
    -Open Turf
    -Go to Settings
    -Select “Install android”
    -Click on the “Install” button
    
    Step2: Activating Turf Keyboard and Turf Launcher
    
    -Open the Android device
    -Go to Settings
    -Go to “Personal” panel
    -Click on “Language and input”
    -On the “Language & input” screen, click on “Current keyboard”
    -Click on “CHOOSE KEYBOARDS”
    -Turn on the “turf Keyboard” item
    -Click on the OK button on the “Attention” window
    -Get back to the “language & input” window and click on “Current Keyboard”
    -Click on the “turf Keyboard” radio button to select it
    -Go back to Settings
    -Go to the “Device” panel
    -Click on the “Home” item
    -Click on “Turf launcher” radio button to select it
    After successfully installing Turf on your Android device, you will be able to use Turf Android features.
  - Note: After you finish working with Turf, make sure to turn off the Turf keyboard and launcher in your android device. This is to ensure that your device activities are not being logged.
- Capture events using Turf Android
  - Follow the following steps to use Turf Android features
    
    -Go to User Testing tab, select the folder where you want to store your data
    -Check the checkbox for Android in the Capture Media ribbon, under the User Testing tab at the top of the screen, to select it
    -Click on Android Settings to select the parameters that you would like to capture
    -Press Record (F7) button to record data
    -Once you are done with recording, click Stop Record (F9) button to stop the recording.

8- User Testing Analytics

Analyze User Testing Data

Introduction

Currently Supported Statistics in Turf

Commonly used measures of descriptive statistics have been included in Turf. These include:

<<Statistic Type>>
Summative	Inferential
Sum	One-way ANOVA
Minimum & Maximum	Mann-Whitney U Test
Range	Two-sample t Test
Mean	Linear Regression
Variance	Pearson Correlation coefficient
Standard Deviation
Kurtosis
Skewness

Summarize, Describe or Present Data
- After collecting data from the user testing, Turf analytics can help you analyze the data and present the results.
  
  Descriptive Analysis of Demographic Data
  
  Query demographic data:
  - Go to Analyze tab
  - Click the Query Files button
  - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
  - Select demographic form (e.g., Demo_1 and Demo_2). The name and path of the selected files will be displayed at the bottom of the Query files dialog
  - Click OK
  
  Select descriptive measures:
  
  Choose the descriptive measure(s) you want to calculate by checking the checkbox of the measure(s), e.g., Mean and standard deviation.
  
  Choose the descriptive measure(s) you want to calculate by checking the checkbox of the measure(s), e.g., Mean and standard deviation.
  
  Select data and calculate:
  
  - Select the data you want to analyze (e.g., Age)
  - Click Run button
  
  Calculate Average SUS Score
  
  The System Usability Survey (SUS) can be used to quickly test the usability of a product. After collecting the surveys with 10 items from all participants, you can convert the raw scores of each survey into an overall SUS score that ranges from 1 to 100. Then you can use the descriptive analytics in Turf to calculate the average SUS score.
  
  Query SUS:
  
  - Click Query Files button
  - Select a Product
  - Demo EHR
  - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
  - Select SUS files, e.g., SUS_1, SUS_2, and SUS_3
  - Click OK
  
  Calculate overall SUS score:
  - Select the three rows
  - Click SUS Score button
  
  Calculate average SUS score:
  
  - Select SUS Score column
  
  - Select Mean by checking the checkbox of mean
  
  - Click Run button
Comparison between Different Groups
- Turf provides statistical methods to compare the usability of different products. Also available within Turf are comparative tests including: two-sample ttest or Mann-Whitney U test to compare two groups of samples and one-way ANOVA to compare three or more groups of samples.
  
  Compare Task Time with Pivot Table/Chart
  
  Step1: Query task time
  - Go to Analyze tab
  - Click Query Files button
  - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
  - Select windows event files, e.g., Win_1 to Win_10
  - Select Task time
  - Click OK
  
  Step2: Add categorical columns
  
  Step2: Add categorical columns
  -Right click on Task time column header and click Insert Column
  -Enter Column Name, e.g., Product Name
  -Enter EHR 1 for the first five rows of Product Name Column to indicate it’s Product 1, enter EHR 2 for the last five rows of Product Name Column to indicate it’s Product 2
  -Click OK
  
  Use the same steps to add another column named User Group.
  
  Step3: Switch to pivot table and chart
  
  - Click Switch to Pivot Table/Chart button
  - Put your mouse over Data Headers
  - Drag Product Name button to Drop Row Field Here, drag User Group button to Drop Column Fields Here and drag Name button to Drop Filter Fields Here
  
  Compare Task Time with One-way ANOVA
  
  Step1: Query task time
  
  - Go to Analyze tab
  - Click Query Files button
  -Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
  - Select windows event files, e.g., Win_1 to Win_10
  - Select Task time
  - Click OK
  
  Step2: Add categorical columns
  
  - Right click on Task time and click Insert Column
  - Right click on Task time and click Insert Column
  - Enter
  
  - Right click on Task time and click Insert Column
  - Enter Column Name, e.g., Product Name
  - Click OK
  - Enter EHR 1 for the first three rows of Product Name Column to indicate it’s Product 1, enter EHR 2 from the fourth to sixth rows of Product Name Column to indicate it’s Product 2, enter EHR 3 for the last four rows of Product to indicate it’s Product 3
  
  Step3: Choose variables and calculate
  - Select Product Name and Task time Column
  - Click One-way ANOVA button
  - Add Product Name as Factor
  - Add Task time to the Dependent list
  - Click OK
  
  Compare Task Time with Mann-Whitney UTest
  
  Step1: Query task time
  
  - Go to Analyze tab
  - Click Query Files button
  - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
  - Select windows event files, e.g., Win_1 to Win_10
  - Select Task time
  - Click OK
  
  Step2: Add categorical columns
  
  - Right click on Task time and click Insert Column
  - Enter Column Name, e.g., Product Name
  - Click OK
  - Enter EHR 1 for the first five rows of Product Name Column to indicate it’s Product 1, enter EHR 2 for the last five rows of Product Name Column to indicate it’s Product 2
  
  <
  
  Step3: Choose variables and calculate
  
  - Select Product Name and Task time Column
  - Click Mann-Whitney U button
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list
  - Click OK
  
  Compare Task Time with Two-Sample t Test
  
  Step 1: Query task time
  
  - Go to Analyze tab
  - Click “Query Files” button
  - Select a Product, e.g. Demo EHR -> Select a project, e.g. Heuristic Evaluation 1 -> Select a folder, e.g.: Electronic prescribing ->
  - Select windows event files, e.g., Win_1 to Win_10
  - Select Task time
  - Click OK
  
  Step2: Add categorical columns
  
  - Right click on Task time and click Insert Column
  - Enter Column Name, e.g., Product Name
  - Click OK
  - Enter EHR 1 for the first five rows of Product Name column to indicate its Product 1, enter EHR 2 for the last five rows of Product Name Column to indicate it’s Product 2
  
  Step3: Choose variables and calculate
  
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list.
  - Select Unpaired and None
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list.
  - Select Unpaired and None
  - Click OK
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list.
  - Select Unpaired and None
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list.
  - Select Unpaired and None
  - Click OK- Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list.
  - Select Unpaired and None
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list.
  - Select Unpaired and None
  - Click OK
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list.
  - Select Unpaired and None
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Select Product Name and Task time Column
  - Click Two-Sample t button.
  - Add Product Name as Group Variable
  - Choose EHR 1 for Group 1, choose EHR 2 for Group 2
  - Add Task time to the Test Variable(s) list.
  - Select Unpaired and None
  - Click OK

9- Reporting Bugs and Feedback
- To submit the bugs or your feedbacks, Email us at SHARPC@uth.tmc.edu. We will try our best to respond to you within one working day.
10- Technical Support
- Technical support is available by appointment only. Email us at SHARPC@uth.tmc.edu for technical support.
11- Glossary
- Errors
  
  Errors may include clicking on an incorrect menu item, incorrect link, or interacting incorrectly with an on-screen control.
  
  Formative Testing
  
  User testing can be part of formative testing of a product that is conducted early in the development lifecycle. In this regard it is used to assess usability of preliminary designs and prototypes in order to facilitate rapid improvements during the design process. Consequently, formative user testing does not require a fully developed system, but needs enough functionality to answer the goals of the usability study. Formative user testing focuses on qualitative data collection in which the user walks through process or explores an interface rather than performing a well-defined scenario.
  
  Heuristic Evaluation
  
  A heuristic evaluation is a usability inspection method for computer software that helps to identify usability problems in the user interface (UI) design. It specifically involves evaluators examining the interface and judging its compliance with recognized usability principles (the "heuristics"). These evaluation methods are now widely taught and practiced in the New Media sector, where UIs are often designed in a short space of time on a budget that may restrict the amount of money available to provide for other types of interface testing.
  
  Online surveys
  
  Online surveys are structured interviews with users, where you display a list of questions online and record users' responses.
  
  Persona
  
  A persona is a fictional person who represents a major user group for your product. Personas help you identify major user groups of your product. You select the characteristics that are most representative of those groups and turn them into a persona.
  
  Prototype
  
  A prototype is a draft version of a product. Prototypes allow you to explore your ideas before investing time and money into development. A prototype can be anything from a paper drawing (low-fidelity), click-through of a few images or pages, or a fully functioning product (high-fidelity).
  
  Scenario
  
  A scenario is a short story about a specific user with a specific goal at your site. Scenarios are the questions, tasks, and stories that users bring to your Web site and that the Web site must satisfy. Scenarios are critical both for designing Web sites and for doing usability testing.
  
  Summative Testing
  
  User testing is also conducted after high-level design has been specified or in summative testing. In this case, it is used to determine how well a user performs various tasks. Summative testing involves collection of both quantitative and qualitative measures. Summative tests can be designed to measure system effectiveness and efficiency as well as user satisfaction with a system.
  
  SUS
  
  In systems engineering, the system usability scale (SUS) is a simple, ten-item attitude Likert scale giving a global view of subjective assessments of usability. It was developed by John Brooke at Digital Equipment Corporation in the UK in 1986 as a tool to be used in usability engineering of electronic office systems.
  
  Task Analysis
  
  Task analysis involves learning about your users' goals, what they want to do on your Web site and how they work. It can also mean learning about specific task users must do to meet those goals and what steps they take to accomplish those tasks. A task analysis complements a user analysis.
  
  Task Failure
  
  User abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion.
  
  Task success
  
  User can complete the task in the allotted time without assistance from the administrator.
  
  Time on task
  
  The length of time in seconds required for user to complete the task.
  
  Turf
  
  Is a software program based on the TURF Unified Framework for Defining, Evaluating, Measuring, and Designing EHR Usability. The acronym TURF comes from the four usability inspection methods that are utilized in the process: Task analysis, User analysis, Representational analysis, and Functional analysis. For details, please check: NIST, or NCBI.
  
  Usability
  
  Usability is defined, in the TURF framework developed by the SHARP-C project funded by ONC, as "how useful, usable, and satisfying a system is for the users to accomplish goals in the work domain by performing certain sequences of tasks."
  
  Usability Testing
  
  Usability testing is a technique used to evaluate a product by testing it with representative users.
  
  Use case
  
  A use case is a description of how users will perform tasks on your Web site. A use case includes two main parts:
  
  1-the steps a user will take to accomplish a particular task on your site.
  
  2-the way the Web site should respond to a user's actions.
  A use case begins with a user's goal and ends when that goal is fulfilled.
  
  User testing
  
  User testing is a technique used to evaluate a product by testing it with representative users. In the test, these users will try to complete typical tasks while observers watch, listen and takes notes. The goal is to identify any usability problems, collect quantitative data on participants' performance (e.g., time on task, error rates), and determine participant's satisfaction with the product.
  
  Validation Testing
  
  Finally, user testing is conducted for validation testing of a product before it is released to the general public. Validation testing measures usability against specific predetermined standards or benchmarks such as “Assess if 80% of users can successfully complete task.” This involves the collection of mainly quantitative data.

12-Statistics Refresher and Guide to Selecting the Proper Statistical Test

Purpose

Many members of the Human-computer Interaction, Human Factors, Ergonomics, or Usability fields will likely have had some training in basic statistics; we realize that this is not a universal truth. We have many colleagues with backgrounds in Graphic Design or other fields that have never been exposed to this material. It is far outside of the scope of this document to teach basic statistics, but we can at least try to focus on the most common statistical methods used in the usability testing of electronic health records (EHR). This document is meant to support those using the Turf software by providing information about the various methods available in the software, how to choose the most appropriate statistical method, and how to properly interpret the results.

The tone of this document is intended to be easy-going because the intention is to provide a reader-friendly guide to statistical analyses, not to turn readers into statisticians. In order to achieve this intent, this document tries to explain things in simpler terms than what might be found in textbooks or scientific journal articles. However, in doing so, there is a chance that parts of this document will make professional statisticians cringe because of how lightly we have treated, described, or defined some of the topics in this document. With that in mind, this text should not be considered an authoritative text on statistics, but it should still be useful to provide readers with sufficient knowledge to be able to select and utilize the most appropriate statistical methods for usability tests and how to properly report results.

Chapter Overview

This chapter covers both the summative and inferential statistics available in Turf. These statistics are found in the following table.

**Statistics used by Turf**
Statistic Type
Summative		Summative	Inferential
Sum	One-way ANOVA
Minimum & Maximum	Mann-Whitney U Test
Range	Two-sample t Test
Mean
Variance
Standard Deviation
Kurtosis
Skewness

Guide to Selecting the Appropriate Statistical Test

Normally a section like this might come at the end of a statistics text. In that this document is intended as a refresher and quick reference, it is placed here to prevent unnecessary searching. Those that need additional help will find additional information in later sections.

Turf currently supports three inferential statistical tests. These are the One-way analysis of variance (ANOVA), the Mann-Whitney U test, and the Two-sample t test. The proper test depends upon the usability experimental design (i.e., within subjects, between subjects, or mixed design), the type of data gathered (i.e. qualitative or quantitative), the measurement type (i.e., nominal, ordinal, interval, and ratio), and the number of variables (e.g., number of comparison groups) to be evaluated. Turf currently only supports statistical analysis capabilities for quantitative data from between subjects experimental designs. This means that in order to utilize the statistical analysis features of Turf, your study will need to use a different group of people for each condition (e.g., old design vs. new design; design A vs. design B).

Using the same people in both groups is a legitimate study design choice, but is known as a within-subjects or repeated-measures design. When comparing performance from only two groups, a dependent-samples t test can typically be used and is relatively simple to calculate. However, comparing more than two conditions (e.g., design A vs. design B vs design C), requires a repeated measures ANOVA, which is considerably more sophisticated than the one-way within-subjects ANOVA provided by Turf. We highly recommend that your research requires a repeated measures experimental design, and having someone in your organization trained in statistics.

If you want to design a usability study around the current statistical functionality of Turf, consider the following: Your study must use a different set of participants for each EHR design being evaluated. You cannot ask participants to be evaluated in both EHR conditions.

Most Statistics Are Fancy Ratios

It is likely that most statistics that you will calculate and report for usability testing will be based on a ratio of some sort; most likely a “mean.” Actually, this is pretty true for a lot of research in general. The field of statistics often has a tendency to turn many people off by requiring them to learn a lot of new, seemingly arbitrary formulas and symbols. However, most statistics that you are likely to encounter and use for usability testing are really just a fancy ratio or the mean of some values. As long as you remember this in addition to the fact that you probably learned all the necessary mathematics required to calculate them in elementary school, you should be able to follow the logic behind the statistics a lot more closely and prevent yourself from letting the formulas and symbols scare you off track.

Descriptive Statistics

Let us start with the concept of a ratio. What is a ratio? mean? A mean, not to be confused with a median or mode, is the arithmetic average of a set of numbers. In very simple terms, it is the sum of a bunch of something (e.g., time in seconds to complete a task) divided by the number of some other unit (e.g., number of participants measured), which results in the number of something per unit. Another way to think about it is to imagine that you collect something from a bunch of contributors and then redistribute the values evenly among all of the contributors. Although this seems like a simple enough concept, many people fail to realize that means are just an equal distribution of some “resource” among all contributors. Understanding this principle will go a long way in helping you see many of the formulas for what they are – just fancy arithmetic means.

Basic Experimental Designs
There are three common experimental designs. They consist of using two different groups of participants to evaluate a single system/design; using the same group of participants to evaluate two different systems/designs; and using three or more groups.

NCCD

Turf Quick Help

1-TURF

What is Turf?

References

2- Installation Guide

3- Settings

To add a new widget:

To delete a widget:

4- Introduction to Heuristic Evaluation

5- User Testing

6- OneDrive

7-Turf for Android

8- User Testing Analytics

9- Reporting Bugs and Feedback

10- Technical Support

11- Glossary

12-Statistics Refresher and Guide to Selecting the Proper Statistical Test