Report - Drummond Group

Transcription

Test Results Summary for 2014 Edition EHR Certification
Version EHR-Test-144 Rev 17-Feb-2014
ONC HIT Certification Program
Part 1: Product and Developer Information
1.1 Certified Product Information
Product Name:
Ibeza eConcepts
Product Version:
2.1
Domain:
Ambulatory
Test Type:
Modular EHR
1.2 Developer/Vendor Information
Developer/Vendor Name:
Ibeza LLC
Address:
2550 S Douglas Rd Coral Gables FL 33134
Website:
www.ibeza.net
Email:
gersson.oviedo@ibeza.net
Phone:
305-357-1711
Developer/Vendor Contact:
Gersson Oviedo
Page 1 of 11
Part 2: ONC-Authorized Certification Body Information
2.1 ONC-Authorized Certification Body Information
ONC-ACB Name:
Drummond Group
Address:
13359 North Hwy 183, Ste B-406-238, Austin, TX 78750
Website:
www.drummondgroup.com
Email:
ehr@drummondgroup.com
Phone:
817-294-7339
ONC-ACB Contact:
Bill Smith
This test results summary is approved for public release by the following ONC-Authorized Certification
Body Representative:
Bill Smith
ONC-ACB Authorized Representative
Signature and Date
2.2
Certification Committee Chair
Function/Title
6/27/2014
Gap Certification
The following identifies criterion or criteria certified via gap certification
§170.314
(a)(1)
(a)(17)
(d)(5)
(d)(9)
(a)(6)
(b)(5)*
(d)(6)
(f)(1)
(a)(7)
(d)(1)
(d)(8)
*Gap certification allowed for Inpatient setting only
x No gap certification
Page 2 of 11
2.3 Inherited Certification
The following identifies criterion or criteria certified via inherited certification
§170.314
(a)(1)
(a)(14)
(c)(3)
(f)(1)
(a)(2)
(a)(15)
(d)(1)
(f)(2)
(a)(3)
(a)(16) Inpt. only
(d)(2)
(f)(3)
(a)(4)
(a)(17) Inpt. only
(d)(3)
(f)(4) Inpt. only
(a)(5)
(b)(1)
(d)(4)
(a)(6)
(b)(2)
(d)(5)
(f)(5) Optional &
Amb. only
(a)(7)
(b)(3)
(d)(6)
(a)(8)
(b)(4)
(d)(7)
(f)(6) Optional &
Amb. only
(a)(9)
(b)(5)
(d)(8)
(g)(1)
(a)(10)
(b)(6) Inpt. only
(d)(9) Optional
(g)(2)
(a)(11)
(b)(7)
(e)(1)
(g)(3)
(a)(12)
(c)(1)
(e)(2) Amb. only
(g)(4)
(a)(13)
(c)(2)
(e)(3) Amb. only
x No inherited certification
Page 3 of 11
Part 3: NVLAP-Accredited Testing Laboratory Information
Report Number: KAM-0626-2472
Test Date(s): 03/11/14, 04/28/14, 05/23/14, 06/05/14, 06/18/14, 06/26/14
3.1 NVLAP-Accredited Testing Laboratory Information
ATL Name:
Drummond Group EHR Test Lab
Accreditation Number:
NVLAP Lab Code 200979-0
Address:
13359 North Hwy 183, Ste B-406-238, Austin, TX 78750
Website:
www.drummondgroup.com
Email:
ehr@drummondgroup.com
Phone:
512-335-5606
ATL Contact:
Beth Morrow
For more information on scope of accreditation, please reference NVLAP Lab Code 200979-0.
Part 3 of this test results summary is approved for public release by the following Accredited Testing
Laboratory Representative:
Kyle Meadors
ATL Authorized Representative
Signature and Date
6/27/2014
Test Proctor
Function/Title
Nashville, TN
Location Where Test Conducted
3.2 Test Information
3.2.1 Additional Software Relied Upon for Certification
Additional Software
EMR Direct phiMail
Applicable Criteria
b.1, b.2
Functionality provided
by Additional Software
Direct HISP
Page 4 of 11
Additional Software
Applicable Criteria
Functionality provided
by Additional Software
No additional software required
3.2.2 Test Tools
Version
Test Tool
Cypress
x
x
2.4.1
ePrescribing Validation Tool
1.0.4
HL7 CDA Cancer Registry Reporting Validation Tool
1.0.3
HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool
1.8
HL7 v2 Immunization Information System (IIS) Reporting Validation
Tool
1.8
HL7 v2 Laboratory Results Interface (LRI) Validation Tool
1.7
HL7 v2 Syndromic Surveillance Reporting Validation Tool
1.7
x
Transport Testing Tool
179
x
Direct Certificate Discovery Tool
3.0.2
x
No test tools required
3.2.3 Test Data
Alteration (customization) to the test data was necessary and is described in
Appendix [insert appendix letter]
No alteration (customization) to the test data was necessary
3.2.4 Standards
3.2.4.1 Multiple Standards Permitted
The following identifies the standard(s) that has been successfully tested
where more than one standard is permitted
Criterion #
Standard Successfully Tested
§170.204(b)(1)
(a)(8)(ii)(A)(2)
HL7 Version 3 Implementation
Guide: URL-Based
Implementations of the
Context-Aware Information
Retrieval (Infobutton) Domain
§170.204(b)(2)
Guide: Context-Aware
Knowledge Retrieval
(Infobutton) Service-Oriented
Architecture Implementation
Guide
Page 5 of 11
Criterion #
x
(a)(13)
IHTSDO SNOMED CT®
International Release July
2012 and US Extension to
SNOMED CT® March 2012
Release
x
(a)(15)(i)
(a)(16)(ii)
§170.207(a)(3)
§170.204(b)(2)
Guide: Context-Aware
Knowledge Retrieval
(Infobutton) Service-Oriented
Architecture Implementation
Guide
§170.210(g)
§170. 210(g)
Network Time Protocol
Version 3 (RFC 1305)
The code set specified at 45
CFR 162.1002(c)(2) (ICD-10CM) for the indicated
conditions
§170.207(i)
(b)(7)(i)
HL7 Version 3 Standard:
Clinical Genomics; Pedigree
§170.204(b)(1)
Guide: URL-Based
Implementations of the
Context-Aware Information
Retrieval (Infobutton) Domain
§170.207(i)
(b)(2)(i)(A)
§170.207(j)
CFR 162.1002(c)(2) (ICD-10CM) for the indicated
conditions
x
§170.207(a)(3)
IHTSDO SNOMED CT®
Release
x
§170.207(a)(3)
IHTSDO SNOMED CT®
Release
Annex A of the FIPS Publication 140-2
(e)(1)(i)
(e)(1)(ii)(A)(2)
[list encryption and hashing algorithms]
§170.210(g)
§170. 210(g)
Annex A of the FIPS Publication 140-2
(e)(3)(ii)
[list encryption and hashing algorithms]
AES
SHA-1
x
Common MU
Data Set (15)
§170.207(a)(3)
IHTSDO SNOMED CT®
Release
§170.207(b)(2)
CFR 162.1002(a)(5) (HCPCS
and CPT-4)
Page 6 of 11
Criterion #
None of the criteria and corresponding standards listed above are
applicable
3.2.4.2 Newer Versions of Standards
The following identifies the newer version of a minimum standard(s) that
has been successfully tested
Newer Version
Applicable Criteria
No newer version of a minimum standard was tested
3.2.5 Optional Functionality
Criterion #
x (a)(4)(iii)
Optional Functionality Successfully Tested
Plot and display growth charts
(b)(1)(i)(B)
Receive summary care record using the standards specified at
§170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C)
Receive summary care record using the standards specified at
§170.202(b) and (c) (SOAP Protocols)
(b)(2)(ii)(B)
Transmit health information to a Third Party using the standards
specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(2)(ii)(C)
Transmit health information to a Third Party using the standards
specified at §170.202(b) and (c) (SOAP Protocols)
(f)(3)
Ambulatory setting only – Create syndrome-based public health
surveillance information for transmission using the standard
specified at §170.205(d)(3) (urgent care visit scenario)
Common MU
Data Set (15)
Express Procedures according to the standard specified at
§170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures
and Nomenclature)
Common MU
Data Set (15)
Express Procedures according to the standard specified at
§170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
No optional functionality tested
Page 7 of 11
3.2.6 2014 Edition Certification Criteria* Successfully Tested
Criteria #
Version
TP** TD***
Criteria #
Version
TP
TD
x
(c)(3)
1.6
x
(d)(1)
1.2
x
(a)(1)
1.2
x
(a)(2)
1.2
x
(a)(3)
1.2
1.4
x
(d)(2)
1.5
x
(a)(4)
1.4
1.3
x
(d)(3)
1.3
x
(a)(5)
1.4
1.3
x
(d)(4)
1.3
x
(a)(6)
1.3
1.4
x
(d)(5)
1.2
x
(a)(7)
1.3
1.3
x
(d)(6)
1.2
x
(a)(8)
1.2
x
(d)(7)
1.2
x
(a)(9)
1.3
1.3
x
(d)(8)
1.2
x
(a)(10)
1.2
1.4
(d)(9) Optional
1.2
x
(a)(11)
1.3
(e)(1)
1.8
1.5
x
(a)(12)
1.3
x
(e)(2) Amb. only
1.2
1.6
x
(a)(13)
1.2
x
(e)(3) Amb. only
1.3
x
(a)(14)
1.2
(f)(1)
1.2
1.2
x
(a)(15)
1.5
(f)(2)
1.3
1.7.1
(a)(16) Inpt. only
1.3
(f)(3)
1.3
1.7
(a)(17) Inpt. only
1.2
(f)(4) Inpt. only
1.3
1.7
x
(b)(1)
1.7
1.4
x
(b)(2)
1.4
1.6
(f)(5) Optional &
Amb. only
1.2
1.2
x
(b)(3)
1.4
1.2
x
(b)(4)
1.3
1.4
(f)(6) Optional &
Amb. only
1.3
1.0.3
x
(b)(5)
1.4
1.7
(g)(1)
1.7
1.9
(b)(6) Inpt. only
1.3
1.7
x
(g)(2)
1.7
1.9
x
(b)(7)
1.4
1.6
x
(g)(3)
1.3
x
(c)(1)
1.6
1.6
x
(g)(4)
1.2
x
(c)(2)
1.6
1.6
1.5
1.2
1.6
No criteria tested
*For a list of the 2014 Edition Certification Criteria, please reference
http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)
**Indicates the version number for the Test Procedure (TP)
***Indicates the version number for the Test Data (TD)
Page 8 of 11
3.2.7 2014 Clinical Quality Measures*
Type of Clinical Quality Measures Successfully Tested:
Ambulatory
x
Inpatient
No CQMs tested
*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov
(navigation: 2014 Clinical Quality Measures)
CMS ID
x
Version
CMS ID
Ambulatory CQMs
Version CMS ID
2
90
136
22
117
137
50
v2
122
x
138
Version
CMS ID
155
x
v2
156
123
139
158
56
124
140
159
61
125
141
160
62
126
142
161
64
127
143
163
65
128
144
164
66
129
145
165
130
146
68
v3
x
69
v2
74
x
131
v2
x
132
v2 x
147
x
v2
166
148
169
133
149
177
77
134
153
179
82
135
154
182
Version
CMS ID
Inpatient CQMs
Version CMS ID
Version
v3
167
75
CMS ID
v2
157
52
x
Version
CMS ID
9
71
107
172
26
72
108
178
30
73
109
185
31
91
110
188
32
100
111
190
53
102
113
55
104
114
60
105
171
Version
Page 9 of 11
3.2.8 Automated Numerator Recording and Measure Calculation
3.2.8.1 Automated Numerator Recording
Automated Numerator Recording Successfully Tested
(a)(1)
(a)(9)
(a)(16)
(b)(6)
(a)(3)
(a)(11)
(a)(17)
(e)(1)
(a)(4)
(a)(12)
(b)(2)
(e)(2)
(a)(5)
(a)(13)
(b)(3)
(e)(3)
(a)(6)
(a)(14)
(b)(4)
(a)(7)
(a)(15)
(b)(5)
x Automated Numerator Recording was not tested
3.2.8.2 Automated Measure Calculation
Automated Measure Calculation Successfully Tested
x
(a)(1)
x
(a)(9)
(a)(16)
(b)(6)
x
(a)(3)
x
(a)(11)
(a)(17)
(e)(1)
x
(a)(4)
x
(a)(12)
x
(b)(2)
x
(e)(2)
x
(a)(5)
x
(a)(13)
x
(b)(3)
x
(e)(3)
x
(a)(6)
x
(a)(14)
x
(b)(4)
x
(a)(7)
x
(a)(15)
x
(b)(5)
Automated Measure Calculation was not tested
3.2.9 Attestation
Attestation Forms (as applicable)
Appendix
x Safety-Enhanced Design*
A
x Quality Management System**
B
x Privacy and Security
C
*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16),
(b)(3), (b)(4)
**Required for every EHR product
3.3 Appendices
Attached below.
Page 10 of 11
Test Results Summary Document History
Version
17-Feb-2014
Description of Change
Edited: section header page 3; contact info
page 4
Date
17-Feb-2014
10-Feb-2014
Modified layout
10-Feb-2014
20-Nov-2013
Updated test tool sections
20-Nov-2013
25-Oct-2013
Corrected numbering of 3.2.8 section
25-Oct-2013
15-Oct-2013
Modified layout slightly
15-Oct-2013
01-Oct-2013
Initial Version
01-Oct-2013
2014 Edition Test Report Summary
Page 11 of 11
USER CENTER DESIGN REPORT –
TEST REPORT UPDATE
This test report was updated in December 2015 to satisfy User Center Design Report specifications by
ONC.
The new Test Report ID is amended as follows:
“Part 3: NVLAP-Accredited Testing Laboratory Information: Report Number” plus the suffix “_Dec2015”.
Igor Lopez
Chief Software Architect - Software Development Manager
Ibeza LLC
2550 Douglas Rd.
Coral Gables, FL 33134
Tel: (305) 838-0880
Fax: (305) 357-1701
Project: eConcepts 2.1
Ibeza LLC EHR Letter of Attestation
Dear DGI Proctor,
Ibeza LLC attests to the validity of the information below to inform you of our compliance for DTR 170.314.g.3 Safety
Enhanced Design, namely User Centered Design practice, used in the following certification criteria:







170.314(a)(1) Computerized provider order entry
170.314(a)(2) Drug-drug, drug-allergy interaction checks
170.314(a)(6) Medication list
170.314(a)(7) Medication allergy list
170.314(a)(8) Clinical decision support
170.314(b)(3) Electronic prescribing
170.314(b)(4) Clinical information reconciliation
The activities in the Software Development process are modeled according to the 6 key principles described in the
internationally appointed manifesto, ISO 9241-210 (ISO 9241-210:2010(E)).
Functional Analysis
Functional Analysis is performed by healthcare professionals with a vast experience in healthcare workflows, which
guarantee that the development has an early focus on users and tasks. Functional Analysis starts with the revision of low
level requirements and impact analysis. The results are discussed with the user to guarantee realistic expectations and
make users active stakeholders. The requirements are created by request and drafted from talking and observing users
doing what they do, listing goals, tasks, and breaking the tasks into steps.
Design
A prototype is designed detailing a functionality’s navigation, interaction behavior and graphic specifications considering
the functional requirements and product design guidelines. The design takes into consideration the usability goals of
effectiveness, efficiency and satisfaction. This task occurs simultaneously with other Software Development tasks, which
contribute to the prototype development. The prototype can be presented to the user to evaluate the design and suggest
improvement needs.
Content
Besides international standardized content embedded in the product, clinical content is adapted with active user
participation based on their practices and needs. Content can also be created to predict the user’s needs, which is done
by extensive research in known referential areas and practices. This content is made available for evaluation by the client
and/or to provide ideas on possible improvements to the current content.
Functional Architecture
Functional Architecture analyzes, supports and validates the implementation in a functional perspective. Considering the
whole system and being an active part in the decision of what is best suited to perform each task.
Software Architecture
Software Architecture analyzes, supports and validates the implementation in a technical perspective. They ensure that
the system will support the design and comply with the constraints with a focus on performance empirical measurement.
Code Development
Code Development (programming and tests) works alongside other teams to make sure development is in concordance
with the User Centered Design assumed by all the other teams.
Quality Control
Team is responsible for assessing the effectiveness of an existing or developmental User Interface. Quality Control
analyzes and validates the implementation in a functional perspective. It follows an iterative design which guarantees
that when problems are found in user testing, they are fixed and further tests are performed. Tests are also performed by
experts, who have knowledge of the application domain, and users perform an analytical evaluation. Acceptance testing
by final users is also conducted before implementation.
References
ISO 9241-210:2010(E), Ergonomics of human-system interaction – Part 210: Human-centered design for interactive
systems
I hereby attest that all above statements are true, as an authorized signing authority on behalf of my organization
________________________________________
Igor Lopez
Chief Software Architect - Software Development Manager
12-01-2015
NISTIR 7742
Customized Common Industry Format
Template for Electronic Health Record
Usability Testing
Robert M. Schumacher
User Centric. Inc,
Svetlana Z. Lowry
Information Access Division
Information Technology Laboratory
National Institute of Standards and Technology
U.S. Department of Commerce
Gary Locke, Secretary
Patrick D. Gallagher, Director
NISTIR 7742
Customized Common Industry Format
Template for Electronic Health Record
Usability Testing
Robert M. Schumacher
User Centric. Inc,
Svetlana Z. Lowry
Information Access Division
Information Technology Laboratory
November 2010
U.S. Department of Commerce
Gary Locke, Secretary
Patrick D. Gallagher, Director
EHR Usability Test Report of eConcepts 2.1
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
eConcepts 2.1
Date of Usability Test
Date of Report
Report Prepared By
05/12/2014 – 05/21/2014
05/22/2014
Gersson Manuel Oviedo
Network Administrator for Ibeza, LLC
305-357-1727
gersson.oviedo@ibeza.net
2550 S Douglas Road
Coral Gables, FL 33134
Table of Contents
1
EXECUTIVE SUMMARY
5
2
3
INTRODUCTION
METHOD
8
8
3.0
TARGET DEMOGRAPHICS
8
3.1
PARTICIPANTS
8
3.2
STUDY DESIGN
10
3.3
TASKS
10
3.4
PROCEDURE
11
3.5
TEST LOCATION
13
3.6
TEST ENVIRONMENT
13
3.7
TEST FORMS AND TOOLS
14
3.8
PARTICIPANT INSTRUCTIONS
14
3.9
USABILITY METRICS
15
4
5
RESULTS
18
4.1
DATA ANALYSIS AND REPORTING
18
4.2
DISCUSSION OF THE FINDINGS
20
APPENDICES
22
5.1
Appendix 1: RECRUITING SCREENER
23
5.2
Appendix 2: PARTICIPANT DEMOGRAPHICS
5.3
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT
26
27
5.4
Appendix 4: MODERATOR’S GUIDE
29
5.5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
32
EXECUTIVE SUMMARY
A usability test of eConcepts 2.1 Clinical decision EHR was
conducted on May 12th 2014 at Beraja Medical Institute by Gersson
Oviedo. The purpose of this test was to test and validate the usability of
the current user interface, and provide evidence of usability in the EHR
under Test (EHRUT).
During the usability test, five (5) healthcare providers
matching the target demographic criteria (see page 11, “Method - Target
Demographics”) served as participants
and used the EHRUT in simulated, but representative tasks.
This study collected performance data on the following tasks typically conducted
on an EHR:
1. 314.a.1 Computerized provider order entry
2. 314.a.2 Drug‐drug, drug‐allergy interaction checks
3. 314.a.6 Medication list
4. 314.a.7 Medication allergy list
5. 314.a.8 Clinical decision support
6. 314.a.16 eMAR Inpatient Only
7. 314.b.3 Electronic prescribing
8. 314.b.4 Clinical information reconciliation
During the 30 minute one-on-one usability test, each participant was
greeted by the test administrator and asked to review and
sign an informed consent/release form (included in Appendix 3); they
were instructed that they could withdraw at any time. Participants
did not have prior experience with the EHR.
The administrator
introduced the test and instructed participants to complete a series of
tasks (given one at a time) using the EHRUT. During the testing, the
administrator timed the test and recorded user performance data on
paper and electronically. The administrator
did not give the participant assistance in how to complete the task.
The Participant’s computer screen was recorded for subsequent analysis.
The following types of data were collected for each participant:
•
Time to complete the tasks
•
Participant’s Comments
•
Number and types of errors
•
Path deviations
•
Participant’s satisfaction ratings of the system
All participant data was de-identified – no correspondence could be
made from the identity of the participant to the data collected. Following
the conclusion of the testing, participants were asked to complete a posttest questionnaire and were thanked for their time.
Various recommended metrics, in accordance with the examples set
forth in the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, were used to evaluate the
usability of the EHRUT. Following is
a summary of the performance and rating data collected on the EHRUT.
TASK PERFORMANCE MEASUREMENT (% Correct, Time Goals, Ease of Task)
Measure
Task
Effectiveness
Goal:
Result
:
σ
σ
Efficiency
Goal:
Result
:
σ
Satisfaction
Goal:
Result
:
σ
σ
CPOE (314.a.1)
Record Medication Order
Change Medication Order
Access Medication Order
Record Laboratory Order
Change Laboratory Order
Access Laboratory Order
Record Radiology/imaging Order
Change Radiology/imaging Order
Access Radiology/imaging Order
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
60” by 100% users
55” by 100% users
60” by 100% users
45” by 100% users
30” by 100% users
25” by 100% users
60” by 100% users
40” by 100% users
60” by 100% users
55” by 100% users
30” by 100% users
20” by 100% users
30” by 100% users
25” by 100% users
30” by 100% users
20” by 100% users
30” by 100% users
15” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Easy or Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
Goal:
Result:
Goal:
Result:
80% Correct
100% Correct
80% Correct
100% Correct
Goal:
Result:
Goal:
Result:
120” by 100% users
65” by 100% users
60” by 100% users
30” by 100% users
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
150” by 100% users
130” by 100% users
60” by 100% users
55” by 100% users
30” by 100% users
20” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
100% Very Easy
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
60” by 100% users
40” by 100% users
60” by 100% users
45” by 100% users
30” by 100% users
10” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
100% Very Easy
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
80% Correct
100% Correct
80% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
150” by 100% users
145” by 100% users
60” by 100% users
50” by 100% users
60” by 100% users
35” by 100% users
30” by 100% users
25” by 100% users
120” by 100% users
105” by 100% users
60” by 100% users
40” by 100% users
30” by 100% users
25” by 100% users
90” by 100% users
75” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
Goal:
Result:
80% Correct
100% Correct
Goal:
Result:
120” by 100% users
85” by 100% users
Goal:
Result:
100% Very Easy
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
90” by 100% users
75” by 100% users
90” by 100% users
70” by 100% users
90” by 100% users
65” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
100% Very Easy
Drug-drug, drug-allergy interactions checks (314.a.2)
Create drug-drug and drug-allergy interventions prior to CPOE completion
Adjustment of severity level of drug-drug interventions
Medication list (314.a.6)
Record Medication List
Change Medication List
Access Medication List
Medication allergy list (314.a.7)
Record Medication Allergy List
Change Medication Allergy List
Access Medication Allergy List
Clinical decision support (314.a.8)
Problem List Interventions
Medication List Interventions
Medication Allergy List Interventions
Demographics Interventions
Lab Tests and Results Interventions
Vital Signs Interventions
Identify User Diagnostic and Therapeutic Reference Information
Configuration of CDS interventions by user
Electronic prescribing (314.b.3)
Create prescriptions
Clinical information reconciliation (314.b.4)
Reconcile patient’s active medication list with another source
Reconcile patient’s active problem list with another source
Reconcile patient’s active medication allergy list with another source
INTRODUCTION
The EHRUT tested for this study was eConcepts 2.1.
Designed to present medical information to healthcare
providers in specialty and general practices, the EHRUT consists of
patient demographics information, past and present medical and
procedure history, document storage, and e-prescribing. The usability
testing attempted to represent realistic exercises and conditions.
The purpose of this study was to test and validate the usability of the
current user interface, and provide evidence of usability in the EHR
Under Test (EHRUT). To this end, measures of effectiveness,
efficiency and user satisfaction, such as time to complete task
and task difficulty rating, were captured during the usability testing.
METHOD
TARGET DEMOGRAPHICS
The target demographics for eConcepts 2.1 are Healthcare professionals with basic
computer skills and knowledge of medical terms. We expect medical professionals
and their assistants (Patient Care Technicians, Residents) to use the final version of
the EHRUT.
PARTICIPANTS
A total of five (5) participants were tested on the EHRUT. Participants in
the test were patient care techs and medical students. Participants were recruited
by Gersson Oviedo and were not monetarily compensated
but thanked for their time. In addition, participants had no direct
connection to the development of or organization producing the
EHRUT(s). Participants were given the opportunity
to have the same orientation and level of training as the actual end users
would have received.
For the test purposes, end-user characteristics were identified and
translated into a recruitment screener used to solicit potential
participants; an example of a screener is provided in Appendix [1].
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a
table of participants by characteristics, including demographics,
professional experience, computing experience and user needs for
assistive technology. Participant names were replaced with Participant
IDs so that an individual’s data cannot be tied back to individual
identities.
Part
ID
1
2
3
4
4
5
Occupation/
Gender Age Education
role
Female 25 College Student Opth Technician
Male
47
University
Diag Tech
Male
31
University
Resident
Female 46
University
Diag Tech
Female 28
University
Resident
Professional
Experience
1.5 yrs
24
1
29
1
Computer
Experience
15 yrs
10
20
8
10
Product
Experience
1 hr
1 hr
1 hr
1 hr
1 hr
Assistive
Technology
Needs
no
no
no
no
no
Five (5) participants (matching the demographics
in the section on Target Demographics – page 8) were recruited and all five (5)
participated in the usability test.
Participants were scheduled for thirty (30) minute sessions with
Fifteen (15) minutes in between each session for debrief by the
administrator(s) and to reset systems to proper test conditions. A
spreadsheet was used to keep track of the participant schedule, and
included each participant’s demographic characteristics
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the
application performed well – that is, effectively, efficiently, and with
satisfaction – and areas where the application failed to meet the needs of
the participants. The data from this test may serve as a baseline for future
tests with an updated version of the same EHR and/or comparison with
other EHRs provided the same tasks are used. In short, this testing serves
as both a means to record or benchmark current usability, but
also to identify areas where improvements must be made.
During the usability test, participants interacted with eConcepts 2.1. Each
participant used the system in the same location, and was provided with
the same instructions. The system was evaluated for effectiveness,
efficiency and satisfaction as defined by measures collected and
analyzed for each participant:
•
Number of tasks successfully completed within the allotted time
without assistance
•
Time to complete the tasks
•
Number and types of errors
•
Path deviations
•
Participant’s verbalizations (comments)
•
Participant’s satisfaction ratings of the system
TASKS
Tasks were constructed that would be realistic and representative of
the kinds of activities a user might do with this EHR. They were
selected and shown here ordered based on their perceived frequency
of use, criticality of function, and those that may be most troublesome
for users.
Initial Setup
What is the testers overall impression of the software?
Administrator must give guidelines for how test will be conducted.
CPOE (314.a.1)
• Record, change, and access Medication Order
• Record, change, and access Laboratory Order
• Record, change, and access Radiology/imaging Order
• Create drug-drug and drug-allergy interventions prior to CPOE completion
• Adjustment of severity level of drug-drug interventions
• Record, change, and access Medication Allergy List
• Record, change, and access Medication List
• Create prescriptions
• Problem List Interventions
• Medication List Interventions
• Medication Allergy List Interventions
• Demographics Interventions
• Lab Tests and Results Interventions
• Vital Signs Interventions
• Identify User Diagnostic and Therapeutic Reference Information
• Configuration of CDS interventions
• Reconcile patient’s active medication list with another source
• Reconcile patient’s active problem list with another source
• Reconcile patient’s active medication allergy list with another source
eMAR (314.a.16)
• Using assistive technology, verify the right patient, medication, dose, route and
time.
PROCEDURES
Upon arrival, participants were greeted; their identity was verified and
matched with a name on the participant schedule. Participants were then
assigned a participant ID. Each participant reviewed and signed an
informed consent and release form (See Appendix 3). A representative
from the test team witnessed the participant’s signature.
To ensure that the test ran smoothly, two staff members participated in
this test, the usability administrator and the data logger.
The administrator moderated the session including administering
instructions and tasks. The administrator also monitored task times,
obtained post-task rating data, and took notes on participant comments.
A second person served as the data logger and took notes on task
success, path deviations, number and type of errors, and comments.
Participants were instructed to perform the tasks (see specific
instructions below):
•
As quickly as possible making as few errors and deviations as
possible.
•
Without assistance; administrators were allowed to give
immaterial guidance and clarification on tasks, but not
instructions on use.
•
Without using a think aloud technique.
For each task, the participants were given a written copy of the task.
Task timing began once the administrator finished reading the question.
The task time was stopped once the participant indicated they had
successfully completed the task. Scoring is discussed below in Section
3.9.
Following the session, the administrator gave the participant the post-test
questionnaire (e.g., the System Usability Scale, see Appendix 5), thanked
them for their time and participation.
Participants' demographic information, task success rate, time on task,
errors, deviations, verbal responses, and post-test questionnaire were
recorded into a spreadsheet.
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a
table and computer for the participant. Only the participant and
administrator were in the test room. All observers and the data logger
worked from a separate room where they could see the participant’s
screen and face shot, and listen to the audio of the session. To ensure
that the environment was comfortable for users, noise levels were kept to
a minimum with the ambient temperature within a normal range. All of
the safety instruction and evacuation procedures were valid, in place, and
visible to the participants.
TEST ENVIRONMENT
The EHRUT would typically be used in a healthcare office or facility.
In this instance, the testing was conducted in an office usually used for office
visits. For testing, the computer used a desktop PC running Windows 7.
The participants used a wireless keyboard and mouse
when interacting with the EHRUT.
The EHRUT used a Dell 24” Monitor operating at 1920 x 1080 desktop
resolution and 32 bit color settings.
The application was running on Google Chrome
browser using a test instance and database over a WAN connection.
Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field
implementation. Additionally, participants were instructed not to change
any of the default system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, the following documents were used, including:
- Informed Consent
- Moderator’s Guide
- Post-test Questionnaire
- Non-Disclosure Agreement
- Final Questions
Examples of these documents can be found in Appendices 3-6. The
Moderator’s Guide was devised so as to be able to capture required
data.
The participant’s interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine. The test
session was electronically transmitted to a nearby observation room
where the data logger observed the test session.
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each
participant (also see the full moderator’s guide in Appendix [B4]):
Thank you for participating in this study. Your input is very
important. Our session today will last about 30 minutes. During
that time you will use an instance of an electronic health record.
I will ask you to complete a few tasks using this system and
answer some questions. You should complete the tasks as
quickly as possible making as few errors as possible. Please try
to complete the tasks on your own following the instructions very
closely. Please note that we are not testing you we are testing
the EHR. Therefore if you have difficulty all this means is that
something needs to be improved in the EHR. I will be here in
case you need help understanding instructions, but I am not able
to instruct you or provide help in how to use the application.
Overall, we are interested in how easy (or how difficult) this
EHR is to use, what in it would be useful to you, and how we
could improve it. I did not have any involvement in its creation,
so please be honest with your opinions. All of the information
that you provide will be kept confidential and your name will not
be associated with your comments at any time. Should you feel
it necessary you are able to withdraw at any time during the
testing.
Following the procedural instructions, participants were shown the EHR
and as their first task, were given five (5) minutes to explore the
system and make comments. Once this time was complete, the
administrator gave the following instructions:
For each task, I will read the description to you and say “Begin.”
At that point, please perform the task and say “Done” once you
believe you have successfully completed the task. I would like to
request that you not talk aloud or verbalize while you are doing
the tasks. I will ask you your impressions about the task once
you are done.
Participants were then given 8 tasks to complete. Tasks are listed in
the moderator’s guide in Appendix [B4].
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving
the Usability of Electronic Health Records, EHRs should support a
process that provides a high level of usability for all users. The goal is for
users to interact with the system effectively, efficiently, and with an
acceptable level of satisfaction. To this end, metrics for effectiveness,
efficiency and user satisfaction were captured during the usability testing.
The goals of the test were to assess:
1. Effectiveness of eConcepts 2.1 by measuring participant
success rates and errors
2. Efficiency of eConcepts 2.1 by measuring the average task
time and path deviations
3. Satisfaction with eConcepts 2.1 by measuring ease of use ratings
DATA SCORING
The following table details how tasks were scored, errors
evaluated, and the time data analyzed.
Measures
Effectiveness:
Task Success
10
Rationale and Scoring
A task was counted as a “Success” if the participant was able to
achieve the correct outcome, without assistance, within the time
allotted on a per task basis.
The total number of successes were calculated for each task and then
divided by the total number of times that task was attempted. The
results are provided as a percentage.
Task times were recorded for successes. Observed task times divided
by the optimal time for each task is a measure of optimal efficiency.
Effectiveness:
Task Failures
If the participant abandoned the task, did not reach the correct answer
or performed it incorrectly, or reached the end of the allotted time
before successful completion, the task would be counted as a
“Failure.” No task times were taken for errors.
The total number of errors was calculated for each task and then
divided by the total number of times that task was attempted. Not all
deviations would be counted as errors. This should also be
expressed as the mean number of failed tasks per participant.
On a qualitative level, an enumeration of errors and error types should
be collected.
Efficiency:
Task Time
Each task was timed from when the administrator said “Begin” until the
participant said, “Done.” If he or she failed to say “Done,” the time was
stopped when the participant stopped performing the task. Only task
times for tasks that were successfully completed were included in the
average task time analysis. Average time per task was calculated for
each task. Variance measures (standard deviation and standard error)
were also calculated.
Satisfaction:
Task Rating
Participant’s subjective impression of the ease of use of the
application was measured by administering both a simple post-task
question as well as a post-session questionnaire. After each task, the
participant was asked to rate “Overall, this task was:” on a scale of 1
(Very Difficult) to 5 (Very Easy). These data are averaged across
participants. 12
Common convention is that average ratings for systems judged easy
to use should be 3.3 or above.
To measure participants’ confidence in and likeability of eConcepts
2.1, the testing team administered the System Usability Scale (SUS)
post-test questionnaire. Questions included, “I think I would like to
use this system frequently,” “I thought the system was easy to use,”
and “I would imagine that most people would learn to use this
system very quickly.” See full System Usability Score questionnaire in
Appendix 5.
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. If a participants were to
fail to follow session and task instructions their data would have been
excluded from the analyses
The usability testing results for the EHRUT are detailed below. The
results should be seen in light of the objectives and goals outlined in
Section 3.2 Study Design. The data should yield actionable
results that, if corrected, yield material, positive impact on user
performance.
Measure
Task
Effectiveness
Goal:
Result
:
σ
σ
Efficiency
Goal:
Result
:
σ
Satisfaction
Goal:
Result
:
σ
σ
CPOE (314.a.1)
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
60” by 100% users
55” by 100% users
60” by 100% users
45” by 100% users
30” by 100% users
25” by 100% users
60” by 100% users
40” by 100% users
60” by 100% users
55” by 100% users
30” by 100% users
20” by 100% users
30” by 100% users
25” by 100% users
30” by 100% users
20” by 100% users
30” by 100% users
15” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
Goal:
Result:
Goal:
Result:
80% Correct
100% Correct
80% Correct
100% Correct
Goal:
Result:
Goal:
Result:
120” by 100% users
65” by 100% users
60” by 100% users
30” by 100% users
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
150” by 100% users
130” by 100% users
60” by 100% users
55” by 100% users
30” by 100% users
20” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
100% Very Easy
Goal:
Result:
Goal:
Result:
Goal:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
60” by 100% users
40” by 100% users
60” by 100% users
45” by 100% users
30” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
100% Very Easy
100% Very Easy
Result:
100% Correct
Result:
10” by 100% users
Result:
100% Very Easy
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
80% Correct
100% Correct
80% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
150” by 100% users
145” by 100% users
60” by 100% users
50” by 100% users
60” by 100% users
35” by 100% users
30” by 100% users
25” by 100% users
120” by 100% users
105” by 100% users
60” by 100% users
40” by 100% users
30” by 100% users
25” by 100% users
90” by 100% users
75” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
100% Very Easy
Goal:
Result:
80% Correct
100% Correct
Goal:
Result:
120” by 100% users
85” by 100% users
Goal:
Result:
100% Very Easy
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
100% Correct
Goal:
Result:
Goal:
Result:
Goal:
Result:
90” by 100% users
75” by 100% users
90” by 100% users
70” by 100% users
90” by 100% users
65” by 100% users
Goal:
Result:
Goal:
Result:
Goal:
Result:
100% Very Easy
100% Very Easy
100% Very Easy
Configuration of CDS interventions by user (may be an admin type function)
The results from the SUS (System Usability Scale) scored the highly subjective
“Satisfaction” rating with the system based on performance with these tasks to
be: 100%. Broadly interpreted, scores under 60 represent systems
with poor usability; scores over 80 would be considered above
average.
DISCUSSION OF THE FINDINGS
EFFECTIVENESS
Based on the success, failure and path deviation data, we can conclude that
test participants had an easy time locating the relevant areas and options that
were required of them for this test. Elements are clearly and intuitively labeled
so participants were able to locate the correct options with little to no errors.
EFFICIENCY
We can judge (based on the observations of the task time and deviation data) that
the users who participated in this study found elements to be clearly labeled and
logically proximate. This enabled them to quickly complete tasks with minimal
movement. The application is also very responsive which contributed to efficiency.
SATISFACTION
Based on the task ratings and SUS results data, we can conclude that users were
very pleased with the speed, ease of use, and clear presentation of eConcepts 2.1
MAJOR FINDINGS
Our data shows that the eConcepts 2.1 received a congratulatory welcome from
our target audience of healthcare professionals. They found the application to be
intuitive and easy to use. Participants shared via the SUS (System Usability Scale)
and in conversation that they found the tasks very easy to perform. Testing
subjects were able to exceed our time expectations which is visible in the data: all
tasks were completed in less time than we had expected.
They also did not appear to be frustrated nor confused at any point during testing.
AREAS FOR IMPROVEMENT
Participants expressed that overall they were pleased with eConcepts 2.1. One
participant requested that, despite limited screen space, procedures should also
appear on the patient dashboard. This change would allow them to see this
information without having to access the Procedure list section.
APPENDICES
The following appendices include supplemental data for this usability test
report. Following is a list of the appendices provided:
1: Sample recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent
Form
4: Example Moderator’s Guide
5: System Usability Scale Questionnaire
6: Incentive receipt and acknowledgment form
Appendix 1: RECRUITING SCREENER
Recruiting Script for Recruiting Firm
Hello, we are recruiting individuals to participate in a usability study for an electronic health record.
We would like to ask you a few questions to see if you qualify and if would like to participate. This
should only take a few minutes of your time. This is strictly for research purposes. If you are
interested and qualify for the study, you will be paid to participate. Can I ask you a few questions?
1.
[If not obvious] Are you male or female? [Recruit a mix of participants]
2. Have you participated in a focus group or usability test in the past xx months? [If yes,
Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability research, web
design […etc.]? [If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research interest in an electronic
health record software or consulting company? [If yes, Terminate]
5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 - to 74; 75 and older]
[Recruit Mix]
6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian,
Black/African-American, Latino/a or Hispanic, etc.]
7.
Do you require any assistive technologies to use a computer? [if so, please describe]
Professional Demographics
8.
What is your current position and title? (Must be healthcare provider)
 RN: Specialty
 Physician: Specialty
 Resident: Specialty
 Administrative Staff
 Other [Terminate]
_
9.
How long have you held this position?
10. Describe your work location (or affiliation) and environment? (Recruit according to the
intended users of the application) [e.g., private practice, health system, government clinic,
etc.]
11. Which of the following describes your highest level of education? [e.g., high school
graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other
(explain)]
Computer Expertise
12. Besides reading email, what professional activities do you do on the computer? [e.g., access
EHR, research; reading news; shopping/banking; digital pictures; programming/word
processing, etc.] [If no computer use at all, Terminate]
13. About how many hours per week do you spend on the computer? [Recruit according to the
demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]
14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
16. In the last month, how often have you used an electronic health record?
17. How many years have you used an electronic health record?
18. How many EHRs do you use or are you familiar with?
19. How does your work environment patient records? [Recruit according to the demographics of
the intended users]
 On paper
 Some paper, some electronic
 All electronic
Contact Information
Those are all the questions I have for you. Your background matches the people we're
looking for.
Would you be able to participate on 05/12/2014?
May I get your contact information?

Name of participant:

Address:

City, State, Zip:

Daytime phone number:

Evening phone number:

Alternate [cell] phone number:

Email address:
Before your session starts, we will ask you to sign a release form allowing us to
videotape your session. The videotape will only be used internally for further study if
needed. Will you consent for your session to be recorded?
This study will take place at Beraja Medical Institute. I will confirm your appointment a
couple of days before your session and provide you with directions to our office. What
time is the best time to reach you?
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Part
ID
1
2
3
4
4
5
.
Occupation/
Gender Age Education
role
Female 25 College Student Opth Technician
Male
47
University
Diag Tech
Male
31
University
Resident
Female 46
University
Diag Tech
Female 28
University
Resident
Professional
Experience
1.5 yrs
24
1
29
1
Computer
Experience
15 yrs
10
20
8
10
Product
Experience
1 hr
1 hr
1 hr
1 hr
1 hr
Assistive
Technology
Needs
no
no
no
no
no
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM
Non-Disclosure Agreement
THIS AGREEMENT is entered into as of _
_, 2014 between
(“the Participant”) and the testing organization Ibeza, LLC
located at 2550 S Douglas Road Coral Gables, FL 33134.
The Participant acknowledges his or her voluntary participation in today’s usability study may
bring the Participant into possession of Confidential Information. The term "Confidential
Information" means all technical and commercial information of a proprietary or confidential
nature which is disclosed by Ibeza, LLC, or otherwise acquired by the Participant, in the
course of today’s study.
By way of illustration, but not limitation, Confidential Information includes trade secrets,
processes, formulae, data, know-how, products, designs, drawings, computer aided design files
and other computer files, computer software, ideas, improvements, inventions, training methods
and materials, marketing techniques, plans, strategies, budgets, financial information, or
forecasts.
Any information the Participant acquires relating to this product during this study is confidential
and proprietary to Ibeza, LLC and is being disclosed solely for the purposes of the
Participant’s participation in today’s usability study. By signing this form the Participant
acknowledges that s/he will not disclose this confidential information obtained today to anyone
else or any other organizations.
Participant’s printed name:
Signature:
Date:
Informed Consent
Ibeza, LLC would like to thank you for participating in this study. The purpose of this study is to
evaluate an electronic health records system. If you decide to participate, you will be asked to perform
several tasks using the prototype and give your feedback. The study will last about 30 minutes. At the
conclusion of the test, you will be compensated for your time.
Agreement
I understand and agree that as a voluntary participant in the present study conducted by Ibeza, LLC I am
free to withdraw consent or discontinue participation at any time. I understand and agree to participate in
the study conducted and recorded by Ibeza, LLC.
I understand and consent to the use and release of the recording by Ibeza, LLC. I understand that the
information and recording is for research purposes only and that my name and image will not be used for
any purpose other than research. I relinquish any rights to the recording and understand the recording may
be copied and used by Ibeza, LLC without further permission.
I understand and agree that the purpose of this study is to make software applications more useful and
usable in the future.
I understand and agree that the data collected from this study may be shared with outside of Ibeza, LLC and
Ibeza, LLC’s clients. I understand and agree that data confidentiality is assured, because only de- identified
data – i.e., identification numbers not names – will be used in analysis and reporting of the results.
I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand
that I can leave at any time.
Please check one of the following:
 YES, I have read the above statement and agree to be a participant.
 NO, I choose not to participate in this study.
Signature:
Date:
Appendix 4: ECONCEPTS 2.1 ADMINISTRATOR’S GUIDE
eConcepts 2.1 Usability Test
Moderator’s Guide
Administrator: Gersson Oviedo
Date
Time
Participant #
Location: Beraja Medical Institute
Prior to testing
 Confirm schedule with Participants
 Ensure EHRUT lab environment is running properly
 Ensure lab and data recording equipment is running properly
Prior to each participant:
 Reset application
 Start session recordings with tool
Prior to each task:
 Reset application to starting point for next task
After each participant:
 End session recordings with tool
After all testing
 Back up all video and data files
Orientation
Thank you for participating in this study. Your input is very important. Our session today will
last about 30 minutes. During that time you will use an instance of an electronic health record
(EHR).
I will ask you to complete a few tasks using this EHR and answer some questions. You should
complete the tasks as quickly as possible making as few errors as possible. Please try to complete
the tasks on your own following the instructions very closely. Please note that we are not testing
you we are testing the EHR. Therefore if you have difficulty all this means is that something needs
to be improved in the EHR. I will be here in case you need help understanding instructions, but I
am not able to instruct you or provide help in how to use the application.
Overall, we are interested in how easy (or how difficult) this EHR is to use, what in it would be
useful to you, and how we could improve it. I did not have any involvement in its creation, so
please be honest with your opinions. All of the information that you provide will be kept
confidential and your name will not be associated with your comments at any time. Should you
feel it necessary you are able to withdraw at any time during the testing.
The product you will be using today is Ibeza’s eConcepts 2.1, beta. Some of the data may
not make sense as it is placeholder data.
We are recording the video and screenshots of our session today. All of the information that you
provide will be kept confidential and your name will not be associated with your comments at any
time.
Do you have any questions or concerns?
Preliminary Questions (X minutes)
What is your job title / appointment?
How long have you been working in this role?
What are some of your main responsibilities?
Tell me about your experience with electronic health records.
Measure
Task
Effectiveness
Goal:
Result
:
σ
σ
Efficiency
Goal:
Result
:
σ
Satisfaction
Goal:
Result
:
σ
σ
CPOE (314.a.1)
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Goal:
Result:
Configuration of CDS interventions by user (may be an admin type function)
Final Questions
What was your overall impression of this system?
What aspects of the system did you like most?
What aspects of the system did you like least?
Were there any features that you were surprised to see?
What features did you expect to encounter but did not see? That is, is there anything that is
missing in this application?
Compare this system to other systems you have used.
Would you recommend this system to your colleagues?
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly
disagree
Strongly
agree
1. I think that I would like to use this
system frequently
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system

Report - Drummond Group

Transcription

Similar documents

IG-2552-14-0099 - UL Transaction Security

Report - Drummond Group

term 1 week 10 2015.pub - Golden Grove Primary School

Education

EHR Final Test Report

Report - Drummond Group

newsletter - Tea Gardens Public School

YDAT Lock-in flyer 10.indd

monday wednesday thursday friday tuesday extended care

Angelo_NMA Podium Presentation 20151215

Preparing for Certification

Understanding and Managing High Blood Pressure

WBE Certified - Perna Design

IG-3721-16-0003 - UL Transaction Security

. - Metropolitan Museum of Art

User testing the CBeebies website: a more child

CHCANYS Data Warehouse

Development of Usability and Context-of-Use - RUC