Performance Testing in Virtualized Environments

Performance Testing in Virtualized
Environments
Emily Apsey
Product Engineer
Introduction
•
Product Engineer on the Performance Engineering Team
•
Overview of team
- Specialty in Virtualization
-
•
Citrix, VMWare, Microsoft
Dedicated to Product Virtualization
-
Configuration
-
Scalability/Density
-
Provide feedback loop across all development teams
Collaborative Testing
•
Vendors
•
Microsoft
-
Active participation by vendors
-
Issaquah labs - Nov 13
-
Hands on configuration and tuning
-
RemoteFX team
-
NVIDIA
-
Hardware
-
SuperMicro and Dell r720
-
K1 and K2 cards
-
Driver (on hypervisor and VM)
-
•
•
EEC lab – Nov 14
VMWare
-
vSGA – Jan 14
-
vGPU – Oct 14
-
Future Reference Architecture Doc
Citrix
-
Majority of our Users
Presentation Overview
•
Why and how we tested using Nvidia GRID cards
-
Test methodology for virtualized test environments
•
Lesson Learned
•
Future Testing
Virtualization – Esri
•
ArcGIS in virtualized environments
-
Users began virtualizing our ArcMap software
-
Improve understanding of performance and scalability of ArcGIS in virtualized
environments
-
Improve information, internally and externally, related to:
-
Virtualization environments
-
What to expect
-
-
Performance
-
Scalability
New technologies
-
NVIDIA K1 and K2 cards
What and Why?
•
ArcGIS Pro
-
New addition to ArcGIS Desktop
suite
-
Desktop- analysis, publication,
visualization
-
New rendering engine
-
-
DirectX11 and OpenGL
-
Currently most heavily used app
-
ArcMap, based on GDI+
-
Heavily used in Citrix XenApp
Must virtualize well
-
Provide great UX
Testing Goals
•
•
Show if a shareable GPU is needed (ArcMap + Pro)
-
ArcMap – single threaded, GDI+ rendering
-
Pro – multi threaded, DX or OpenGL rendering
GPU needed for both 2D and 3D workflows?
-
•
•
How to monitor and communicate performance/ scalability
Optimal VM configurations
-
vCPU, Memory, GPU Profile for optimal user experience
-
What and how to communicate information to users, IT administrators
-
How many VMs can a single GRID card accommodate?
Administration
-
Know a little before you go on-site or create a demo of ArcGIS Desktop in virtualized
environments
Graphics Options in Virtualization
GRID vGPU Profiles
GPU Profile
•
Citrix and VMWare vGPU
-
Amount of Frame Buffer
-
-
•
Application requires at least 1 GB of FB
Identify Optimal profile for both VMWare and Citrix Hypervisors
-
Dependent on data and workflow
-
Effects density
Microsoft Hyper-V
-
Does not use GPU Profiles
-
Some control by screen resolution
Data and Workflows
•
2D Denver
-
Local government – Vector data model
-
•
3D Virtual City
-
•
Database- Points, line and polygons
Philly – Raster data, Z values extruded, building textures
Zoom/Pan to a series of standardized bookmarks within App
-
Time rendering (Sum of all bookmarks)
-
Frames Per Second (FPS)
-
Average – Sample every 100 milliseconds
-
Minimum – Indicates jitter, poor user experience
Testing Tools
•
•
Visual Studio vs. Light-weight application.
-
VS -performance overhead on each VM: disk, startup, GPU
-
UI maps maintenance overhead
Login VSI
-
•
Trail period, can be scriptable
Accurate Performance Metrics within Application
-
Internally developed ArcGIS Pro Add-In
-
Exposure of valuable metrics
-
Propagating events into an API method.
-
Very accurate performance metrics
-
Used by developers
-
Light weight
ArcGIS Pro Add-In
•
Specifically designed for performance
•
Script and reproducible
•
Self contained
•
Packaging data and test package to business partners
•
Great for Demos!
ArcGIS Pro Add-In UI
Automate Test Execution
•
Test Controller
-
Tests are executed via command line
-
Calls upon a python script which creates windows scheduled tasks
-
Opens ArcGIS Pro application and project
-
Rendering events logged
-
Automatically closes ArcGIS Pro when test completes
•
Scalability – across multiple VM’s
•
1, 4, 8, 12, 16 VM’s
-
TestList.csv
-
Test Cleanup
-
Collect logs from VM’s to File Share or SAN
-
Unique Names – VM Name
Collecting Data
•
•
ArcGIS ProAddIn Log
-
Draw Times, FPS – Avg and Min
-
Named Metrics
Open Project Time
Hung Threads
GPU Utilization, GPU Memory Utilization
ESXiTop (VMWare)
-
•
-
Nvidia SMI
-
•
Licensing Time
ArcMon (ArcGIS Pro) – Internal Use only
-
•
-
CPU Utilization (Host and VM)
Python based parsers
-
Export to log which is useable for reporting purposes
-
Drawback - No automated UI to show system metrics
ArcGIS Pro Add-In Log
Python
based
parser
2D
1
4
8
SumBkAvg
92.51
87.78
100.62
12
106.86
4.13
69.30
37.54
16
111.95
4.95
74.11
23.70
1
4
8
SumBkAvg
81.00
79.66
86.95
4.35
4.33
FPSAvg
54.02
53.41
47.38
FPSMin
13.74
14.65
10.71
12
92.00
6.19
43.23
9.64
16
103.42
8.00
40.91
7.66
3D
StDev
4.44
6.75
FPSAvg
66.65
66.60
65.99
FPSMin
42.06
42.93
37.54
StDev
Additional Hands-On Testing
•
Further validate automated test findings
•
Usability tests were executed manually by team members
•
Further qualifying the UX
-
Validating rendering quality to human eye
-
Overall responsiveness of the application
User Experience Ranking
1 to 5
Unacceptably jerky; Poor to annoying user experience
6
7
8
9 to 10
Jerky, but useable
Moderately smooth, moderate jerkiness
Smooth, minor jerkiness
Very smooth; little to no difference between a non-VDI solution. No apparent jerkiness, lag/tearing, or texturing delays.
Host CPU Utilization
Scalability of 240Q Profile - 3D Data
100%
90%
80%
70%
60%
50%
16 VMs:Host CPU
40%
14 VMs:Host CPU
12 VMs:Host CPU
30%
20%
10%
0%
10 VMs:Host CPU
16:13:02
16:13:18
16:13:33
16:13:49
16:14:04
16:14:20
16:14:35
16:14:51
16:15:07
16:15:22
16:15:38
16:15:54
16:16:09
16:16:25
16:16:40
16:16:56
16:17:11
16:17:27
16:17:43
16:17:58
16:18:14
16:18:29
16:18:45
16:19:01
16:19:16
16:19:32
16:19:47
16:20:03
16:20:18
16:20:34
16:20:50
16:21:05
16:21:21
16:21:36
16:21:52
16:22:08
16:22:23
16:22:39
16:22:54
16:23:10
16:23:25
16:23:41
Host Utilization Rates
16VMs - 6vCPU: 3D data
100
90
80
70
60
50
GPU1 %Util
GPU2 %Util
GPU3 %Util
40
GPU4 %Util
Cpu(Total)\% Util Time
30
20
10
0
Answers obtained from testing
•
ArcGIS Pro
-
GPU is required for 3D data; K2 card preferred
-
GPU aids 2D data delivery by relieving CPU; considered recommended.
GPU Utilization
GPU can be configured to be shareable (vGPU)
-
-
Exception XenApp; requires Pass Through GPU
Optimal VM configuration can vary by HyperVisor (encoding/decoding)
ArcGIS Pro – XenApp 7.6
XenApp 7.6 - 4 Users 3D
16vCPU, 16 GB of Memory
Requires GPU Pass-Through
- K2 GPU Memory = 4GB
100
OS does not do well splitting
GPU resources across
different terminal sessions
90
80
70
XenDesktop recommended
- Dedicated resources
- vGPU profile manages FB
- Dictates VM density
60
50
40
Esri collaborating with Nvidia
and Citrix
30
20
10
0
XenApp76\% Processor Time
XenApp76\% GPU Usage
XenApp76\% GPU Memory Usage
Recommended GPU Profiles
•
Dependent on user data and workflows
3D Data
2D Data
2D Data
Answers obtained from testing (continued)
•
ArcMap
•
Rendering engine does not
utilize GPU
-
Map docs
GPU is not heavily utilized
-
Only when user explicitly
executes hardware
acceleration operations
Rendering Sum Times (mins)
Lesson Learned
•
Online data introducing network latency
-
•
Eliminate ArcGIS Online data for initial benchmarks
Licensing
-
License Server vs. Application licensing
-
Eliminates logins into Application
•
FPS sampling – Avg and Min
•
Think time (5 seconds vs 10 seconds)
-
•
Aggressively intensive, further refine to closely model user workflows
Nvidia Drivers – HyperVisor and VM, same release
Future testing
•
Mixed workflows
-
Initial testing focused on rendering pipeline
-
Add analytical operations (CPU intensive)
-
•
Mixed Virtualization Vendor Solutions
-
•
VMWare ESXi XenDesktop
Storage – SAN
-
•
How GPU Utilization and density is affected when users are simultaneously running a CPU
intensive geoprocessing task?
More realistic – large infrastructure don’t use SSDs
Cloud based VDI (DaaS)
-
Increasing questions and interest
-
Distributors – Europe, NZ, US
-
Esri Redlands – Ed Services, Customer Holistic Testing
Questions?
•
Emily Apsey
-
eapsey@esri.com