TSCO-10.0-Managing Capacity in an OpenStack environment
Transcription
TSCO-10.0-Managing Capacity in an OpenStack environment
Connect with BMC TrueSight Capacity Optimization v10.0 Managing capacity in an OpenStack environment - Int'l Toll: 719-386-0002 - US/CAN Toll free: 888-287-5534 - Conference ID: 9962526 - Participant Passcode: 333044 International Toll Free Argentina: 0800 666 3168 Australia: 1 800 106 747 Austria: 0800 291 959 Belgium: 0 800 75 269 Brazil: 0 800 891 6846 Bulgaria: 00 800 120 1113 Chile: 123 0020 9728 China, Northern Region: 10 800 714 1203 China, Southern Region: 10 800 140 1182 Colombia: 01 800 518 0791 Costa Rica: 0800 015 0618 Czech Republic: 800 700 296 Denmark: 80 886 221 France: 0 800 908 251 Germany: 0 800 181 9045 Greece: 00 800 161 2205 5966 Hong Kong: 800 901 125 Hungary: 06 800 162 52 India: 000 800 1006 982 Indonesia: 001 803 017 5966 Ireland: 1 800 760 613 Israel: 1 80 924 6043 Italy: 800 873 746 Japan: 00531 16 0846 Local Latvia: 8000 2413 Lithuania: 8 800 3 05 27 Luxembourg: 800 2 7667 Malaysia: 1800 81 3716 Mexico: 01 800 522 5328 Monaco: 800 93 418 Netherlands: 0 800 023 5307 New Zealand: 0 800 451 052 Norway: 800 196 67 Panama: 00 800 226 5966 Peru: 0800 55 436 Philippines: 1 800 111 008 64 Poland: 00 800 111 49 60 Portugal: 800 819 731 Russian Federation: 810 800 2706 1012 Singapore, Singapore: 800 101 2004 Slovakia: 0800 606 718 Slovenia: 0 800 80205 South Africa: 0 800 980 990 South Korea, Korea, Republic Of: 003 0813 1965 Spain: 900 947 606 Sweden: 02 079 9908 Switzerland: 0 800 564 399 Taiwan: 00 801 126 975 Thailand: 001 800 156 205 5966 Trinidad and Tobago: 1 800 205 5966 United Kingdom: 0 808 101 1148 Uruguay: 0004 019 0190 Venezuela: 0 800 100 8303 Australia, Brisbane: +61 (0) 7 3123 0044 Australia, Canberra: +61 (0) 2 6111 2009 Australia, Melbourne: +61 (0) 3 9951 0936 Australia, Sydney: +61 (0) 2 9038 0411 Austria, Vienna: +43 (0) 1 2675 902 Bahrain, Manama: +973 1619 9828 Belgium, Brussels: +32 (0) 2 303 2271 Brazil, Sao Paolo: +55 11 3181 3952 Bulgaria, Sofia: +359 (0) 2 491 6409 Canada, Montreal: +1 514 669 6114 Canada, Toronto: +1 647 426 9211 China: +86 400 120 2695 China: +86 400 120 26 95 Czech Republic, Prague: +420 234 147 001 Denmark, Copenhagen: +45 78 78 79 61 Estonia, Tallinn: +372 622 5740 Finland, Helsinki: +358 (0) 9 7479 0099 France, Lille: +33 (0) 359 69 03 40 France, Lyon: +33 (0) 426 10 30 22 France, Paris: +33 (0) 1 70 71 29 51 Germany, Berlin: +49 (0) 30 2555 5428 Germany, Frankfurt: +49 (0) 69 1200 9825 Germany, Munich: +49 (0) 89 1436 7909 Hong Kong, Hong Kong: +852 3008 0381 Hungary, Budapest: +36 1 577 9956 India, Bangalore: +91 (0) 80 6127 5132 India, Mumbai: +91 (0) 22 6150 2332 Ireland, Dublin: +353 (0) 1 437 0558 Israel, Tel Aviv: +972 (0) 3 721 9307 Italy, Milan: +39 02 8978 1996 Italy, Rome: +39 06 8743 4379 Japan, Tokyo: +81 (0) 3 4589 9498 Latvia, Riga: +371 6601 3681 Lithuania, Vilnius: +370 5205 5591 Luxembourg, Luxembourg: +352 2786 0223 Malaysia, Kuala Lumpur: +60 (0) 3 7724 0845 Mexico, Mexico City: +52 55 4777 2669 Netherlands, Amsterdam: +31 (0) 20 262 0092 New Zealand, Auckland: +64 (0) 9 929 1882 Norway, Oslo: +47 21 95 32 31 Poland, Warsaw: +48 (0) 22 295 36 29 Portugal, Lisbon: +351 21 120 1905 Romania, Bucharest: +40 (0) 21 529 1342 Russian Federation, Moscow: +7 495 620 9816 Singapore, Singapore: +65 6416 9955 Slovakia (Slovak Republic), Bratislava: +421 (0) 2 3278 6630 Slovenia, Ljubljana: +386 (0) 1 888 8399 South Africa, Johannesburg: +27 11 589 8380 Spain, Madrid: +34 91 080 0151 Sweden, Stockholm: +46 (0) 8 4030 4951 Switzerland, Geneva: +41 (0) 22 555 0255 Switzerland, Zurich: +41 (0) 44 556 8426 Taiwan, Taipei: +886 (0) 2 2650 7290 United Kingdom, London: +44 (0) 20 8150 0793 Connect with BMC TrueSight Capacity Optimization 10.0 Managing capacity in an OpenStack environment — Speaker Sudheer Apte, Product Development Architect April 28, 2015 Connect with BMC TrueSight Capacity Optimization Agenda Managing capacity in an OpenStack Environment • Introduction to OpenStack • OpenStack configurations and evolution • Collecting data into TrueSight Capacity Optimization • Managing Capacity of OpenStack Introduction to OpenStack — Connect with BMC TrueSight Capacity Optimization 10.0 What is OpenStack? “Cloud operating system” Infrastructure as a Service (IaaS) software. Manage compute, storage, network. Open source, public APIs Heat = service orchestration What can it do? IaaS Examples: - start, stop, resume an instance - create, attach a volume - take snapshots Horizon dashboard For cloud provider and for tenant user OpenStack terminology OpenStack term Meaning Server, instance Virtual machine Compute node, hypervisor node Virtual host. Can be KVM, XenServer, VMware cluster… Image Virtual machine template Volume, block volume Virtual disk attached to a virtual machine Ephemeral volume One having the same lifetime as its virtual machine Persistent volume One created independently and attached to VM Block storage service Service to create, delete, and manage block volumes Image service Service to create, delete, and manage VM templates Object storage service Service for URL-accessible files, like Amazon S3 OpenStack configurations and evolution — Connect with BMC TrueSight Capacity Optimization 10.0 OpenStack Foundation Platinum, Gold Plus over 100 corporate sponsors and supporters Key OpenStack service APIs Purpose OpenStack component service API Identity (Authentication, Authorization) Keystone Compute Nova Object storage service Swift Image storage service Glance Block storage service Cinder Network Neutron (old name: Quantum) Horizon Swift Neutron Nova Cinder Keystone Glance Example with Nova networking Public network Flat network Cloud controller node Database Scheduler Message queue Identity API services Image Block storage Management network Compute node Block storage node Hypervisor SCSI target API service Cinder-volume Nova network NFS server Nova implementations and features - 1 Nova implementations and features - 2 Nova implementations and features - 3 Cinder implementations – 1 Cinder implementations – 2 Cinder implementations – 3 Ephemeral disk vs “block volume” Block volume ephemeral disk guest Cinder storage host KVM host storage pool type = “network” directory /cinder/vols Partitions Devices /root sda /data /home In releases up to kilo: KVM host sees an iSCSI disk, which is presented to guest. In releases from kilo: It is possible to import the iSCSI disk as a “network” type, which is directly presented to the guest. Storage Evolution of OpenStack Feature Keystone Nova 2013 2014 2015 Folsom, Grizzly, Havana Havana, Icehouse Juno,Kilo, Liberty Request verification UUID tokens PKI tokens, domains Integration with auth LDAP with write access Read-only LDAP Hypervisor drivers Libvirt (KVM), XenServer, HyperV, VMware, PowerVM PowerVM removed Quotas Project (tenant) level User-level; default Storage attach GlusterFS via FUSE GlusterFS native; iSER Scheduling Utilization-aware scheduler Glance Back-ends Sheepdog, Cinder GridFS, VMware datastore Networking Floating IP module Cisco, Nova plugin Brocade plugin Routers, Load balancers HAProxy-based OpenStack components Quantum now called Neutron Attach technology Fibre Channel Back-end drivers EMC, 3Par, GlusterFS, many more Cinder Federated auth Configurations and evolution Summary Conclusions Multiple components built and released by distributed teams Open source software (OSS) projects • GNU/Linux • Libvirt • KVM • Nova, Cinder, Swift, Glance, Keystone, … Plug-ins and drivers contributed by large constellation of vendors Bundled “distributions” of OpenStack created by multiple groups and companies • Always contain Nova • Almost always based on KVM hypervisor Supporting additional OSS components are becoming popular • Ceph (distributed storage) “Do you support OpenStack?” • Not a simple question • Look at exact combination deployed • • • • Understand additional supporting components • • • Hypervisor Object storage Ephemeral versus block storage policy Commercial OSS Exact releases deployed • • In test In production Collecting data into TrueSight Capacity Optimization 10.0 — Connect with BMC TrueSight Capacity Optimization 10.0 Connect with BMC TrueSight Capacity Optimization Agenda Managing capacity in an OpenStack Environment • Introduction to OpenStack • OpenStack configurations and evolution • Collecting data into TrueSight Capacity Optimization • Managing Capacity of OpenStack Data sources relevant to OpenStack Openstack Admin UI Nova API Openstack controllers Nova DB scheduler Available in 10.0 OOTB Hypervisors, VMs, relationships and some configuration Host Aggregates, Availability Zones, Regions, relationships Tenants, VMs, relationships and some configuration • • • Nova API OS access (depending on controller node type) MQ • • OS performance data OpenStack components (database, MQ, services) process and workload performance data Available in 10.0 OOTB Ceilometer components Ceilometer store Ceilometer API Hypervisor Clusters/hosts VM Guest OS Hypervisor API Ceilometer (optional component) • • • Hypervisor performance data where collected and configured VM performance data where collected and configured Other (storage) performance data where collected and configured Hypervisor access (depending on hypervisor type) • • Hypervisor performance data VM performance data Available in 10.0 OOTB: • • Via GW + Agents – most platforms Via RHEV-M – for RHEV-M KVM OS access (depending on hypervisor type) • OS, application performance data Available in 10.0 OOTB OOTB Connector for OpenStack (Nova) OpenStack – Open-source Cloud OS - also packaged by ISVs (e.g., RedHat) – Support for multiple hypervisor technologies, including RedHat Enterprise Virtualization (RHEV), Linux KVM and VMware vSphere – Pooling constructs (e.g. Host Aggregates, Availability Zones) – Tenants (aka “Projects”) – Admin dashboard for Cloud Admin – Self-provisioning web interface for End Users New OpenStack API Extractor ETL – – Supports OpenStack versions Grizzly, Havana and Ice House Leverages OpenStack Nova APIs to import infrastructure entities and tenants into TSCO: • Basic configuration metrics (e.g., number of vCPUs requested) - more conf and performance metrics need to be collected from other ETLs, depending on which hypervisor is being adopted • Cloud topology relationships, that is: – Infrastructure hierarchy (host aggregate – host – VM) – Tenant hierarchy (tenant – VM) TrueSight Capacity Optimization 10.0 Data flows for managing OpenStack cloud capacity – KVM example Openstack controller nodes (Linux) Cloud Admin Nova DB Message queue scheduler Image service Object store Network controller Compute node (KVM) TSCO Console Openstack Admin UI Capacity Planner BMC TrueSight Capacity Optimization Nova API Keystone Block storage service Import resource and tenant hierarchies TSCO OpenStack ETL TSCO CDB Agent Volume controller import system and workload data Compute controller TSCO Gateway Server VM TSCO vis parser ETL Agent Agent BMC CONFIDENTIAL ©2015 BMC Software OpenStack hierarchy in TSCO Hierarchy representing OpenStack relationships: mapping from Host Aggregates to Hosts and from Host to VMs Data aggregated at Cloud level from underlying Host Aggregate level Data aggregated at Host Aggregate level from underlying Hosts data Hierarchy representing Tenants (Projects) and their relationship to VMs from OpenStack environment Tenant (Project) and VMs in use Cloud hierarchy Host Aggregate hierarchy Tenant hierarchy OpenStack entities in workspace Collected from OpenStack Configuring OpenStack connector Setup connection parameters • Identity service URL: value of OS_AUTH_URL parameter from stack.sh on Keystone server (identity service used by OpenStack for authentication) • Username and password (for an authenticated user with "admin" role): value of OS_USERNAME parameter (typically "admin“) from stack.sh on Keystone server • Tenant name: value of OS_TENANT_NAME parameter (typically "admin" tenant) from stack.sh on Keystone server Sharing entity catalog with performance ETLs Share Entity Catalog with other ETLs collecting performance/capacity data Capacity Agents and Gateway Server Console ETL Engine Server VIS file integration Gateway Server Other data sources Agent data collection Database Server Application Server Extracting agent data – platform selection Easier configuration of the collected metrics – including Virtual Nodes Automatically configuring vis parsers “Auto-create” option for Gateway Server configuration Viewing Gateway Servers and connectors Red Hat Enterprise Virtualization Manager (RHEV-M) Datacenter D1 Virtual clusters contain physical hosts – Running VMs Storage clusters contain storage hosts – – Implementing “storage domains” Storage domains are used by Virtual clusters for storage Virtual Cluster VC1 VH1 VH2 VM1 VM3 VM2 VM4 uses Datacenters contain clusters – – Both Virtual Clusters and Storage Clusters Organize clusters into larger units for management Storage Cluster SC1 SH1 SH2 Storage Domain D1 RHEV – Extractor Service Datacenter Virtual Cluster Virtual host Virtual machines Domain Tree imported by RHEV – Extractor Service Storage domain Storage cluster Storage host KVM Views with RHEV – Extractor Service RHEV data gets populated into existing KVM Views – There are metrics that RHEV connector does not support the Network, File System and Disk metrics from the Host and Partition detail pages (displayed as “No data available”) Comparison of Metrics Capacity Agent vs. RHEV – Extractor Service Virtual host KVM BCO Metric RESNAME ALIAS_NAME BYAPP_ACTIVE_PROC BYAPP_CPU_UTIL BYAPP_CPU_UTIL_SYSTEM BYAPP_CPU_UTIL_USER BYAPP_DISK_READ_RATE BYAPP_DISK_WRITE_RATE BYAPP_PAGE_FAULT_RATE BYAPP_THREADS BYBENCHMARK_COMP BYBENCHMARK_DATE BYBENCHMARK_SOURCE BYBENCHMARK_VALUE BYCPU_CPU_UTIL BYCPU_CPU_UTIL_IDLE BYCPU_CPU_UTIL_SYSTEM BYCPU_CPU_UTIL_USER BYCPU_CPU_UTIL_WAIO BYDISK_BLOCK_SIZE BYDISK_MODEL BYDISK_NONPAGING_IO_RATE BYDISK_PAGING_IO_RATE BYDISK_PAGING_TYPE BYDISK_PHYS_DEV BYDISK_PHYS_IO_RATE BYDISK_PHYS_IO_READ_RATE BPA RHEV x x x x x x x x x x x x x x x x x x x x x x x x x x x x Virtual host KVM BCO Metric BYDISK_PHYS_IO_WRITE_RATE BYDISK_QUEUE_SIZE BYDISK_SERVICE_TIME BYDISK_SIZE BYDISK_TRANSFER_RATE BYDISK_TYPE BYDISK_UTIL BYFS_FREE BYFS_FREE_INODES BYFS_SIZE BYFS_TOTAL_INODES BYFS_USED_INODES_PCT BYFS_USED_SPACE_PCT BYIF_BANDWIDTH BYIF_COLLISION_RATE BYIF_DUPLEX_STATUS BYIF_ERROR_RATE BYIF_IN_BYTE_RATE BYIF_IN_PKT_RATE BYIF_OUT_BYTE_RATE BYIF_OUT_PKT_RATE BYUSR_ACTIVE_PROC BYUSR_CPU_UTIL BYUSR_CPU_UTIL_SYSTEM BYUSR_CPU_UTIL_USER BYUSR_DISK_READ_RATE BYUSR_DISK_WRITE_RATE BPA RHEV x x x x x x x x x x x x x x x x x x x x x x x x x x x Virtual host KVM BCO Metric BYUSR_PAGE_FAULT_RATE CPU_CORES_PER_SOCKET CPU_DESCRIPTION CPU_MAX_PROCESSORS CPU_MHZ CPU_MODEL CPU_MT_TYPE CPU_NUM CPU_RUN_QUEUE CPU_THREADS_PER_CORE CPU_TOTAL_MHZ CPU_UTIL CPU_UTIL_IDLE CPU_UTILMHZ CPU_UTILSPEC CPU_UTIL_SYSTEM CPU_UTIL_USER CPU_UTIL_WAIO CPU_VENDOR DESCRIPTION DISK_IO_RATE DISK_NUM DISK_READ_RATE DISK_SIZE DISK_TRANSFER_RATE DISK_WRITE_RATE GM_NUM BPA x x x x x x x x x x x x x x x x x x x x x x x x x x x RHEV x x x x x x x x x x x Comparison of Metrics – 2 Capacity Agent vs. RHEV – Extractor Service Virtual host KVM BCO Metric GM_ON_NUM HOST_ID HW_DESCRIPTION HW_MODEL HW_VENDOR LCPU_NUM MEM_BUFFERED MEM_CACHED MEM_COMMITED_AS MEM_FREE MEM_PAGE_IN_RATE MEM_PAGE_OUT_RATE MEM_PAGE_SIZE MEM_REAL_CONSUMED MEM_REAL_UTIL MEM_SYSTEM MEM_USED MEM_USER MEM_UTIL MEM_UTIL_CACHED MEM_UTIL_SYSTEM MEM_UTIL_USER NET_BIT_RATE NETIF_NUM NET_IP_ADDRESS NETWORK_DOMAIN OS_TYPE BPA RHEV x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x Virtual host KVM BCO Metric OS_VER TOTAL_FS_SIZE TOTAL_FS_USED TOTAL_FS_UTIL TOTAL_LDISK_SIZE TOTAL_LDISK_USED TOTAL_REAL_MEM UPTIME_PCT W_CPU_UTIL W_DESCRIPTION W_MEM_PRIVATE_ACTIVE W_MEM_PRIVATE_INACTIVE W_MEM_SHARED_ACTIVE W_MEM_SHARED_INACTIVE W_RESPONSE_TIME W_TRANSACTION_RATE SWAP_SPACE_FREE SWAP_SPACE_TOT SWAP_SPACE_USED SWAP_SPACE_UTIL TOTAL_REAL_MEM BPA RHEV x x x x x x x x x x x x x x x x x x x x x x Virtual guest KVM BCO Metric RESNAME CPU_CORES_PER_SOCKET CPU_LIMIT_MHZ CPU_NUM CPU_RESERVED_MHZ CPU_SHARES CPU_UTIL CPU_UTIL_OVERHEAD CPU_UTILMHZ DISK_TRANSFER_RATE HOST_ID HOST_NAME LDISK_NUM MEM_ALLOCATION MEM_CONSUMED MEM_FREE MEM_RESERVED MEM_UTIL NET_BIT_RATE NET_IN_BYTE_RATE NET_IN_ERROR_RATE NET_OUT_BYTE_RATE NET_OUT_ERROR_RATE NET_MAC_ADDRESS OS_TYPE TOTAL_LDISK_SIZE TOTAL_REAL_MEM BPA RHEV x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x Managing OpenStack Capacity — Connect with BMC TrueSight Capacity Optimization 10.0 Capacity use cases for clouds Category Use cases Typical role of TrueSight CO Capacity visibility Views and reports showing aggregated CPU, memory, disk, network, used and spare capacity Analyses, Views Capacity analysis Manual analysis; Early warning of capacity risks; detection of opportunities for savings Workspace; Capacity Pool Views: recommendations and alerts Cloud metering Show-back usage reporting for tenants. Chargeback. Capacity planning Sizing of servers and services; hardware sizing P2V, V2V studies Integrated demand and capacity mgmt Reserve capacity for projects in advance; plan hardware deployments Reservations. Placement during cloud provisioning Placement advice during instance creation Capacity-aware placement advice (CAPA) Rebalancing or “defrag” Batch-mode optimization of a large environment by moving workloads to “rebalance” clusters. Recommendations for source cluster; V2V study to find destination cluster. Types of OpenStack nodes • Compute nodes: appear as KVM hosts; relationships extracted via Nova API. • • Instances: appear as KVM guests; relationships extracted via Nova API. • • • Payload resources Grouped into Availability Zones and Regions. Belong to individual Tenants. Cloud storage nodes: could appear as: • • • Grouped into Host Aggregates, Availability Zones, and Regions. KVM guests. (Option 1) OR Standalone Linux machines (Option 2) Cloud controller nodes: could appear as: • • KVM guests. (Option 1) OR Standalone Linux machines (Option 2) Control resources Capacity visibility for payload resources Compute (Nova) • CPU, memory, ephemeral disk space on compute nodes • • Size, used, available Aggregated by: • • • Host aggregates – groups of hosts Availability zones – user-visible groups of hosts Regions – administrator-defined geographic partitions Storage (Cinder) • Storage volumes – size, used, available • Depends on storage host types (cinder backends) • RHEV storage nodes running Gluster • Ceph storage clusters • iSCSI on LVM Capacity analysis: Analyses, Models Time – Line Analysis: CPU and Memory Utilization % over last 2 months at Host Aggregate level Time Forecasting Model: for Memory Utilization % over the next 2 months at Cloud level Capacity analysis: By-exception reporting Capacity visibility for control resources Controller node • Generic name for any host that runs OpenStack services • Great flexibility in deployment on machines • Could run some services on compute nodes, too Machine type • Any Linux machine • Could be a VM Typical modes for resource usage Instance provisioning • Authentication steps not shown Compute server Nova API provision Instance create update Scheduler update Nova DB Glance service GET image Swift service GET object Control resources visible as process usage on controller nodes Find active intervals in Gateway Server Console Process statistics for top-10 processes UNIX process aggregation report By-user summary for workloads Workload analysis in Workspace Capacity in OpenStack environments Summary All standard TrueSight Capacity Optimization use cases for clouds • Data collection OOTB in most cases • Payload resources can be managed using OOTB views, chargeback, reservations, etc. • Control resources often require some manual interpretation • Controller nodes can be configured in many ways Conclusions “Do you support OpenStack?” • Many possible deployment options • Rapidly evolving Understand key drivers • Payload capacity • Controller capacity BMC TrueSight Capacity Optimization 10.0 — Reference Material TrueSight Capacity Optimization 10.0 New Welcome Page and How-to videos How to videos More to come … TrueSight Capacity Optimization v10.0 BMC Communities https://communities.bmc.com/co mmunity/bmcdn/service_assuranc e/capacity_management TrueSight Capacity Optimization 10.0 Documentation https://docs.bmc.com/docs/display /btco100/Home One single tree with updated and new sections More to come… TrueSight Capacity Optimization 10.0 Collaterals http://www.bmc.com/itsolutions/capacity-optimization.html new 10.0 datasheet new TrueSight solution brief new videos (10.0 demo about reservations and Engage session from Product Mgmt) new research papers (Forrester) (more to come) TrueSight Capacity Optimization 10.0 AMIGO program now available https://communities.bmc.com/docs/DOC-33065 Thank You — Bring IT to Life.™