Consumption Analytics Documentation

Home > Documentation for older versions > Cloud Cruiser 4 > Collecting, transforming, and publishing > VMware vSphere

VMware vSphere

The vSphere Snapshot Collector and vSphere Performance Collector, both retrieve data from VMware vCenter servers and ESX hosts using the VMware vSphere 5.1 Web Services API. They are intended to be used together to pair snapshot data with historical performance data. You can collect snapshot data at any frequency (HPE recommends hourly) for the current day and then merge it with related performance data in a daily flow.

For an understanding of the vSphere Web Services API, see the VMware vSphere API Reference Documentation. You will likely need this documentation as a reference for the hierarchy and data types of the vSphere data model as you set up collection.


The Snapshot and Performance Collectors work together.

To set up regular collection and loading of data

  1. Create a vSphere data source named vSphere with your vSphere credentials.
    The sample workbook expects a data source with this name. You can rename it later. For information on data source fields, see Data source properties.
  2. Create a new workbook from the vSphere_Template template.
    This workbook includes the following components:
    • vSphere_Snapshot collection to run hourly to collect snapshots of inventory data
    • vSphere_Performance collection to run nightly to collect performance data.
    • Main flow to transform data from both collections and publish it to the Cloud Cruiser database
  3. Edit the two Publish steps in the flow and add the account ID mappings required by the Virtual Machine and Storage schemas.
  4. Add any transformation to the flow required by your business rules.
  5. Schedule the collections and flows roughly as follows:
    Job name Select Date Frequency Start Time
    <workbookName>\vSphere_Snapshot.collect Current Day Hourly 0
    <workbookName>\vSphere_Performance.collect Previous Day Daily 00:05
    <workbookName>\Main Previous Day Daily 22:30

    If the Sampling Period for your vSphere Performance collection is larger, schedule its job further after midnight. The performance data samples are rolled up on the vSphere server a short time after each period ends, and the longer the period the more data there is to roll up. There is no guarantee as to how soon the data rollup will complete, making the data available to Cloud Cruiser.

Tracking entities for performance collection

To ensure that objects that are deleted during a day still have historical data collected for that day, the vSphere Performance Collector does its historical collection based on a list of entities which were discovered during snapshot collection rather than the entities that exist at the time it runs.

On each run, the vSphere Snapshot Collector logs the type, ID, and name of every entity collected to the CSV file <processDir>/snapshot/entities/entities_<date>.csv. This file is required input for the vSphere Perf Collector, which collects historical performance data for only the entities listed in it.

Correlating snapshot and performance data

Data from a vSphere Snapshot collection and a vSphere Performance collection combines to give you the full set of measures for a given vSphere entity. To enable this, your ETL workbook must perform the following tasks to transform rows for Datastore entities into rows for VirtualMachine entities. The template vSphere workbook does this:

  1. Create a lookup file that maps EntityID to EntityName.
  2. For datastore records, set the EntityID dimension to their corresponding VM's EntityID by prepending vm- to the datastore's Instance value.
  3. For the same records, set their EntityType to VirtualMachine.
  4. For the same records, copy their EntityName value to the Datastore dimension to preserve it.
  5. For the same records, set their EntityName by looking up their EntityID in the lookup file you created.
Last modified


This page has no custom tags.


This page has no classifications.

 (c) Copyright 2017-2020 Hewlett Packard Enterprise Development LP