Consumption Analytics Documentation

Home > Documentation for older versions > Cloud Cruiser 3 > Setting Up Collection > Native collectors > Microsoft Azure

Microsoft Azure

Cloud Cruiser's Microsoft Azure Collector, introduced in Cloud Cruiser 3.2, is a native collector that retrieves usage and cost data from Azure, Microsoft's public cloud platform. For information about collecting data from a private Windows Azure Pack cloud, see Microsoft Windows Azure Pack.

NOTE: In this release, only Enterprise Agreement accounts are supported.

This section contains the following articles:


The Azure Collector downloads usage reports from the Azure Enterprise Portal website. It supports only Azure accounts linked to an Enterprise Agreement and uses a Microsoft Account, not an Organizational Account, to connect to Azure. Interaction with the Enterprise Portal works as follows:

  • Version 3.3 and later: When you run a collection job, the Azure Collector requests a report for the selection date, waits for the report to become available on the Portal, and then downloads it. The report covers all accounts in your enterprise.
  • Version 3.2: The collector can only download the reports that are available in the Portal. You must manually request a report for a given day as an Enterprise Administrator and wait for it to become available for download on the Portal before you can run a job to collect data for that day.

The Portal does not offer real-time data: typically usage reports are available for dates five days ago and earlier, though this interval can vary. To always have the most recent data available in Cloud Cruiser, you need to daily collect for a date five or more days previous (seven days is a safer interval to use). You could choose to request and collect usage reports less often if you don't need the most recent data.

If the Enterprise Portal has available usage reports for overlapping date ranges and more than one covers the date that the Azure Collector is seeking, the topmost one is used. The collector reads the list of available usage files from the top (last requested) to the bottom (first requested) and downloads the first one it encounters that includes data for the select date of the running job.

The usage files downloaded by the collector are not placed in your <working_dir>/usage_files directory. The collector only reads the files into memory, never saving them to disk.

To set up regular collection and loading of data

This procedure varies by the version of Cloud Cruiser you are using.

Version 3.2

Version 3.3 and later

  1. Create an Azure data source.
    See Creating a data source.
  2. Save the sample jobs azure-public-excel-collect and azure-public-excel-load as production jobs.
    If you want to collect a different set of identifiers than those listed in Identifiers, modify the feed configuration as explained in Mapping data to Cloud Cruiser.
  3. Reconcile your account structure with the format of the account IDs produced by the load job.
    If they are different you can reconcile them by changing the transform step of the job to produce different account IDs, by changing your account structure to match the IDs produced by the job, or a combination of both. The job produces account IDs based on a three-level account structure: Account Name (17), Subscription Name (42), and Component (30).
  4. Run your collection job using a select date range starting from your earliest Azure usage and ending seven days ago.
    This might take awhile if your date range is wide.
  5. When that run finishes, run your load job against the same date range.
    You have now brought your system up to date and are ready for ongoing collection and loading.
  6. Schedule the collection job to run daily starting tomorrow with a select date of Current Day Minus 7 Days.
    For information about scheduling jobs, see Scheduling a job.
  7. Schedule the load job to run daily with the same select date, starting 30 minutes after the collection job.

Sample jobs

The following sample job files are available in the <install_dir>/job_samples directory:

  • azure-public-excel-collect.xml: Downloads usage report data and writes it to CC Record files. Because the Azure Collector extends the Excel Collector, this job uses the same SpreadSheetFeedConfig bean for feed configuration.
  • azure-public-excel-load.xml: Aggregates and transforms this data, then loads charges into the Cloud Cruiser database.
    This job also creates a passthrough resource or updates an existing one for each resource in the current dataset, and it creates or updates a customer record at all levels of the account structure for each account ID in the dataset.

Bean reference

This section lists the beans you need to write batch jobs that collect and process data from Microsoft Azure and provides pointers to their reference information.


API documentation



com.cloudcruiser.batch.collect <install_dir>/docs/api/batch/com/cloudcruiser/batch/collect/package-frame.html SpreadSheetFeedConfig


It is common for the Microsoft Azure Collector to run successfully without collecting all the data you expect to see for the select date, or even without collecting any data at all. This is usually due to two factors inherent in Azure:

  • Availability of Azure usage data is delayed and unpredictable. Data usually becomes available several days after the usage occurred, at varying intervals, and the usage data for a particular day is often incomplete when first released. The procedure for setting up collection and loading accounts for this, but this doesn't guarantee that complete Azure data will be available when you collect.
  • There is no API available for Azure usage data, so the collector logs in to the Enterprise Portal and performs the same actions that a person would to request a usage report. If the site is temporarily down or if Microsoft changes how the Download Usage Data page is coded, the collector will fail to collect Azure data.

To troubleshoot missing data

  1. From the Cloud Cruiser computer, log in to the Enterprise Portal using the same account as in your Cloud Cruiser data source, request and download a usage report for the dates in question, and then examine the report.
    • If the site is down or hangs and you are unable to download a usage report: Cloud Cruiser is also unable to download it. Wait for Microsoft to resolve the problem with the site. When you are able to manually download a report, then try collecting with Cloud Cruiser again.
    • If you are forced to take steps to verify your identity upon login: Cloud Cruiser is encountering the same obstacle upon logging in, but is unable to get around it. After you have completed these steps so that Azure allows a normal login from this computer, try collecting with Cloud Cruiser again.
    • If the report is missing the same data as Cloud Cruiser: Azure has not yet made complete data available and Cloud Cruiser is working normally. Wait until Azure makes complete data available, then continue to the next step.
    • If the report contains more data than collected by Cloud Cruiser: It's likely that more data was made available between the run of the collection job and your live download of the report. Continue to the next step.
  2. Delete the data loads for the dates in question, then run your collection and load jobs again for those dates.
    If Cloud Cruiser collects more data than before and now matches your downloaded report, then most likely the complete data was not available when the collection job originally ran and Cloud Cruiser is working normally.
  3. If Cloud Cruiser still does not have complete data for the dates in question, contact HPE Support.
Last modified



This page has no classifications.

 (c) Copyright 2017-2020 Hewlett Packard Enterprise Development LP