Cloud Cruiser became HPE Consumption Analytics on Nov. 1st. You'll still see the old name in places while we update this site.

 

 

Consumption Analytics Documentation

Home > Cloud Cruiser 3 > Setting Up Collection > Native collectors > Microsoft Azure

Microsoft Azure

HPE Consumption Analytics Portal's Microsoft Azure Collector, introduced in HPE Consumption Analytics Portal 3.2, is a native collector that retrieves usage and cost data from Azure, Microsoft's public cloud platform. For information about collecting data from a private Windows Azure Pack cloud, see Microsoft Windows Azure Pack.

NOTE: In this release, only Enterprise Agreement accounts are supported.

This section contains the following articles:

Overview

The Azure Collector downloads usage reports from the Azure Enterprise Portal website. It supports only Azure accounts linked to an Enterprise Agreement and uses a Microsoft Account, not an Organizational Account, to connect to Azure. Interaction with the Enterprise Portal works as follows:

  • Version 3.3 and later: When you run a collection job, the Azure Collector requests a report for the selection date, waits for the report to become available on the Portal, and then downloads it. The report covers all accounts in your enterprise.
  • Version 3.2: The collector can only download the reports that are available in the Portal. You must manually request a report for a given day as an Enterprise Administrator and wait for it to become available for download on the Portal before you can run a job to collect data for that day.

The Portal does not offer real-time data: typically usage reports are available for dates five days ago and earlier, though this interval can vary. To always have the most recent data available in HPE Consumption Analytics Portal, you need to daily collect for a date five or more days previous (seven days is a safer interval to use). You could choose to request and collect usage reports less often if you don't need the most recent data.

If the Enterprise Portal has available usage reports for overlapping date ranges and more than one covers the date that the Azure Collector is seeking, the topmost one is used. The collector reads the list of available usage files from the top (last requested) to the bottom (first requested) and downloads the first one it encounters that includes data for the select date of the running job.

The usage files downloaded by the collector are not placed in your <working_dir>/usage_files directory. The collector only reads the files into memory, never saving them to disk.

To set up regular collection and loading of data

This procedure varies by the version of HPE Consumption Analytics Portal you are using.

Version 3.2

Version 3.3 and later

  1. Create an Azure data source.
    See Creating a data source.
  2. Save the sample jobs azure-public-excel-collect and azure-public-excel-load as production jobs.
    If you want to collect a different set of identifiers than those listed in Identifiers, modify the feed configuration as explained in Mapping data to Cloud Cruiser.
  3. Reconcile your account structure with the format of the account IDs produced by the load job.
    If they are different you can reconcile them by changing the transform step of the job to produce different account IDs, by changing your account structure to match the IDs produced by the job, or a combination of both. The job produces account IDs based on a three-level account structure: Account Name (17), Subscription Name (42), and Component (30).
  4. Run your collection job using a select date range starting from your earliest Azure usage and ending seven days ago.
    This might take awhile if your date range is wide.
  5. When that run finishes, run your load job against the same date range.
    You have now brought your system up to date and are ready for ongoing collection and loading.
  6. Schedule the collection job to run daily starting tomorrow with a select date of Current Day Minus 7 Days.
    For information about scheduling jobs, see Scheduling a job.
  7. Schedule the load job to run daily with the same select date, starting 30 minutes after the collection job.

Sample jobs

The following sample job files are available in the <install_dir>/job_samples directory:

  • azure-public-excel-collect.xml: Downloads usage report data and writes it to CC Record files. Because the Azure Collector extends the Excel Collector, this job uses the same SpreadSheetFeedConfig bean for feed configuration.
  • azure-public-excel-load.xml: Aggregates and transforms this data, then loads charges into the HPE Consumption Analytics Portal database.
    This job also creates a passthrough resource or updates an existing one for each resource in the current dataset, and it creates or updates a customer record at all levels of the account structure for each account ID in the dataset.

Bean reference

This section lists the beans you need to write batch jobs that collect and process data from Microsoft Azure and provides pointers to their reference information.

Package

API documentation

Beans

com.cloudcruiser.batch.collect.microsoft.azurepublic​

<install_dir>/docs/api/batch/com/cloudcruiser/batch/collect/microsoft/azurepublic/package-frame.html

com.cloudcruiser.batch.collect <install_dir>/docs/api/batch/com/cloudcruiser/batch/collect/package-frame.html SpreadSheetFeedConfig

Troubleshooting

It is common for the Microsoft Azure Collector to run successfully without collecting all the data you expect to see for the select date, or even without collecting any data at all. This is usually due to two factors inherent in Azure:

  • Availability of Azure usage data is delayed and unpredictable. Data usually becomes available several days after the usage occurred, at varying intervals, and the usage data for a particular day is often incomplete when first released. The procedure for setting up collection and loading accounts for this, but this doesn't guarantee that complete Azure data will be available when you collect.
  • There is no API available for Azure usage data, so the collector logs in to the Enterprise Portal and performs the same actions that a person would to request a usage report. If the site is temporarily down or if Microsoft changes how the Download Usage Data page is coded, the collector will fail to collect Azure data.

To troubleshoot missing data

  1. From the HPE Consumption Analytics Portal computer, log in to the Enterprise Portal using the same account as in your HPE Consumption Analytics Portal data source, request and download a usage report for the dates in question, and then examine the report.
    • If the site is down or hangs and you are unable to download a usage report: HPE Consumption Analytics Portal is also unable to download it. Wait for Microsoft to resolve the problem with the site. When you are able to manually download a report, then try collecting with HPE Consumption Analytics Portal again.
    • If you are forced to take steps to verify your identity upon login: HPE Consumption Analytics Portal is encountering the same obstacle upon logging in, but is unable to get around it. After you have completed these steps so that Azure allows a normal login from this computer, try collecting with HPE Consumption Analytics Portal again.
    • If the report is missing the same data as HPE Consumption Analytics Portal: Azure has not yet made complete data available and HPE Consumption Analytics Portal is working normally. Wait until Azure makes complete data available, then continue to the next step.
    • If the report contains more data than collected by HPE Consumption Analytics Portal: It's likely that more data was made available between the run of the collection job and your live download of the report. Continue to the next step.
  2. Delete the data loads for the dates in question, then run your collection and load jobs again for those dates.
    If HPE Consumption Analytics Portal collects more data than before and now matches your downloaded report, then most likely the complete data was not available when the collection job originally ran and HPE Consumption Analytics Portal is working normally.
  3. If HPE Consumption Analytics Portal still does not have complete data for the dates in question, contact HPE Support.
Last modified

Tags

Classifications

This page has no classifications.
© Copyright 2018 Hewlett Packard Enterprise Development LP