Consumption Analytics Documentation

Home > Documentation for older versions > Cloud Cruiser 3 > Setting Up Collection > Native collectors > Amazon Web Services

Amazon Web Services

Cloud Cruiser’s Amazon Web Services (AWS) Collector is a native collector added in version 3.2 that retrieves consumption data using the AWS reporting API.

This section contains the following articles:


Amazon’s Programmatic Billing Access provides hourly detailed billing reports in CSV format, zipping the CSV files and storing them in an S3 (Simple Storage Service) bucket. The AWS Collector downloads these files to <working_dir>/usage_files/aws_detailed_reports and processes them to import usage data into Cloud Cruiser.

The files are named aws-detailed-<accountId>-<year>-<month>_<utcTimestamp>, for example

AWS generates a detailed billing report every hour. For complete information about the contents of a detailed billing report, see Understand Your Usage with Detailed Billing Reports in the AWS documentation. For information about which fields Cloud Cruiser collects from these reports, see Mapping data to Cloud Cruiser.

By default the sample job collects unblended charges, which do not reflect discounts from pricing tiers and reserved instances. This provides more accurate results on a day-to-day basis but will not match the final billing report for a month from AWS. The following procedure includes steps to achieve accurate end-of-month results using blended charges, a capability added in version 3.4.

To set up regular collection and loading of data

  1. Set up your AWS account to generate billing data and give access to Cloud Cruiser.
    See Setting up your account.
  2. Create a data source using the credentials you just created.
    See Creating a data source.
  3. Save the sample job aws_collect as a production job
  4. Make the following edits to the feed configuration.
    • Specify how to handle user tags.
      See User tags.
    • Uncomment the useBlendedValues property and change its value from true to ${param.blended:false}.
      This enables you to pass the value of this property as an argument as instructed in step 7.3. The full line of code should be:
      <property name="useBlendedValues" value="${param.blended:false}"/>
  5. (optional) If you are connecting through a proxy server, specify its information in the job.
    See AWSConnectionConfiguration.
  6. Schedule the job to run daily with the previous day as the select date.
    Complete data for the previous day might not be available until late evening, so HPE recommends scheduling the job to run at approximately 10pm.
  7. At the end of every month:
    1. Wait until you have downloaded the final report for the month, usually available from AWS a few days into the next month.
      This report, downloaded by your daily AWS job, is located in <working_dir>/usage_files/aws_detailed_reports. It will have a numeric ID in the InvoiceID column, whereas the partial-month reports have the value Estimated in that column.
    2. Go to Data > Job Loads, then select and delete all AWS data loads for the month.
      If you haven't changed the process name in your job, you can find these loads quickly by setting an Accounting Date range and filtering Process Name to aws_collect.
    3. Go to Data > Job Maintenance and run your AWS job. Use the full month as your date range and specify blended=true in the Arguments field.
    These steps replace the unblended charges originally loaded for the month with blended charges, which are now accurate.

Reserved Instances

AWS detailed billing reports include charges both for Reserved Instance purchase fees and for virtual machines and databases to which a Reserved Instance was applied. These rows have a value of Y in the ReservedInstance field, which becomes an identifier in Cloud Cruiser.

To help you distinguish between Reserved Instances and other instances when viewing Cloud Cruiser​ reports, you can set a value for the collector's reservedInstanceResourcePrefix property. This prepends a string to the resource name in records where ReservedInstance is set to Y. For example, if you set the prefix to RI- then charges for box usage in the US-West region would be stored as the resource USW1-BoxUsage for an On-Demand Instance and as RI-USW1-BoxUsage for a Reserved Instance.

Alternatively, you can simply drill down on the ReservedInstance identifier in reports to determine which charges pertain to Reserved Instances.

Sample job

A sample job file is located in the <install_dir>/job_samples directory as aws_collect.xml. It should be run for the previous day, not the current day; otherwise, data could be lost for the day.

Bean reference

This section lists the beans you need to write batch jobs that collect and process data from AWS and provides pointers to their reference information.


API documentation



Last modified



This page has no classifications.

 (c) Copyright 2017-2020 Hewlett Packard Enterprise Development LP