Consumption Analytics Documentation

Home > Documentation for older versions > Cloud Cruiser 4 > Collecting, transforming, and publishing > Amazon Web Services > Advanced AWS collection

Advanced AWS collection

If you are using the CloudSmart-Now solution for AWS, you can collect, transform, and publish data with minimal configuration. If the default configuration does not suit your needs, use the information in this topic to guide your decisions and changes.

Consider whether to mark reserved instances

AWS detailed billing reports include charges both for Reserved Instance purchase fees and for virtual machines and databases to which a Reserved Instance was applied. These rows have a value of Y in the ReservedInstance field, which becomes a dimension in Cloud Cruiser.

To help you distinguish between Reserved Instances and other instances when viewing Cloud Cruiser​ reports, you can set a value for the collector's Reserved Inst Res Prefix property, which prepends a string to the ComputedServiceName dimension name in records where ReservedInstance is set to Y. For example, if you set the prefix to RI- then charges for box usage in the US-West region would be collected as USW1-BoxUsage for an On-Demand Instance and as RI-USW1-BoxUsage for a Reserved Instance. To set the Reserved Inst Prefix property, in your AWS workbook click Collections > List, select your collection, click Edit, and then open the Amazon Web Service Properties tab.

Alternatively, you can simply drill down on the ReservedInstance dimension in reports to determine which charges pertain to Reserved Instances.

Consider how to use AWS tags

AWS enables you to add user-defined tags to a service for cost allocation. For example, you might want to specify the owner of a VM or the IT service that it supports. These tags are included in the detailed billing reports, with each tag adding a column titled
user:<tagName>. For example, a user tag named Department would be reported by AWS as user:Department.

Each service for which a given tag has been set has the tag value in that column. By default the AWS Collector collects each of these as a dimension. You can modify this behavior with the Strip User Tag Prefix and Excluded Tags properties. To set these properties, in your AWS workbook click Collections > List, select your collection, click Edit, and then open the Amazon Web Service Properties tab.

Consider the following guidelines before creating AWS tags:

  • Determine what downstream analysis you want tags to support, and use tags appropriate for that analysis. Define tags that meet the uses of your entire team, including your technical staff, who might be leveraging tags for technical infrastructure purposes.
  • To ensure that services are tagged correctly immediately upon provisioning, educate anyone who has the authority to provision services about proper tag usage.
  • Understand that once a tag has been applied to an AWS resource, changing the tag in your AWS billing report does not retroactively change tags already applied. 
  • Configure AWS to include tags in your cost allocation reports. For more information see Setting Up Your Monthly Cost Allocation Report in the AWS documentation.

For more information about AWS tags, see Use Cost Allocation Tags in the AWS documentation.

Consider whether to use blended values

AWS provides both blended and unblended rates and costs in its invoices. Unblended values don't reflect discounts from pricing tiers and reserved instances. When collecting daily during a month unblended values provide more accurate results,​ but these values will not match the final billing report for a month from AWS.

Set both Use Blended Values and End of Month Mode to No in a collection scheduled daily, and set both to Yes in a collection scheduled monthly after the final invoice is available (on or after the 10th of the following month).

Collecting Amazon Marketplace data

If you want to collect Amazon Marketplace data instead of the standard AWS billing reports, modify the collection job as described in the following procedure.

To collect Amazon Marketplace data

  1. Navigate to Data > Processing > Advanced Job Maintenance.
  2. In the list of jobs, expand Workbooks and then click the AWS\AWS1.collect job.
    The XML for that job opens in the display pane.
  3. Above the <batch:job id="AWS1" parent="ccjob"> line, add the following lines:
    <bean id="feedConfig" class="">
       <!-- If set, controls whether Amazon-assigned prefix for user tags ("user:") is stripped. -->
       <!-- Default value is true. -->
       <!-- <property name="stripUserTagPrefix" value="false"/> -->
       <!-- sets list of user tags to exclude from processing -->
       <!-- <property name="excludedUserTags"> -->
       <!-- <list> -->
       <!-- <value>MyTag1</value> -->
       <!-- <value>MyTag2</value> -->
       <!-- </list> -->
       <!-- </property> -->
       <!-- if true, use blended cost/rate (as opposed to unblended cost/rate). Defaults to false (unblended) -->
       <property name="useBlendedValues" value="${param.useBlendedValues:false}"/>
       <!-- mapping of file names as they are stored on Amazon (S3) to locally saved names. The ${variable} expressions -->
       <!-- are substituted at runtime to form a complete regular expression pattern. Each exression is expected to be a -->
       <!-- named java regular expression group. ${awsSccount}, ${reportYear}, ${reportMonth} and ${lastModifiedTimestamp} -->
       <!-- are required and map to predefined regular expressions (which can be overridden). Extra expressions are allowed. -->
       <property name="s3FileToSavedFilePatternMap"> 
             <!-- <entry key="${awsAccount}-${reportWithTagsBasename}-${reportYear}-${reportMonth}\.csv\.zip" -->
             <!-- value="${awsAccount}-${reportWithTagsBasename}-${reportYear}-${reportMonth}-${lastModifiedTimestamp}"/> -->
                <entry key="${awsAccount}-${marketplaceReportBasename}-${reportYear}-${reportMonth}\.csv\.zip" value="${awsAccount}-${marketplaceReportBasename}-${reportYear}-${reportMonth}-${lastModifiedTimestamp}"/>
       <!-- Default regular expressions for ${awsSccount}, ${reportYear}, ${reportMonth} and ${lastModifiedTimestamp} -->
       <property name="accountRegex" value="(?&lt;awsAccount&gt;[0-9]+)"/> 
       <property name="reportYearRegex" value="(?&lt;reportYear&gt;[0-9]{4})"/> 
       <property name="reportMonthRegex" value="(?&lt;reportMonth&gt;[0-9]{1,2})"/> 
       <property name="lastModifiedRegex" value="(?&lt;lastModifiedTimestamp&gt;[0-9]{8}T[0-9]{6})"/> 
       <!-- additional expressions can be defined here -->
       <property name="patternExpressionsMap">
             <!-- <entry key="reportWithTagsBasename" -->
             <!-- value="(?&lt;reportWithTagsBasename&gt;aws-billing-detailed-line-items-with-resources-and-tags)"/> -->
                <entry key="marketplaceReportBasename" value="(?&lt;marketplaceReportBasename&gt;aws-billing-detailed-line-items-AWSMarketplace)"/>
  4. Click Save.

Advanced collection properties

For each collection in the AWS workbook, you can modify the following properties in the Amazon Web Services Properties section of the Edit Collection dialog box:

  • Template: Determines the format used by the S3 Filename Format field. Click Select, and then select either Default or Marketplace, depending on the type of AWS data you want to collect.
  • Use Blended Values: Whether to collect the BlendedRate and BlendedCost values from the AWS report as the rate and cost for each service. When set to No, the collector collects UnBlendedRate and UnBlendedCost.​ For more information, see Consider whether to use blended values.​
  • Strip User Tag Prefix: Whether to remove user: from the name of each dimension created from a user-defined tag. For more information, see Consider how to use AWS tags.
  • End of Month Mode: Whether to run the collector in End of Month Mode, which collects from a final monthly invoice to update the previous month's charges. When set to No, the collector collects month-to-date charges from incomplete daily invoices.​ For more information, see Consider whether to use blended values.​
  • Bucket Name: The Amazon S3 bucket that you set up for programmatic billing
  • Excluded Tags: The user tags to ignore, separated by commas or spaces. These do not become dimensions. For more information, see Consider how to use AWS tags.
  • Reserved Inst Res Prefix: String to prepend to the names of measures when the value of the ReservedInstance dimension is Y, indicating that the measure is a Reserved Instance. For more information, see Consider whether to mark reserved instances.
  • Max Error Retry: The number of times for the collector to retry connecting to AWS after an error on first attempt.
  • Socket Timeout (msec): The number of milliseconds to wait before the socket times out. A socket timeout occurs when there is an interruption in the flow of data packets between the data source and Cloud Cruiser, usually because of bandwidth issues.
  • Connection Timeout (msec): The number of milliseconds to wait before the connection times out. A connection timeout occurs when the data source cannot be reached, usually because the server is down.
  • S3 Filename Format: The format of the file named used for data collection. If no format is specified, the collection uses the format of the Default format.
    • Default: ${awsAccount}-aws-billing-detailed-line-items-with-resources-and-tags-${reportYear}-${reportMonth}\.csv\.zip
    • Markekplace: ${awsAccount}-aws-billing-detailed-line-items-AWSMarketplace-${reportYear}-${reportMonth}\.csv\.zip
Last modified


This page has no custom tags.


This page has no classifications.

 (c) Copyright 2017-2020 Hewlett Packard Enterprise Development LP