Skip to main content

This site will be migrated to HPE Support Center by 1 November 2023. The new URLs for the HPE GreenLake User Guide and HPE GreenLake Partner User Guide will be posted here by 3 October. More information coming soon!

Documentation - HPE Usage and Analytics

Collecting data from Amazon Web Services (AWS)

  1. Last updated
  2. Save as PDF

HPE Consumption Analytics platform retrieves Amazon Web Services (AWS) consumption data using the AWS Billing API. This not only provides richer reporting in HPE Consumption Analytics platform, but also enables Insights that alert you to take action based on these metrics.

HPE Consumption Analytics platform does not support AWS accounts in India that are operated by Amazon Internet Services Pvt. Ltd (AISPL), a local reseller in India. To determine which company your account is with, refer to the AWS documentation.

HPE Consumption Analytics platform reads metrics for the following namespaces:

  • EC2
  • Redshift
  • RDS
  • EBS
  • ElastiCache

If you're using Amazon CloudWatch, HPE Consumption Analytics platform can collect performance and utilization data through the CloudWatch API. Because HPE Consumption Analytics platform collects data nightly, it collects unblended rates, which are more accurate on a day-to-day basis than blended rates.

Every night, HPE Consumption Analytics platform collects usage, billing, and performance data from one or more of your Amazon web Service (AWS) accounts. If you have linked accounts, HPE Consumption Analytics platform collects usage and billing data from the payer account, and CloudWatch performance data from each of your linked accounts. During this nightly collection HPE Consumption Analytics platform collects data for the current and previous months. This ensures that if something in your AWS bill changes, you always have the most current data available in your HPE Consumption Analytics platform analytics.

HPE Consumption Analytics platform also collects tax information from the AWS bill, where those records appear with the description Tax of type <type>. In HPE Consumption Analytics platform, this description appears in the Product field. The Resource ID for these records begins with the prefix Tax-. If the tax is a monthly charge, it appears in HPE Consumption Analytics platform with the usage for the last day of the month.

For more information about how HPE Consumption Analytics platform maps cloud providers' source data to HPE Consumption Analytics platform normalized fields, see Field mapping.

Setting up the AWS account to work with Consumption Analytics

Before you can create an AWS collection, you must set up the AWS account to work with HPE Consumption Analytics platform.  How you need to set it up depends on the type of credentials you want to use, and the data source from which you want the HPE Consumption Analytics platform to collect your data.

You must set up the management AWS account to work with HPE Consumption Analytics platform, not a member account (see AWS Organizations terminology and concepts in the AWS documentation).  HPE Consumption Analytics platform will find all linked accounts automatically once you enter credentials in a later step.

As you set up your AWS account(s) in AWS, note the items in bold in this section - you'll need them later when creating an AWS collection in HPE Consumption Analytics platform.

Choosing the credentials to use

The step-by-step instructions in this section explain how to set up your AWS data collection with access key and secret. 

These steps do not apply to data collections using delegated access, so if you plan to use delegated access credentials, you can skip to the next section and set up your data source.

If you create your HPE Consumption Analytics platform AWS data collection using key and secret credentials, then all linked accounts will also use key and secret credentials. If you create the data collection using delegated access credentials, then all linked accounts within will also use delegated access credentials.

The authentication method can be changed in the future if necessary.  In this case, it will be necessary to change your AWS account setup and collection configuration in HPE Consumption Analytics platform accordingly.

To use key and secret credentials

  1. Using Amazon's IAM service, create an AWS user with an access key ID, a secret key, and secure access to the specified S3 bucket (cost bucket) containing the data you want to collect. HPE Consumption Analytics platform needs only the name of the cost bucket, and not the full directory path. The cost bucket name must be at least 3 characters long, and can contain only letters, numbers, periods (.), hyphens (-), and underscores (_).
  2.  Set up a role in your AWS account, using Amazon's Identitiy and Access Management (IAM) service.

This role provides a mechanism to allow a third party to access your AWS resources without needing to share long-term credentials.  You need to give this role the appropriate permissions to allow HPE Consumption Analytics platform to discover the linked accounts within the AWS account and access it to collect usage information.

For further details on permissions requirements, including sample policies, see AWS permissions needed by the HPE Consumption Analytics Portal.

  1. If you want HPE Consumption Analytics platform to collect and report on the resource utilization, application performance, and operational health data available through the Amazon CloudWatch service, ensure that you apply the AmazonS3ReadOnlyAccess and CloudWatchReadOnlyAccess policies to your AWS user account.  You also need to add permission for AWS Organizations (list accounts), to enable HPE Consumption Analytics platform to locate the linked accounts associated with the collection. 

The resources HPE Consumption Analytics platform attempts to read are the ARNs that appear in the billing reports that support CloudWatch metrics.  The tool only needs access to metrics, not logs; the only request executed is GetMetricStatistics. Also, the account owner can limit the resources we are allowed to access if desired (see AWS permissions needed by the HPE Consumption Analytics Portal).

  1. If you want HPE Consumption Analytics platform to collect your AWS tags, ensure that those tags are included in your Cost Allocation Report (see Setting Up Your Monthly Cost Allocation Report in the AWS documentation).
  2. Create an AWS user for each of your linked accounts as per step 2.
  3. Continue with selecting your data source below. 

Choosing the data source to use

When you create an AWS data collection in HPE Consumption Analytics platform, you have the option to collect data from either the AWS Detailed Billing Reports (DBR) or the AWS Cost and Usage Report (CUR).  Either way, you must set up your data source in AWS before you can create your data collection in the HPE Consumption Analytics platform.

The Detailed Billing Reports feature in AWS is deprecated in favor of the Cost and Usage Report.  It is unavailable for new AWS customers as of July 2019.

To collect data from an AWS Cost and Usage Report

Create a Cost and Usage report in AWS for the account for which you want to collect data, by following the instructions in Creating Cost and Usage Reports in the AWS documentation.

To enable your HPE Consumption Analytics platform collection:

  • Specify the desired Report name
  • Select Include resource IDs
  • Select (or configure) the desired S3 Bucket.
  • Specify the Report path prefix
  • Set the time granularity to hourly.

To collect data from AWS Detailed Billing Reports

Turn on the legacy Detailed Billing Reports in AWS for the account for which you want to collect data (see the instructions in Understand Your Usage with Detailed Billing Reports in the AWS documentation).

If you are using consolidated billing, perform this step for the payer account, not the linked accounts. For more information about consolidated billing, see Pay Bills for Multiple Accounts with Consolidated Billing in the AWS documentation.

To enable your HPE Consumption Analytics platform collection:

  • Select the Monthly report and the Detailed billing report with services and tags

Once you've set up your AWS account for the credentials and data source you want theHPE Consumption Analytics platform to use, continue with the steps in the next section to create your AWS collection.

If you intend to create a collection with delegated access credentials, keep your AWS session open, because there are a few more steps to do in AWS as you create the collection in HPE Consumption Analytics platform. 

Creating an AWS collection in the HPE Consumption Analytics platform

When you create an AWS data collection in HPE Consumption Analytics platform, you have the option to create it (and all linked accounts in the collection) with Access Key ID and Secret Key authentication, or with Delegated Access authentication.  The step-by-step instructions that follow show you how to create an AWS data collection in the HPE Consumption Analytics platform with either access key and secret or delegated access. 
CAP_AWS_deleg_fig1.png

To create an AWS collection

If you enter invalid credentials, you will see the following message:

CAP_AWS_deleg_fig3c_add_error_msg.png

In this case, correct the error and try again.

  1. Click Data in the left navigation panel.
  2. On the Collections tab of the Data page, click the + icon.
    The New Data Collection: SOURCE page displays.  This is the first page of the data collection wizard: 
    CAP_AWS_deleg_fig2_add.png
  3. Click the Amazon Web Services button.
    The New Data Collection: Credentials screen displays.
    CAP_AWS_deleg_fig3_add.png

The instructions on the right hand side of the screen will update based on which radio buttons you select for AWS authentication and reports.

  1. To use delegated access credentials, skip to step 5.
    To use key and secret credentials, enter the Access Key ID, Secret Key, and Cost Bucket for your AWS billing account, then skip to step 10.
  2. To use delegated access credentials, select Delegated Access, then copy the value from the External ID field on this screen to your clipboard. 

You can copy the external ID manually, or use the copy button to the right of the field. 

You need the external ID, along with the AWS account ID for the HPE Consumption Analytics platform, to create the Amazon Resource Name (ARN) in your AWS account (the ARN is needed to create your collection if you configure it to collect metrics as well as cost and usage).

For the AWS account ID, please contact HPE Support using the Contact Support link in the bottom right corner of this screen.

  1. Using the external ID and the HPE Consumption Analytics platform account ID, create a delegated access role in your AWS account with AmazonS3ReadOnlyAccess for the Cost Bucket as well as any Linked Accounts  (see the instructions at  Creating a Role to Delegate Permissions to an IAM User and How to Use an External ID When Granting Access to Your AWS Resources to a Third Party). You also need to add permission for AWS Organizations (list accounts), to enable HPE Consumption Analytics platform to locate the linked accounts associated with the collection.
  2. If you want the HPE Consumption Analytics platform to collect and report on the resource utilization, application performance, and operational health data available through the Amazon CloudWatch service, apply the CloudWatchReadOnlyAccess policy to your AWS account.   

The resources HPE Consumption Analytics platform attempts to read are the ARNs that appear in the billing reports that support CloudWatch metrics.  The tool only needs access to metrics, not logs; the only request executed is GetMetricStatistics. Also, the account owner can limit the resources we are allowed to access if desired (see AWS permissions needed by the HPE Consumption Analytics Portal). 

  1. Once you've finished setting up your AWS account to work with the HPE Consumption Analytics platform, navigate to the New Data Collection: Credentials page of the wizard as shown in steps 1-3 above.
  2. Select Delegated Access, then enter the Access ARN, and Cost Bucket for your AWS billing account: 

    CAP_AWS_deleg_fig3a_add.png
  3. Select the appropriate radio button for your desired data source.
    • If you selected Detailed Billing Report, skip to step 12.
    • If you selected Cost and Usage Report, continue with step 11.
  4. Enter the Report path prefix and Report name from the Cost & Usage Report you created in AWS:
    CAP_AWS_RN_20200511-1.png

  5. Click Next.
    HPE Consumption Analytics platform verifies your credentials, then displays all of your linked accounts, including your payer account, in the New Data Collection: Accounts page of the wizard.

    Using key and secret credentials:
    CAP_AWS_deleg_fig4_add_accts_key+secret.png

    Using delegated access credentials:
    CAP_AWS_deleg_fig4_add_accts.png

  6. On the New Data Collection: Accounts page, choose one of the three options in the Action column for each linked account (the default is Collect cost, usage, and metrics).
    CAP_AWS_deleg_fig4a_add_actions.png

If there are multiple accounts to be ignored, you can simply skip entering credentials (step 12) in those rows.  When you click Next, in step 13, the following warning will display at the bottom of the screen, and you can ignore all of them in one step by clicking Ignore all accounts without valid credentials.

CAP_AWS_deleg_fig4b_add_ARN_error.png

  1. For delegated access, skip to step 15.
    For key and secret, enter the key in the Access Key ID column and the secret in the Secret Key column for all accounts (except for any to be ignored), then skip to step 16.
    CAP_AWS_deleg_fig4b_add_key+secret.png
  2. For delegated access, enter the ARN in the Access ARN column for all accounts that will collect metrics as well as cost and usage.  
    CAP_AWS_deleg_fig4b_add_ARN_ignore.png
  3. If any linked account is missing from the list and you want to receive CloudWatch metrics for it, click the Add button, enter its AWS Account ID and an Account Name, and enter the credentials for it per step 14 or 15, as appropriate.
  4. Click Next.
    The New Data Collection: Finish page displays.
    CAP_AWS_deleg_fig5_add_finish.png
  5. Enter the following information:
    • Name: Enter a unique name for the collection.
    • Comments: Optionally, enter additional information about the collection. For example, you might describe the types of data you are collecting, or details about an AWS instance.
    • Earliest Date to Collect: Select the first day's data you want to collect. HPE Consumption Analytics platform collects data from that date forward. For example, if you had your cloud in a pre-production mode for a short time while you fine-tuned the services you offer, consider setting the date to the first day your cloud was in production.
    • Source Currency: The currency in which your cloud provider reports your charges. For AWS, this is always US Dollars, so you cannot change this setting. If the Source Currency and your account's display currency do not match, the following choices appear. These affect your entire account, not just this collection. For information about them, see Currency and locale
      • Display Currency
      • Locale
  6. Click Finish
    The new collection appears in your list of collections. Any errors or warnings related to your collection appear on the Health Checks page. For more information, see Health checks.

 For a list of the specific HPE Consumption Analytics platform metrics collected from AWS and other providers, see Performance and utilization metrics.

Understanding CloudWatch rate limits

If you notice a gap in your AWS performance data in HPE Consumption Analytics platform charts and reports, you might have reached the CloudWatch API limit. Enter an AWS Support case to have your limit increased.

Amazon implements limits on customer accounts to prevent runaway usage, including a monthly limit of 1 million CloudWatch API requests. (For information about this and other CloudWatch limits, see CloudWatch, CloudWatch Events, and CloudWatch Logs Limits in the AWS documentation.) The more AWS resources you are using, the more CloudWatch API requests are needed to collect metrics data for those resources. Unfortunately, the limit does not scale with the number of resources you have in production, and it is possible to reach the limit with fewer than 300 AWS resources. If you reach the limit, HPE Consumption Analytics platform can no longer collect CloudWatch data.

You must have a Developer, Business, or Enterprise support plan with AWS to submit a technical support case. If you have the Basic plan, you cannot submit a technical support case. 

To request an AWS rate increase

  1. Navigate to AWS Support (sign in if necessary).
  2. Create your case.
    • Enter your account information and make the appropriate selections for your case (see note below).
    • In the Description field, explain that you are using a CloudWatch monitoring tool that makes many ListMetrics and GetMetricStatistics requests.
      • If you do not expect a significant increase in the number AWS resources you use, ask Amazon to double your limit.
      • If you think your resource use will continue to grow, ask for a 4x or 5x increase.

To speed a resolution with AWS Support, include your AWS region as part of your request.

  1. Submit your case.