Cloud Cruiser became HPE Consumption Analytics on Nov. 1, 2018. You'll still see the old name in places while we update this site.

 

 

Consumption Analytics Documentation

Home > HPE Consumption Analytics Portal Documentation > Configuring the HPE Consumption Analytics Portal > Creating Collections > Collecting data from Amazon Web Services (AWS)

Collecting data from Amazon Web Services (AWS)

HPE Consumption Analytics Portal retrieves Amazon Web Services (AWS) consumption data using the AWS Billing API.  This not only provides richer reporting in HPE Consumption Analytics Portal, but also enables Insights that alert you to take action based on these metrics. 

HPE Consumption Analytics Portal does not support AWS accounts in India that are operated by Amazon Internet Services Pvt. Ltd (AISPL), a local reseller in India. To determine which company your account is with, refer to the AWS documentation.

HPE Consumption Analytics Portal reads metrics for the following namespaces:

  • EC2
  • Redshift
  • RDS
  • EBS
  • ElastiCache

If you're using Amazon CloudWatch, HPE Consumption Analytics Portal can collect performance and utilization data through the CloudWatch API. Because HPE Consumption Analytics Portal collects data nightly, it collects unblended rates, which are more accurate on a day-to-day basis than blended rates.

Every night, HPE Consumption Analytics Portal collects usage, billing, and performance data from one or more of your Amazon web Service (AWS) accounts. If you have linked accounts, HPE Consumption Analytics Portal collects usage and billing data from the payer account, and CloudWatch performance data from each of your linked accounts. During this nightly collection HPE Consumption Analytics Portal collects data for the current and previous months. This ensures that if something in your AWS bill changes, you always have the most current data available in your HPE Consumption Analytics Portal analytics.

HPE Consumption Analytics Portal also collects tax information from the AWS bill, where those records appear with the description Tax of type <type>. In HPE Consumption Analytics Portal, this description appears in the Product field. The Resource ID for these records begins with the prefix Tax-. If the tax is a monthly charge, it appears in HPE Consumption Analytics Portal with the usage for the last day of the month.

For more information about how HPE Consumption Analytics Portal maps cloud providers' source data to HPE Consumption Analytics Portal normalized fields, see Field mapping.

Setting up the AWS account to work with Consumption Analytics

Before you can create an AWS collection, you must set up the AWS account to work with HPE Consumption Analytics Portal.  How you need to set it up depends on the type of credentials you want to use, and the data source from which you want the HPE Consumption Analytics Portal to collect your data.

As you set up your AWS account(s) in AWS, note the items in bold in this section - you'll need them later when creating an AWS collection in HPE Consumption Analytics Portal.

Choosing the credentials to use

The step-by-step instructions in this section explain how to set up your AWS data collection with access key and secret. 

These steps do not apply to data collections using delegated access, so if you plan to use delegated access credentials, you can skip to the next section and set up your data source.

If you create your HPE Consumption Analytics Portal AWS data collection using key and secret credentials, then all linked accounts will also use key and secret credentials. If you create the data collection using delegated access credentials, then all linked accounts within will also use delegated access credentials.

The authentication method can be changed in the future if necessary.  In this case, it will be necessary to change your AWS account setup and collection configuration in HPE Consumption Analytics Portal accordingly.

To use key and secret credentials

  1. Using Amazon's IAM service, create an AWS user with an access key ID, a secret key, and secure access to the specified S3 bucket (cost bucket) containing the data you want to collect. HPE Consumption Analytics Portal needs only the name of the cost bucket, and not the full directory path. The cost bucket name must be at least 3 characters long, and can contain only letters, numbers, periods (.), hyphens (-), and underscores (_).
  2.  Set up a role in your AWS account, using Amazon's Identity and Access Management (IAM) service.

    This role provides a mechanism to allow a third party to access your AWS resources without needing to share long-term credentials.  You need to give this role the appropriate permissions to allow HPE Consumption Analytics Portal to discover the linked accounts within the AWS account and access it to collect usage information.

    For further details on permissions requirements, including sample policies, see AWS permissions needed by the HPE Consumption Analytics Portal.

  3. If you want HPE Consumption Analytics Portal to collect and report on the resource utilization, application performance, and operational health data available through the Amazon CloudWatch service, ensure that you apply the AmazonS3ReadOnlyAccess and CloudWatchReadOnlyAccess policies to your AWS user account.  You also need to add permission for AWS Organizations (list accounts), to enable HPE Consumption Analytics Portal to locate the linked accounts associated with the collection.

    The resources HPE Consumption Analytics Portal attempts to read are the ARNs that appear in the billing reports that support CloudWatch metrics.  The tool only needs access to metrics, not logs; the only request executed is GetMetricStatistics. Also, the account owner can limit the resources we are allowed to access if desired (see AWS permissions needed by the HPE Consumption Analytics Portal).

  4. If you want HPE Consumption Analytics Portal to collect your AWS tags, ensure that those tags are included in your Cost Allocation Report (see Setting Up Your Monthly Cost Allocation Report in the AWS documentation).
  5. Create an AWS user for each of your linked accounts as per step 2.
  6. Continue with selecting your data source below.

Choosing the data source to use

When you create an AWS data collection in HPE Consumption Analytics Portal, you have the option to collect data from either the AWS Detailed Billing Reports (DBR) or the AWS Cost and Usage Report (CUR).  Either way, you must set up your data source in AWS before you can create your data collection in the HPE Consumption Analytics Portal.

The Detailed Billing Reports feature in AWS is deprecated in favor of the Cost and Usage Report.  It is unavailable for new AWS customers as of July 2019.

To collect data from an AWS Cost and Usage Report

Create a Cost and Usage report in AWS for the account for which you want to collect data, by following the instructions in Creating Cost and Usage Reports in the AWS documentation.

To enable your HPE Consumption Analytics Portal collection:

  • Specify the desired Report name
  • Select Include resource IDs
  • Select (or configure) the desired S3 Bucket.
  • Specify the Report path prefix
  • Set the time granularity to hourly.

 

To collect data from AWS Detailed Billing Reports

Turn on the legacy Detailed Billing Reports in AWS for the account for which you want to collect data (see the instructions in Understand Your Usage with Detailed Billing Reports in the AWS documentation).

If you are using consolidated billing, perform this step for the payer account, not the linked accounts. For more information about consolidated billing, see Pay Bills for Multiple Accounts with Consolidated Billing in the AWS documentation.

To enable your HPE Consumption Analytics Portal collection:

  • Select the Monthly report and the Detailed billing report with services and tags

 

Once you've set up your AWS account for the credentials and data source you want theHPE Consumption Analytics Portal to use, continue with the steps in the next section to create your AWS collection.

If you intend to create a collection with delegated access credentials, keep your AWS session open, because there are a few more steps to do in AWS as you create the collection in HPE Consumption Analytics Portal.

Creating an AWS collection in the HPE Consumption Analytics Portal

When you create an AWS data collection in HPE Consumption Analytics Portal, you have the option to create it (and all linked accounts in the collection) with Access Key ID and Secret Key authentication, or with Delegated Access authentication.  The step-by-step instructions that follow show you how to create an AWS data collection in the HPE Consumption Analytics Portal with either access key and secret or delegated access.

CAP_AWS_deleg_fig1.png

To create an AWS collection

If you enter invalid credentials, you will see the following message:

CAP_AWS_deleg_fig3c_add_error_msg.png

In this case, correct the error and try again.

  1. Click Data in the left navigation panel.
  2. On the Collections tab of the Data screen, click the + icon.
    The New Data Collection screen displays.
    CAP_AWS_deleg_fig2_add.png

  3. Click the Amazon Web Services button.
    The New Data Collection: Credentials screen displays.
    CAP_AWS_deleg_fig3_add.png

    The instructions on the right hand side of the screen will update based on which radio buttons you select for AWS authentication and reports.

  4. To use delegated access credentials, skip to step 5.
    To use key and secret credentials, enter the Access Key ID, Secret Key, and Cost Bucket for your AWS billing account, then skip to step 10.
  5. To use delegated access credentials, select Delegated Access, then copy the value from the External ID field on this screen to your clipboard.

    You can copy the external ID manually, or use the copy button to the right of the field. 

    You need the external ID, along with the AWS account ID for the HPE Consumption Analytics Portal, to create the Amazon Resource Name (ARN) in your AWS account (in turn, the ARN is needed to create your collection.

    For the AWS account ID, please contact HPE Support using the Contact Support link in the bottom right corner of this screen.

  6. Using the external ID and the HPE Consumption Analytics Portal account ID, create a delegated access role in your AWS account with AmazonS3ReadOnlyAccess for the Cost Bucket as well as any Linked Accounts  (see the instructions at  Creating a Role to Delegate Permissions to an IAM User and How to Use an External ID When Granting Access to Your AWS Resources to a Third Party). You also need to add permission for AWS Organizations (list accounts), to enable HPE Consumption Analytics Portal to locate the linked accounts associated with the collection.
  7. If you want the HPE Consumption Analytics Portal to collect and report on the resource utilization, application performance, and operational health data available through the Amazon CloudWatch service, apply the CloudWatchReadOnlyAccess policy to your AWS account.  

    The resources HPE Consumption Analytics Portal attempts to read are the ARNs that appear in the billing reports that support CloudWatch metrics.  The tool only needs access to metrics, not logs; the only request executed is GetMetricStatistics. Also, the account owner can limit the resources we are allowed to access if desired (see AWS permissions needed by the HPE Consumption Analytics Portal).

  8. Once you've finished setting up your AWS account to work with the HPE Consumption Analytics Portal, navigate to the New Data Collection: Credentials screen as shown in steps 1-3 above.
  9. Select Delegated Access, then enter the Access ARN, and Cost Bucket for your AWS billing account:
    CAP_AWS_deleg_fig3a_add.png

  10. Select the appropriate radio button for your desired data source.
    • If you selected Detailed Billing Report, skip to step 12.
    • If you selected Cost and Usage Report, continue with step 11.
  11. Enter the Report path prefix and Report name from the Cost & Usage Report you created in AWS:
    CAP_AWS_RN_20200511.png

  12. Click Next.
    HPE Consumption Analytics Portal verifies your credentials, then displays all of your linked accounts, including your payer account, in the Manage Amazon Web Services Linked Accounts page.

    Using key and secret credentials:
    CAP_AWS_deleg_fig4_add_accts_key+secret.png

    Using delegated access credentials:
    CAP_AWS_deleg_fig4_add_accts.png

  13. On the Manage Amazon Web Services Linked Accounts  page, choose one of the three options in the Action column for each linked account (the default is Collect cost, usage, and metrics). 
    CAP_AWS_deleg_fig4a_add_actions.png

    If there are multiple accounts to be ignored, you can simply skip entering credentials (step 12) in those rows.  When you click Next, in step 13, the following warning will display at the bottom of the screen, and you can ignore all of them in one step by clicking Ignore all accounts without valid credentials.

    CAP_AWS_deleg_fig4b_add_ARN_error.png

  14. For delegated access, skip to step 15.
    For key and secret, enter the key in the Access Key ID column and the secret in the Secret Key column for all accounts (except for any to be ignored), then skip to step 16.
    CAP_AWS_deleg_fig4b_add_key+secret.png

  15. For delegated access, enter the ARN in the Access ARN column for all accounts (except for any to be ignored).  
    CAP_AWS_deleg_fig4b_add_ARN_ignore.png

  16. If any linked account is missing from the list and you want to receive CloudWatch metrics for it, click the Add button, enter its AWS Account ID and an Account Name, and enter the credentials for it per step 14 or 15, as appropriate.
  17. Click Next.
    The Data Collection Options page displays.
    CAP_AWS_deleg_fig5_add_finish.png

  18. Enter the following information:
    • Name: Enter a unique name for the collection.
    • Comments: Optionally, enter additional information about the collection. For example, you might describe the types of data you are collecting, or details about an AWS instance.
    • Earliest Date to Collect: Select the first day's data you want to collect. HPE Consumption Analytics Portal collects data from that date forward. For example, if you had your cloud in a pre-production mode for a short time while you fine-tuned the services you offer, consider setting the date to the first day your cloud was in production.
    • Source Currency: The currency in which your cloud provider reports your charges. For AWS, this is always US Dollars, so you cannot change this setting. If the Source Currency and your account's display currency do not match, the following choices appear. These affect your entire account, not just this collection. For information about them, see Currency and locale
      • Display Currency
      • Locale
  19. Click Finish
    The new collection appears in your list of collections. Any errors or warnings related to your collection appear on the Health Checks page. For more information, see Health checks.

 

For a list of the specific HPE Consumption Analytics Portal metrics collected from AWS and other providers, see Performance and utilization metrics.

Understanding CloudWatch rate limits

If you notice a gap in your AWS performance data in HPE Consumption Analytics Portal charts and reports, you might have reached the CloudWatch API limit. Enter an AWS Support case to have your limit increased.

Amazon implements limits on customer accounts to prevent runaway usage, including a monthly limit of 1 million CloudWatch API requests. (For information about this and other CloudWatch limits, see CloudWatch, CloudWatch Events, and CloudWatch Logs Limits in the AWS documentation.) The more AWS resources you are using, the more CloudWatch API requests are needed to collect metrics data for those resources. Unfortunately, the limit does not scale with the number of resources you have in production, and it is possible to reach the limit with fewer than 300 AWS resources. If you reach the limit, HPE Consumption Analytics Portal can no longer collect CloudWatch data.

You must have a Developer, Business, or Enterprise support plan with AWS to submit a technical support case. If you have the Basic plan, you cannot submit a technical support case.

To request an AWS rate increase

  1. Navigate to AWS Support (sign in if necessary).
  2. Create your case.
    • Enter your account information and make the appropriate selections for your case (see note below).
    • In the Description field, explain that you are using a CloudWatch monitoring tool that makes many ListMetrics and GetMetricStatistics requests.
      • If you do not expect a significant increase in the number AWS resources you use, ask Amazon to double your limit.
      • If you think your resource use will continue to grow, ask for a 4x or 5x increase.

    To speed a resolution with AWS Support, include your AWS region as part of your request.

  3. Submit your case.
Last modified

Tags

Classifications

This page has no classifications.

 (c) Copyright 2017-2020 Hewlett Packard Enterprise Development LP