Collecting data from Amazon Web Services (AWS)

Setting up the AWS account to work with Consumption Analytics

Before you can create an AWS collection, you must set up the AWS account to work with HPE Consumption Analytics Portal.  How you need to set it up depends on the type of credentials you want to use, and the data source from which you want the HPE Consumption Analytics Portal to collect your data.

Creating an AWS collection in the HPE Consumption Analytics Portal

When you create an AWS data collection in HPE Consumption Analytics Portal, you have the option to create it (and all linked accounts in the collection) with Access Key ID and Secret Key authentication, or with Delegated Access authentication.  The step-by-step instructions that follow show you how to create an AWS data collection in the HPE Consumption Analytics Portal with either access key and secret or delegated access.

To create an AWS collection

1. Click Data in the left navigation panel.
2. On the Collections tab of the Data screen, click the + icon.
   The New Data Collection screen displays.
   


3. Click the Amazon Web Services button.
   The New Data Collection: Credentials screen displays.
   

   The instructions on the right hand side of the screen will update based on which radio buttons you select for AWS authentication and reports.

4. To use delegated access credentials, skip to step 5.
   To use key and secret credentials, enter the Access Key ID, Secret Key, and Cost Bucket for your AWS billing account, then skip to step 10.
5. To use delegated access credentials, select Delegated Access, then copy the value from the External ID field on this screen to your clipboard.

   You can copy the external ID manually, or use the copy button to the right of the field. 

   You need the external ID, along with the AWS account ID for the HPE Consumption Analytics Portal, to create the Amazon Resource Name (ARN) in your AWS account (in turn, the ARN is needed to create your collection.

   For the AWS account ID, please contact HPE Support using the Contact Support link in the bottom right corner of this screen.

6. Using the external ID and the HPE Consumption Analytics Portal account ID, create a delegated access role in your AWS account with AmazonS3ReadOnlyAccess for the Cost Bucket as well as any Linked Accounts  (see the instructions at  Creating a Role to Delegate Permissions to an IAM User and How to Use an External ID When Granting Access to Your AWS Resources to a Third Party). You also need to add permission for AWS Organizations (list accounts), to enable HPE Consumption Analytics Portal to locate the linked accounts associated with the collection.
7. If you want the HPE Consumption Analytics Portal to collect and report on the resource utilization, application performance, and operational health data available through the Amazon CloudWatch service, apply the CloudWatchReadOnlyAccess policy to your AWS account.  

   The resources HPE Consumption Analytics Portal attempts to read are the ARNs that appear in the billing reports that support CloudWatch metrics.  The tool only needs access to metrics, not logs; the only request executed is GetMetricStatistics. Also, the account owner can limit the resources we are allowed to access if desired (see AWS permissions needed by the HPE Consumption Analytics Portal).

8. Once you've finished setting up your AWS account to work with the HPE Consumption Analytics Portal, navigate to the New Data Collection: Credentials screen as shown in steps 1-3 above.
9. Select Delegated Access, then enter the Access ARN, and Cost Bucket for your AWS billing account:
   


10. Select the appropriate radio button for your desired data source.
    • If you selected Detailed Billing Report, skip to step 12.
    • If you selected Cost and Usage Report, continue with step 11.
11. Enter the Report path prefix and Report name from the Cost & Usage Report you created in AWS:
    


12. Click Next.
    HPE Consumption Analytics Portal verifies your credentials, then displays all of your linked accounts, including your payer account, in the Manage Amazon Web Services Linked Accounts page.
    
    Using key and secret credentials:
    
    
    Using delegated access credentials:
    


13. On the Manage Amazon Web Services Linked Accounts  page, choose one of the three options in the Action column for each linked account (the default is Collect cost, usage, and metrics). 
    

    If there are multiple accounts to be ignored, you can simply skip entering credentials (step 12) in those rows.  When you click Next, in step 13, the following warning will display at the bottom of the screen, and you can ignore all of them in one step by clicking Ignore all accounts without valid credentials.

    

14. For delegated access, skip to step 15.
    For key and secret, enter the key in the Access Key ID column and the secret in the Secret Key column for all accounts (except for any to be ignored), then skip to step 16.
    


15. For delegated access, enter the ARN in the Access ARN column for all accounts (except for any to be ignored).  
    


16. If any linked account is missing from the list and you want to receive CloudWatch metrics for it, click the Add button, enter its AWS Account ID and an Account Name, and enter the credentials for it per step 14 or 15, as appropriate.
17. Click Next.
    The Data Collection Options page displays.
    


18. Enter the following information:
    • Name: Enter a unique name for the collection.
    • Comments: Optionally, enter additional information about the collection. For example, you might describe the types of data you are collecting, or details about an AWS instance.
    • Earliest Date to Collect: Select the first day's data you want to collect. HPE Consumption Analytics Portal collects data from that date forward. For example, if you had your cloud in a pre-production mode for a short time while you fine-tuned the services you offer, consider setting the date to the first day your cloud was in production.
    • Source Currency: The currency in which your cloud provider reports your charges. For AWS, this is always US Dollars, so you cannot change this setting. If the Source Currency and your account's display currency do not match, the following choices appear. These affect your entire account, not just this collection. For information about them, see Currency and locale
      • Display Currency
      • Locale
19. Click Finish
    The new collection appears in your list of collections. Any errors or warnings related to your collection appear on the Health Checks page. For more information, see Health checks.

Understanding CloudWatch rate limits

If you notice a gap in your AWS performance data in HPE Consumption Analytics Portal charts and reports, you might have reached the CloudWatch API limit. Enter an AWS Support case to have your limit increased.

Amazon implements limits on customer accounts to prevent runaway usage, including a monthly limit of 1 million CloudWatch API requests. (For information about this and other CloudWatch limits, see CloudWatch, CloudWatch Events, and CloudWatch Logs Limits in the AWS documentation.) The more AWS resources you are using, the more CloudWatch API requests are needed to collect metrics data for those resources. Unfortunately, the limit does not scale with the number of resources you have in production, and it is possible to reach the limit with fewer than 300 AWS resources. If you reach the limit, HPE Consumption Analytics Portal can no longer collect CloudWatch data.

To request an AWS rate increase

1. Navigate to AWS Support (sign in if necessary).
2. Create your case.
   • Enter your account information and make the appropriate selections for your case (see note below).
   • In the Description field, explain that you are using a CloudWatch monitoring tool that makes many ListMetrics and GetMetricStatistics requests.
     • If you do not expect a significant increase in the number AWS resources you use, ask Amazon to double your limit.
     • If you think your resource use will continue to grow, ask for a 4x or 5x increase.

   To speed a resolution with AWS Support, include your AWS region as part of your request.

3. Submit your case.