Cloud Cruiser became HPE Consumption Analytics on Nov. 1st. You'll still see the old name in places while we update this site.

 

 

Consumption Analytics Documentation

Home > HPE Consumption Analytics Portal > Configuring the HPE Consumption Analytics Portal > Collections > Uploading usage and billing data

Uploading usage and billing data

If the collections available in HPE Consumption Analytics Portal do not suit your needs, you can upload your cloud usage and billing data as JSON files through the HPE Consumption Analytics Portal API. To do so, complete the following general steps:

  1. Format your JSON files as described in Formatting JSON files for upload.
  2. Create an Upload Usage API collection in HPE Consumption Analytics Portal to generate an authorization token. For more information, see Creating an Upload Usage API collection.
  3. Write a script or program that uses the HPE Consumption Analytics Portal API to upload your JSON files via HTTPS. For more information, see Uploading data programmatically.

Formatting JSON files for upload

You need a JSON file for each accounting date of the data you want to upload. If you later upload a file for the same accounting date, the previous data is overwritten.

You can build JSON files from whatever source you like. However, each JSON file you want to upload must include the following fields as headers in the first row of your file.

Required field Data type
Usage Start

Date/time, formatted using one of the following formats:

  • yyyy-MM-dd HH:mm
  • yyyy-MM-dd HH:mm:ss
  • yyyy-MM-ddTHH:mm:ss
  • yyyy-MM-ddTHH:mm:ssZ, where Z can be replaced with a UTC time zone offset, such as +03.
Usage End Date/time, formatted using the same format as the Usage Start field.
Account ID String
Account Name String
Provider String
Provider Type String, with a value of either Public or Private
Product String
Meter ID String
Meter Name String
Quantity Decimal
Units String

You can add other fields after these fields. For example, HPE recommends that you add Service Category and Resource ID.

Optional field Data type
Account Owner ID STRING
Account Owner STRING
Provider Region STRING
Provider Zone STRING
Service Category STRING
Resource ID STRING
Resource Name STRING
Resource Type STRING
Rate FLOAT
Cost FLOAT
VM Provider Type STRING
VM Provider Series STRING
Operating System STRING
CPU Utilization FLOAT
Network In MB FLOAT
Network Out MB FLOAT
Storage Read Bytes INT
Storage Write Bytes INT
Read Throughput INT
Write Throughput INT
VM Family STRING
VM Size STRING
VM CPU Count INT
VM GPU Count INT
VM Memory GB FLOAT
VM Disk GB FLOAT
VM Disk Type STRING
VM Max Num Disk INT

You can also include optional fields called extraTypedFields. For example, this could be project names, departments, etc. extraTypedFields includes the value type for each each field: FLOAT, INT or STRING.

For example:

{
   "systemFields":{
      "Usage Start":"2017-03-01 00:00",
      "Usage End":"2017-03-01 23:59",
      "Account ID":"{B902801E-03F2-4793-B016-263A989BC2E6}",
      "Account Name":"BULKSALES-AZURE-PROD",
      "Provider":"Azure",
      "Provider Type":"Public",
      "Product":"Networking: All Data Transfer Out (GB) - Zone 1",
      "Meter Id":"3c5324ad-eb8c-44c6-af9a-6741ae75fc90",
      "Meter Name":"Data Transfer Out at 500 Mbps (GB)",
      "Quantity":0.000874,
      "Units":"GB",
      "Cost":0.000026,
      "Service Category":"Database",
      "Resource ID":"rpqfosupvn - ProvisionRepository"
   },
   "extraTypedFields":{
      "a":{
         "type":"FLOAT",
         "value":1.23
      },
      "b":{
         "type":"INT",
         "value":2
      },
      "c":{
         "type":"STRING",
         "value":"abc"
      }
   }
}

Creating an Upload Usage API collection

Create an Upload Usage API collection in HPE Consumption Analytics Portal to generate an authorization token you can use in API calls to upload your JSON files.

To create an Upload Usage API collection

  1. On the Data > Collections screen, click the Add icon .
    The New Data Collection screen appears.
  2. Click Upload Usage API.
  3. On the Options page, enter the following information, and then click Next:
    • Name: Enter a unique name for the collection.
    • Comments: Optionally, enter additional information about the collection. For example, you might describe the source of your data.
  4. Note the Authorization Token HPE Consumption Analytics Portal generates. You will need to include this in the header of your API request, as described in Uploading data programmatically.
  5. Click Finish.
  6. If you need to generate a new token, open your collection for editing and then click Generate New Token.

Uploading data programmatically

You can access the API by using HTTPS. Each request needs to provide credentials information in an Authorization header that includes Bearer followed by the authorization token generated by HPE Consumption Analytics Portal, as described in Creating an Upload Usage API collection.

Use the following API reference information to guide your script or program.

Request

Resource URL

PUT https://na1.cloudcruiser.com/api/upload/v1/schema/usage/daily/<date>

Where the <date> (formatted as YYYY-MM-DD) is the accounting date of the data in the file.

By default, data uploaded to HPE Consumption Analytics Portal will be in Coordinated Universal Time (UTC). If you want to use a different time zone for your data you can append the ?zoneOffset=<n> parameter to your PUT request URL, where <n> is the amount of offset you want used for your data. (For a list of offest hours for countries, territories, and regions, see https://en.wikipedia.org/wiki/List_of_UTC_time_offsets.) You can use any of the following formats for your offset:

+h
-h
+hh
-hh
+hh:mm
-hh:mm
+hhmm
-hhmm
+hh:mm:ss
-hh:mm:ss
+hhmmss
-hhmmss

For example, if you want your uploaded data to be in Pacific Daylight Time, which has a UTC offset of -7, your PUT request URL would have the following format:

PUT https://na1.cloudcruiser.com/api/upload/v1/schema/usage/daily/<date>/?=zoneOffset=-7

If you want to provide a more granular time for your data you could specify the minutes and seconds, as in the following example:

PUT https://na1.cloudcruiser.com/api/upload/v1/schema/usage/daily/<date>/?=zoneOffset=-07:01:30

Header

An Authorization header with Bearer, followed by a space, followed by the authorization token generated by your HPE Consumption Analytics Portal collection.

Content-Type

multipart/form-data, with name="data" specified for the file portion of the request

The name parameter is merely an identifier for the file part of the request. The actual JSON file being uploaded can have any name.

Body

A JSON file to upload, as described in Formatting JSON files for upload.

Response

HPE Consumption Analytics Portal returns status and error messages in API responses to show whether your upload succeeded or failed. For example, if the upload is successful the HPE Consumption Analytics Portal API server returns a JSON response similar to the following:

{
  "success": true,
  "message": "Completed upload of 34567 record(s)",
  "status": 200
}

If authentication fails due to a missing or invalid API token, the response indicates the failure. For example:

{
  "success": false,
  "status": 401,
  "message": "Usage upload authentication failed: The specified token did not match the expected value"
}

Example

The Java program below uploads a JSON file to HPE Consumption Analytics Portal. The Java file is also attached to this topic.

Call

Use the following call to run the program, where:

  • sToken is the authorization token generated by your HPE Consumption Analytics Portal collection
  • sJsonFile is the full path to the JSON file you want to upload
  • sDate is the accounting date of the data being uploaded
PushApiFunctions pushApiTest = new PushApiFunctions();
pushApiTest.pushDataWithPushApi(sToken, sJsonFile, sDate);
Script
package library;
import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
public class PushApiFunctions 
{    
    public static void main (String[] args)
    {
        String sToken = "ayJhbGciOiJIUzUxMiJ9.eyJ0ZW5hbnRJZCI6ImIzMGM3OGI1LWU3NTMtNGY5ZC04YmUxLWVkNmFlYjE2ZDA3OCIsImRhdGFzb3VyY2VJZCI6IjY2M2QwNTg0LTI2ZTctNDUxNy1iZWI4LWE1YmQzj60kw.jre1IsImlzcyI6ImNsb3VkY3J1aXNlci5jb20iLCJpYXQiOjE0NTg3NTU5MzJ9.9coa79APysbKiPwYSJ2j2Tb68sknrVLZxly6EGB9XDw1bFYzKZ915E8Ws2NjMgd3IUTEvFiBtcUA987hggOjrw";
        String sJsonFile = "C:\\misc\\PushAPI\\test.json";
        String sDate  = "2016-03-01";
        PushApiFunctions pushApiTest = new PushApiFunctions();
        pushApiTest.pushDataWithPushApi(sToken, sJsonFile,sDate);
    }
    
    public  String  pushDataWithPushApi(String sAuthToken, String sFile,String sDate) 
    {
        String crlf = "\r\n";
        String twoHyphens = "--";
        String boundary =  "*****";
        File fileToUpload = new File(sFile);
        String sUrl = "https://na1.cloudcruiser.com/api/upload/v1/schema/usage/daily/" + sDate;
    
    
        try
        {
            HttpsURLConnection httpUrlConnection = null;
            URL url = new URL(sUrl);
            httpUrlConnection = (HttpsURLConnection) url.openConnection();
            httpUrlConnection.setUseCaches(false);
            httpUrlConnection.setDoOutput(true);
            httpUrlConnection.setRequestMethod("PUT");
            httpUrlConnection.setRequestProperty("Connection", "Keep-Alive");
            httpUrlConnection.setRequestProperty("Cache-Control", "no-cache");
            httpUrlConnection.setRequestProperty("Authorization", "Bearer " + sAuthToken);
            httpUrlConnection.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);
        
            DataOutputStream request = new DataOutputStream(httpUrlConnection.getOutputStream());
            request.writeBytes(twoHyphens + boundary + crlf);    
            request.writeBytes("Content-Disposition: form-data; name=\"data\"; filename=\""+ fileToUpload.getName() +"\" \nContent-Type: application/octet-stream\n\n");
            request.writeBytes(crlf);
        
            // Write the actual file contents
            FileInputStream inputStreamToOut = new FileInputStream(fileToUpload);
            int bytesRead;
            byte[] dataBuffer = new byte[1024];
            while((bytesRead = inputStreamToOut.read(dataBuffer)) != -1) {
                request.write(dataBuffer, 0, bytesRead);
               // System.out.println(dataBuffer);
            }    
            request.writeBytes(crlf);
            request.writeBytes(twoHyphens + boundary + twoHyphens + crlf);
           
            request.flush();
            request.close();
            inputStreamToOut.close();
        
            InputStream responseStream = new BufferedInputStream(httpUrlConnection.getInputStream());
            BufferedReader responseStreamReader = 
                new BufferedReader(new InputStreamReader(responseStream));
        
            String line = "";
            StringBuilder stringBuilder = new StringBuilder();
        
            while ((line = responseStreamReader.readLine()) != null) {
                stringBuilder.append(line).append("\n");
            }
            responseStreamReader.close();
        
            String response = stringBuilder.toString();
             System.out.println("response = " + response);
        
            responseStream.close();
            httpUrlConnection.disconnect();
            return response;
        }
        catch (Exception e) 
          {
            e.printStackTrace();
            return e.toString();
          }   
    }
}
Last modified

Tags

Classifications

This page has no classifications.
© Copyright 2018 Hewlett Packard Enterprise Development LP