Uploading usage and billing data
If the collections available in HPE Consumption Analytics Portal do not suit your needs, you can upload your cloud usage and billing data as JSON files through the HPE Consumption Analytics Portal API. To do so, complete the following general steps:
- Format your JSON files as described in Formatting JSON files for upload.
- Create an Upload Usage API collection in HPE Consumption Analytics Portal to generate an authorization token. For more information, see Creating an Upload Usage API collection.
- Write a script or program that uses the HPE Consumption Analytics Portal API to upload your JSON files via HTTPS. For more information, see Uploading data programmatically.
Formatting JSON files for upload
You need a JSON file for each accounting date of the data you want to upload. If you later upload a file for the same accounting date, the previous data is overwritten.
You can build JSON files from whatever source you like. However, each JSON file you want to upload must include the following fields as headers in the first row of your file.
Required field | Data type |
---|---|
Usage Start |
Date/time, formatted using one of the following formats:
|
Usage End | Date/time, formatted using the same format as the Usage Start field. |
Account ID | String |
Account Name | String |
Provider | String |
Provider Type | String, with a value of either Public or Private |
Product | String |
Meter ID | String |
Meter Name | String |
Quantity | Decimal |
Units | String |
You can add other fields after these fields. For example, HPE recommends that you add Service Category and Resource ID.
Optional field | Data type |
---|---|
Account Owner ID | STRING |
Account Owner | STRING |
Provider Region | STRING |
Provider Zone | STRING |
Service Category | STRING |
Resource ID | STRING |
Resource Name | STRING |
Resource Type | STRING |
Rate | FLOAT |
Cost | FLOAT |
VM Provider Type | STRING |
VM Provider Series | STRING |
Operating System | STRING |
CPU Utilization | FLOAT |
Network In MB | FLOAT |
Network Out MB | FLOAT |
Storage Read Bytes | INT |
Storage Write Bytes | INT |
Read Throughput | INT |
Write Throughput | INT |
VM Family | STRING |
VM Size | STRING |
VM CPU Count | INT |
VM GPU Count | INT |
VM Memory GB | FLOAT |
VM Disk GB | FLOAT |
VM Disk Type | STRING |
VM Max Num Disk | INT |
You can also include optional fields called extraTypedFields. For example, this could be project names, departments, etc. extraTypedFields includes the value type for each each field: FLOAT, INT or STRING.
For example:
{ "systemFields":{ "Usage Start":"2017-03-01 00:00", "Usage End":"2017-03-01 23:59", "Account ID":"{B902801E-03F2-4793-B016-263A989BC2E6}", "Account Name":"BULKSALES-AZURE-PROD", "Provider":"Azure", "Provider Type":"Public", "Product":"Networking: All Data Transfer Out (GB) - Zone 1", "Meter Id":"3c5324ad-eb8c-44c6-af9a-6741ae75fc90", "Meter Name":"Data Transfer Out at 500 Mbps (GB)", "Quantity":0.000874, "Units":"GB", "Cost":0.000026, "Service Category":"Database", "Resource ID":"rpqfosupvn - ProvisionRepository" }, "extraTypedFields":{ "a":{ "type":"FLOAT", "value":1.23 }, "b":{ "type":"INT", "value":2 }, "c":{ "type":"STRING", "value":"abc" } } }
Creating an Upload Usage API collection
Create an Upload Usage API collection in HPE Consumption Analytics Portal to generate an authorization token you can use in API calls to upload your JSON files.
To create an Upload Usage API collection
- On the Data > Collections screen, click the Add icon
.
The New Data Collection screen appears. - Click Upload Usage API.
- On the Options page, enter the following information, and then click Next:
- Name: Enter a unique name for the collection.
- Comments: Optionally, enter additional information about the collection. For example, you might describe the source of your data.
- Note the Authorization Token HPE Consumption Analytics Portal generates. You will need to include this in the header of your API request, as described in Uploading data programmatically.
- Click Finish.
- If you need to generate a new token, open your collection for editing and then click Generate New Token.
Uploading data programmatically
You can access the API by using HTTPS. Each request needs to provide credentials information in an Authorization header that includes Bearer followed by the authorization token generated by HPE Consumption Analytics Portal, as described in Creating an Upload Usage API collection.
Use the following API reference information to guide your script or program.
Request
Resource URL
PUT https://na1.cloudcruiser.com/api/upload/v1/schema/usage/daily/<date>
Where the <date>
(formatted as YYYY-MM-DD
) is the accounting date of the data in the file.
By default, data uploaded to HPE Consumption Analytics Portal will be in Coordinated Universal Time (UTC). If you want to use a different time zone for your data you can append the ?zoneOffset=<n>
parameter to your PUT
request URL, where <n>
is the amount of offset you want used for your data. (For a list of offest hours for countries, territories, and regions, see https://en.wikipedia.org/wiki/List_of_UTC_time_offsets.) You can use any of the following formats for your offset:
+h
-h
+hh
-hh
+hh:mm
-hh:mm
+hhmm
-hhmm
+hh:mm:ss
-hh:mm:ss
+hhmmss
-hhmmss
For example, if you want your uploaded data to be in Pacific Daylight Time, which has a UTC offset of -7, your PUT
request URL would have the following format:
PUT https://na1.cloudcruiser.com/api/upload/v1/schema/usage/daily/<date>/?=zoneOffset=-7
If you want to provide a more granular time for your data you could specify the minutes and seconds, as in the following example:
PUT https://na1.cloudcruiser.com/api/upload/v1/schema/usage/daily/<date>/?=zoneOffset=-07:01:30
Header
An Authorization header with Bearer
, followed by a space, followed by the authorization token generated by your HPE Consumption Analytics Portal collection.
Content-Type
multipart/form-data, with name="data"
specified for the file portion of the request
The name
parameter is merely an identifier for the file part of the request. The actual JSON file being uploaded can have any name.
Body
A JSON file to upload, as described in Formatting JSON files for upload.
Response
HPE Consumption Analytics Portal returns status and error messages in API responses to show whether your upload succeeded or failed. For example, if the upload is successful the HPE Consumption Analytics Portal API server returns a JSON response similar to the following:
{ "success": true, "message": "Completed upload of 34567 record(s)", "status": 200 }
If authentication fails due to a missing or invalid API token, the response indicates the failure. For example:
{ "success": false, "status": 401, "message": "Usage upload authentication failed: The specified token did not match the expected value" }
Example
The Java program below uploads a JSON file to HPE Consumption Analytics Portal. The Java file is also attached to this topic.
Call
Use the following call to run the program, where:
sToken
is the authorization token generated by your HPE Consumption Analytics Portal collectionsJsonFile
is the full path to the JSON file you want to uploadsDate
is the accounting date of the data being uploaded
PushApiFunctions pushApiTest = new PushApiFunctions(); pushApiTest.pushDataWithPushApi(sToken, sJsonFile, sDate);
Script
package library; import java.io.BufferedInputStream; import java.io.BufferedReader; import java.io.DataOutputStream; import java.io.File; import java.io.FileInputStream; import java.io.InputStream; import java.io.InputStreamReader; import java.net.URL; import javax.net.ssl.HttpsURLConnection; public class PushApiFunctions { public static void main (String[] args) { String sToken = "ayJhbGciOiJIUzUxMiJ9.eyJ0ZW5hbnRJZCI6ImIzMGM3OGI1LWU3NTMtNGY5ZC04YmUxLWVkNmFlYjE2ZDA3OCIsImRhdGFzb3VyY2VJZCI6IjY2M2QwNTg0LTI2ZTctNDUxNy1iZWI4LWE1YmQzj60kw.jre1IsImlzcyI6ImNsb3VkY3J1aXNlci5jb20iLCJpYXQiOjE0NTg3NTU5MzJ9.9coa79APysbKiPwYSJ2j2Tb68sknrVLZxly6EGB9XDw1bFYzKZ915E8Ws2NjMgd3IUTEvFiBtcUA987hggOjrw"; String sJsonFile = "C:\\misc\\PushAPI\\test.json"; String sDate = "2016-03-01"; PushApiFunctions pushApiTest = new PushApiFunctions(); pushApiTest.pushDataWithPushApi(sToken, sJsonFile,sDate); } public String pushDataWithPushApi(String sAuthToken, String sFile,String sDate) { String crlf = "\r\n"; String twoHyphens = "--"; String boundary = "*****"; File fileToUpload = new File(sFile); String sUrl = "https://na1.cloudcruiser.com/api/upload/v1/schema/usage/daily/" + sDate; try { HttpsURLConnection httpUrlConnection = null; URL url = new URL(sUrl); httpUrlConnection = (HttpsURLConnection) url.openConnection(); httpUrlConnection.setUseCaches(false); httpUrlConnection.setDoOutput(true); httpUrlConnection.setRequestMethod("PUT"); httpUrlConnection.setRequestProperty("Connection", "Keep-Alive"); httpUrlConnection.setRequestProperty("Cache-Control", "no-cache"); httpUrlConnection.setRequestProperty("Authorization", "Bearer " + sAuthToken); httpUrlConnection.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary); DataOutputStream request = new DataOutputStream(httpUrlConnection.getOutputStream()); request.writeBytes(twoHyphens + boundary + crlf); request.writeBytes("Content-Disposition: form-data; name=\"data\"; filename=\""+ fileToUpload.getName() +"\" \nContent-Type: application/octet-stream\n\n"); request.writeBytes(crlf); // Write the actual file contents FileInputStream inputStreamToOut = new FileInputStream(fileToUpload); int bytesRead; byte[] dataBuffer = new byte[1024]; while((bytesRead = inputStreamToOut.read(dataBuffer)) != -1) { request.write(dataBuffer, 0, bytesRead); // System.out.println(dataBuffer); } request.writeBytes(crlf); request.writeBytes(twoHyphens + boundary + twoHyphens + crlf); request.flush(); request.close(); inputStreamToOut.close(); InputStream responseStream = new BufferedInputStream(httpUrlConnection.getInputStream()); BufferedReader responseStreamReader = new BufferedReader(new InputStreamReader(responseStream)); String line = ""; StringBuilder stringBuilder = new StringBuilder(); while ((line = responseStreamReader.readLine()) != null) { stringBuilder.append(line).append("\n"); } responseStreamReader.close(); String response = stringBuilder.toString(); System.out.println("response = " + response); responseStream.close(); httpUrlConnection.disconnect(); return response; } catch (Exception e) { e.printStackTrace(); return e.toString(); } } }