Consumption Analytics Documentation

Home > Documentation for older versions > Cloud Cruiser 4 for HPE GreenLake Flex Capacity > Cloud Cruiser architecture and workflow > Customer-specific processing

Customer-specific processing

This section describes the processing flow for customers, starting with the per-customer data files produced by the Triage step discussed in Data collection and general processing, followed by processing that is specific to the customer’s use case and billing model. The outputs of the customer-specific processing are mapped to schemas. Schemas are in mapped to services, which are assigned to a Rate Plan. Lastly, monthly costs are calculated, making the data available for reporting.

Understanding workbooks, worksheets, and collections

In the FC framework Cloud Cruiser has established the following paradigm:

  • Processing for a given customer is handled by a dedicated workbook.
  • A workbook contains worksheets, with one worksheet per device category (specifically, <category> from the attachment filename)
  • Data in each sheet is populated by a collection.

Workbooks

A dedicated workbook is required for each named customer account. The following image shows some examples:

workbookList.jpg

Note the leading customer ID number in each workbook name (00002, 00003, and so on). This convention helps maintain consistency throughout the application. Always include the customer ID as the leading characters of a workbook name.

The following image shows the 00021-AceRun workbook:

aceRunWorkbook.jpg

Create a new workbook as part of the process of onboarding a new FC customer. See Onboarding FC customers for more information.

Worksheets

A worksheet is a standalone flow of processing steps that operate on collected data. Processing steps in a worksheet usually culminate in either a Publish step that loads data either into the database or to a dataset (a data file that can be consumed by other worksheets).

Worksheets are shown as tabs at the bottom of the workbook. For example, the 00021-AceRun workbook contains two worksheets: _3PAR and Blades. As their names suggest, these worksheets process and publish 3PAR and Blade data.

A worksheet requires data, which can come from one of two places:

  • A dataset (data produced by another worksheet)
  • A collection (data produced by collectors)

When designing a workbook, it is customary to start by setting up worksheets that are directly driven by collections. As a workbook grows in complexity, data from one worksheet will be passed to another worksheet (where it could potentially be merged in with datasets from other worksheets).

Collections

As described earlier, most worksheets are driven by collections. In the FC framework, all collections originate from the Triage process. This process results in a hierarchy of  CCR data files, which are organized in the
cc-working/processing/_byCustomer directory.

Within  the _byCustomer directory are all of the collected customer-specific data files, organized first by customer, then by date. For example:

  • _byCustomer
    • <customer_alias>
      • <data_file>
      • ...
    • ALCATEL-LUCENT
      • 20150401.ccr
      • 20150402.ccr
      • ...
    • adidas
      • 20150401.ccr
      • 20150402.ccr
      • ...

Note that directories within _byCustomer equate to the <customer_alias> label recovered from the original usage file email attachment. At this point Cloud Cruiser has not yet correlated the <customer_alias> to an actual billable customer account.

Within the dated CCR files Cloud Cruiser has all of the usage data recovered from the downloaded scripts, sorted by customer and date. These files will be read into customer-specific worksheets for processing. The following text sows a sample piece of a dated CCR file:

20150401,20150401,00:00:00,23:59:59,12,@collection,Collect-3PAR-ShowPDcols_10.255.255.71,@customer,RealDolmen,    
   ArrayModel,HP_3PAR 7400,ArrayName,mrdc3par0001,ArraySerial,28155,CagePos,14:13:0,Capacity,450, 
   IP_Address,10.255.255.71,Id,135,RPM,10,Serial,1628155,Type,FC,3,Used,130,Usable,408,Free,278

Key properties from this record include:

  • @collection--The collection that recovered the usage from the downloaded usage file. It is effectively unique to the filename of the email attachment.
  • @customer--The <customer_alias> field from the downloaded usage file.
  • IP_Address--Recovered from the email attachment filename.
  • Other properties--All other properties are specific to the device being metered, and will be carried to the workbooks so they can be loaded into the database for reporting.

The master collection

Step-by-step instructions for setting up the master collection in a workbook are described in Configuring a collection. This section described why each step is necessary.

To create a master collection, in a new customer workbook select Collections > New from the worksheet ribbon. From the drop-down list, select CC Record file. The Edit Collection dialog box appears.

AceRunCollection.jpg

Define the following settings on the Name and Source tab:

Field Description
Collection Name A unique name for this collection. Cloud Cruiser recommends using the workbook name prefaced by the customer ID.
Active This should be Yes, as the preferred way to disable daily processing is to disable the schedule instead of the collection.
Name A unique name for the data source. Use the workbook name.
Source File or Command. In this case, the CCR data we are importing is already present on the system as a file.
File Name

The file already exists on the system. To point the collector at the appropriate path, enter the following:

${env.processingDir}/_byCustomer/<customer_alias>/${env.selectDate}.ccr

There are four important components to this path:

  • ${env.processingDir}--This parameter points at the directory in which processing files are read and written. In the FC deployment, this parameter is equivalent to /mnt/cc-working/processing.
  • _byCustomer--The directory in which the earlier _ProcessUsage workbook structures the customer data.
  • <customer_alias>--The <customer_alias> field from the email attachment. For example, AceRun. To conclusively determine this value, consult the file system directly.
  • ${env.selectDate}.ccr--This parameter varies day to day as the daily schedules run.
Comments For maintainability, leave a comment for other system administrators.

Define the following sections on the General Properties tab:

Field Description
Data Sample Limit The maximum number of rows to display in the ETL simulator. For performance reasons, Cloud Cruiser recommends limiting this to 5000.
Exemption Limit The maximum number of records on which to fail. Cloud Cruiser recommends leaving this at 0. There should be no collection exceptions.
Exemption Log Limit The maximum number of exceptions to archive to the log files. This is not applicable if the Exception Limit is 0.
Keep Hourly Output This is not applicable because the preceding collection flow aggregates data to the day.
Multiple Feeds Because the dated CCR files in the _byCustomer directory contain data from multiple usage files, set this to Yes.
Feed Dimension

This property appears when Multiple Feeds is set to Yes. It allows you to specify a dimension that to use for sorting the input file into individual feeds. Use @collection for this purpose.

Setting the Multiple Feeds and Feed Dimension fields effectively sorts the per-customer usage back into feeds that correlate 1:1 to the usage files that were originally received.

For example:

aceRunGeneral.jpg

No settings are required in the CC Record File Properties tab.

Processing flow

Cloud Cruiser processes the data it receives in daily reports and makes it available for reporting as described in this section.

Measures to services

Collectors gather usage data from different resources. Workbooks import this usage data from the collectors and transform it as needed. This usage data is known as measures and is published into schemas in Cloud Cruiser. Services can be defined for each measure collected. When you define a service, you select measures from a list of all schemas, so when a measure is published it is associated with one or more services. Services can filter measures based on dimensions in the same record (these are called dimension filters).

Services to rate plans

When services are defined, they can be included in rate plans. Rate plans provide the information needed to determine how much to charge for each service.

Charging

Charging occurs when measures are published to a schema in a workbook. Because measures are associated with services, and services are associated with rate plans, the charge engine can calculate what to charge for each measure. The charge process is started each time a record is published to a schema.

Reporting

Reporting provides a way to view usage data and charges collected and calculated by Cloud Cruiser. These reports have several parameters that can be used to filter reports so they only show specific data. For more information, see Reporting.

Reports automatically restrict data based on the user’s privileges based on customer records that specify what data can be viewed by each user.

You can export reports to several formats, including PDF, Excel, and CSV. You can also configure reports to be automatically emailed to recipients.

Insights

You can also use Insights reports to view usage data and charges collected and calculated by Cloud Cruiser. Analytic dashboards and reports are interactive, so users can click on different parts of the reports and drill down to see data for selected items. You can customize these dashboards and reports as needed. You can also use Insights to show trends and do forecasting.

Last modified

Tags

This page has no custom tags.

Classifications

This page has no classifications.

 (c) Copyright 2017-2020 Hewlett Packard Enterprise Development LP