Cloud Cruiser became HPE Consumption Analytics on Nov. 1st. You'll still see the old name in places while we update this site.

 

 

Consumption Analytics Documentation

Home > Cloud Cruiser 3 > Getting Started > Collection and processing > XML job scripting language > Steps in a job

Steps in a job

The job is a series of one or more steps. Each step in a job must be reachable as either the first step, or referenced as ‘next’ from some other step in the following format:

<batch:step id="Collect" next="Assemble">

All jobs must declare a parent of ccjob to properly integrate with the overall application framework in the following format:

<batch:job id="loadUsageData" parent="ccjob">

Tasklets are the most common strategy for processing within a step. A tasklet class is provided for each distinct processing stage for collecting, transforming and loading data. All tasklets include an active flag:

<property name="active" value="true" />

Tasklets that are not active (where the value is false) are skipped during job execution.

The following sections describe the main steps of a collection job, each with example XML content.

Collection step

Collection is the first step of the collection job flow. Processing begins with the retrieval of data from the various sources (feeds) in the cloud. The data from a feed is converted into a generic CC Record (.ccr ) file format.

CCRecordCollectTasklet collects data from an external source and writes the collected data under a feed directory for the active process.

In the following XML example, the feedName property specifies inventory as the feed directory for output, and the collector property gathers the data from the data source dataSource1 .

<batch:step id="collect_server1" next="collect_server2">
  <batch:tasklet>
  ​  <bean class="com.cloudcruiser.batch.CCRecordCollectTasklet">
  ​    <property name="active" value="true" />
  ​    <property name="feedName" value="inventory" />
  ​    <property name="collector">
  ​      <bean class="com.cloudcruiser.batch.collect.SmartJdbcCollector">
  ​        <property name="feedConfig" ref="feedConfig_inv" />
          ​<property name="dataSourceConfig" ref="dataSource1" />
        </bean>
      </property>
    </bean>
  </batch:tasklet>
</batch:step>

The next collection step gathers different data from the same source and writes it to feed performance:

<batch:step id="collect_server2" next="assemble">
  <batch:tasklet>
  ​  <bean class="com.cloudcruiser.batch.CCRecordCollectTasklet">
  ​    <property name="active" value="true" />
  ​    <property name="feedName" value="performance" />
  ​    <property name="collector">
  ​      <bean class="com.cloudcruiser.batch.collect.SmartJdbcCollector">
  ​        <property name="feedConfig" ref="feedConfig_perf" />
  ​        <property name="dataSourceConfig" ref="dataSource1" />
  ​      </bean>
  ​    </property>
  ​  </bean>
  ​</batch:tasklet>
</batch:step>

If you need to sort data from a single collection step into multiple feeds, grouping similar records so that you can perform different processing on each group, use CCRecordSiftingCollectTasklet​ instead of CCRecordCollectTasklet. This tasklet reads a feed name from an identifier in each record and writes the record to that feed. For more information, see the Java API documentation in <install_dir>/docs/api/batch/com/cloudcruiser/batch/CCRecordSiftingCollectTasklet.html .

Assembly step

The assembly step merges data from multiple feeds together into a single input file. In continuing with the example, inventory and performance are combined. This step is required even when only a single feed is collected to provide the input file that the next step expects.

CCRecordAssemblyTasklet locates feed files (typically created by CCRecordCollectTasklet) for a particular selection date based on the date in the filename, in the format yyyymmdd, and combines their contents into one file prior to processing. By using the optional enableMissingFeedSupport property and its three companion missingDaysXXX properties, you can create estimated usage data for dates where a feed is missing.

<batch:step id="assemble" next="aggregate">
  <batch:tasklet>
  ​  <bean class="com.cloudcruiser.batch.CCRecordAssemblyTasklet">
  ​    <property name="active" value="true"/>
  ​    <property name="maxFilesPerFeed" value="1"/>
  ​    <property name="createFeedIdentifier" value="true"/>
  ​    <property name="enableMissingFeedSupport" value="true"/>
  ​    <property name="missingDaysUsageAdjustment" value="true"/>
  ​    <property name="missingDaysResourceId" value="missingDays"/>
  ​    <property name="missingDaysLimit" value="2"/>
  ​  </bean>
  ​</batch:tasklet>
</batch:step>

This creates an intermediate date-stamped assembly.ccr file.

Aggregation step (optional)

The aggregation step is optional. In aggregation, individual data records are merged together based on their identifiers and the time frame the record pertains to.

CCRecordAggregationTasklet merges the assembly.ccr file’s data records within the same time frame that have matching identifiers.

<batch:step id="aggregate" next="transform">
  <batch:tasklet>
  ​  <bean class="com.cloudcruiser.batch.CCRecordAggregationTasklet" />
  ​</batch:tasklet>
</batch:step>

This creates an intermediate date-stamped aggregate.ccr file.

Transformation step

Transformation is a series of specialized record processors used to manipulate the data. In most cases, this includes combining usage identifiers (processing codes that have been assigned to the various cloud resources) into an account ID (a code that uniquely identifies the consumer of the resource). This creates an a date-stamped transformed.ccr file.

For more information, see Using transform processors.

Charge step

Charge is the final step of the collection job flow. CCRecordChargeTasklet accumulates resource allocations and usage by account, calculates charges, and loads this information into the system database. The following is an example charge step:

<batch:step id="charge">
  <batch:tasklet>
    <bean class="com.cloudcruiser.batch.CCRecordChargeTasklet">
      <property name="active" value="true" />
    </bean>
  </batch:tasklet>
</batch:step>

This creates and saves a date-stamped .ccr file to the database that gets parsed out and processed, so that the standard reports can report against that data.

Last modified

Tags

This page has no custom tags.

Classifications

This page has no classifications.
© Copyright 2018 Hewlett Packard Enterprise Development LP