Home > Documentation for older versions > Cloud Cruiser 4 > Collecting, transforming, and publishing

Collecting, transforming, and publishing

CloudSmart-Now solutions
Standard collection methods
1. Native collectors
2. Universal collectors

This section of articles helps you collect usage data from various sources, transform and enrich it with financial and organizational data, and publish it to the Cloud Cruiser database where you can use it in reports and Cloud Cruiser Insights modules.

For end-to-end instructions, see ETL process overview. Additionally, Steps for all collectors covers useful concepts that apply no matter which type of data source you collect from.

CloudSmart-Now solutions

Cloud Cruiser includes the CloudSmart-Now solutions, which work out of the box with minimal configuration, enabling you to collect, process, and publish billing data faster than with standard collection methods. CloudSmart-Now solutions are available for the following data sources:

CloudSmart-Now solutions offer the following benefits over standard, manually-configured collection methods:

Workbook templates that include all of the collections and ETL workflow necessary to take advantage of the other listed benefits.
A simple, five-level account structure: Business Unit, Department, Owner, Environment, and Resource ID.
For private cloud data sources, services to handle common chargeback use cases. Usage is automatically matched to one of these services. (Public cloud providers use passthrough services.)
For private cloud services, baseline rates. These rates provide a realistic starting point for the rates you assign your services, and will speed your implementation. (Public cloud providers use passthrough rates.)
Private cloud VMs are charged according to standardized sizes based on their number of CPUs and amount of allocated RAM. A VM that doesn’t exactly match both defined amounts for a size is charged as a nonstandard version of that size.
Easy methods to allocate cost to departments and individual owners for collected resources. Private clouds use a CSV file you maintain to map resources to owners and departments. For AWS, Cloud Cruiser can collect the department and owner from tag values, or through the same methods available for private cloud vendors. For Microsoft Azure, Cloud Cruiser can collect the department and owner from the Department Name and AccountOwnerId fields in the Azure bill, or through the same methods available for private cloud usage.
Tailored analytics reports out-of-the-box.

Standard collection methods

Cloud Cruiser's standard collection methods provide broad collection capabilities that you configure specifically for your data-collection needs.

Native collectors

Native collectors are specialized collection modules that are typically developed by HPE to deal with a complex, proprietary data source. Native collectors have the advantage of being able to perform more advanced functions, such as interfacing to third-party APIs, exchanging credentials and security tokens, and more. Cloud Cruiser includes native collectors for the following data sources:

Universal collectors

Universal collectors gather data from common data formats such as flat files, Excel spreadsheets, or databases. These generic collectors are flexible enough to handle a very wide array of data and allow you to collect from data sources for which there is no native collector available. Cloud Cruiser includes universal collectors for the following data sources:

How-to articles

ETL process overview
Amazon CloudWatch

Collector properties
Advanced configuration

Installation

Managing lookup tables in the Cloud Cruiser Portal
Managing lookup tables using flat files
Managing lookup tables using flat files

Email notification
Data sources: Every collection requires information to tell it how to find the data how to collect, such as a URL or filename, and sometimes also needs credentials to get access to the data. This information is saved in an object called a data source. The different collectors in Cloud Cruiser need different information, so there is a type of data source corresponding to each. A data source is an independent object in the Cloud Cruiser database, not part of a specific collection or ETL workbook. Therefore, it can be used multiple times. For example, one XML data source can be used by three XML collections in three separate workbooks.

Advanced Windows Azure Pack collection
Changing the REST API endpoint in Windows Azure Pack
Enabling Windows Azure Pack to display Cloud Cruiser data
Troubleshooting Windows Azure Pack collection

Creating collections
Creating workbooks
Publishing data to a schema
Run a workbook for a range of dates

Creating Microsoft Azure data sources
Collecting Microsoft Azure tags and data

Reference articles

Data source properties

Requirements
Configuration: All configuration options supported by the Network Traffic Collector are defined in a configuration file. By default, the collector will look for a file called ccnetcollect.conf located in the same directory as the collector itself. You must edit this file to suit the environment. Any changes made to the file become active when the collector is restarted. You can force the collector to use an alternative configuration file by using the -f option as follows: ccnetcollect -f myconfig.conf To generate a sample configuration file, which will contain information about all the available configuration options, use the command line option -sampleconfig. This will output the sample configuration to the terminal, so to capture it as a file use redirection as follows: ccnetcollect -sampleconfig > myconfig.conf A configuration file consists of configuration items of the form item = value. All whitespace is ignored, and any text on a line after a # character is also ignored. The item specified on each line can be in upper case, lower case, or a mixture of the two. The collector will perform a number of sanity checks on the values specified in the configuration file, as well as establish that no required items are missing. In the event of missing or invalid data, diagnostic information will be written to the logfile specified in the configuration file, or to syslog if no logfile is specified. If any required options are missing, the collector will inform you how many are missing and refer you to the log file for the specific list of missing options.
Sample configuration file
Deployment
Support information
Limitations