Consumption Analytics Documentation
Home > Documentation for older versions > Cloud Cruiser 4 > Collecting, transforming, and publishing > Steps for all collectors > Data sources

Data sources

Overview

Every collection requires information to tell it how to find the data how to collect, such as a URL or filename, and sometimes also needs credentials to get access to the data. This information is saved in an object called a data source.

The different collectors in Cloud Cruiser need different information, so there is a type of data source corresponding to each. A data source is an independent object in the Cloud Cruiser database, not part of a specific collection or ETL workbook. Therefore, it can be used multiple times. For example, one XML data source can be used by three XML collections in three separate workbooks.

Creating a data source

This section explains how to create a data source on its own. You can also create one as part of creating a collection.

To create a data source

  1. Go to Data > Setup > Sources.
  2. Click plus_icon.png at the bottom of the list.
    The Add New Data Source dialog appears.
  3. Enter a Name for the data source, use the drop-down menu select a Type from the drop-down list, and then click OK.
  4. In the content pane, enter values for all fields.
    The fields needed by a data source vary by data source type. For a full description of the fields needed, see the data source article for your collector under Collecting, transforming, and publishing.

    When connecting to an SQL database using a JDBC data source, the value for the URL field is in the format jdbc:jtds:<server_type>://<server>[:<port>][/<database>][;<property>=<value>[;...]]. For example, jdbc:jtds:sqlserver://localhost:1433/ccdb;domain=CCRUISER.

    If you are connecting to the Cloud Cruiser SQL server, the URL is found in the <installation_dir>\apache-tomcat-7.0.54\webapps\ROOT\WEB-INF\classes\database.properties file.

  5. (Optional) If you are creating a data source for Amazon Web Services or Microsoft Azure EA Billing, and you want Cloud Cruiser to connect to your data source through a proxy server, use the Proxy fields to specify the proxy information.
  6. (Optional) Click Test Connection to test the data source parameters, then make corrections if the test fails.
  7. Click Save.

Using wildcards to read multiple files

For data source types that read files, you can use wildcard characters in the File Name field to specify multiple files. Cloud Cruiser reads these files as one concatenated file. You can use the following wildcard characters:

? Matches one character
* Matches zero or more characters
** Matches zero or more directories in a path

Examples:

com/t?st.jsp Matches com/test.jsp but also com/tast.jsp or com/txst.jsp 
com/*.jsp Matches all .jsp files in the com directory 
com/**/test.jsp Matches all test.jsp files underneath the com path 
org/springframework/**/*.jsp Matches all .jsp files underneath the org/springframework path 
org/**/servlet/bla.jsp Matches org/springframework/servlet/bla.jsp but also org/springframework/testing/servlet/bla.jsp and org/servlet/bla.jsp

Importing and exporting a data source

You can import and export data sources from the Cloud Cruiser portal or using the command line. This is useful if you want to copy a data source from a Cloud Cruiser system to another system. For example, from a staging environment to a production environment.

If you upload a data source with the same name as an existing data source in the database, it overwrites the existing data source with the version of the driver jar files attached to the latest version.

From the Cloud Cruiser portal

You can import and export data sources from the Cloud Cruiser portal.

To import a data source from the Cloud Cruiser portal

  1. Go to Data > Setup > Sources.
  2. From the list of data sources, right-click the data source you want to import. A drop-down list appears.
  3. Select Import data source. The Upload File dialog appears.
  4. Click Browse and select the file you want to import.
  5. Click OK and click Save. An information message appears confirming a successful update.

To export a data source from the Cloud Cruiser portal

  1. Go to Data > Setup > Sources.
  2. From the list of data sources, right-click the data source you want to export. A drop-down list appears.
  3. Select Export data source. A CSV file downloads with the specified information.
  4. Click Open to open the file or click Save to save the file to your desired location.

From the command line

You can import and export data sources from the command line on Windows and Linux systems.

To import a data source on Windows systems

  1. Open a command prompt.
  2. Go to the <install_dir>/bin directory and locate the dataSourceSetup.bat executable file.
  3. Run dataSourceSetup.bat -i import -f <file_name>
    Where:
  • -i import = Imports the data source.
  • -f = The name of the file to export.

If the file name you are importing contains spaces, then you must add double quotes around the file name to make sure the file imports successfully.

To import a data source on Linux systems

  1. Open a terminal. 
  2. Go to the <install_dir>/bin directory and locate the dataSourceSetup.sh executable file.
  3. Run dataSourceSetup.sh -i import -f <file_name>
    Where:
  • -i import = Imports the data source.
  • -f = The name of the file to export.

To export a data source from the command line

You can export data sources from the command line on Windows and Linux systems.

To export a data source on Windows systems

  1. Open a command prompt. 
  2. Go to the <install_dir>/bin directory and locate the dataSourceSetup.bat executable file.
  3. Run dataSourceSetup.bat -e export -f <file_name> -d <datasourcename>
    Where:
  • -e export = Exports the data source.
  • -d = The name of the data source to export.
  • -f = The name of the export file.

To export a data source on Linux systems

  1. Open a terminal. 
  2. Go to the <install_dir>/bin directory and locate the dataSourceSetup.bat executable file.
  3. Run dataSourceSetup.sh -e export -f <file_name> -d <datasourcename>
    Where:
  • -e export = Exports the data source.
  • -d = The name of the data source to export.
  • -f = The name of the export file.

Database driver for a JDBC data source

If you create a JDBC data source, you need to specify a database driver for it. For a Microsoft SQL Server data source, the driver you need is located at <install_dir>/lib/jtds-1.2.6.jar. If the driver you need is not available in the Driver field when you create your JDBC data source, complete the following procedure to make it available.

To set up or modify a database driver

  1. Open your JDBC data source for editing in Cloud Cruiser.
  2. Click Manage Drivers, then click the + icon to add a new driver.
  3. Enter a Driver Name and, optionally, fill out other fields with relevant information.
  4. Click Add Files to upload JDBC driver files.
    The JDBC driver is a file with the extension .jar (some drivers need more than one file).
  5. Choose your files, then click OK.
  6. Click Save.

To import a JDBC driver from the command line

You can import JDBC drivers from the command line on Windows and Linux systems.

To import a JDBC driver on Windows systems

  1. Open a command prompt.
  2. Go to <install_dir>/ccapp/bin and locate the dataSourceSetup.bat executable file.

  3.  Run the command:

 dataSourceSetup -j -d <C:\test_upload>

or

dataSourceSetup -jdbc_import -d <C:\test_upload>

Where:

  • -j or -jdbc_import = Imports the JDBC driver.
  • -d = The path and file to import.

 

To import a JDBC driver on Linux systems

  1. Open a terminal.
  2. Go to <install_dir>/ccapp/bin and locate the dataSourceSetup.sh executable file.

  3.  Run the command:

 dataSourceSetup -j -d <C:\test_upload>

or

dataSourceSetup -jdbc_import -d <C:\test_upload>

Where:

  • -j or -jdbc_import = Imports the JDBC driver.
  • -d = The path and file to import.

To export JDBC drivers from the command line

You can export JDBC drivers from the command line on Windows and Linux systems. 

To export a JDBC driver on Windows systems

  1. Open a command prompt.
  2. Go to <install_dir>/ccapp/bin and locate the dataSourceSetup.bat executable file.
  3. Run the command:

dataSourceSetup -e -f <c:\temp\filename> -n <datasourcename>

or

dataSourceSetup -jdbc_export -d <C:\test_upload>

Where:

  • -e = Imports the JDBC drivers.
  • -f = The path and name of the file to export.
  • -n = The name of the data source to export.
  • -jdbc-export = Imports the JDBC drivers.

All directories for options -f and -d must exist.  

To export a JDBC driver on Linux systems

  1. Open a terminal.
  2. Go to <install_dir>/ccapp/bin and locate the dataSourceSetup.sh executable file.
  3. Run the command:

dataSourceSetup -e -f <c:\temp\filename> -n <datasourcename>

or

dataSourceSetup -jdbc_export -d <C:\test_upload>

Where:

  • -e = Imports the JDBC drivers.
  • -f = The path and name of the file to export.
  • -n = The name of the data source to export.
  • -jdbc-export = Imports the JDBC drivers.

All directories for options -f and -d must exist.  

Last modified

Tags

Classifications

This page has no classifications.
    Powered by MindTouch ®

     (c) Copyright 2017-2020 Hewlett Packard Enterprise Development LP