Version
2. Data Catalog
Data Catalog allows you to search and explore data sets published by your IT team. It is usually the first place to visit when you are starting to work on a new Wrangler job and would like to start looking for the data you need.
Figure 1. Main page of the Explorer view of the Data Catalog showing connectors available in the Catalog.
Content of the Data Catalog is managed by your IT team. To see how to publish data into Data Catalog, please review the Libraries section in our Server documentation.
Data Catalog indexes metadata related to your data sets - connector names, column names, their descriptions, data types and more. You can then explore and search through metadata collected by the Data Catalog to quickly find the data set you need for your job.
Data sets are provided by connectors which are installed and configured as CloverDX libraries by your IT team (for technical documentation on libraries, please review our technical reference for libraries).
Data Catalog consists of two screens: Explorer and My sources. Explorer screen allows you to browse and search through all connectors indexed by the Data Catalog while My sources screen allows you to view data sources you’ve created and used in your Wrangler jobs.
Explorer view
The Explorer screen allows you to do full text search in metadata collected by the Data Catalog. This way, you can easily find specific connectors or data sets based on their descriptions, field names or even parameters.
Figure 2. Data Catalog Explorer view
When hovering over connectors listed on the Explorer screen, you will see different actions depending on whether you have permissions to access given data source or not:
| Action | Visibility | Description |
|---|---|---|
View |
Always |
Opens Connector details screen with additional information about the connector. |
Add to My sources |
If you have permissions |
Opens Add to My sources wizard that will allow you to enter connector parameters and create new data source to save in your My sources screen. |
Use in a new job |
If you have permissions |
A shortcut that allows you to quickly create new job as well as new data source. This is the most common action to take when you want to work with your data. |
Get access |
If you don’t have permissions |
This link is only visible if you don’t have access to given connector and it provides more details about getting the access. |
If you have permissions for given data source connector, you can use it in your jobs right away. If you don’t have permissions to use given connector, you must contact your Wrangler administrator. They will be able to configure access permissions in CloverDX Server to grant you access. See more details about access permissions for connectors in Server documentation.
Searching in the Data Catalog
Data Catalog indexes all metadata it collects and allows you to search for your data based on names of libraries, connectors, data structures or even columns provided by various connectors.
To search, enter your search terms into the search box at the top of the Explorer view in Data Catalog. The page will update live with the data matching your search terms:
Figure 3. Search results displayed after searching for "payment" in the Data Catalog.
The search results page will show you where the keyword you were looking for was found so that you can better determine if the data or connector you found matches what you are looking for. You can then click on View button to view connector details or use the data right away by clicking on Add to My sources or Use in a new job buttons.
Note that search will also look through connectors that you may not have access to (but they are still visible to you). In that case, you will have to contact your IT administrator to grant you permissions to access and use the connector.
Connector details
Connector details screen shows you all metadata collected about select connected by the Data Catalog. For connectors that do not require additional configuration it will show you preview of the data. If connector requires additional settings, you will have to use Add to My sources wizard first to configure it and then you will be able to see the preview in My sources screen.
Figure 4. Connector details screen with data preview
Additional details about your data are available on Columns tab:
Figure 5. Connector details screen with Columns tab open showing list of all columns in the data source
My sources view
My sources view provides details of all data sources that you’ve configured and used in your jobs. It list all types of sources - files uploaded to your Wrangler workspace as well as sources delivered via libraries installed on CloverDX Server.
Figure 6. My sources view in Data catalog showing a preview of a data source alongside list of configured sources
My sources table shows you the sources you’ve created in the past and also allows you to easily create new ones either by uploading a new file via Upload area or by selecting source from Data Catalog via Open catalog button.
Table showing the list of all sources provides basic information for each source:
-
Name is the name of your source.
-
Origin to help you determine the type of source. Source can be file-based (created from files uploaded to your workspace) or connector-based (created from connectors in your Data Catalog).
-
Updated date shows you when the source was created or its parameters changes. Note that this does not show the latest data date.
-
Actions directly in the table or in three dots menu shown on hover for each item:
-
Use source (white arrow in blue circle) to create a new job from given source,
-
Delete (in three dots menu) to delete the source. Note that deleting the source will invalidate the jobs that are using that source.
-
View in Data Catalog (in three dots menu for non-file sources) to open a Data Catalog page with details for given source.
-
When you select a source in the table, the right half of the screen will show you additional details about that source and will allow you to edit its configuration. You will be able to also see preview of data provided by the data source.
To edit the configuration, click on the Edit button in the Configuration section. This will open a dialog that will allow you to change parameters of your data source. Exact layout of this dialog will depend on the source type - you’ll either get a CSV data source settings dialog or dialog to configure source provided by a library similar to one in Add to My sources wizard.
Adding connector to My sources
There are several ways of adding data source to My sources view:
-
Click the Add to My sources or Use in a new job buttons in Explorer view in Data Catalog
-
Click the Add to My sources or Use in a new job buttons when exploring connector details in Data Catalog
-
Uploading a file via drop area at the top of My sources page
-
Uploading a file via drop area when you create a new job via Create a new job action on My jobs page
The first two will lead to Add to My sources wizard that will allow you to configure settings for your data source. The dialog will look different for each data source since it will show the parameters for that data source:
Figure 7. Connector configuration dialog in Add to My sources wizard