Version CloverDX 6.7.0
MySQLBulkWriter
Short description
MySQLBulkWriter is a high-speed MySQL table loader. It uses a MySQL native client.
|
When to use MySQLBulkWriter
This component requires installation and configuration of a database native client. The client must be installed on the same machine as CloverDX Server (when working in a server project) or CloverDX Designer (when working in a local project). Due to this overhead, we recommend using this component only if you require significantly higher loading performance than with DatabaseWriter which should be used in typical scenarios. |
| Data output | Input ports | Output ports | Transformation | Transf. required | Java | CTL | Auto-propagated metadata |
|---|---|---|---|---|---|---|---|
database |
0-1 |
0-1 |
⨯ |
⨯ |
⨯ |
⨯ |
⨯ |
Ports
| Port type | Number | Required | Description | Metadata |
|---|---|---|---|---|
Input |
0 |
Records to be loaded into the database |
Any |
|
Output |
0 |
⨯ |
For information about incorrect records |
Metadata
MySQLBulkWriter does not propagate metadata.
Error metadata cannot use Autofilling functions.
| Field number | Field name | Data type | Description |
|---|---|---|---|
0 |
<any_name1> |
integer |
The number of the incorrect record (records are numbered starting from 1). |
1 |
<any_name2> |
integer |
The number of the incorrect column. |
2 |
<any_name3> |
string |
Error message |
MySQLBulkWriter attributes
| Attribute | Req | Description | Possible values |
|---|---|---|---|
Basic |
|||
The path to MySQL utility |
yes |
The name of the MySQL utility, including the path. Must be installed and configured on the same machine where CloverDX runs. The MySQL command line tool must be available. |
|
Host |
Host where the database server is located. |
localhost (default) | other host |
|
Database |
yes |
The name of the database into which the records should be loaded. |
|
Database table |
yes |
The name of the database table into which the records should be loaded. |
|
User name |
yes |
Database user |
|
Password |
yes |
The password for a database user. |
|
Advanced |
|||
Path to control script |
The name of the command file containing the |
||
Lock database table |
By default, the database is not locked and multiuser access is allowed.
If set to |
false (default) | true |
|
Ignore rows |
The number of rows of a data file that should be skipped. By default, no records are skipped. Valid only for input file with data. |
0 (default) | 1-N |
|
Column delimiter |
The delimiter used for each column in the data. Field values must not include this delimiter as their part. By default, a tabulator is used. |
\t (default) | other character |
|
Loader input file |
The name of an input file to be loaded, including path. For more information, see Loader input file. |
||
Control script parameters |
All parameters that can be used as parameters by the |
||
| 1 |
If the input port is not connected, Loader input file must be specified and contain data. For more information, see Loader input file. |
Details
MySQLBulkWriter loads data into a database table using native MySQL database client.
It reads data either from the input port or from an input file.
You can attach the optional output port and read records which have been reported as rejected.
Reading from input port (input port connected) dumps the data into a temporary file which is then used by mysql utility.
You can set the temporary file explicitly by setting the Loader input file attribute or leave it blank to use default.
Reading from a file (no input connected) uses the Loader input file attribute as a path to your data file.
The attribute is mandatory in this case.
The file needs to be in a format recognized by the mysql utility (see MySQL LOAD DATA).
This component executes the MySQL native command-line client (bin/mysql or bin/mysql.exe). The client must be installed on the same machine as the graph is running on.
MySQLBulkWriter is a bulk loader suitable for uploading many records to database.
To insert several records, you can also use DatabaseWriter, which does not require mysql utility.
Path to Control Script
The name of a command file containing the LOAD DATA INFILE statement (See MySQL LOAD DATA), including path.
-
If it is not set, the command file is created in CloverDX temporary directory and it is deleted after the load finishes.
-
If it is set, but the specified command file does not exist, a temporary command file is created with the specified name and path and it is not deleted after the load finishes.
-
If it is set and the specified command file exists, this file is used instead of command file created by CloverDX.
Loader input file
The name of the input file to be loaded, including the path.
-
If it is not set, a loader file is created in CloverDX or OS temporary directory (on Windows) or stdin is used instead of temporary file (on Unix). The file is deleted after the load finishes.
-
If it is set, a specified file is created. It is not deleted after data is loaded and it is overwritten on each graph run.
-
If the input port is not connected, this file must be specified, must exist and must contain data that should be loaded into the database. The file is not deleted nor overwritten.
Notes and limitations
Maps and lists
You should not write maps and lists.
MySQL 8.0.2 and newer
When using MySQLBulkWriter to write into MySQL 8.0.2 and newer, you must enable the local_infile property. If disabled, MySQLBulkWriter fails with the following error:
Mysql utility has failed.
Also, the following error reported by the MySQL database appears in the job log:
The used command is not allowed with this MySQL version
Examples
Loading data to MySQL database
Load records (productID, amount) to table products of MySQL database dbo on mysql.example.com with username is smithj and password smithy.
Solution
Install the mysql utility.
Set up the following attributes of MYSQLBulkWriter.
| Attribute | Value |
|---|---|
Path to mysql utility |
C:/Program Files/MySQL/MySQL Server 5.6/bin/mysql.exe |
Host |
mysql.example.com |
Database |
dbo |
Database table |
products |
User name |
smithj |
Password |
smithy |
In case the server uses a non-standard port number, specify the Port parameter in the Control script parameters attribute.
Compatibility
| Version | Compatibility Notice |
|---|---|
5.3.0 |
MySQLDataWriter was renamed to MySQLBulkWriter. |