MongoDBWriter

CloverDX Designer > Component Reference > Writers > MongoDBWriter

Short Description

Ports

Metadata

MongoDBWriter Attributes

Details

Mapping

Error Handling in Bulk Operations

Examples

Short Description

MongoDBWriter stores, removes, or updates data in the MongoDB database using the Java driver.

MongoDBWriter can manipulate with documents in a MongoDB collection. New documents can be inserted, existing documents can be updated or removed.

Component	Data output	Input ports	Output ports	Transformation	Transf. required	Java	CTL	Auto-propagated metadata
MongoDBWriter	database	1	0-2	✓>	✓>	⨯>	✓>	✓

Component

Data output

Input ports

Output ports

Transformation

Transf. required

Java

CTL

Auto-propagated metadata

MongoDBWriter

database

0-2

✓>

⨯>

✓>

✓

Ports

Port type	Number	Required	Description	Metadata
Input	0	✓	Input data records to be mapped to component attributes.	any
Output	0	⨯	Results	any
1	⨯	Errors	any

Port type

Number

Required

Description

Metadata

Input

✓

Input data records to be mapped to component attributes.

any

Output

⨯

Results

any

⨯

Errors

any

Metadata

MongoDBWriter does not propagate metadata.

This component has metadata templates. The templates are described in the documentation of MongoDBReader in the section Metadata.

MongoDBWriter Attributes

Attribute Req Description Possible values

Attribute	Req	Description	Possible values
Basic
Connection	✓	The ID of the MongoDB connection to be used.
Collection name	✓ ^[1]	The name of the target collection.
Operation		The operation to be performed. MongoDBWriter can perform: Bulk write operations (recommended) insertOne, updateOne, updateMany, replaceOne, deleteOne, deleteMany Basic operations insert, remove, save, update, update_multi, upsert	See the description.
Query		Selects a subset of documents from a collection. The selection criteria may contain query operators. Ignored by the `insertOne`, `insert` and `save` operations.	BSON document
New value	✓ ^[1]	Specifies the document to be stored in the database. Ignored by delete operations (`deleteOne`, `deleteMany` and `remove`). For update operations, New value may contain update operators.	BSON document
Input mapping	✓	Defines the mapping of input records to component attributes.
Output mapping		Defines mapping of results to standard output port.
Error mapping		Defines the mapping of errors to the error output port.
Advanced
Batch size		Number of records that can be sent to database in one batch. Bulk write operations may significantly increase performance. However, the whole batch is stored in memory, so increasing Batch size also increases memory requirements.	bulk write: 100 (default) \| basic: 1 (default)
Stop processing on fail		If `true`, a failure causes the component to skip all subsequent operations and send the information about skipped executions to the error output port. Note: this function works only if the edge is connected to the component’s error port.	true (default) \| false
Field pattern		Specifies the format of placeholders that can be used within the Query and New value attributes. The value of the attribute must contain "`field`" as a substring, e.g. "`<field>`", "`#{field}`", etc. During the execution, each placeholder is replaced using a simple string substitution with the value of the respective input field, e.g. the string "`@{name}`" will be replaced with the value of the input field called "`name`" (assuming the default format of the placeholders).	@{field} (default) \| any string containing "`field`" as a substring.

Basic

Connection

✓

The ID of the MongoDB connection to be used.

Collection name

✓ ^[1]

The name of the target collection.

Operation

The operation to be performed. MongoDBWriter can perform:

Bulk write operations (recommended)

insertOne, updateOne, updateMany, replaceOne, deleteOne, deleteMany
Basic operations

insert, remove, save, update, update_multi, upsert

See the description.

Query

Selects a subset of documents from a collection. The selection criteria may contain query operators.

Ignored by the insertOne, insert and save operations.

BSON document

New value

✓ ^[1]

Specifies the document to be stored in the database.

Ignored by delete operations (deleteOne, deleteMany and remove).

For update operations, New value may contain update operators.

BSON document

Input mapping

✓

Defines the mapping of input records to component attributes.

Output mapping

Defines mapping of results to standard output port.

Error mapping

Defines the mapping of errors to the error output port.

Advanced

Batch size

Number of records that can be sent to database in one batch.

Bulk write operations may significantly increase performance. However, the whole batch is stored in memory, so increasing Batch size also increases memory requirements.

bulk write: 100 (default) | basic: 1 (default)

Stop processing on fail

If true, a failure causes the component to skip all subsequent operations and send the information about skipped executions to the error output port. Note: this function works only if the edge is connected to the component’s error port.

true (default) | false

Field pattern

Specifies the format of placeholders that can be used within the Query and New value attributes. The value of the attribute must contain "field" as a substring, e.g. "<field>", "#{field}", etc.

During the execution, each placeholder is replaced using a simple string substitution with the value of the respective input field, e.g. the string "@{name}" will be replaced with the value of the input field called "name" (assuming the default format of the placeholders).

@{field} (default) | any string containing "field" as a substring.

¹	The attribute is required, unless specified in the Input mapping.

Details

Operations

Mapping

Error Handling in Bulk Operations

Notes and Limitations

Format of the date Field Value

Operations

There are two types of operations available for this component: bulk write and basic operations. Bulk write operations are supported since driver and DB version 3.2.

Table 49. List of operations
Operation	Description
Bulk write operations (recommended)
`insertOne`	Adds the value of the New value attribute as a new document to the target Collection. If the document does not contain the `_id` field, a generated one will be added. See also db.collection.insertOne().
`updateOne`	Updates a single document matching the Query, with the values specified in the New value attribute, which must contain update operators. The `Upsert` parameter is available for this operation. See also db.collection.updateOne().
`updateMany`	Updates multiple documents matching the Query, with the values specified in the New value attribute, which must contain update operators. The `Upsert` parameter is available for this operation. See also db.collection.updateMany().
`replaceOne`	Replaces a single document matching the Query. The `Upsert` parameter is available for this operation. See also db.collection.replaceOne().
`deleteOne`	Removes a single document matching the Query. See also db.collection.deleteOne().
`deleteMany`	Removes all documents matching the Query. See also db.collection.deleteMany().
Parameters for bulk write operations
Upsert	Only applicable to `updateOne`, `updateMany` and `replaceOne`. If enabled, the operation inserts a new document into the collection if no document matches the Query. Generated object IDs for upserted documents will be returned as the `objectId` field in the output mapping.
Ordered	Executes operations in the order they arrive within every batch. If enabled, the first error causes the following operations in the same batch to be skipped.
Basic operations
`insert`	Adds the value of the New value attribute as a new document to the target Collection. If the document does not contain the `_id` field, a generated one will be added. See also db.collection.insert().
`remove`	Removes objects that match the Query from the Collection. See also db.collection.remove().
`save`	Similar to `insert`. Adds the document specified as the New value attribute to the Collection or replaces an existing document with the same `_id`. See also db.collection.save().
`update`	Updates at most one document that matches the Query, with the values specified in the New value attribute which may contain update operators. See also db.collection.update().
`update_multi`	Updates multiple documents matching the Query with the values specified in the New value attribute which must contain update operators. See also db.collection.update() - multi.
`upsert`	If no existing document matches the Query, inserts a new document into the Collection, otherwise performs an update. The New value attribute may contain update operators. See also db.collection.update() - upsert.

Mapping

Editing any of the Input, Output or Error mapping opens the Transform Editor.

Input mapping

The editor allows you to override selected attributes of the component with the values of the input fields.

Field Name	Attribute	Type
collection	Collection	string
query	Query	string
newValue	Projection	string

Field Name

Attribute

Type

Possible values

collection

Collection

string

query

Query

string

newValue

Projection

string

Output mapping

The editor allows you to map the results and the input data to the output port.

If Output mapping is empty, fields of input record and result record are mapped to output by name.

Field Name Type Description

Field Name	Type	Description
numAffected	integer	The number of affected documents, only set by the `update`, `update_multi` and `upsert` operations.
objectId	string	The object ID of the document. Bulk write operations: set by the `insertOne` operation, and the `updateOne`, `updateMany` and `replaceOne` operations for `upsert`. Basic operations: set by the `insert` and `save` operations. (Not populated in the bulk insert mode.)
batchNumber	long	The sequence number of the current batch, starting from 0.
deletedCount	integer	The number of documents deleted by the current batch.
insertedCount	integer	The number of documents inserted by the current batch.
matchedCount	integer	The number of documents matched by the current batch.
modifiedCount	integer	The number of documents modified by the current batch.

numAffected

integer

The number of affected documents, only set by the update, update_multi and upsert operations.

objectId

string

The object ID of the document.

Bulk write operations: set by the insertOne operation, and the updateOne, updateMany and replaceOne operations for upsert.

Basic operations: set by the insert and save operations. (Not populated in the bulk insert mode.)

batchNumber

long

The sequence number of the current batch, starting from 0.

deletedCount

integer

The number of documents deleted by the current batch.

insertedCount

integer

The number of documents inserted by the current batch.

matchedCount

integer

The number of documents matched by the current batch.

modifiedCount

integer

The number of documents modified by the current batch.

Error mapping

The editor allows you to map the errors and the input data to the error port.

If Error mapping is empty, fields of input record and result record are mapped to output by name.

Field Name	Type	Description
errorMessage	string	The error message.
stackTrace	string	The stack trace of the error.
batchNumber	long	The sequence number of the current batch, starting from 0.
deletedCount	integer	The number of documents deleted by the current batch.
insertedCount	integer	The number of documents inserted by the current batch.
matchedCount	integer	The number of documents matched by the current batch.
modifiedCount	integer	The number of documents modified by the current batch.

Field Name

Type

Description

errorMessage

string

The error message.

stackTrace

string

The stack trace of the error.

batchNumber

long

The sequence number of the current batch, starting from 0.

deletedCount

integer

The number of documents deleted by the current batch.

insertedCount

integer

The number of documents inserted by the current batch.

matchedCount

integer

The number of documents matched by the current batch.

modifiedCount

integer

The number of documents modified by the current batch.

Error Handling in Bulk Operations

Each input record produces one output record either on the standard, or error output port. A record is sent to the error output port if an error occurs or the operation is skipped. In such a case, see the errorMessage field in Error mapping for details and possible solution.

Notes and Limitations

MongoDBWriter does not write maps and lists. It converts maps and lists to string and writes the string.

Format of the `date` Field Value

A date value is in an ISO-8601 date format.

{
    withTZ : { "$date": "2018-11-22T14:25:11.541+02:00" },
    millis: { "$date": "2018-11-22T14:25:11.541" },
    seconds: { "$date": "2018-11-22T14:25:11" },
    dateLocal: { "$date": "2018-11-22" },
    dateUTC: { "$date": "2018-11-22+00:00" }
}

Examples

Writing records to MongoDB

Insert records (productID, productName, description) to collection newProducts.

Solution

Create MongoDB Connection to target database.

Set up the following attributes:

Attribute	Value
Connection	MyMongoDBConnection
Collection name	newProducts
Operation	insertOne
New value	{ productID : @{productID}, productName : "@{productName}", description: "@{description}"}

Attribute

Value

Connection

MyMongoDBConnection

Collection name

newProducts

Operation

insertOne

New value

{ productID : @{productID}, productName : "@{productName}", description: "@{description}"}

MongoDBWriter

Short Description

Ports

Metadata

MongoDBWriter Attributes

Details

Operations

Mapping

Input mapping

Output mapping

Error mapping

Error Handling in Bulk Operations

Notes and Limitations

Format of the date Field Value

Examples

Writing records to MongoDB

Solution

See also

Format of the `date` Field Value