Version

    CopyFiles

    Jobflow Component

    Short Description
    Ports
    Metadata
    CopyFiles Attributes
    Details
    Examples
    See also

    Short Description

    CopyFiles can be used to copy files and directories.

    CopyFiles can copy multiple sources into one destination directory, or a regular source file to a target file. Directories can be copied recursively. Optionally, existing files may be skipped or updated based on the modification date of the files.

    [Note]Note

    To be able to use this component, you need a separate jobflow license.

    ComponentInputsOutputsAuto-propagated metadata
    CopyFiles0-10-2✓

    Ports

    Port typeNumberRequiredDescriptionMetadata
    Input0⨯Input data records to be mapped to component attributes.Any
    Output0⨯ResultsAny
    1⨯ErrorsAny

    Metadata

    CopyFiles does not propagate metadata from left to right or from right to left.

    This component has metadata template available. See Details for details on template fields of CopyFiles or metadata templates for general details on metadata templates.

    CopyFiles Attributes

    AttributeReqDescriptionPossible values
    Basic
    Source file URLyes [1] Path to the source file or directory (see Supported URL Formats for File Operations).  
    Target file URLyes[1]

    Path to the destination file or directory (see Supported URL Formats for File Operations). When it points to a directory, the source will be copied into the directory.

    It must be a path to a single file or directory.

    		 
    RecursivenoCopies directories recursively.false (default) | true
    Overwriteno

    Specifies whether existing files shall be overwritten.

    In always mode, the target will be overwritten.

    In update mode, the target will be overwritten only when the source file is newer than the destination file.

    In never mode, the target will not be overwritten.

    		always (default) | update | never
    Create parent directoriesno

    Attempts to create non-existing parent directories.

    When the Create parent directories option is enabled and the Target file URL ends with a slash ('/'), it is treated as a parent directory, i.e. the source directory or file is copied into the target directory, even if it does not exist.

    		false (default) | true
    Input mapping[2]Defines mapping of input records to component attributes.
    Output mapping[2]Defines mapping of results to the standard output port. 
    Error mapping [2]Defines mapping of errors to the error output port. 
    Redirect error outputnoIf enabled, errors will be sent to the output port instead of the error port.false (default) | true
    Verbose outputno

    If enabled, one input record may cause multiple records to be sent to the output (e.g. as a result of wildcard expansion). Otherwise, each input record will yield just one cumulative output record.

    		false (default) | true
    Advanced
    Stop processing on failno

    By default, a failure causes the component to skip all subsequent input records and send the information about skipped input records to the error output port. This behavior can be turned off by this attribute. Note: this function works only if an edge is connected to the component's error port.

    		true (default) | false

    [1]  The attribute is required, unless specified in the Input mapping.

    [2]  Required if the corresponding edge is connected.

    Details

    Editing any of the Input, Output or Error mapping opens the Transform Editor.

    Input mapping

    The editor allows you to override selected attributes of the component with the values of the input fields.

    Field NameAttributeTypePossible values
    sourceURLSource file URLstring 
    targetURLTarget file URLstring 
    recursiveRecursivebooleantrue | false
    overwriteOverwritestring"always" | "update" | "never"
    makeParentDirsCreate parent directoriesbooleantrue | false
    Output mapping

    The editor allows you to map the results and the input data to the output port.

    If Output mapping is empty, fields of the input record and result record are mapped to the output by name.

    Field NameTypeDescription
    sourceURLstringURL of the source file.
    targetURLstringURL of the destination.
    resultURLstring

    A new URL of the successfully copied file. Only set in Verbose output mode.

    resultboolean

    True if the operation has succeeded (can be false when Redirect error output is enabled).

    errorMessagestring

    If the operation has failed, the field contains the error message (used when Redirect error output is enabled).

    stackTracestring

    If the operation has failed, the field contains the stack trace of the error (used when Redirect error output is enabled).

    Error mapping

    The editor allows you to map errors and input data to the error port.

    If Error mapping is empty, fields of the input record and result record are mapped to the output by name.

    Field NameTypeDescription
    resultbooleanWill always be set to false.
    errorMessagestringThe error message.
    stackTracestringThe stack trace of the error.
    sourceURLstringURL of the source file.
    targetURLstringURL of the destination.

    Stop processing on fail

    If you get file URLs from the input port having Stop processing on fail set to true and an error occurs, the record causing the error and the following ones are skipped.

    Examples

    Copying Files from one Directory to Another
    Using Stop Processing on Fail
    Non-recursive copying directory with mixed content

    Copying Files from one Directory to Another

    Copy .txt files from ${DATAIN_DIR}/FO/ to ${DATAOUT_DIR}/FO/. The target directory may not exist. If the files in the target directory exist, they should be overwritten.

    Solution

    Use the attributes Source file URL, Target file URL and Create parent directories.

    AttributeValue
    Source file URL${DATAIN_DIR}/FO/*.txt
    Target file URL${DATAOUT_DIR}/FO/
    Create parent directoriestrue

    Using Stop Processing on Fail

    Copy files f1.txt f2.txt and f3.txt. If you cannot copy a file, continue with the following one. 

    -rw-rw-r--. 1 clover clover 0 Feb 23 18:11 f1.txt
-rw-------. 1 user23 user23 0 Feb 23 18:11 f2.txt
-rw-rw-r--. 1 clover clover 0 Feb 23 18:11 f3.txt
    Solution with Source File URL attribute

    The solution without reading file URLs from an input edge is the same as in the previous example. Use the attributes Source file URL, Target file URL and Create parent directories.

    Note that only files f1.txt and f2.txt will be copied. The file f2.txt cannot be copied as the user clover does not have read access to the file.

    Solution with Input Edge

    If you read URLs of files from an input edge one by one, use the attributes Target file URL, Create parent directories, Input mapping and Stop processing on fail.

    AttributeValue
    Target file URL${DATAOUT_DIR}/FO/
    Create parent directoriestrue
    Input mapping 
    Stop processing on failfalse

    In Input mapping, you map Source URL.

    Note that you need to uncheck the attribute Stop processing on fail. Otherwise only the file f1.txt would be copied. (Assuming that the files come in ascending order.)

    Non-recursive copying directory with mixed content

    This example shows non-recursive copying of files from one directory to another.

    Copy files and directories from ${DATAIN_DIR}/abc to ${DATAOUT_DIR}/def. If a directory in ${DATAIN_DIR}/abc contains a file or a directory, these files and directories should not be copied.

    Source directory

    ${DATAIN_DIR}/abc/file.txt
${DATAIN_DIR}/abc/dirA/file2.txt

    Expected result

    ${DATAIN_DIR}/def/file.txt
${DATAIN_DIR}/def/dirA
    Solution

    Use ListFiles to list the files and directories to be copied. In ListFiles, set the File URL attribute to the source directory.

    AttributeValue
    File URL${DATAIN_DIR}/abc/

    In CopyFiles, copy the files and directories from the list received from an input edge.

    AttributeValue
    Target file URL${DATAIN_DIR}/def/
    Create parent directoriestrue
    Input mapping
    //#CTL2

// Transforms input record into output record.
function integer transform() {
    $out.0.sourceURL = $in.0.URL;

    return ALL;
}
    [Note]Note

    The ListFiles component is necessary to list all files and directories. If you non-recursively copy files with CopyFiles only, the copying stops when a first subdirectory is reached.