You can use DataMigrator to build a flow that automatically processes source files as they appear in a designated directory. This can be useful if, for example, source files are transferred by FTP from various locations throughout the day and need to be processed immediately.
Before you create and run the data flows discussed in this section, you must create the sample data and synonyms. For more information, see How to Create Sample Procedures and Data for Iteration. This will create source and target synonyms and four source data files used in the example.
To implement and run a real time flow:
This flat file synonym corresponds to the layout of the source files that will appear in the designated directory. The flat file source is loaded to the target of your choice in real time.
Refer to the sample flow DMFLORD in the IBISAMP directory for the completed example.
For DataMigrator to automatically process files when they arrive in a directory, you need to designate a directory for that purpose. It could be any directory that your server can access. For this example, we will create an application directory called listen.
In addition, if you want to keep a copy of the files after they are processed you need to designate a directory for that purpose too. For this example, we will create an application directory called archive.
The Create New Application dialog box opens.
De-select the Add directory to APPPATH check box.
Click OK.
Next you need a synonym with File Collection Parameters to control how the file listener process files. We will start with a synonym that describes just the record layout of the source files and add the parameters.
This creates a synonym that listens on the listen directory for files with names that start with dmord and an extension of data polling every 5 seconds until 60 seconds have passed without a new file. As files are processed they are archived to a directory archive. Files are processed as they arrive until 5 files or 50 records are processed.
A flow with a File Listener source can be either a Data Flow or a Direct Load Flow. For this example, use a Direct Load Flow.
In the DMC:
The Properties dialog box opens.
The Transformations window opens.
Click the Automap button and click OK.
You can use DataMigrator to build a flow that automatically processes source files as soon as a specific Trigger file is detected in an application directory. A trigger is typically used when processing needs to be delayed. For example, when using FTP to copy files, if you wait until this secondary file appears while a large file is being copied into a directory, the trigger file indicates when processing can begin.
The following example shows how to create such synonym in a baseapp application using an Adapter for Delimited Flat File. You can also create such synonym using an Adapter for a Flat File or in any other application directory.
The Select adapter to configure or Select connection to create synonym dialog box opens.
The Select Data File dialog box opens,
DataMigrator creates the Master File for the File Listener synonym for the synonym later, based on this file.
Enter the following values:
The options should appear as shown in the following image.
The Define Collection dialog box opens.
The options should appear as shown in the following image.
The Select Synonym Candidates dialog box opens.
In this example, we will start the flow and then copy source files to the listen directory.
In the DMC:
dmordbos.dat, dmorddal.dat, dmordorl.dat, dmorstl.da
Tip: For a Windows based server, you can use the Edit and Copy/Paste functions.
Note: As the files are processed, you should see them disappear from the listen directory. This could take up to the timeout interval specified in the synonym.
With the Discard option set to Archive, after each file is processed, it is copied to the archive directory, as shown in the following image.
These files can be deleted, as they are no longer needed.
iWay Software |