1.Since OIC is a PaaS offering, there has been checks on the file sizes that can be read with
FTP Adapter. OIC limits the largest file that can be read in-memory using FTP Adapter to
maximum of 5 MB.
2.Fortunately, OIC has ability to chunk read the large source file. This feature is called the Stage Read. Stage Read is a map reduce feature within OIC. Prior to making use of Stage Read in the OIC flow, one must make sure to download the file locally to OIC where OIC can read the file in chunks. The reason why the file must be downloaded to OIC prior to chunk reading it is because chunk reading from FTP Server is very expensive.
3.FTP Adapter in OIC does support the minimum age property but sometimes the source application may be trying to append to data file if there were many records. So using minimum age may not work in such scenarios and hence the need of having a trigger file.
4.How do we read large files upto 1Gb?
Ans:The ‘Read file’ approach is fine for files upto 10MB in size. However, Integration Cloud does not allow loading files larger then 10MB into memory for performance reasons. For reading larger files, developers should use the “Download file” option of FTP Connection. This allows file to be directly downloaded into ICS local file system without reading into memory.
FTP download can be used for files upto 1GB in size!
Once the file is available within Integration Cloud, then the Stage activity with operation ‘Read file in Segments’ can be used to perform chunked processing of the file contents.
Stage ‘Read file in Segments’ allows us to specify the segment size, in number of records, to process say 20 or 50 records per read cycle. The stage activity along with parallel processing option provides further enhanced performance.
5.Where can we decrypt a file?
Ans:Decryption can be done during 'download file' option of FTP Connection. Since ‘download file’ does not read contents into memory, the file read is effectively an opaque file. (no schema can be supplied during download file). But once the file is downloaded and decrypted, we can use the now familiar Stage read or segment read operation to read the structured data within the file.
2.Fortunately, OIC has ability to chunk read the large source file. This feature is called the Stage Read. Stage Read is a map reduce feature within OIC. Prior to making use of Stage Read in the OIC flow, one must make sure to download the file locally to OIC where OIC can read the file in chunks. The reason why the file must be downloaded to OIC prior to chunk reading it is because chunk reading from FTP Server is very expensive.
3.FTP Adapter in OIC does support the minimum age property but sometimes the source application may be trying to append to data file if there were many records. So using minimum age may not work in such scenarios and hence the need of having a trigger file.
4.How do we read large files upto 1Gb?
Ans:The ‘Read file’ approach is fine for files upto 10MB in size. However, Integration Cloud does not allow loading files larger then 10MB into memory for performance reasons. For reading larger files, developers should use the “Download file” option of FTP Connection. This allows file to be directly downloaded into ICS local file system without reading into memory.
Once the file is available within Integration Cloud, then the Stage activity with operation ‘Read file in Segments’ can be used to perform chunked processing of the file contents.
5.Where can we decrypt a file?
Ans:Decryption can be done during 'download file' option of FTP Connection. Since ‘download file’ does not read contents into memory, the file read is effectively an opaque file. (no schema can be supplied during download file). But once the file is downloaded and decrypted, we can use the now familiar Stage read or segment read operation to read the structured data within the file.
6.When can we encrypt a file?
Ans:Encryption can be performed during “Write file” option of FTP Connection. Note that both structured and opaque file can be encrypted! Choose the “enable PGP Security” as shown below
Note Note that encryption and decryption are currently available only on FTP Connection. The encryption and decryption keys should be provided in FTP connection used.
Encryption and Decryption are planned in other adapters like SOAP and Fusion HCM.
7.How can we read a structured file ?
Ans:Files from source FTP or File server can be read into Integration Cloud as a structured file or as an opaque file. Structured files can be xml, json or a simple comma separated file (csv).
7.How can we read a structured file ?
Ans:Files from source FTP or File server can be read into Integration Cloud as a structured file or as an opaque file. Structured files can be xml, json or a simple comma separated file (csv).
These could be a list of invoices, employee records in XML or json format which have a defined structure. Read file operation of FTP (and File) can read records from such files. The schema (or structure) can be specified by providing a schema file (xsd). Integration cloud can also generate a schema file using sample XML, JSON or CSV file, when the developer does not have the schema file.
The Read File operation reads the records from source file into memory. Subsequent mapper activities will present the schema structure allowing developer to map required values as required by the integration flow.
Figure below shows a mapping of structured elements
8.How can we read images and pdf files?
Ans:Consider a scenario when source file from FTP is a pdf file of a contract or a zip file containing jpeg images of inventory items. These are examples of opaque files. No Schema is provided during modelling and the file is read as opaque into memory. Figure below shows configuration to read an opaque file.
The output is available as a FileReference. Mapper activity allows us to map these file references to target systems. The target maybe another FTP server or a SOAP webservice which requires the image payload in base64encoded format.
Figure below shows a mapping of FTP FileReference
There is a special purpose function available in ICS, which can take a FileReference as input and convert the contents of the file into base64encoded string. This is the encodeReferenceToBase64 function we have seen earlier.
Note that SOAP connection also supports MTOM attachment which is more performant. When talking to SOAP service that accepts MTOM attachments, the FileReference can be directly assigned (or mapped) to the MTOM attachment of SOAP operation. There is no need of the base64encoding function in this case. See below figure showing SOAP adapter invoke configured with MTOM attachments option.
No comments:
Post a Comment