In my project, I am extracting data from Oracle DB using ExecuteSQL (using Select query joining 4 tables ) and ingesting data to POSTGRESDB using PutDataBaserecord processor
Issue : Execute SQL is taking 4-6 minutes to extract 50K records from DB. So for 1 million records it will take over 1 hour. At the same time PutDBrecordprocessor is inserting records in few seconds
Is there any other way to extract data from DB?
Execute SQL : Attributes given is like below MaxWaittime :10seconds Use Avro logical format :true Compression Format :None Max rows per flow file : 50000 (I have tried with 10K as well) Outputbatch size : 50000
Nifi is running in kubernetes with single cluster