In my project, I am extracting data from Oracle DB using ExecuteSQL (using Select query joining 4 tables ) and ingesting data to POSTGRESDB using PutDataBaserecord processor

Issue : Execute SQL is taking 4-6 minutes to extract 50K records from DB. So for 1 million records it will take over 1 hour. At the same time PutDBrecordprocessor is inserting records in few seconds

Is there any other way to extract data from DB?

Execute SQL : Attributes given is like below MaxWaittime :10seconds Use Avro logical format :true Compression Format :None Max rows per flow file : 50000 (I have tried with 10K as well) Outputbatch size : 50000

Nifi is running in kubernetes with single cluster

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.