I have created a file with a list of approximately 8 million import statements to be imported into an SQLite database. I have been redirecting errors to a file. Every few thousand lines there is an error, which let me know the progress of the import. As of yesterday the import was at line 5 million, but as far as I can tell nothing new has happened. [The import process - cat listofsqlstatements.sql| sqlite3 mydb.db 2>errors.txt
- is still running.]
The error log has not changed;
the file size of the db seems unchanged (using du database.db
);
and an - sqlite3 mydb.db "select distinct * from sometable"|wc -l
does not show any change.
However, looking at htop
I see the import process has high CPU use.
What might be going on?
How might I determine this?
Can I check to see what the last data imported to the DB was?
Add a comment
|
1 Answer
Use strace -p
to attach to the cat
process. See if it's still reading from the file.
By the way, if you're not doing your import inside a transaction, it will run several hundred times slower than it should.
-
So true. I've revised the code that generates the import statements. Now, I'll just use the sqlite3 .import with a csv file delimited by |.– d-cubedFeb 3, 2013 at 23:41