Very slow sqlite performance?

[Lesiok]

I've used SQLite (via the QSqlite driver) to create databases that are 300+ GB in size. When I use SSD (solid state disk) drives as the source and output drives, it takes a few hours to read and process 450 GB of input ASCII data and create the 360 GB database. The DB creation uses "INSERT" rather than "UPDATE", I prepare the query once with placeholders for values, then I bind the values just prior to each insert. The DB has one table and 11 indexes plus the auto index for the table's primary key. The final DB has around 69 million rows in the main table.

Key-based searching is almost instantaneous after the database has been created. All of this is done using the QSqlite driver and QSqlDatabase / QSqlQuery.

So, basically SQLite isn't the bottleneck.

D_stranz, try to load data in packs, for example 1000 INSERTs in transaction. Something like this :

db.transaction();
1000 x INSERT;
db.commit();

db.transaction();
1000 x INSERT;
db.commit();

To copy to clipboard, switch view to plain text mode 

You can be very positively surprised