Results 1 to 7 of 7

Thread: Fetch millions of records using QtSql

  1. #1
    Join Date
    Jun 2010
    Posts
    91
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Fetch millions of records using QtSql

    Hi,

    I have requirement where there is an Oracle table which has almost 40 million data. This data I need to process and has to be written in various flat files which is to be given to client. The problem is whenever I fetches records from this table, it gives memory fault error after 10-15 million data fetched. Hence I am looking for solution which will fetch resultset in batches, clear memory and get next resultset so that I can avoid memory error.

    Thanks in advance

    Manish

  2. #2
    Join Date
    Jan 2006
    Location
    Belgium
    Posts
    1,938
    Thanked 268 Times in 268 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows
    Wiki edits
    20

    Default Re: Fetch millions of records using QtSql

    For large datasets try to use transactions (if possible of Oracle of course).

    Assuming that you can process the results in batch sizes of 100 records for example, you can easily use SQL to limit the results. See SQL limit on google.

  3. #3
    Join Date
    Jun 2010
    Posts
    91
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Fetch millions of records using QtSql

    Thanks for prompt reply.
    SQL limit clause can be used in PL/SQL, not sure if can be used directly in SQL.
    If possible can you elaborate how we can code using transaction.

    Thanks

    Manish

  4. #4
    Join Date
    Feb 2007
    Location
    Karlsruhe, Germany
    Posts
    469
    Thanks
    17
    Thanked 90 Times in 88 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Fetch millions of records using QtSql

    This sounds like you are having a memory leak and not a design problem. If you look at the taskmanager is it your application or the database-server that accumulates memory? Some time ago i didn't properly clear the query objects which resulted in a memory pileup in the sql-server. also try only fetching and not processing. maybe the problem is your output-code?

    Johannes

  5. #5
    Join Date
    Aug 2009
    Location
    Belgium
    Posts
    310
    Thanks
    10
    Thanked 31 Times in 25 Posts
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Fetch millions of records using QtSql

    Hi,

    Maybe use a QSqlQuery and use setForwardOnly(). setForwardOnly() should prevent loading of the entire result in memory at once.

    Regards,
    Marc

  6. #6
    Join Date
    Jun 2010
    Posts
    91
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Fetch millions of records using QtSql

    Thanks for reply.
    I will try with setForwardOnly option and will post details.
    However I just want to add that there is database or application issue. It is cache which is getting full due to huge number of records from resultset.

    Manish

  7. #7
    Join Date
    Jun 2010
    Posts
    91
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Fetch millions of records using QtSql

    Thanks marcvanriet.

    It has worked. setForwardOnly has solved memory leak problem. Now I am able to fetch 100 million records without any problem.

    Manish

Similar Threads

  1. Graphics view with millions of items
    By JovianGhost in forum Qt Programming
    Replies: 13
    Last Post: 26th March 2010, 23:58
  2. QMYSQL3: Unable to fetch data
    By crystlrain in forum Qt Programming
    Replies: 2
    Last Post: 11th November 2009, 07:24
  3. Replies: 6
    Last Post: 8th June 2009, 21:44
  4. Adding millions of GraphicsItems is slow
    By rk123 in forum Qt Programming
    Replies: 1
    Last Post: 28th April 2009, 07:17
  5. QSQLITE Unable to fetch row in create
    By xgoan in forum Newbie
    Replies: 3
    Last Post: 18th October 2006, 14:39

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.