Page 1 of 2 12 LastLast
Results 1 to 20 of 33

Thread: reading from a file

  1. #1
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default reading from a file

    hi, I need to put in 'buffer' ALL content of a file. Is it possibile. Is this faster way to manage a file?
    This below should reads 1000 char but I need to read until the EOF. Is there a way?
    Qt Code:
    1. char* buffer=0;
    2. while ( ifile.get(buffer,1000) ) {
    3. cout << " buffer " << buffer << endl;
    4. }
    To copy to clipboard, switch view to plain text mode 
    Regards

  2. #2
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    EDIT: (faster way?) I have a very large text file where every line is very long! thanks
    Regards

  3. #3
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    Quote Originally Posted by mickey View Post
    This below should reads 1000 char but I need to read until the EOF. Is there a way?
    What is the type of that ifile variable? You can check the file size and allocate a buffer of appropriate size.

  4. #4
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    Qt Code:
    1. std::fstream file;
    2. std::fstream ifile = file.open(file_name.c_str();
    To copy to clipboard, switch view to plain text mode 
    1.uhm, but anyway the 'get' cause a crash at runtime....
    2.But is there a way to assign in one "hit" all text file to a char*? (I thought use a vector<string> too, but If I have a very large file and large lines maybe vector goes slow...)
    thanks
    Regards

  5. #5
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    Quote Originally Posted by mickey View Post
    the 'get' cause a crash at runtime....
    How did you allocate the buffer?

    Quote Originally Posted by mickey View Post
    But is there a way to assign in one "hit" all text file to a char*? (I thought use a vector<string> too, but If I have a very large file and large lines maybe vector goes slow...)
    You could use seekg() and tellg() to check the file size first and then use a large enough buffer, but it might lead to race condition if some other process is modifying that file.

  6. #6
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    thanks for reply;
    I just see for the tellp/g solution and it seems ok; But could I put file into vector<string> too? (I mean: is it efficient a vector? eg: my file is 2GB size large)
    Regards

  7. #7
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    Quote Originally Posted by mickey View Post
    is it efficient a vector?
    If you are going to use push_front/push_back methods it won't be. On the other hand, if you are going to allocate it in one go, it won't differ much from reading data line by line.

    Quote Originally Posted by mickey View Post
    eg: my file is 2GB size large
    How much RAM do you have in your machine? You certainly don't want your machine to start swapping.

    If you want to optimize something, first use a profiler to check where the bottleneck is.

    Remember: "premature optimization is the root of all evil".

  8. #8
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    Hi I have 512mb. and with this below my machine swapping! So haven't I hoping to use file larger than my Ram?
    Qt Code:
    1. ifile.read (memblock, size);
    To copy to clipboard, switch view to plain text mode 
    Sorry: are you saying that working on "memblock" is much less efficient than using push_back to put the file in a vector?

    thanks
    Regards

  9. #9
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    Quote Originally Posted by mickey View Post
    Hi I have 512mb. and with this below my machine swapping! So haven't I hoping to use file larger than my Ram?
    You can use files larger than your RAM, but in parts that fit in the memory. Getting data from disk is *a lot* slower than getting it from the memory, so you should avoid swapping at all cost.

    Quote Originally Posted by mickey View Post
    are you saying that working on "memblock" is much less efficient than using push_back to put the file in a vector?
    On the contrary. push_back() might result in relocation of all of the data in the vector.

  10. #10
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    hi, What does that mean?
    ...... but in parts that fit in the memory.
    So is "memblock = new char [size];" is the best solution?
    thanks
    Regards

  11. #11
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    Quote Originally Posted by mickey View Post
    What does that mean?
    If you have 512MB of RAM, you shouldn't try to read more than that.

    Quote Originally Posted by mickey View Post
    So is "memblock = new char [size];" is the best solution?
    std::vector and std::string will be OK too, but only if you won't change their size too often.

  12. #12
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    hi, I tried read with the solution above with memblock on a 200mb text file. it's goes fast.
    Then I tried to read file and put it into a vector:
    Qt Code:
    1. vector<string> text;
    2. copy (it_file, eos, inserter (text, text.begin()) );
    To copy to clipboard, switch view to plain text mode 
    this goes very slow (I didn't see the end of the copy!)
    I tried this operation of "copy" because I need to sort the lines of the file too. Then I thought to put the lines in a vector and then call sort() on it. But the "reading" is too slow and I need speed.
    I'm thinking of implement my own sort on memblock. But maybe it won't so fast as vector.sort().......What do you suggest, please?
    Regards

  13. #13
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    What is the format of the file you want to read and what operations you need to perform?

  14. #14
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    hi, my file contains lines of strings (it's a text); At the moment I'd like order every line of the file ( line1 < line2)
    Regards

  15. #15
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    First of all there's a small application called sort that will do that for you, but I'm not sure if it's available on windows.

    If you still want to do it yourself, it sounds like a job for merge sort. You can save some time by starting with the biggest blocks that fit into your RAM and sorting them using quick sort or similar algorithm.

    If you know the maximum line length or at least its approximate value, you can create std::vector< std::string > of some constant size and preallocate space in every string (you can do that using two-parameter version of std::vector's constructor).

  16. #16
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    Quote Originally Posted by jacek View Post
    If you know the maximum line length or at least its approximate value, you can create std::vector< std::string > of some constant size and preallocate space in every string (you can do that using two-parameter version of std::vector's constructor).
    I'm confused; I know the maximum dimension of lines (1KB) but I cannot know the file size... When I say vector <string> I mean every element of vector is a line of the file.
    Are you saying to do somthing like: vector<string> line (10, ""); ?? (if it's so I have allocate 10 lines.. and I don't know how many lines could have the file).
    Regards

  17. #17
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    Quote Originally Posted by mickey View Post
    I cannot know the file size...
    Luckily it isn't required for merge sort.

    Quote Originally Posted by mickey View Post
    Are you saying to do somthing like: vector<string> line (10, ""); ?? (if it's so I have allocate 10 lines.. and I don't know how many lines could have the file).
    Yes. To be exact, something like:
    Qt Code:
    1. std::string temp;
    2. temp.reserve( MAX_LINE_LEN );
    3. std::vector< std::string > lines( NLINES, temp );
    4.  
    5. // or if std::string tries to be smart:
    6.  
    7. std::vector< std::string > lines( NLINES );
    8. for( ... ) {
    9. line->reserve( MAX_LINE_LEN );
    10. }
    To copy to clipboard, switch view to plain text mode 
    Where MAX_LINE_LEN = 1024 and NLINES is around 256k. Remember that the point is to avoid reallocations and swapping.

    This way you can read 256k lines into preallocated space, which should be fast enough. Then you can sort that vector using std::sort, dump everyting into a temporary file and read another set of lines. And so on until the end of file. When you are finished you can free the memory and continue with merge sort algorithm.

  18. #18
    Join Date
    Jan 2006
    Location
    travelling
    Posts
    1,116
    Thanks
    8
    Thanked 127 Times in 121 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    Quote Originally Posted by jacek View Post
    If you have 512MB of RAM, you shouldn't try to read more than that.
    This doesn't make much sense... In most cases (users who dropped text-based interface) the OS/Desktop/backgroud tasks... occupy about half of the available memory (sometimes more). Thus it is highly recommened not to load a full file which is bigger than 40% of your available memory (yet it is possible). In such cases the best way is to read pieces of the file, perform some tasks and then discard them before loading some other new pieces... It should be a little tricky to work this way to sort lines of text but it's still doable.
    Current Qt projects : QCodeEdit, RotiDeCode

  19. #19
    Join Date
    Jan 2006
    Posts
    976
    Thanks
    53
    Qt products
    Qt3
    Platforms
    Windows

    Default Re: reading from a file

    OK! But I don't still understand this:
    1. I don't know the number of the lines ( I know only its dimesion with instructions below)
    Qt Code:
    1. ifile.seekg (0, ios::end);
    2. size = ifile.tellg();
    To copy to clipboard, switch view to plain text mode 
    So I cannot allocate space. Instead I know maximum lenght (dimension) of one line. So I can allocate the size for string line.
    Then, I have to sort only line by line (not the words inside a line and then line by line)
    2. After that, in your opinion, can I use
    Qt Code:
    1. memblock = new char [size];
    2. ifile.read (memblock, size);
    To copy to clipboard, switch view to plain text mode 
    to do all work ( I still understand it'll more speed) ??
    Regards

  20. #20
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: reading from a file

    Quote Originally Posted by fullmetalcoder View Post
    This doesn't make much sense...
    What doesn't make sense exactly?

    Quote Originally Posted by fullmetalcoder View Post
    In such cases the best way is to read pieces of the file, perform some tasks and then discard them before loading some other new pieces... It should be a little tricky to work this way to sort lines of text but it's still doable.
    Isn't that what I wrote in my two previous posts?

Similar Threads

  1. qt-3.3.8 fail in scratchbox
    By nass in forum Installation and Deployment
    Replies: 0
    Last Post: 25th May 2007, 16:21
  2. Interesting little Segfault w/r to signal/slot connection
    By Hydragyrum in forum Qt Programming
    Replies: 24
    Last Post: 12th September 2006, 20:22
  3. Reading a unicode names from a file???
    By darpan in forum Qt Programming
    Replies: 7
    Last Post: 3rd May 2006, 18:28
  4. Replies: 6
    Last Post: 27th February 2006, 13:47
  5. Problem with reading a file
    By Buhmann in forum Qt Programming
    Replies: 11
    Last Post: 17th February 2006, 14:02

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.