reading from a file

[mickey]

std::fstream file;
std::fstream ifile = file.open(file_name.c_str();

std::fstream file;
std::fstream ifile = file.open(file_name.c_str();

To copy to clipboard, switch view to plain text mode 

1.uhm, but anyway the 'get' cause a crash at runtime....
2.But is there a way to assign in one "hit" all text file to a char*? (I thought use a vector<string> too, but If I have a very large file and large lines maybe vector goes slow...)
thanks

[jacek]

the 'get' cause a crash at runtime....

How did you allocate the buffer?

But is there a way to assign in one "hit" all text file to a char*? (I thought use a vector<string> too, but If I have a very large file and large lines maybe vector goes slow...)

You could use seekg() and tellg() to check the file size first and then use a large enough buffer, but it might lead to race condition if some other process is modifying that file.

[mickey]

thanks for reply;
I just see for the tellp/g solution and it seems ok; But could I put file into vector<string> too? (I mean: is it efficient a vector? eg: my file is 2GB size large)

[jacek]

is it efficient a vector?

If you are going to use push_front/push_back methods it won't be. On the other hand, if you are going to allocate it in one go, it won't differ much from reading data line by line.

eg: my file is 2GB size large

How much RAM do you have in your machine? You certainly don't want your machine to start swapping.

If you want to optimize something, first use a profiler to check where the bottleneck is.

Remember: "premature optimization is the root of all evil".

[mickey]

Hi I have 512mb. and with this below my machine swapping! So haven't I hoping to use file larger than my Ram?

ifile.read (memblock, size);

ifile.read (memblock, size);

To copy to clipboard, switch view to plain text mode 

Sorry: are you saying that working on "memblock" is much less efficient than using push_back to put the file in a vector?

thanks

[jacek]

Hi I have 512mb. and with this below my machine swapping! So haven't I hoping to use file larger than my Ram?

You can use files larger than your RAM, but in parts that fit in the memory. Getting data from disk is *a lot* slower than getting it from the memory, so you should avoid swapping at all cost.

are you saying that working on "memblock" is much less efficient than using push_back to put the file in a vector?

On the contrary. push_back() might result in relocation of all of the data in the vector.

[mickey]

hi, What does that mean?

...... but in parts that fit in the memory.

So is "memblock = new char [size];" is the best solution?
thanks

[jacek]

What does that mean?

If you have 512MB of RAM, you shouldn't try to read more than that.

So is "memblock = new char [size];" is the best solution?

std::vector and std::string will be OK too, but only if you won't change their size too often.

[mickey]

hi, I tried read with the solution above with memblock on a 200mb text file. it's goes fast.
Then I tried to read file and put it into a vector:

vector<string> text;
copy (it_file, eos, inserter (text, text.begin()) );

vector<string> text;
copy (it_file, eos, inserter (text, text.begin()) );

To copy to clipboard, switch view to plain text mode 

this goes very slow (I didn't see the end of the copy!)
I tried this operation of "copy" because I need to sort the lines of the file too. Then I thought to put the lines in a vector and then call sort() on it. But the "reading" is too slow and I need speed.
I'm thinking of implement my own sort on memblock. But maybe it won't so fast as vector.sort().......What do you suggest, please?

[jacek]

What is the format of the file you want to read and what operations you need to perform?

[mickey]

hi, my file contains lines of strings (it's a text); At the moment I'd like order every line of the file ( line1 < line2)

[jacek]

First of all there's a small application called sort that will do that for you, but I'm not sure if it's available on windows.

If you still want to do it yourself, it sounds like a job for merge sort. You can save some time by starting with the biggest blocks that fit into your RAM and sorting them using quick sort or similar algorithm.

If you know the maximum line length or at least its approximate value, you can create std::vector< std::string > of some constant size and preallocate space in every string (you can do that using two-parameter version of std::vector's constructor).

[fullmetalcoder]

If you have 512MB of RAM, you shouldn't try to read more than that.

This doesn't make much sense... In most cases (users who dropped text-based interface) the OS/Desktop/backgroud tasks... occupy about half of the available memory (sometimes more). Thus it is highly recommened not to load a full file which is bigger than 40% of your available memory (yet it is possible). In such cases the best way is to read pieces of the file, perform some tasks and then discard them before loading some other new pieces... It should be a little tricky to work this way to sort lines of text but it's still doable.

[mickey]

OK! But I don't still understand this:
1. I don't know the number of the lines ( I know only its dimesion with instructions below)

ifile.seekg (0, ios::end);
size = ifile.tellg();

ifile.seekg (0, ios::end);
size = ifile.tellg();

To copy to clipboard, switch view to plain text mode 

So I cannot allocate space. Instead I know maximum lenght (dimension) of one line. So I can allocate the size for string line.
Then, I have to sort only line by line (not the words inside a line and then line by line)
2. After that, in your opinion, can I use

memblock = new char [size];
ifile.read (memblock, size);

memblock = new char [size];
ifile.read (memblock, size);

To copy to clipboard, switch view to plain text mode 

to do all work ( I still understand it'll more speed) ??

[jacek]

1. I don't know the number of the lines ( I know only its dimesion with instructions below)

If you are going to use merge sort, you don't have to know this.

After that, in your opinion, can I use
[...]
to do all work ( I still understand it'll more speed) ??

The version with std::vector will be easier to implement and you if preallocate memory, it shouldn't be significantly slower.

[mickey]

EDIT: OK. Such as other user, I couldn't understand the trick to work on block of 256k

[mickey]

The thing becames interesting....

The version with std::vector will be easier to implement and you if preallocate memory, it shouldn't be significantly slower.

I still haven't clear a point: How do I put lines of file X in the vector line?

while( getline (ifile, temp, '\n') ) {
		lines[l] = temp;
                        l++;
}

while( getline (ifile, temp, '\n') ) {
		lines[l] = temp;
                        l++;
}

To copy to clipboard, switch view to plain text mode 

When I speak of "speed" I mean that I know read a file line by line isn't so efficent..(!?)
Then you said to work on 3 files: but file access and work on files is so efficent? (maybe it'll be better use 2 vector instead of 2 files?)
thanks

[jacek]

This doesn't make much sense...

What doesn't make sense exactly?

In such cases the best way is to read pieces of the file, perform some tasks and then discard them before loading some other new pieces... It should be a little tricky to work this way to sort lines of text but it's still doable.

Isn't that what I wrote in my two previous posts?

[fullmetalcoder]

What doesn't make sense exactly?

Telling that a file bigger than memory shouldn't be loaded in order to prevent swap use because the "swap limit" is actually much lower than the actual physical memory size...

[jacek]

Telling that a file bigger than memory shouldn't be loaded in order to prevent swap use because the "swap limit" is actually much lower than the actual physical memory size...

In other words, you say that upper bound is incorrect, because it's too big?