QTextStream readLine() problem

[walito]

Hi guys..
I have some problems reading a file.
always I run the program, this crash in the same position of file.
I used gdb to debug and I had this error:

#0 0x00002aaaabd4c40b in memcpy () from /lib64/libc.so.6
#1 0x00002aaaaaf9c172 in QString::append (this=0x639260, str=@0x7fff529efe10) at /usr/include/bits/string3.h:52
#2 0x00002aaaaafe4277 in QTextStreamPrivate::fillReadBuffer (this=0x6391d0, maxBytes=-1) at ../../src/corelib/tools/qstring.h:255
#3 0x00002aaaaafe4f25 in QTextStreamPrivate::scan (this=0x6391d0, ptr=0x7fff529f3f08, length=0x7fff529f3f1c, maxlen=0, delimiter=QTextStreamPrivate::EndOfLine) at io/qtextstream.cpp:696
#4 0x00002aaaaafe5a81 in QTextStream::readLine (this=<value optimized out>, maxlen=0) at io/qtextstream.cpp:1584
#5 0x0000000000401ed2 in main (argc=2, argv=0x7fff529f41b8) at main.cpp:53


the problem, I think is in.readLine() but I don't know why.

Could you help me?




if(sql_file.open(QIODevice::ReadOnly | QIODevice::Text))
{
QTextStream in(&sql_file);
QString line;
int pos = 0;

do {
line = in.readLine(); <------- problem here
qDebug() << "pos: " << pos << " length: " << line.length();

++pos;
//process_line(line);
} while (!line.isNull());

sql_file.close();
}


Thanks in advance

[high_flyer]

What do you do with pos? Do you use it to seek in the file?
Can you show more code responsible for reading and navigating in file?

[walito]

yes, of course.
The "pos" var is only to see something and always crash in the same position.



QCoreApplication a(argc, argv);

QStringList args = QCoreApplication::arguments();

QRegExp rxArgPath("^--sql_path=(.*)");

for (int i = 1; i < args.size(); ++i)
{
//qDebug() << args.at(i);

if(rxArgPath.indexIn(args.at(i), 0) == 0)
{
if(!rxArgPath.cap(1).isEmpty())
{
QFile sql_file(rxArgPath.cap(1));

if(sql_file.exists())
{
if(sql_file.open(QIODevice::ReadOnly | QIODevice::Text))
{
qDebug() << "reading file " << rxArgPath.cap(1);

QTextStream in(&sql_file);
QString line;
int pos = 0;

do {
line = in.readLine();
qDebug() << "pos: " << pos << " length: " << line.length();

++pos;
} while (!line.isNull());

sql_file.close();
}
}
else
{
qWarning() << "Er: the file " << rxArgPath.cap(1) << "not exists";
}
}
else
{
qWarning() << "Er: Add the path --sql_path";
}
}
}

return a.exec();


thanks in advance

[high_flyer]

Ahh... this all thing is in your main...
I think that working with QIODevice needs a running event loop (but I am not sure).
Put this in to a class, and run this code after the event loop is running, that will be a good idea even if its not the problem.
Maybe some one else can confirm if this is the problem though.

EDIT:
[re-edit] I take back what I said here.

[walito]

thz for your reply.

I moved the code into a class, and I have the same problem :(

The file size is 39G, but Im reading line by line, Ithink should have no problem.

#0 0x00002aaaabfc640b in memcpy () from /lib64/libc.so.6
#1 0x00002aaaab216172 in QString::append (this=0x6387d0, str=@0x7fff838bbc80) at /usr/include/bits/string3.h:52
#2 0x00002aaaab25e277 in QTextStreamPrivate::fillReadBuffer (this=0x638740, maxBytes=-1) at ../../src/corelib/tools/qstring.h:255
#3 0x00002aaaab25ef25 in QTextStreamPrivate::scan (this=0x638740, ptr=0x7fff838bfd78, length=0x7fff838bfd8c, maxlen=0, delimiter=QTextStreamPrivate::EndOfLine) at io/qtextstream.cpp:696
#4 0x00002aaaab25fa81 in QTextStream::readLine (this=<value optimized out>, maxlen=0) at io/qtextstream.cpp:1584
#5 0x0000000000402831 in readFile::read (this=0x7fff838bfecf, file_path=@0x7fff838bff60) at read_file.cpp:34
#6 0x0000000000401e1b in main (argc=2, argv=0x7fff838c0078) at main.cpp:40

Added after 35 minutes:

mm I found the problem..

I modified this line :

line = in.readLine();
to this line:

line = in.readLine(100);

mm, the line is very large, how can I read the big line?

Thanks

[ChrisW67]

The size of the line is not the cause of this problem (unless the line length overflows a size_t). The following code, for example, has no issues other than speed:

#include <QtGui>
#include <QDebug>

int main(int argc, char *argv[])
{
QApplication app(argc, argv);

// 40 MB of the letter A
QByteArray b(40 * 1024 * 1024, 'A');
QFile f("test.txt");
if (f.open(QIODevice::WriteOnly)) {
f.write(b);
f.close();
}

// Read it line at a time
if (f.open(QIODevice::ReadOnly | QIODevice::Text)) {
QTextStream s(&f);
QString line;
line = s.readLine();
qDebug() << line.length() << line.left(10) << line.right(10);
}
}


Does the problem persist if you completely rebuild the application? That is, is this a binary mismatch between components of the compiled program.

Edit: I misread your earlier post. If the file is 39 gigabytes and all one line, i.e. no line breaks, and size_t is 32-bits then this may be the issue. This would be an exceptionally unusual text file though. If any one line is longer than the largest RAM allocation available then you will have problems too.

[high_flyer]

Do you have line breaks in the file?
As far as I know, QTextStream can identify both linux and windows break lines, so no matter what line break encoding you have, it should still get it right...

What you can also try is to go more low level and use QIODevice::readLine() - see the docs, they have an example too.
This will allow you to make a small loop, that will read X bytes per read until the whole line is read.
Depending of what you need to do with the data you read, you can either parse the line parts in each iteration, and reuse your buffer, or, if you need the full string at hand, you can store the read parts in a QStringList for example and parse these...
The exact way of doing this depends on exactly what you want to do with the read data.

[walito]

thz for your reply.

I was testing and I think the problem is the memory, because the readline() method don't free him buffer and the program crash (memcpy)

How can I free the memory?

I'm doing "watch -n1 free -m" and the memory start to down until the program crash.
I thought the readline() free the memory

Thanks in advance

[high_flyer]

you didn't answer a very important question:
do you have line breaks in the file?

[walito]

yes yes, exists \n at the end of each row.

that's weird

[ChrisW67]

I was testing and I think the problem is the memory, because the readline() method don't free him buffer and the program crash (memcpy)

Have you any evidence for this assertion?


How can I free the memory?

Allow the QTextStream and QString to go out of scope.


I'm doing "watch -n1 free -m" and the memory start to down until the program crash.

Something in your program, that is not in the code we have been shown is allocating memory and not freeing it: perhaps storing data about each line. Eventually, a memory allocation fails and the attempt to copy something into it crashes. That this occurs inside Qt code does not make it a Qt problem.

If you want to test for actual memory leaks, which are exceptionally unlikely in the relevant Qt code, then use valgrind or something similar. System level tools like free are not useful in finding memory leaks. See this thread for more:
http://www.qtcentre.org/threads/37424-300KB-of-RAM-lost-after-New-and-Delete?highlight=free+memory+leak


I thought the readline() free the memory

QTextStream and QString both manage their memory.


#include <QtCore>
#include <QDebug>

int main(int argc, char *argv[])
{
QCoreApplication app(argc, argv);

// 3 million lines of 100 characters into temp file
QFile f("/tmp/test.txt");
QByteArray data(99, 'A');
data.append('\n');
qDebug() << "Writing";
if (f.open(QIODevice::WriteOnly)) {
for (int line = 0; line < 3000000; ++line) {
f.write(data);
}
f.close();
}

// Read it line at a time
qDebug() << "Reading...";
if (f.open(QIODevice::ReadOnly | QIODevice::Text)) {
QTextStream s(&f);
QString line;
while (!f.atEnd())
line = s.readLine();
f.close();
}
}


$ valgrind --leak-ckeck=yes ./test
...
Writing
Reading...
==6051==
==6051== HEAP SUMMARY:
==6051== in use at exit: 10,966 bytes in 144 blocks
==6051== total heap usage: 3,055,495 allocs, 3,055,351 frees, 3,058,183,080 bytes allocated
==6051==
...
==6051== LEAK SUMMARY:
==6051== definitely lost: 0 bytes in 0 blocks // NO LEAKS
==6051== indirectly lost: 0 bytes in 0 blocks //
==6051== possibly lost: 3,136 bytes in 29 blocks
==6051== still reachable: 7,830 bytes in 115 blocks
==6051== suppressed: 0 bytes in 0 blocks
==6051== Reachable blocks (those to which a pointer was found) are not shown.
==6051== To see them, rerun with: --leak-check=full --show-reachable=yes
==6051==
==6051== For counts of detected and suppressed errors, rerun with: -v
==6051== Use --track-origins=yes to see where uninitialised values come from
==6051== ERROR SUMMARY: 19 errors from 18 contexts (suppressed: 0 from 0)

All of the small possible 'leaks' here are coming from libraries outside Qt (glib in my case).

Have you tried rebuilding the entire application yet?
Can you produce a small test program and set of data that breaks for you?

[walito]

Thanks for your reply.

Nice program ( valgrind ) I'll try to test.

so, I solved the problem using QFile::readline() method, is that's ok? work fine for me.

I'll try testing using QTextStream::readline(), I need know the problem.

Thanks all for your help.

I'll keep you posted