PDA

View Full Version : QtConcurrent mappedReduced QFuture Core Dump



shawno
16th July 2010, 03:29
I'm finding all files and then grepping them for a regular expression. I count the times the regular expression occurs in a given file in the background using QFuture and QFutureWatcher with QtConcurrent::mappedReduced(). The context of the call to mappedReduced() is as follows:


m_hitv[idx] = new Hits;
connect(&m_hitv[idx]->watcher, SIGNAL(finished()), this, SLOT(hitMapReady()));
m_hitv[idx]->futureHits = QtConcurrent::mappedReduced(paths, &Mapped::countHits, &Mapped::reduceHits);
m_hitv[idx]->watcher.setFuture(m_hitv[idx]->futureHits);

The implementation of class Mapped is:


/************************************************** ****************************
* NAME: mapped.cpp
*
* PURPOSE: Functor parameters for QtConcurrent::mappedReduced()
*
* DATE: June 29, 2010
************************************************** *****************************/
#include "debug.h"

#include <QString>
#include <QFile>
#include <QRegExp>
#include <QIODevice>
#include <QTextStream>
#include <QThread>

#include "mapped.h"
#include "tab.h"

/************************************************** ****************************
* Mapped(): Create a Mapped object.
************************************************** ****************************/
Mapped::Mapped(const QString &_abspath)
:
tab(NULL),
m_abspath(_abspath)
{}
/************************************************** ****************************
* CountHits(): Count lines in file that contain at least one pattern (m_text).
************************************************** ****************************/
WordCount Mapped::countHits (const Mapped &_mapped) {
qd(4) << "countHits(): " << "this thread is " << QThread::currentThread();
QString path = _mapped.getAbspath();
QFile file(path);

QRegExp rx(_mapped.getPtrn());
rx.setPatternSyntax(_mapped.getSntx());
rx.setCaseSensitivity(_mapped.getKase());
const QRegExp crx(rx); // performance tweak

int m = 0;
WordCount wc;
wc[path] = 0;

if (file.open(QIODevice::ReadOnly)) {
QString line;
QTextStream in(&file);
while (!in.atEnd()) {
if (_mapped.getTab()->getAbortMapped()) {
qd(6) << "aborted counting hits on " << _mapped.getTab();
break;
}
if (m == INT_MAX)
m = 0;
if (++m % 100000 == 0)
qd(6) << "thread " << QThread::currentThread() << " counting hits...";

line = in.readLine();

if (crx.indexIn(line, 0) != -1)
wc[path] += 1;
}
}
else
fprintf(stderr, "could not open %s", path.toStdString().c_str());

return wc;
}
/************************************************** ****************************
* ReduceHits(): reduce hit maps to one map to return to caller.
************************************************** ****************************/
void Mapped::reduceHits (WordCount &result, const WordCount &w) {
QMapIterator<QString, int> i(w);
while (i.hasNext()) {
i.next();
result[i.key()] += i.value();
}
}

Even though it may take the application 2-3 hours of chugging through my C drive, I ALWAYS get a core dump (on a random file and folder) that occurs in:


line = in.readLine();

due to a a corrupt QString a little higher in the stack. All I can think of is the fact that the object, which lives in the main thread, that calls mappedReduced() has a QThreadPool member, which it uses to call the run() function of a couple different child objects. So, would there be a problem if an object living in the main thread creates a QFuture on a call to mappedReduced() AND also happens to own its own QThreadPool which it uses to farm threads out to various child objects? One of these child objects runs in the thread that signals back to the main thread slot that calls mappedReduced().

The only other relevant info, possibly, is that my app is resource-intensive because it traverses the filesystem and searches inside every file on the filesystem for a regular expression. The number of QFutureWatcher objects that pile up viz. have not yet signalled with ResultReadyAt() may vary from 100 to 5000 depending on the number of threads I allow.

I tried wrapping the line in which the core occurs in a mutex to no avail. What am I doing wrong?