Recursive directory scan with abort function in a non blocking manner

[Lykurg]

Hi,

this is the task: Scan a folder recursive for - lets say - png files and give their addresses back in a QStringList. Furthermore the count of the found images should be displayed in a label or whatever during the scan. And of course the gui is not allowed to freeze and you should be able to abort the scan.

And the solution is ... ? Or better asked, what is the best, easiest and best-performance solution?

Right now I have a simple subclasses thread and do like:

class ThreadReadDir : public QThread
{
    Q_OBJECT
  public:
    explicit ThreadReadDir(QObject* parent = 0)
      : QThread(parent)
    {
      m_stopped = false;
    }
 
    void stop()
    {
      m_stopped = true;
    }
 
    void readDir(QDir dir)
    {
      m_dir = dir;
      if (!m_stopped)
        start();
    }
 
    void run()
    {
      readDir_private(m_dir);
    }
 
    QStringList files() const
    {
      return m_files;
    }
 
  Q_SIGNALS:
    void currentFileCountChanged(int);
 
  private:
    QStringList m_files;
    QDir m_dir;
    bool m_stopped;
 
    void readDir_private(QDir dir)
    {
      if (m_stopped)
        return;
      QStringList files = dir.entryList(QStringList() << "*.png", QDir::Files, QDir::Name);
      for (int i = 0; i < files.count(); ++i)
        m_files << dir.absolutePath() + QDir::separator() + files.value(i);
      Q_EMIT currentFileCountChanged(m_files.count());
 
      if (m_stopped)
        return;
      QStringList directories = dir.entryList(QStringList(), QDir::AllDirs | QDir::NoDotAndDotDot, QDir::Name);
      for (int i = 0; i < directories.count(); ++i)
      {
        if (m_stopped)
          return;
        readDir_private(dir.absolutePath() + QDir::separator() + directories.at(i));
      }
    }
};

class ThreadReadDir : public QThread
{
    Q_OBJECT
  public:
    explicit ThreadReadDir(QObject* parent = 0)
      : QThread(parent)
    {
      m_stopped = false;
    }

    void stop()
    {
      m_stopped = true;
    }

    void readDir(QDir dir)
    {
      m_dir = dir;
      if (!m_stopped)
        start();
    }

    void run()
    {
      readDir_private(m_dir);
    }

    QStringList files() const
    {
      return m_files;
    }

  Q_SIGNALS:
    void currentFileCountChanged(int);

  private:
    QStringList m_files;
    QDir m_dir;
    bool m_stopped;

    void readDir_private(QDir dir)
    {
      if (m_stopped)
        return;
      QStringList files = dir.entryList(QStringList() << "*.png", QDir::Files, QDir::Name);
      for (int i = 0; i < files.count(); ++i)
        m_files << dir.absolutePath() + QDir::separator() + files.value(i);
      Q_EMIT currentFileCountChanged(m_files.count());

      if (m_stopped)
        return;
      QStringList directories = dir.entryList(QStringList(), QDir::AllDirs | QDir::NoDotAndDotDot, QDir::Name);
      for (int i = 0; i < directories.count(); ++i)
      {
        if (m_stopped)
          return;
        readDir_private(dir.absolutePath() + QDir::separator() + directories.at(i));
      }
    }
};

To copy to clipboard, switch view to plain text mode 

With

// Constructor
ThreadReadDir* m_filethread = new ThreadReadDir(this);
QObject::connect(m_filethread, SIGNAL(currentFileCountChanged(int)), this, SLOT(setFileCount(int)));
 
// After scan request
m_filethread->readDir(QDir(/*...*/));
 
//
void XXX::setFileCount(int cnt)
{
  ui->statRead->setText(QString("%1 Files").arg(cnt));
}
 
//
void XXX::OnAbortRequest()
{
  m_filethread->stop();
  m_filethread->wait();
}

// Constructor
ThreadReadDir* m_filethread = new ThreadReadDir(this);
QObject::connect(m_filethread, SIGNAL(currentFileCountChanged(int)), this, SLOT(setFileCount(int)));

// After scan request
m_filethread->readDir(QDir(/*...*/));

//
void XXX::setFileCount(int cnt)
{
  ui->statRead->setText(QString("%1 Files").arg(cnt));
}

//
void XXX::OnAbortRequest()
{
  m_filethread->stop();
  m_filethread->wait();
}

To copy to clipboard, switch view to plain text mode 

It works, but it is a hell of work for such a simple request and therefore I wonder if there is a easier and better to maintain way to achieve that. QtConcurrent::run would be fine if one could cancel it.


Thanks for sharing your thoughts,

Lykurg

[franz]

The QtConcurrent::map*() functions are cancellable through the QFuture object they return. Maybe it's possible to mock something up using that? (it would probably be a little hackish though )

[MTK358]

Not sure if this is relavent, but the file extension is NOT a certain way to tell the type of a file. For example i use Linux, and file extensions have no meaning to it. The only real reason for them is for the user to tell the file type easily, and because *some* programs pay attention to them.

Unlike in Windows, in Linux I can call a PNG file "image.jpg" and it will work just fine, because *nix and most *nix programs look at the contents of the file, not the extension.

michael@michael-desktop ~ $ file image.jpg
image.jpg: JPEG image data, JFIF standard 1.01                              
michael@michael-desktop ~ $ mv image.jpg image.png            
michael@michael-desktop ~ $ file image.png
image.png: JPEG image data, JFIF standard 1.01

michael@michael-desktop ~ $ file image.jpg
image.jpg: JPEG image data, JFIF standard 1.01                              
michael@michael-desktop ~ $ mv image.jpg image.png            
michael@michael-desktop ~ $ file image.png
image.png: JPEG image data, JFIF standard 1.01

To copy to clipboard, switch view to plain text mode 

[hobbyist]

I'd probably go with QThread as well. Something like

class Scanner : public QThread
{
  Q_OBJECT
 
public:
  explicit Scanner(const QDir &dir, QObject *parent = 0)
    : iterator(dir, QDirIterator::Subdirectories), QThread(parent), stopped(false)
  {
  }
 
  void run()
  {
    while(!stopped && iterator.hasNext()) {
      QString entry(iterator.next());
      if(QFileInfo(entry).suffix().toLower() == "png") {
	fileList << entry;
	emit fileCountChanged(fileList.count());
      }
    }
  }
 
  QStringList files() const
  {
    return fileList;
  }
 
public slots:
  void stop()
  {
    stopped = true;
  }
 
signals:
  void fileCountChanged(int);
 
private:
  QDirIterator iterator;
  QStringList fileList;
  bool stopped;
};

class Scanner : public QThread
{
  Q_OBJECT
  
public:
  explicit Scanner(const QDir &dir, QObject *parent = 0)
    : iterator(dir, QDirIterator::Subdirectories), QThread(parent), stopped(false)
  {
  }

  void run()
  {
    while(!stopped && iterator.hasNext()) {
      QString entry(iterator.next());
      if(QFileInfo(entry).suffix().toLower() == "png") {
	fileList << entry;
	emit fileCountChanged(fileList.count());
      }
    }
  }

  QStringList files() const
  {
    return fileList;
  }

public slots:
  void stop()
  {
    stopped = true;
  }

signals:
  void fileCountChanged(int);

private:
  QDirIterator iterator;
  QStringList fileList;
  bool stopped;
};

To copy to clipboard, switch view to plain text mode 

Then simply start(), stop() and wait(), as in the original code snippet above.

[Lykurg]

Oh, I love Qt! And I should probably look more often in the docs, also for working solutions since I wasn't aware of QDirIterator. This will save me doing the recursive function. Nice! Thanks for pointing that out. (One small improvement if somebody want use it later: since the iterator uses a QFileInfo internal better use that (QDirIterator::fileInfo) instead of creating a second one.)

@franz: the map* functions are nice but personal I find them hard to maintain (especial in that case). A capsuled Thread is also easier to reuse.
@MTK358: The suffixes are of course meaningless. If you want be sure that it is really a png or any other file you have to do "deeper" tests.


And one follow up: Did you normally subclass QThread or use QObject::moveToThread()? (I am just curious how others did it.)

[MTK358]

The suffixes are of course meaningless. If you want be sure that it is really a png or any other file you have to do "deeper" tests.

Yes, sure. If I would make a simple example I would just have it search for *.png, too. Anyway, if it matters, in Wikipedia it says that a PNG file should always begin with:

89 50 4E 47 0D 0A 1A 0A

89 50 4E 47 0D 0A 1A 0A

To copy to clipboard, switch view to plain text mode