QFile::copy copies only 3kb

[camol]

Hi I've noticed a very strange problem a use QFileSystemWatcher to monitor one folder, when new files are copied to this folder the signal directoryChanged is emitted. I wrote a slot, which works well, it gives me the confidence that when ALL files are copied to the folder only then the right action is taken:

void DServer::newFiles_()
{
    QDir input(QDir::homePath() + "/INPUT DICOM");
 
    if(input.count() > lastInput)
    {
        QTextStream(stdout)<<"not all files" << lastInput <<endl;
        lastInput = input.count();
        return;
    }
    else
    {
        QDir inputDir(QDir::homePath() + "/INPUT DICOM");
        QTextStream(stdout) << "BEFORE IF   "<<inputDir.count() <<endl;
        if(inputDir.count() > 1)
        {
            QTextStream(stdout)<<"Got all files"<< lastInput <<endl;
            inputFiles = getPaths(QDir::homePath() + "/INPUT DICOM");
            emit this->_moveFiless();
            lastPatientName.clear();
            lastInput = 1;
        }
        else
            return;
    }
}

void DServer::newFiles_()
{
    QDir input(QDir::homePath() + "/INPUT DICOM");

    if(input.count() > lastInput)
    {
        QTextStream(stdout)<<"not all files" << lastInput <<endl;
        lastInput = input.count();
        return;
    }
    else
    {
        QDir inputDir(QDir::homePath() + "/INPUT DICOM");
        QTextStream(stdout) << "BEFORE IF   "<<inputDir.count() <<endl;
        if(inputDir.count() > 1)
        {
            QTextStream(stdout)<<"Got all files"<< lastInput <<endl;
            inputFiles = getPaths(QDir::homePath() + "/INPUT DICOM");
            emit this->_moveFiless();
            lastPatientName.clear();
            lastInput = 1;
        }
        else
            return;
    }
}

To copy to clipboard, switch view to plain text mode 

However this slot

void DServer::moveFiless_()
{
    QString input("/INPUT DICOM");
    QDate date = QDate::currentDate();
    QTime time = QTime::currentTime();
 
    for(int i=0; i<inputFiles.count(); i++)
    {
        QString old_input_list = inputFiles[i];
        inputFiles[i].remove(0,(QDir::homePath().count()) + input.count());
 
        int length = inputFiles[i].length();
        int startindex = inputFiles[i].lastIndexOf("/",-1) + 1;
        QString dir_path = inputFiles[i];
 
        dir_path.remove(startindex, length);
 
        QDir dir;
        if(dir.mkpath(QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path) == true)
        {
            //anonymize here
            anonymous("/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i], old_input_list);
            dir.mkpath(QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path);
date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + input_list[i] + ".tmp");
 
            QFile::copy(old_input_list, (QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i] + ".tmp"));
            QFile::copy(old_input_list, (QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i]));
            QFile::remove(old_input_list);
            QTextStream(stdout)<<"source file:  "<< old_input_list <<endl;
            dir.rmpath(QDir::homePath() + "/INPUT DICOM" + dir_path);
        }
        else
        {
            QTextStream(stdout) <<"Couldn't make this dir: ' "<< QDir::homePath() + "/DICOM STORAGE" + dir_path <<endl;
        }
    }
}

void DServer::moveFiless_()
{
    QString input("/INPUT DICOM");
    QDate date = QDate::currentDate();
    QTime time = QTime::currentTime();

    for(int i=0; i<inputFiles.count(); i++)
    {
        QString old_input_list = inputFiles[i];
        inputFiles[i].remove(0,(QDir::homePath().count()) + input.count());

        int length = inputFiles[i].length();
        int startindex = inputFiles[i].lastIndexOf("/",-1) + 1;
        QString dir_path = inputFiles[i];

        dir_path.remove(startindex, length);

        QDir dir;
        if(dir.mkpath(QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path) == true)
        {
            //anonymize here
            anonymous("/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i], old_input_list);
            dir.mkpath(QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path);
date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + input_list[i] + ".tmp");

            QFile::copy(old_input_list, (QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i] + ".tmp"));
            QFile::copy(old_input_list, (QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i]));
            QFile::remove(old_input_list);
            QTextStream(stdout)<<"source file:  "<< old_input_list <<endl;
            dir.rmpath(QDir::homePath() + "/INPUT DICOM" + dir_path);
        }
        else
        {
            QTextStream(stdout) <<"Couldn't make this dir: ' "<< QDir::homePath() + "/DICOM STORAGE" + dir_path <<endl;
        }
    }
}

To copy to clipboard, switch view to plain text mode 

doesn't work well, the files from source are copied but they only have from 2.6-3.0 kB, when the original ones have about 80kB, that means the QFile::copy works but not good.

[JohannesMunk]

Just reread your post..

Is "if(input.count() > lastInput)" really correct? Don't you expect the input.count() to increase until all files have been copied?

In that case you have to keep waiting while it is below the threshold lastinput.

Joh

[camol]

But the signal directoryChanged() is emitted many times during copying the files to the monitored folder and as far as I know I'm interested in the last one becouse it indicates that the last file occured in the watched folder and after that I am performing the copy action. if(input.count > lastinput) informs if current number of files in the monitored directory is greater then it was during the last run of this slot, so that means the files are still being copied to the folder.

I tried even sleep for 5 seconds to be sure that all files are in the directory but still all files are copied but only 2.8kB of them.

[JohannesMunk]

Ah! Sorry about that. Now I get it. But it wont work like that, because QFileSystemWatcher fires also when a file is modified. Thus during the copy process your method will be executed multiple times for each file. The watcher supresses to frequent updates but nonetheless: You will get updates where the filecount has not changed but only the file has been modified (a few additional bytes have been copied). Thus you think the file-list is already copied completely and make a copy of incomplete files.

Joh

[camol]

Do you have any idea to solve that? I really need this functionality which I have described above

[wysota]

Connect a timer to the file watcher and reset the timer whenever you receive a signal from the watcher. At some point you will stop receiving signals and the timer will timeout. At this point you will know that for some time there haven't been any updates to the file. This is not foul proof but should be enough for you.

[camol]

Can you bring some example code of this becouse I am trying to imagine the situation but I can't figure how it should work this. Big thanks in advance for your time here for both of you.

[wysota]

void Class::setup() {
  timer.setSingleShot(true);
  timer.setInterval(1000); // 1s
  connect(...);
}
 
void Class::onSignalFromFileSystemWatcher() {
  timer.start();
}
 
void Class::onTimerTimeout() {
  QFile::copy(...);
}

void Class::setup() {
  timer.setSingleShot(true);
  timer.setInterval(1000); // 1s
  connect(...);
}

void Class::onSignalFromFileSystemWatcher() {
  timer.start();
}

void Class::onTimerTimeout() {
  QFile::copy(...);
}

To copy to clipboard, switch view to plain text mode 

[camol]

It didn't work still is copied only about 3kB of each file, and also this solution makes sth that I wanted to prevent my application from. Becouse I copy the files from monitored folder to folder created which name is the current time and now for example if I put many files in the monitored folder the whole operation lasts long and the result of that is 2-3 folders with names with different times. My code above didn't do this it only created one folder for each operation of putting files to the monitored folder

[JohannesMunk]

I would suggest, you drop this. And provide the user with a simple button, where he can trigger an import after having copied all files.

Your concept of monitoring will fail because of so many things. How can you be sure, that the user will not copy files in several packets? That the source medium won't take 10secs to spin up. That the user realizes he has got the wrong files...

What is this for anyway?

Joh

[camol]

I am building a server aplication which runs and stays on all the time. It stores files in hierachy of folders thats why when a user brings new files for the server the server is automaticaly archiving them in the right order without any other actions from the user becouse it is an non-gui application. That it is why I need it so much.

I've tried this at the beginning of the slot connected to directoryChanged:
this->inputWatcher->blockSignals(true); //to prevent more the one signal
sleep(40); //to make sure that all files are in the watched folder


but still it doesn't work


I don't get it I tried even to make this functionality with a timer but it still the same