QFile::copy copies only 3kb

[camol]

Hi I've noticed a very strange problem a use QFileSystemWatcher to monitor one folder, when new files are copied to this folder the signal directoryChanged is emitted. I wrote a slot, which works well, it gives me the confidence that when ALL files are copied to the folder only then the right action is taken:

void DServer::newFiles_()
{
    QDir input(QDir::homePath() + "/INPUT DICOM");
 
    if(input.count() > lastInput)
    {
        QTextStream(stdout)<<"not all files" << lastInput <<endl;
        lastInput = input.count();
        return;
    }
    else
    {
        QDir inputDir(QDir::homePath() + "/INPUT DICOM");
        QTextStream(stdout) << "BEFORE IF   "<<inputDir.count() <<endl;
        if(inputDir.count() > 1)
        {
            QTextStream(stdout)<<"Got all files"<< lastInput <<endl;
            inputFiles = getPaths(QDir::homePath() + "/INPUT DICOM");
            emit this->_moveFiless();
            lastPatientName.clear();
            lastInput = 1;
        }
        else
            return;
    }
}

void DServer::newFiles_()
{
    QDir input(QDir::homePath() + "/INPUT DICOM");

    if(input.count() > lastInput)
    {
        QTextStream(stdout)<<"not all files" << lastInput <<endl;
        lastInput = input.count();
        return;
    }
    else
    {
        QDir inputDir(QDir::homePath() + "/INPUT DICOM");
        QTextStream(stdout) << "BEFORE IF   "<<inputDir.count() <<endl;
        if(inputDir.count() > 1)
        {
            QTextStream(stdout)<<"Got all files"<< lastInput <<endl;
            inputFiles = getPaths(QDir::homePath() + "/INPUT DICOM");
            emit this->_moveFiless();
            lastPatientName.clear();
            lastInput = 1;
        }
        else
            return;
    }
}

To copy to clipboard, switch view to plain text mode 

However this slot

void DServer::moveFiless_()
{
    QString input("/INPUT DICOM");
    QDate date = QDate::currentDate();
    QTime time = QTime::currentTime();
 
    for(int i=0; i<inputFiles.count(); i++)
    {
        QString old_input_list = inputFiles[i];
        inputFiles[i].remove(0,(QDir::homePath().count()) + input.count());
 
        int length = inputFiles[i].length();
        int startindex = inputFiles[i].lastIndexOf("/",-1) + 1;
        QString dir_path = inputFiles[i];
 
        dir_path.remove(startindex, length);
 
        QDir dir;
        if(dir.mkpath(QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path) == true)
        {
            //anonymize here
            anonymous("/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i], old_input_list);
            dir.mkpath(QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path);
date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + input_list[i] + ".tmp");
 
            QFile::copy(old_input_list, (QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i] + ".tmp"));
            QFile::copy(old_input_list, (QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i]));
            QFile::remove(old_input_list);
            QTextStream(stdout)<<"source file:  "<< old_input_list <<endl;
            dir.rmpath(QDir::homePath() + "/INPUT DICOM" + dir_path);
        }
        else
        {
            QTextStream(stdout) <<"Couldn't make this dir: ' "<< QDir::homePath() + "/DICOM STORAGE" + dir_path <<endl;
        }
    }
}

void DServer::moveFiless_()
{
    QString input("/INPUT DICOM");
    QDate date = QDate::currentDate();
    QTime time = QTime::currentTime();

    for(int i=0; i<inputFiles.count(); i++)
    {
        QString old_input_list = inputFiles[i];
        inputFiles[i].remove(0,(QDir::homePath().count()) + input.count());

        int length = inputFiles[i].length();
        int startindex = inputFiles[i].lastIndexOf("/",-1) + 1;
        QString dir_path = inputFiles[i];

        dir_path.remove(startindex, length);

        QDir dir;
        if(dir.mkpath(QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path) == true)
        {
            //anonymize here
            anonymous("/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i], old_input_list);
            dir.mkpath(QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path);
date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + input_list[i] + ".tmp");

            QFile::copy(old_input_list, (QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i] + ".tmp"));
            QFile::copy(old_input_list, (QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i]));
            QFile::remove(old_input_list);
            QTextStream(stdout)<<"source file:  "<< old_input_list <<endl;
            dir.rmpath(QDir::homePath() + "/INPUT DICOM" + dir_path);
        }
        else
        {
            QTextStream(stdout) <<"Couldn't make this dir: ' "<< QDir::homePath() + "/DICOM STORAGE" + dir_path <<endl;
        }
    }
}

To copy to clipboard, switch view to plain text mode 

doesn't work well, the files from source are copied but they only have from 2.6-3.0 kB, when the original ones have about 80kB, that means the QFile::copy works but not good.

[JohannesMunk]

Just reread your post..

Is "if(input.count() > lastInput)" really correct? Don't you expect the input.count() to increase until all files have been copied?

In that case you have to keep waiting while it is below the threshold lastinput.

Joh

[camol]

But the signal directoryChanged() is emitted many times during copying the files to the monitored folder and as far as I know I'm interested in the last one becouse it indicates that the last file occured in the watched folder and after that I am performing the copy action. if(input.count > lastinput) informs if current number of files in the monitored directory is greater then it was during the last run of this slot, so that means the files are still being copied to the folder.

I tried even sleep for 5 seconds to be sure that all files are in the directory but still all files are copied but only 2.8kB of them.

[JohannesMunk]

Ah! Sorry about that. Now I get it. But it wont work like that, because QFileSystemWatcher fires also when a file is modified. Thus during the copy process your method will be executed multiple times for each file. The watcher supresses to frequent updates but nonetheless: You will get updates where the filecount has not changed but only the file has been modified (a few additional bytes have been copied). Thus you think the file-list is already copied completely and make a copy of incomplete files.

Joh

[camol]

Do you have any idea to solve that? I really need this functionality which I have described above

[wysota]

Connect a timer to the file watcher and reset the timer whenever you receive a signal from the watcher. At some point you will stop receiving signals and the timer will timeout. At this point you will know that for some time there haven't been any updates to the file. This is not foul proof but should be enough for you.

[camol]

Can you bring some example code of this becouse I am trying to imagine the situation but I can't figure how it should work this. Big thanks in advance for your time here for both of you.

[wysota]

void Class::setup() {
  timer.setSingleShot(true);
  timer.setInterval(1000); // 1s
  connect(...);
}
 
void Class::onSignalFromFileSystemWatcher() {
  timer.start();
}
 
void Class::onTimerTimeout() {
  QFile::copy(...);
}

void Class::setup() {
  timer.setSingleShot(true);
  timer.setInterval(1000); // 1s
  connect(...);
}

void Class::onSignalFromFileSystemWatcher() {
  timer.start();
}

void Class::onTimerTimeout() {
  QFile::copy(...);
}

To copy to clipboard, switch view to plain text mode 

[camol]

It didn't work still is copied only about 3kB of each file, and also this solution makes sth that I wanted to prevent my application from. Becouse I copy the files from monitored folder to folder created which name is the current time and now for example if I put many files in the monitored folder the whole operation lasts long and the result of that is 2-3 folders with names with different times. My code above didn't do this it only created one folder for each operation of putting files to the monitored folder

[JohannesMunk]

I would suggest, you drop this. And provide the user with a simple button, where he can trigger an import after having copied all files.

Your concept of monitoring will fail because of so many things. How can you be sure, that the user will not copy files in several packets? That the source medium won't take 10secs to spin up. That the user realizes he has got the wrong files...

What is this for anyway?

Joh

[camol]

I am building a server aplication which runs and stays on all the time. It stores files in hierachy of folders thats why when a user brings new files for the server the server is automaticaly archiving them in the right order without any other actions from the user becouse it is an non-gui application. That it is why I need it so much.

I've tried this at the beginning of the slot connected to directoryChanged:
this->inputWatcher->blockSignals(true); //to prevent more the one signal
sleep(40); //to make sure that all files are in the watched folder


but still it doesn't work


I don't get it I tried even to make this functionality with a timer but it still the same

[camol]

Does anybody know what is going on? I really need this.

[SixDegrees]

Try simplifying your code dramatically so it only copies a single file. Or just write a small test program to do that, without the timers and other drivers. See if that works. Then build up to what you have now.

I'd also examine the path strings you're constructing, although it sounds like those probably work if the files are produced at all.

Reworking the copy routine in plain vanilla C++ is another alternative that would involve less than a dozen lines of code.

[camol]

I think file by file won't work also becouse I've tried putting files in folders and in subfolders etc, and the signal directory change wasn't emited as many times, and also doing file by file will create many time name folders like I described above. Strings of paths are done ok becouse the files are succesfully copied to the destination folder in right order but only the size of copied files is invalid.

[SixDegrees]

The point is to break your procedure down into small, testable pieces and verify that all of the individual bits are working as expected. You could also try replacing the QFile.copy() call with a system call to your OS's 'cp' or 'mv' command, with the same intention. I'd also try printing out every filename as your innermost loop is executed; it probably won't reveal any flaws, but it costs you nothing to try and it might illuminate something.

[camol]

Still the same I've tried the system cp but the result was the same. Even many different combinations of sleep and blocking the signal from QFileSystemWatcher also didn't help I'm doomed I really need this functionality ;(

[wysota]

To me it seems you are just starting the copy operation too early. You get correct timestamps but wrong behaviour. Use your filesystem api (QFileInfo::created() and friends) to determine the time but copy the file when it is not being written to anymore.

[JohannesMunk]

Assuming the user copies files only by one thread, the following would be a brute force solution to this problem:

The idea is to find out which file has actually been added from the current call to the last one. If one has been added, you can safely copy all others.

To do this, I suggest you store the list of files at the end of each call. When the current file count increased compared to that previously stored file list, you know, that all files of the previous file list are finished writing and can be copied. Maintain a list of already copied files to, to easily find out, which ones you actually need to copy at each step.

Joh

[squidge]

Still the same I've tried the system cp but the result was the same. Even many different combinations of sleep and blocking the signal from QFileSystemWatcher also didn't help I'm doomed I really need this functionality ;(

Blocking the signal? Surely you want all of the signals so you know when the directory/file has finished being updated so you can perform your operations after the last access? (after a set delay)

[camol]

I thought that if I use the code I wrote above and when I get into if condition with lastInput then I should block the signals from watcher in order to be sure that that signals won't be emmited when I am coping and deleting the source file. But it didnt' give the result