QFile::copy copies only 3kb

Hi I've noticed a very strange problem a use QFileSystemWatcher to monitor one folder, when new files are copied to this folder the signal directoryChanged is emitted. I wrote a slot, which works well, it gives me the confidence that when ALL files are copied to the folder only then the right action is taken:

Code:

---

void DServer::newFiles_()
{
    QDir input(QDir::homePath() + "/INPUT DICOM");
 
    if(input.count() > lastInput)
    {
        QTextStream(stdout)<<"not all files" << lastInput <<endl;
        lastInput = input.count();
        return;
    }
    else
    {
        QDir inputDir(QDir::homePath() + "/INPUT DICOM");
        QTextStream(stdout) << "BEFORE IF   "<<inputDir.count() <<endl;
        if(inputDir.count() > 1)
        {
            QTextStream(stdout)<<"Got all files"<< lastInput <<endl;
            inputFiles = getPaths(QDir::homePath() + "/INPUT DICOM");
            emit this->_moveFiless();
            lastPatientName.clear();
            lastInput = 1;
        }
        else
            return;
    }
}

---

However this slot

Code:

---

void DServer::moveFiless_()
{
    QString input("/INPUT DICOM");
    QDate date = QDate::currentDate();
    QTime time = QTime::currentTime();
 
    for(int i=0; i<inputFiles.count(); i++)
    {
        QString old_input_list = inputFiles[i];
        inputFiles[i].remove(0,(QDir::homePath().count()) + input.count());
 
        int length = inputFiles[i].length();
        int startindex = inputFiles[i].lastIndexOf("/",-1) + 1;
        QString dir_path = inputFiles[i];
 
        dir_path.remove(startindex, length);
 
        QDir dir;
        if(dir.mkpath(QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path) == true)
        {
            //anonymize here
            anonymous("/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i], old_input_list);
            dir.mkpath(QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path);
date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + input_list[i] + ".tmp");
 
            QFile::copy(old_input_list, (QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i] + ".tmp"));
            QFile::copy(old_input_list, (QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/"  + time.toString("hh:mm:ss") + inputFiles[i]));
            QFile::remove(old_input_list);
            QTextStream(stdout)<<"source file:  "<< old_input_list <<endl;
            dir.rmpath(QDir::homePath() + "/INPUT DICOM" + dir_path);
        }
        else
        {
            QTextStream(stdout) <<"Couldn't make this dir: ' "<< QDir::homePath() + "/DICOM STORAGE" + dir_path <<endl;
        }
    }
}

---

doesn't work well, the files from source are copied but they only have from 2.6-3.0 kB, when the original ones have about 80kB, that means the QFile::copy works but not good.

Just reread your post..

Is "if(input.count() > lastInput)" really correct? Don't you expect the input.count() to increase until all files have been copied?

In that case you have to keep waiting while it is below the threshold lastinput.

Joh

But the signal directoryChanged() is emitted many times during copying the files to the monitored folder and as far as I know I'm interested in the last one becouse it indicates that the last file occured in the watched folder and after that I am performing the copy action. if(input.count > lastinput) informs if current number of files in the monitored directory is greater then it was during the last run of this slot, so that means the files are still being copied to the folder.

I tried even sleep for 5 seconds to be sure that all files are in the directory but still all files are copied but only 2.8kB of them.

Ah! Sorry about that. Now I get it. But it wont work like that, because QFileSystemWatcher fires also when a file is modified. Thus during the copy process your method will be executed multiple times for each file. The watcher supresses to frequent updates but nonetheless: You will get updates where the filecount has not changed but only the file has been modified (a few additional bytes have been copied). Thus you think the file-list is already copied completely and make a copy of incomplete files.

Joh

Do you have any idea to solve that? I really need this functionality which I have described above :(

Connect a timer to the file watcher and reset the timer whenever you receive a signal from the watcher. At some point you will stop receiving signals and the timer will timeout. At this point you will know that for some time there haven't been any updates to the file. This is not foul proof but should be enough for you.

Can you bring some example code of this becouse I am trying to imagine the situation but I can't figure how it should work this. Big thanks in advance for your time here for both of you.

Code:

---

void Class::setup() {
  timer.setSingleShot(true);
  timer.setInterval(1000); // 1s
  connect(...);
}
 
void Class::onSignalFromFileSystemWatcher() {
  timer.start();
}
 
void Class::onTimerTimeout() {
  QFile::copy(...);
}

---

It didn't work still is copied only about 3kB of each file, and also this solution makes sth that I wanted to prevent my application from. Becouse I copy the files from monitored folder to folder created which name is the current time and now for example if I put many files in the monitored folder the whole operation lasts long and the result of that is 2-3 folders with names with different times. My code above didn't do this it only created one folder for each operation of putting files to the monitored folder

I would suggest, you drop this. And provide the user with a simple button, where he can trigger an import after having copied all files.

Your concept of monitoring will fail because of so many things. How can you be sure, that the user will not copy files in several packets? That the source medium won't take 10secs to spin up. That the user realizes he has got the wrong files...

What is this for anyway?

Joh

I am building a server aplication which runs and stays on all the time. It stores files in hierachy of folders thats why when a user brings new files for the server the server is automaticaly archiving them in the right order without any other actions from the user becouse it is an non-gui application. That it is why I need it so much.

I've tried this at the beginning of the slot connected to directoryChanged:
this->inputWatcher->blockSignals(true); //to prevent more the one signal
sleep(40); //to make sure that all files are in the watched folder


but still it doesn't work :(


I don't get it I tried even to make this functionality with a timer but it still the same

Does anybody know what is going on? :( I really need this.

Try simplifying your code dramatically so it only copies a single file. Or just write a small test program to do that, without the timers and other drivers. See if that works. Then build up to what you have now.

I'd also examine the path strings you're constructing, although it sounds like those probably work if the files are produced at all.

Reworking the copy routine in plain vanilla C++ is another alternative that would involve less than a dozen lines of code.

I think file by file won't work also becouse I've tried putting files in folders and in subfolders etc, and the signal directory change wasn't emited as many times, and also doing file by file will create many time name folders like I described above. Strings of paths are done ok becouse the files are succesfully copied to the destination folder in right order but only the size of copied files is invalid.

The point is to break your procedure down into small, testable pieces and verify that all of the individual bits are working as expected. You could also try replacing the QFile.copy() call with a system call to your OS's 'cp' or 'mv' command, with the same intention. I'd also try printing out every filename as your innermost loop is executed; it probably won't reveal any flaws, but it costs you nothing to try and it might illuminate something.

Still the same I've tried the system cp but the result was the same. Even many different combinations of sleep and blocking the signal from QFileSystemWatcher also didn't help I'm doomed I really need this functionality ;(

To me it seems you are just starting the copy operation too early. You get correct timestamps but wrong behaviour. Use your filesystem api (QFileInfo::created() and friends) to determine the time but copy the file when it is not being written to anymore.

Assuming the user copies files only by one thread, the following would be a brute force solution to this problem:

The idea is to find out which file has actually been added from the current call to the last one. If one has been added, you can safely copy all others.

To do this, I suggest you store the list of files at the end of each call. When the current file count increased compared to that previously stored file list, you know, that all files of the previous file list are finished writing and can be copied. Maintain a list of already copied files to, to easily find out, which ones you actually need to copy at each step.

Joh

Quote:

---

Originally Posted by camol

Still the same I've tried the system cp but the result was the same. Even many different combinations of sleep and blocking the signal from QFileSystemWatcher also didn't help I'm doomed I really need this functionality ;(

---

Blocking the signal? Surely you want all of the signals so you know when the directory/file has finished being updated so you can perform your operations after the last access? (after a set delay)

I thought that if I use the code I wrote above and when I get into if condition with lastInput then I should block the signals from watcher in order to be sure that that signals won't be emmited when I am coping and deleting the source file. But it didnt' give the result

There is no magic wand here. Don't expect a single call to exist that does all the work for you. You need to design an algorithm that will fulfill all your requirements. You know what the problem is so you need to find a way to resolve it. You need an approach that will 1) detect new files in the directory and the time they appeared there, 2) detect when a new file is completely copied into the directory, 3) copy the file to its proper place. Once you do that you need to implement the algorithm in code. And don't expect the solution to be a single linear block of code. It's likely you'll have to implement a class that does that using signals and slots.

I've tried sth like this:

Code:

---

void DServer::newFiles_()
{
    this->inputWatcher->blockSignals(true);
    fileTimer->start();
}
 
void DServer::newFilesTimer_()
{
    output = "";
    ps.start("du -sk /home/camol/INPUT_DICOM");
}
 
void DServer::cmdExec_data_available()
{
    output += ps.readAllStandardOutput() + "\n";
}
 
void DServer::cmdExec_finish(int pl)
{
    QTextStream(stdout) << "INPUT FOLDER SIZE: " << output.remove("/home/camol/INPUT_DICOM")<<endl;
    ps.close();
    QTextStream(stdout) << "INPUT FOLDER SIZE int: " << output.toInt()<<endl;
    if(output.toInt() > lastInput)
        this->lastInput = output.toInt();
    else
    {
        QTextStream(stdout) << "mam wszystko: " << lastInput<<endl;
        inputFiles = getPaths(QDir::homePath() + "/INPUT_DICOM");
        this->fileTimer->stop();
        emit this->_copy();
    }
}

---

ps is a QProcess object and it works it constantly updates the size of the folder during copying the files but still copying the files doesn't work :(

Now I am completely out of ideas, I've just splitted the code and made the archiving-copy side, which was presented, as a separate application which I am running when all files are ready to be copied. However it still doesn't work and I don't know why :(

Are you able to sucessfully and properly detect when the file is ready to be copied? How do you do that? Because blocking signals from the file watcher is probably not going to give you the result you expect.

Now I'm not blocking anything, I made a separate program and I run it when copied all files. Basically I tried the manual approach to the problem but I still get the same result, firstly I thought that was the problem of QFileSytemWatcher signal etc. but it didn't get the right result.

So the following also copies 3kB of the file? Try it for different files in different directories.

Code:

---

int main(int argc, char **argv){
  QFIle::copy(argv[1], "/tmp/testfile");
  return 0;
}

---

You've enlighten but but sth to test and before I do those copies I am doing my method for doing the anonimization of the files when comment it the files are copied well, I moved it and try to anonimize the copied files but it also "corupts" the files :/ The function looks like this:

Code:

---

void ArchiveDicom::anonymous(QString path, QString file)
{
    DcmFileFormat fileformat;
    OFCondition status = fileformat.loadFile(file.toStdString().c_str());
 
    if (status.good())
    {
        OFString patientName;
        if (fileformat.getDataset()->findAndGetOFString(DCM_PatientName, patientName).good())
        {
            QString currentPatientName(patientName.c_str());
 
            if(currentPatientName == lastPatientName)
            {
                writeToAnonym(lastGenNum, path, fileformat, currentPatientName, file);
            }
            else
            {
                std::string randNum = generateNumber();
                writeToAnonym(randNum, path, fileformat, currentPatientName, file);
                lastPatientName = currentPatientName;
                lastGenNum = randNum;
            }
        }
        else
            std::cerr << "Error: cannot access Patient's Name!" << endl;
     }
     else
         std::cerr << "Error: cannot read DICOM file (" << status.text() << ")" << endl;
 }

---

Basically the function changes the TAG, in the .dcm file, which consists Patient Name and replaces it with some random numbers then stores the patient name which was readed.

So does my example work or not?

I found that my method works well in copying but there is sth wrong with the function I wrote above.

Without knowing what your functions do it's not possible to provide any assistance.

Quote:

---

Originally Posted by camol

You've enlighten but but sth to test and before I do those copies I am doing my method for doing the anonimization of the files when comment it the files are copied well, I moved it and try to anonimize the copied files but it also "corupts" the files :/ The function looks like this:

Code:

---

void ArchiveDicom::anonymous(QString path, QString file)
{
    DcmFileFormat fileformat;
    OFCondition status = fileformat.loadFile(file.toStdString().c_str());
 
    if (status.good())
    {
        OFString patientName;
        if (fileformat.getDataset()->findAndGetOFString(DCM_PatientName, patientName).good())
        {
            QString currentPatientName(patientName.c_str());
 
            if(currentPatientName == lastPatientName)
            {
                writeToAnonym(lastGenNum, path, fileformat, currentPatientName, file);
            }
            else
            {
                std::string randNum = generateNumber();
                writeToAnonym(randNum, path, fileformat, currentPatientName, file);
                lastPatientName = currentPatientName;
                lastGenNum = randNum;
            }
        }
        else
            std::cerr << "Error: cannot access Patient's Name!" << endl;
     }
     else
         std::cerr << "Error: cannot read DICOM file (" << status.text() << ")" << endl;
 }

---

Basically the function changes the TAG, in the .dcm file, which consists Patient Name and replaces it with some random numbers then stores the patient name which was readed.

---

I wrote this above what it does and I can't figure out why it is wrong

Well, I don't know what generateNumber(), writeToAnonym, DcmFileFormat, OFCondition and OFString do.

Three last items are members of DCMTK library for DICOM files. GenerateNumber-generates simply a random number which is placed instead of patient name in the TAG .dcm file, and writeToAnonym writes the path of the file and what was in the TAG in .dcm file to the txt file.

And at which point your code starts malfunctioning?

What you exactly mean? I am just manually putting files in folders to my INPUT folder and the just run from console my from which does the anonimization and copies the files to another directory and I see that is sth wrong with that function in writeToAnonym:

Code:

---

void ArchiveDicom::writeToAnonym(std::string num, QString path, DcmFileFormat fileformat, QString currentPatientName, QString file)
{
    if (anonymFile->open(QIODevice::ReadWrite | QIODevice::Append) == true)
    {
        QTextStream out(anonymFile);
        out << path << " " << currentPatientName << endl;
        anonymFile->close();
        fileformat.getDataset()->putAndInsertString(DCM_PatientName, num.c_str());
        fileformat.saveFile(file.toStdString().c_str());
    }
    else
    {
        QTextStream(stdout)<<"couldn't open anonym.an file, so I didn't anonymize the file"<<endl;
    }
}

---

this line is stinky fileformat.saveFile(file.toStdString().c_str()); I got this from the site from the example from DCMTK usage, when I comment this everything works so, fortunetelly coping is ok but without this line the changes described above won't take place.

Quote:

---

Originally Posted by camol

What you exactly mean?

---

I mean your code is not a black box. You can do more than just feed it with data and see if the result is correct. I assume you have some algorithm that the code should implement and at different stages of the algorithm the code produces some intermediate results. So I'm asking at which point those results start differing from what should be happening according to your algorithm.

I see. I try simply anonymize the files. After opening->reading PatientName tag everything is ok, but the I write to the file in the place of PatientName then I have to save the file and I get the file corrupted.


I think I got I will report my result in short time