PDA

View Full Version : QFile::copy copies only 3kb



camol
7th April 2011, 21:49
Hi I've noticed a very strange problem a use QFileSystemWatcher to monitor one folder, when new files are copied to this folder the signal directoryChanged is emitted. I wrote a slot, which works well, it gives me the confidence that when ALL files are copied to the folder only then the right action is taken:




void DServer::newFiles_()
{
QDir input(QDir::homePath() + "/INPUT DICOM");

if(input.count() > lastInput)
{
QTextStream(stdout)<<"not all files" << lastInput <<endl;
lastInput = input.count();
return;
}
else
{
QDir inputDir(QDir::homePath() + "/INPUT DICOM");
QTextStream(stdout) << "BEFORE IF "<<inputDir.count() <<endl;
if(inputDir.count() > 1)
{
QTextStream(stdout)<<"Got all files"<< lastInput <<endl;
inputFiles = getPaths(QDir::homePath() + "/INPUT DICOM");
emit this->_moveFiless();
lastPatientName.clear();
lastInput = 1;
}
else
return;
}
}


However this slot


void DServer::moveFiless_()
{
QString input("/INPUT DICOM");
QDate date = QDate::currentDate();
QTime time = QTime::currentTime();

for(int i=0; i<inputFiles.count(); i++)
{
QString old_input_list = inputFiles[i];
inputFiles[i].remove(0,(QDir::homePath().count()) + input.count());

int length = inputFiles[i].length();
int startindex = inputFiles[i].lastIndexOf("/",-1) + 1;
QString dir_path = inputFiles[i];

dir_path.remove(startindex, length);

QDir dir;
if(dir.mkpath(QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path) == true)
{
//anonymize here
anonymous("/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + inputFiles[i], old_input_list);
dir.mkpath(QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + dir_path);
date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + input_list[i] + ".tmp");

QFile::copy(old_input_list, (QDir::homePath() + "/TEMPORARY STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + inputFiles[i] + ".tmp"));
QFile::copy(old_input_list, (QDir::homePath() + "/DICOM STORAGE/" + date.toString("dd.MM.yyyy") + "/" + time.toString("hh:mm:ss") + inputFiles[i]));
QFile::remove(old_input_list);
QTextStream(stdout)<<"source file: "<< old_input_list <<endl;
dir.rmpath(QDir::homePath() + "/INPUT DICOM" + dir_path);
}
else
{
QTextStream(stdout) <<"Couldn't make this dir: ' "<< QDir::homePath() + "/DICOM STORAGE" + dir_path <<endl;
}
}
}

doesn't work well, the files from source are copied but they only have from 2.6-3.0 kB, when the original ones have about 80kB, that means the QFile::copy works but not good.

JohannesMunk
8th April 2011, 07:45
Just reread your post..

Is "if(input.count() > lastInput)" really correct? Don't you expect the input.count() to increase until all files have been copied?

In that case you have to keep waiting while it is below the threshold lastinput.

Joh

camol
8th April 2011, 09:31
But the signal directoryChanged() is emitted many times during copying the files to the monitored folder and as far as I know I'm interested in the last one becouse it indicates that the last file occured in the watched folder and after that I am performing the copy action. if(input.count > lastinput) informs if current number of files in the monitored directory is greater then it was during the last run of this slot, so that means the files are still being copied to the folder.

I tried even sleep for 5 seconds to be sure that all files are in the directory but still all files are copied but only 2.8kB of them.

JohannesMunk
8th April 2011, 15:17
Ah! Sorry about that. Now I get it. But it wont work like that, because QFileSystemWatcher fires also when a file is modified. Thus during the copy process your method will be executed multiple times for each file. The watcher supresses to frequent updates but nonetheless: You will get updates where the filecount has not changed but only the file has been modified (a few additional bytes have been copied). Thus you think the file-list is already copied completely and make a copy of incomplete files.

Joh

camol
8th April 2011, 15:51
Do you have any idea to solve that? I really need this functionality which I have described above :(

wysota
8th April 2011, 16:16
Connect a timer to the file watcher and reset the timer whenever you receive a signal from the watcher. At some point you will stop receiving signals and the timer will timeout. At this point you will know that for some time there haven't been any updates to the file. This is not foul proof but should be enough for you.

camol
8th April 2011, 16:27
Can you bring some example code of this becouse I am trying to imagine the situation but I can't figure how it should work this. Big thanks in advance for your time here for both of you.

wysota
8th April 2011, 16:29
void Class::setup() {
timer.setSingleShot(true);
timer.setInterval(1000); // 1s
connect(...);
}

void Class::onSignalFromFileSystemWatcher() {
timer.start();
}

void Class::onTimerTimeout() {
QFile::copy(...);
}

camol
9th April 2011, 16:02
It didn't work still is copied only about 3kB of each file, and also this solution makes sth that I wanted to prevent my application from. Becouse I copy the files from monitored folder to folder created which name is the current time and now for example if I put many files in the monitored folder the whole operation lasts long and the result of that is 2-3 folders with names with different times. My code above didn't do this it only created one folder for each operation of putting files to the monitored folder

JohannesMunk
9th April 2011, 16:11
I would suggest, you drop this. And provide the user with a simple button, where he can trigger an import after having copied all files.

Your concept of monitoring will fail because of so many things. How can you be sure, that the user will not copy files in several packets? That the source medium won't take 10secs to spin up. That the user realizes he has got the wrong files...

What is this for anyway?

Joh

camol
9th April 2011, 16:18
I am building a server aplication which runs and stays on all the time. It stores files in hierachy of folders thats why when a user brings new files for the server the server is automaticaly archiving them in the right order without any other actions from the user becouse it is an non-gui application. That it is why I need it so much.

I've tried this at the beginning of the slot connected to directoryChanged:
this->inputWatcher->blockSignals(true); //to prevent more the one signal
sleep(40); //to make sure that all files are in the watched folder


but still it doesn't work :(


I don't get it I tried even to make this functionality with a timer but it still the same

camol
9th April 2011, 19:12
Does anybody know what is going on? :( I really need this.

SixDegrees
9th April 2011, 19:29
Try simplifying your code dramatically so it only copies a single file. Or just write a small test program to do that, without the timers and other drivers. See if that works. Then build up to what you have now.

I'd also examine the path strings you're constructing, although it sounds like those probably work if the files are produced at all.

Reworking the copy routine in plain vanilla C++ is another alternative that would involve less than a dozen lines of code.

camol
9th April 2011, 19:35
I think file by file won't work also becouse I've tried putting files in folders and in subfolders etc, and the signal directory change wasn't emited as many times, and also doing file by file will create many time name folders like I described above. Strings of paths are done ok becouse the files are succesfully copied to the destination folder in right order but only the size of copied files is invalid.

SixDegrees
9th April 2011, 19:44
The point is to break your procedure down into small, testable pieces and verify that all of the individual bits are working as expected. You could also try replacing the QFile.copy() call with a system call to your OS's 'cp' or 'mv' command, with the same intention. I'd also try printing out every filename as your innermost loop is executed; it probably won't reveal any flaws, but it costs you nothing to try and it might illuminate something.

camol
10th April 2011, 13:57
Still the same I've tried the system cp but the result was the same. Even many different combinations of sleep and blocking the signal from QFileSystemWatcher also didn't help I'm doomed I really need this functionality ;(

wysota
10th April 2011, 21:44
To me it seems you are just starting the copy operation too early. You get correct timestamps but wrong behaviour. Use your filesystem api (QFileInfo::created() and friends) to determine the time but copy the file when it is not being written to anymore.

JohannesMunk
10th April 2011, 21:56
Assuming the user copies files only by one thread, the following would be a brute force solution to this problem:

The idea is to find out which file has actually been added from the current call to the last one. If one has been added, you can safely copy all others.

To do this, I suggest you store the list of files at the end of each call. When the current file count increased compared to that previously stored file list, you know, that all files of the previous file list are finished writing and can be copied. Maintain a list of already copied files to, to easily find out, which ones you actually need to copy at each step.

Joh

squidge
10th April 2011, 21:58
Still the same I've tried the system cp but the result was the same. Even many different combinations of sleep and blocking the signal from QFileSystemWatcher also didn't help I'm doomed I really need this functionality ;(

Blocking the signal? Surely you want all of the signals so you know when the directory/file has finished being updated so you can perform your operations after the last access? (after a set delay)

camol
11th April 2011, 00:06
I thought that if I use the code I wrote above and when I get into if condition with lastInput then I should block the signals from watcher in order to be sure that that signals won't be emmited when I am coping and deleting the source file. But it didnt' give the result

wysota
11th April 2011, 10:28
There is no magic wand here. Don't expect a single call to exist that does all the work for you. You need to design an algorithm that will fulfill all your requirements. You know what the problem is so you need to find a way to resolve it. You need an approach that will 1) detect new files in the directory and the time they appeared there, 2) detect when a new file is completely copied into the directory, 3) copy the file to its proper place. Once you do that you need to implement the algorithm in code. And don't expect the solution to be a single linear block of code. It's likely you'll have to implement a class that does that using signals and slots.

camol
15th April 2011, 13:21
I've tried sth like this:



void DServer::newFiles_()
{
this->inputWatcher->blockSignals(true);
fileTimer->start();
}

void DServer::newFilesTimer_()
{
output = "";
ps.start("du -sk /home/camol/INPUT_DICOM");
}

void DServer::cmdExec_data_available()
{
output += ps.readAllStandardOutput() + "\n";
}

void DServer::cmdExec_finish(int pl)
{
QTextStream(stdout) << "INPUT FOLDER SIZE: " << output.remove("/home/camol/INPUT_DICOM")<<endl;
ps.close();
QTextStream(stdout) << "INPUT FOLDER SIZE int: " << output.toInt()<<endl;
if(output.toInt() > lastInput)
this->lastInput = output.toInt();
else
{
QTextStream(stdout) << "mam wszystko: " << lastInput<<endl;
inputFiles = getPaths(QDir::homePath() + "/INPUT_DICOM");
this->fileTimer->stop();
emit this->_copy();
}
}


ps is a QProcess object and it works it constantly updates the size of the folder during copying the files but still copying the files doesn't work :(

camol
20th April 2011, 00:16
Now I am completely out of ideas, I've just splitted the code and made the archiving-copy side, which was presented, as a separate application which I am running when all files are ready to be copied. However it still doesn't work and I don't know why :(

wysota
20th April 2011, 08:31
Are you able to sucessfully and properly detect when the file is ready to be copied? How do you do that? Because blocking signals from the file watcher is probably not going to give you the result you expect.

camol
20th April 2011, 10:20
Now I'm not blocking anything, I made a separate program and I run it when copied all files. Basically I tried the manual approach to the problem but I still get the same result, firstly I thought that was the problem of QFileSytemWatcher signal etc. but it didn't get the right result.

wysota
20th April 2011, 10:55
So the following also copies 3kB of the file? Try it for different files in different directories.


int main(int argc, char **argv){
QFIle::copy(argv[1], "/tmp/testfile");
return 0;
}

camol
20th April 2011, 11:17
You've enlighten but but sth to test and before I do those copies I am doing my method for doing the anonimization of the files when comment it the files are copied well, I moved it and try to anonimize the copied files but it also "corupts" the files :/ The function looks like this:



void ArchiveDicom::anonymous(QString path, QString file)
{
DcmFileFormat fileformat;
OFCondition status = fileformat.loadFile(file.toStdString().c_str());

if (status.good())
{
OFString patientName;
if (fileformat.getDataset()->findAndGetOFString(DCM_PatientName, patientName).good())
{
QString currentPatientName(patientName.c_str());

if(currentPatientName == lastPatientName)
{
writeToAnonym(lastGenNum, path, fileformat, currentPatientName, file);
}
else
{
std::string randNum = generateNumber();
writeToAnonym(randNum, path, fileformat, currentPatientName, file);
lastPatientName = currentPatientName;
lastGenNum = randNum;
}
}
else
std::cerr << "Error: cannot access Patient's Name!" << endl;
}
else
std::cerr << "Error: cannot read DICOM file (" << status.text() << ")" << endl;
}


Basically the function changes the TAG, in the .dcm file, which consists Patient Name and replaces it with some random numbers then stores the patient name which was readed.

wysota
20th April 2011, 11:49
So does my example work or not?

camol
20th April 2011, 12:08
I found that my method works well in copying but there is sth wrong with the function I wrote above.

wysota
20th April 2011, 12:23
Without knowing what your functions do it's not possible to provide any assistance.

camol
20th April 2011, 12:29
You've enlighten but but sth to test and before I do those copies I am doing my method for doing the anonimization of the files when comment it the files are copied well, I moved it and try to anonimize the copied files but it also "corupts" the files :/ The function looks like this:



void ArchiveDicom::anonymous(QString path, QString file)
{
DcmFileFormat fileformat;
OFCondition status = fileformat.loadFile(file.toStdString().c_str());

if (status.good())
{
OFString patientName;
if (fileformat.getDataset()->findAndGetOFString(DCM_PatientName, patientName).good())
{
QString currentPatientName(patientName.c_str());

if(currentPatientName == lastPatientName)
{
writeToAnonym(lastGenNum, path, fileformat, currentPatientName, file);
}
else
{
std::string randNum = generateNumber();
writeToAnonym(randNum, path, fileformat, currentPatientName, file);
lastPatientName = currentPatientName;
lastGenNum = randNum;
}
}
else
std::cerr << "Error: cannot access Patient's Name!" << endl;
}
else
std::cerr << "Error: cannot read DICOM file (" << status.text() << ")" << endl;
}


Basically the function changes the TAG, in the .dcm file, which consists Patient Name and replaces it with some random numbers then stores the patient name which was readed.





I wrote this above what it does and I can't figure out why it is wrong

wysota
20th April 2011, 12:34
Well, I don't know what generateNumber(), writeToAnonym, DcmFileFormat, OFCondition and OFString do.

camol
20th April 2011, 12:52
Three last items are members of DCMTK library for DICOM files. GenerateNumber-generates simply a random number which is placed instead of patient name in the TAG .dcm file, and writeToAnonym writes the path of the file and what was in the TAG in .dcm file to the txt file.

wysota
20th April 2011, 17:58
And at which point your code starts malfunctioning?

camol
20th April 2011, 19:50
What you exactly mean? I am just manually putting files in folders to my INPUT folder and the just run from console my from which does the anonimization and copies the files to another directory and I see that is sth wrong with that function in writeToAnonym:


void ArchiveDicom::writeToAnonym(std::string num, QString path, DcmFileFormat fileformat, QString currentPatientName, QString file)
{
if (anonymFile->open(QIODevice::ReadWrite | QIODevice::Append) == true)
{
QTextStream out(anonymFile);
out << path << " " << currentPatientName << endl;
anonymFile->close();
fileformat.getDataset()->putAndInsertString(DCM_PatientName, num.c_str());
fileformat.saveFile(file.toStdString().c_str());
}
else
{
QTextStream(stdout)<<"couldn't open anonym.an file, so I didn't anonymize the file"<<endl;
}
}


this line is stinky fileformat.saveFile(file.toStdString().c_str()); I got this from the site from the example from DCMTK usage, when I comment this everything works so, fortunetelly coping is ok but without this line the changes described above won't take place.

wysota
20th April 2011, 20:02
What you exactly mean?
I mean your code is not a black box. You can do more than just feed it with data and see if the result is correct. I assume you have some algorithm that the code should implement and at different stages of the algorithm the code produces some intermediate results. So I'm asking at which point those results start differing from what should be happening according to your algorithm.

camol
20th April 2011, 20:52
I see. I try simply anonymize the files. After opening->reading PatientName tag everything is ok, but the I write to the file in the place of PatientName then I have to save the file and I get the file corrupted.


I think I got I will report my result in short time