PDA

View Full Version : writing huge binary data



Alex22
11th December 2015, 18:07
Hi,
there is a 2D array 500*100,000 like this:




double (*data)[100000];
data = new double[500][100000];

for(int i=0; i<500; i++)
{
for(int j=0; j<100000; j++)
{
if(j==0)
{
data[i][j]=0xdd00bb00;//i want this to be my header
continue;
}
data[i][j] = (i+1)*(j+1);
}
}
// now data was initialized



I wrote this code:



void write(QString filename)
{
QFile myfile(filename);

if(!myfile.open(QIODevice::WriteOnly))
{
qDebug()<<"can not open to write";
return;
}

QDataStream out(&myfile);

//now I want to write all of "data" in binary mode

myfile.flush();
myfile.close();
}


thanks for any help

anda_skoa
12th December 2015, 11:44
Well, if you don't have any format constraints, just loop over the entries and write them.
Maybe write the dimensions before, so you can read them before reading the data, unless your data has defined fixed size.

Cheers,
_

Alex22
12th December 2015, 12:31
please read under post (sorry because of slow speed of the sit post was been repeated)!!!

Alex22
12th December 2015, 12:31
@anda_skoa thanks. when i use this to write, it gives a filename.txt in zero size:



double (*data)[100000];
data = new double[500][100000];

for(int i=0; i<500; i++)
{
for(int j=0; j<100000; j++)
{
if(j==0)
{
data[i][j]=0xdd00bb00;//i want this to be my header
continue;
}
data[i][j] = (i+1)*(j+1);
}
}
// now data was initialized

QFile myfile("d:/filename.txt");

if(!myfile.open(QIODevice::WriteOnly))
{
qDebug()<<"can not open to write";
return;
}

QDataStream out(&myfile);

out.writeRawData((char *) (&(data[0][0])), 100000*500*sizeof(double)); // did you mean i write here??? but .txt size is zero and nothing is written

myfile.flush();
myfile.close();


thanks for any help

anda_skoa
12th December 2015, 14:35
Well, since you indicated usage of QDataStream, I assumed you wanted to use it, so I meant looping through data and writing the doubles.

Now the question is, why instantiate a QDataStream if you are then bypassing it?
If you want to go down the route of doing a memory dump, why not just use QFile directly?

In any case, have you checked the return values of the operations? Also what size does QFile report after writing and after closing?

Cheers,
_

P.S.: naming a binary file ".txt" doesn't make sense.

Alex22
12th December 2015, 15:33
@anda_skoa, thanks. I found an amazing thing happens here. for data[m][n] (double (*data)[n]; data = double[m][n]) when m>n it gives me right myfile.txt but when m<n it gives a file in size of zero!!! for example when m=599 and n=600 it gives me a file in size of zero and returns:"The program has unexpectedly finished." but when m=601 and n=600 it gives me a file in size of 2,344kB

even when m is so larger than n, there is no false but if n is only one bigger that m, its not ok!!!

for example:


#include <QApplication>
#include <QFile>
#include <QDebug>
#include <QDataStream>


int main(int argc, char *argv[])
{
QApplication a(argc, argv);

QFile myfile("d:/test.txt");

if(!myfile.open(QIODevice::WriteOnly))
{
qDebug() << "can not open to write";
return 1;
}

double (*data)[600];
data = new double[599][600];//now is not ok but if i define data = new double[601][600] it gives me right file!!!

for(int i=0; i<599; i++)
{
for(int j=0; i<600; i++)
{
data[i][j] = (i+1)*(j+1);
}
myfile.write((char *)(&(data[i][600])), sizeof(double));
}

//myfile.write((char *)(&(data[0][0])), 599*600*sizeof(double)); when m>n this is true also

myfile.flush();
myfile.close();

return 0;
}


You are right I need no QDataStrem and I can use QFile directly.

thanks for anymore help

anda_skoa
12th December 2015, 15:47
and returns:"The program has unexpectedly finished."

Ah, no.
It crashed.

You could run the program in the debugger and look where it crashes.

I would suggest to either use a one dimensional array and calculate the offset based on i and j or initialize each entry in your array of rows individually.

Cheers,
_

Alex22
12th December 2015, 16:29
Thanks Anda,
I wrote only in one row and now it gives a file in size of 390,625kB (for below example), but almost every byte inside of this file is zero. my code is:



#include <QApplication>
#include <QFile>
#include <QDebug>

int main(int argc, char *argv[])
{
QApplication a(argc, argv);

QFile myfile("d:/test.txt");

if(!myfile.open(QIODevice::WriteOnly))
{
qDebug() << "can not open to write";
return 1;
}

double *d;
d = new double[500*100000];

int i = 0;
while(i<100000)
{
for(int j=0; j<500; j++)
{
d[j]=(i+1)*(j+1);
}
i = i + 500;
}

myfile.write((char *) d, 500*100000*sizeof(double));
myfile.flush();
myfile.close();

return 0;
}


Thanks for your helping

anda_skoa
12th December 2015, 17:24
You are always writing to the first 500 doubles.

Cheers,
_

Alex22
12th December 2015, 19:31
@anda_sko, sorry I must write [j+(i*500)] and i++. now all thing is OK and i could write from a 2D array to a text.