Page 1 of 2 12 LastLast
Results 1 to 20 of 23

Thread: Calculate MD5 sum of a big file

  1. #1
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Calculate MD5 sum of a big file

    Hi all,

    I need a md5 sum of a file with a big size. I found this:

    Qt Code:
    1. QFile file("/home/chris/backup.img");
    2.  
    3. if (file.open(QIODevice::ReadOnly)) {
    4. QByteArray fileData = file.readAll();
    5. QByteArray hashData = QCryptographicHash::hash(fileData, QCryptographicHash::Md5);
    6.  
    7. qDebug() << hashData.toHex();
    8. }
    To copy to clipboard, switch view to plain text mode 

    but with a file size > 8 GB it's a memory overkill!

    Are there any other methods?

    Thanks!
    Chris

  2. #2
    Join Date
    Dec 2008
    Location
    Poland
    Posts
    383
    Thanks
    52
    Thanked 42 Times in 42 Posts
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows Android

    Default Re: Calculate MD5 sum of a big file

    QCryptographicHash::hash() is static "helper" function, and what You are looking for is QCryptographicHash::addData().

    This was answered here i.e. http://www.qtcentre.org/threads/3567...an-entire-file
    In the near future - corporate networks reach out to the stars. Electrons and light flow throughout the universe.
    The advance of computerization however, has not yet wiped out nations and ethnic groups.

  3. #3
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    That works, thanks for the hint!

    Is there a way to calculate md5 sum from a dvd (/dev/cdrom) with this method? It musts read the dvd raw data, not files on disc!

  4. #4
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,734
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Calculate MD5 sum of a big file

    Open and read the /dev/cdrom device directly: you get the bytes from one end of the disc to the other.

  5. #5
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    How? QFile doesn't work!

  6. #6
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,734
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Calculate MD5 sum of a big file

    Sure it does, but QIODevice::atEnd() is not reliable on block special files so you have to adopt a slightly different approach:
    Qt Code:
    1. QFile in("/dev/cdrom");
    2. if (in.open(QIODevice::ReadOnly)) {
    3. qDebug() << "Opened ok";
    4. char buf[2048];
    5. int bytesRead;
    6. while ((bytesRead = in.read(buf, 2048)) > 0) {
    7. qDebug() << "Read" << bytesRead;
    8. }
    9. qDebug() << "Final bytesRead value" << bytesRead;
    10. in.close();
    11. }
    To copy to clipboard, switch view to plain text mode 
    This will fail to open if there is no disc in the drive. In a real program you would use a larger buffer, distinguish between bytesRead == 0 and -1 etc.

  7. #7
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    Thanks for the hint, but it doesn't work for me! The md5 sum from image and dvd are not the same! On command line they are identical! Or is the code wrong?

    Qt Code:
    1. QCryptographicHash hash(QCryptographicHash::Md5);
    2. QFile in("/dev/cdrom");
    3.  
    4. if (in.open(QIODevice::ReadOnly)) {
    5. char buf[2048];
    6.  
    7. while (in.read(buf, 2048) > 0) {
    8. hash.addData(buf, 2048);
    9. }
    10.  
    11. in.close();
    12. qDebug() << hash.result().toHex();
    13. }
    14. else {
    15. qDebug() << "Failed to open device!";
    16. }
    To copy to clipboard, switch view to plain text mode 

  8. #8
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,364
    Thanks
    3
    Thanked 5,014 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Calculate MD5 sum of a big file

    How many bytes did you manage to read from the device?
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  9. #9
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    ...the while loop reads all data from dvd!?

  10. #10
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,364
    Thanks
    3
    Thanked 5,014 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Calculate MD5 sum of a big file

    Quote Originally Posted by realdarkman71 View Post
    ...the while loop reads all data from dvd!?
    Was this supposed to be an answer to my question? If so, then it's not what I expected. I expected that you will change your code to count how many bytes were really read, not that you expect the loop to read "all" the data.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  11. #11
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    Ok, but how? ...sorry!

  12. #12
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,364
    Thanks
    3
    Thanked 5,014 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Calculate MD5 sum of a big file

    Read the docs to learn what QFile::read() returns. Then use your newly acquired knowledge to count how many bytes in total were read from the file.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  13. #13
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    Ok, I used this:

    Qt Code:
    1. QCryptographicHash hash(QCryptographicHash::Md5);
    2. QFile in("/dev/cdrom");
    3.  
    4. if (in.open(QIODevice::ReadOnly)) {
    5. char buf[2048];
    6. int bytesRead;
    7. qint64 overallBytesRead = 0;
    8.  
    9. while ((bytesRead = in.read(buf, 2048)) > 0) {
    10. overallBytesRead += bytesRead;
    11. hash.addData(buf, 2048);
    12. }
    13.  
    14. in.close();
    15. qDebug() << "overall bytes read:" << overallBytesRead;
    16. qDebug() << hash.result().toHex();
    17. }
    18. else {
    19. qDebug() << "Failed to open device!";
    20. }
    To copy to clipboard, switch view to plain text mode 

    After completed, overallBytesRead is "8738865152", but the file size of image is "8738846720". He reads more bytes as on dvd is! I'm confused!

  14. #14
    Join Date
    Mar 2008
    Location
    Kraków, Poland
    Posts
    1,531
    Thanked 281 Times in 276 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    Line 11 should be :
    Qt Code:
    1. hash.addData(buf, bytesRead);
    To copy to clipboard, switch view to plain text mode 

  15. #15
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,734
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Calculate MD5 sum of a big file

    This is not a failing of Qt, but a failing of understanding the data you are handling. The DVD/CD will almost always contain extra padding data at the end of the supplied image to meet requirements of the specifications. You should read (up to) as many bytes from the DVD as are in the image you are trying to compare with. So, for example, a Gentoo image:
    Qt Code:
    1. // Original image file
    2. chrisw@newton ~ $ dd if=install-amd64-minimal-20110609.iso bs=2048 | md5sum
    3. 64900+0 records in // <<< this is the number of blocks in the image
    4. 64900+0 records out
    5. 132915200 bytes (133 MB) copied, 4.39815 s, 30.2 MB/s
    6. 3acf53667fcf1d03e98068ee4af5f4a3 -
    7.  
    8. // Reading all data from the raw device... fails
    9. chrisw@newton ~ $ dd if=/dev/cdrom bs=2048 | md5sum
    10. 64963+0 records in
    11. 64963+0 records out
    12. b0700288a316b71dee09ed87dce3b160 - // <<<< Not good
    13. 133044224 bytes (133 MB) copied, 39.3402 s, 3.4 MB/s
    14.  
    15. // reading correct number of blocks from the device matches
    16. chrisw@newton ~ $ dd if=/dev/cdrom bs=2048 count=64900 | md5sum
    17. 64900+0 records in
    18. 64900+0 records out
    19. 3acf53667fcf1d03e98068ee4af5f4a3 - // <<< Sweet :)
    20. 132915200 bytes (133 MB) copied, 37.9759 s, 3.5 MB/s
    To copy to clipboard, switch view to plain text mode 

  16. #16
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    Okay, I understand! But how can I do this in my code? This works for me:

    Qt Code:
    1. QCryptographicHash hash(QCryptographicHash::Md5);
    2. QFile in("/dev/cdrom");
    3. QFileInfo fileInfo("/home/chris/backup.img");
    4. qint64 imageSize = fileInfo.size();
    5.  
    6. if (in.open(QIODevice::ReadOnly)) {
    7. char buf[2048];
    8. int bytesRead;
    9. qint64 overallBytesRead = 0;
    10.  
    11. while ((bytesRead = in.read(buf, 2048)) > 0) {
    12. overallBytesRead += bytesRead;
    13. hash.addData(buf, 2048);
    14.  
    15. if (overallBytesRead == imageSize) {
    16. break;
    17. }
    18. }
    19.  
    20. in.close();
    21. qDebug() << "overall bytes read:" << overallBytesRead;
    22. qDebug() << hash.result().toHex();
    23. }
    24. else {
    25. qDebug() << "Failed to open device!";
    26. }
    To copy to clipboard, switch view to plain text mode 

    Is that right?

  17. #17
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,734
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Calculate MD5 sum of a big file

    This works because the image will be an exact number of 2048 byte blocks. If you change that buffer size then you might need to handle reading a last partial block in the read at line 11 and line 13. For example if you used at 10000 byte buffer with my Gentoo image of 132,915,200 bytes the last block will be 5200 bytes and you do not want to read more than that or your hash will be affected.

    I'd be inclined to do it this way:
    Qt Code:
    1. QCryptographicHash hash(QCryptographicHash::Md5);
    2. QFile in("/dev/cdrom");
    3. QFileInfo fileInfo("/home/chris/backup.img");
    4. qint64 imageSize = fileInfo.size();
    5.  
    6. const int bufferSize = 10000;
    7. if (in.open(QIODevice::ReadOnly)) {
    8. char buf[bufferSize];
    9. int bytesRead;
    10.  
    11. int readSize = qMin(imageSize, bufferSize);
    12. while (readSize > 0 && (bytesRead = in.read(buf, readSize)) > 0) {
    13. imageSize -= bytesRead;
    14. hash.addData(buf, bytesRead);
    15. readSize = qMin(imageSize, bufferSize);
    16. }
    17.  
    18. in.close();
    19. qDebug() << hash.result().toHex();
    20. }
    21. else {
    22. qDebug() << "Failed to open device!";
    23. }
    To copy to clipboard, switch view to plain text mode 

    There is always more than one way to do it.

  18. #18
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,364
    Thanks
    3
    Thanked 5,014 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Calculate MD5 sum of a big file

    The smartest thing would probably be to detect the end of the image data instead of relying on having the image size given upfront. Padding probably begins with some fixed pattern or there is some ioctl call that can return the real data size. Otherwise it wouldn't be possible to create the image in the first place.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  19. #19
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,734
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Calculate MD5 sum of a big file

    In the case of my Gentoo CDROM the extra 63 blocks on the CD are all zero. Unfortunately, so are at least the last 100 blocks of the original image. The end of my VirtualBox additions CD image and a data DVD is similar. I don't claim to know the complete in-and-outs of the various standards but it does seem that zero padding is done at several stages in the mastering and writing of an image (-pad option in mkisofs for example).

    If the point of the exercise is to compare an image to the version on a disc then knowing the image size up front is hardly unreasonable.

  20. #20
    Join Date
    Oct 2010
    Location
    Earth
    Posts
    50
    Thanks
    3
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Calculate MD5 sum of a big file

    Okay, I know the image size at this point. I read "image size" / 2048 blocks from DVD in a "for" loop. That works, check sums are identical!

    Thank you very much to all for your help!!!

Similar Threads

  1. Replies: 7
    Last Post: 15th February 2012, 14:51
  2. Qtcreator won't calculate correctly
    By thefatladysingsopera in forum Newbie
    Replies: 3
    Last Post: 4th June 2011, 00:43
  3. calculate draw time
    By johnsoga in forum Qt Programming
    Replies: 1
    Last Post: 30th May 2009, 03:03
  4. how to calculate difference b/w two times
    By dummystories in forum Newbie
    Replies: 1
    Last Post: 9th March 2009, 14:58
  5. Replies: 0
    Last Post: 6th March 2009, 09:19

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.