High performance large file reading on OSX

[mikeee7]

Hello,

my aim is to read very large files (10GB to 30GB) into main memory (my MacPro has 32GB installed, running OSX 10.5.8).

This is basically very easy to achieve using Qt:

QFile file(filename);
file.open(QIODevice::ReadOnly);
file.readAll();
file.close();

QFile file(filename);
file.open(QIODevice::ReadOnly);
file.readAll();
file.close();

To copy to clipboard, switch view to plain text mode 

However, what happens on OSX is that the so called Unified Buffer Cache (UBC) automatically caches all reads and when reading a file larger than 16GB the systems ends up with 16GB of the file in the active memory and 16GB of the same data in the inactive memory (the cache) and starts to swap heavily. This is a well known problem and the UBC can be disabled with a simple additional system call:

#include <fcntl.h>
QFile file(filename);
file.open(QIODevice::ReadOnly);
fcntl(file.handle(), F_NOCACHE, 1); // <- disable UBC for that file
file.readAll();
file.close();

#include <fcntl.h>
QFile file(filename);
file.open(QIODevice::ReadOnly);
fcntl(file.handle(), F_NOCACHE, 1); // <- disable UBC for that file
file.readAll();
file.close();

To copy to clipboard, switch view to plain text mode 

This successfully disables caching and allows loading files up to 30GB into main memory.

However, the problem now is that reading performance with the cache disabled drastically drops from ~200mb/s to ~25mb/s.

Does anyone have any hints on how to have the UBC disabled but still achieve high reading throughput?

Best regards & thanks a lot,
mikeee7

[caduel]

have you tried mmap, or if you want to be platform independent Boost.Interprocess' equivalent (see e.g. http://boost.cowic.de/rc/pdf/interprocess.pdf, pages 20ff)?

[mikeee7]

Hi caduel,

thanks for your suggestions -- memory mapping the file would be an option that could circumvent the problem. However we have a large existing codebase that is based on QDataStream, QFile and so on for loading those large files which we would like to continue to use.

The function call

fcntl(file.handle(), F_NOCACHE, 1);

fcntl(file.handle(), F_NOCACHE, 1);

To copy to clipboard, switch view to plain text mode 

successfully disables caching but why does read performance suffer that much -- is Qt doing something else behind the scenes?

When using standard C calls for reading a large file, caching can be disabled while retaining the original high read performance:

int fd = open(src.toStdString().c_str(), O_RDONLY);
fcntl(fd, F_GLOBAL_NOCACHE, 1);
size_t bufferSize = 1024ul*1024ul*2000ul;
char* buffer = new char[bufferSize];
read(fd, buffer, bufferSize);
close(fd);
delete[] buffer;

int fd = open(src.toStdString().c_str(), O_RDONLY);
fcntl(fd, F_GLOBAL_NOCACHE, 1);
size_t bufferSize = 1024ul*1024ul*2000ul;
char* buffer = new char[bufferSize];
read(fd, buffer, bufferSize);
close(fd);
delete[] buffer;

To copy to clipboard, switch view to plain text mode 

Any ideas why disabling caching slows down Qt but not the original C calls?