Read-only accessing files from Vista's Program Files (bypassing Virtual Store)

[Gnurou]

Hi everybody,

I'm developing a multi-platform (Linux/OSX/Windows) Qt application that heavily relies on read-only SQLite databases. Using Qt's SQL module, everything works fine. But lately Vista users have reported that the program could not find the databases when they installed it into Program Files. Indeed, I realized they were redirected by the virtual store "feature" into some user-specific, writable directory, which of course does not contain the databases which have been installed into Program Files.

I don't really know how I can fix that problem. I'm only a Linux user (I cross-compile the Windows binaries and have no Vista at hand for testing), and the way I understand it, Vista thinks my program is going to write into these databases and therefore redirects it to the virtual store. Well, I just want to *read* them. Would anybody have a suggestion as to how I could get a read-only access to these databases files without being redirected by the virtual store feature?

[tinsuke]

That's not the way the VirtualStore works. Take a look at the Data Redirection section of: http://technet.microsoft.com/pt-br/m...80(en-us).aspx

The Data Redirection (use of VirtualStore) occurs when one write operation would fail (it actually fails, but), then the write operation is performed at a COPY of the original file at place to where the user has access (the VirtualStore). All this happens in a transparent way.

The reason why your application can't OPEN database files for READ-ONLY operations can't be the Data Redirection policy. I guess it has something to do with permissions. One easy solution (even if the problem is related to the use o VirtualStore) would be to ask your Vista users to open your application by right-clicking the link and selecting Run as Administrator (or selecting the Run as Administrator box at the link properties).

Hope it helps,
tinsukE

[Gnurou]

Thanks for your answer.

So from your explanation and the document you link, it seems like that virtual store thing actually performs a copy-on-write at the filesystem layer. This is strange as it contradicts the behavior my users reported, which is that the program worked just fine if they installed it somewhere else than Program Files.

The installer program puts the database files into the same directory as the binary, so I expect I can reach them using the following code:

dbFile.setFileName(QDir(QCoreApplication::applicationDirPath()).filePath("kanjidic2.db"));

dbFile.setFileName(QDir(QCoreApplication::applicationDirPath()).filePath("kanjidic2.db"));

To copy to clipboard, switch view to plain text mode 

And indeed that works, excepted in this case. Maybe I should point out that I'm using QtSQL, so therefore I have no control about whether I'm opening my files read-only or read-write (guess it always tries the latter). But then again, by the behavior you described this should not cause any problem.

But following other links from your documents I figured out that I can disable that virtual store thing using an application manifest file. As said in http://msdn.microsoft.com/en-us/library/bb756929.aspx :


Application Marking Virtualize?
Unmarked Yes
asInvoker No

So just requiring "asInvoker" executing level should be enough to disable virtualization. I'll ask one user who reported the problem to test this.

[wysota]

Maybe the problem is related to the whitespace in the name "Program Files"? Check if the same happens if you store the application inside a "Some Directory" folder (containing a space as well).

[Gnurou]

wysota: I don't think this is related, as other vista users (those who run with their administrator account, probably) reported no problem. Others who had problems installing into "Program Files" reported success when installing under "My Documents".

[tinsuke]

An application of mine had the same problem about using the VirtualStore, because it asked, on uninstallation, if the user would like to keep his database info. If it was running on Vista and the user selected "No", the Program Files copy was erased and the VirtualStore one was kept, what resulted, on further installations, in a "virtually restored" database, even if the original one was a brand new database.

One intermediary solution was to use a manifest file with the fields you showed. The actual solution is to save the database at the user folder, which, by the way, enables using one database per user.

[Gnurou]

One intermediary solution was to use a manifest file with the fields you showed.

Well, unfortunately the manifest file thing did not work. Maybe I did something wrong though. My manifest file content is as follows:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <assemblyIdentity version="1.0.0.0" processorArchitecture="X86" name="tagainijisho" type="win32"/>
  <description>A free Japanese dictionary and study assistant</description>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level="asInvoker" uiAccess="false"/>
        </requestedPrivileges>
       </security>
  </trustInfo>
</assembly>

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <assemblyIdentity version="1.0.0.0" processorArchitecture="X86" name="tagainijisho" type="win32"/>
  <description>A free Japanese dictionary and study assistant</description>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level="asInvoker" uiAccess="false"/>
        </requestedPrivileges>
       </security>
  </trustInfo>
</assembly>

To copy to clipboard, switch view to plain text mode 

To include it, I just added the following line into the .rc file:

CREATEPROCESS_MANIFEST_RESOURCE_ID RT_MANIFEST "tagainijisho.exe.manifest"

CREATEPROCESS_MANIFEST_RESOURCE_ID RT_MANIFEST "tagainijisho.exe.manifest"

To copy to clipboard, switch view to plain text mode 

Which is processed by windres through qmake as my project file includes this line:

RC_FILE = tagainijisho.rc

RC_FILE = tagainijisho.rc

To copy to clipboard, switch view to plain text mode 

Everything is processed by mingw32 under Linux (cross-compilation environment).

The RC file also includes the program icon which is visible, so I think it is correctly used here. But maybe I did something wrong here. Can windows experts give me some comments?

The actual solution is to save the database at the user folder, which, by the way, enables using one database per user.

I think this is clearly not the solution here. These databases are read-only - no user is ever going to change them, so there is no reason for them to have their own copy. Moreover they are rather large (around 100Mb total). The user also has his settings stored into a private database, which is stored in the directory returned by QDesktopServices::storageLocation(QDesktopServices :: DataLocation). Here things are working perfectly, as the storage is done in the user profile and Vista does not abusively get into the way.

[tinsuke]

I really don't know whats going wrong with your app. It should be able to see and even write (on a virtually stored copy) at the database file. You could give a try to changing

<requestedExecutionLevel level="asInvoker" uiAccess="false"/>

<requestedExecutionLevel level="asInvoker" uiAccess="false"/>

To copy to clipboard, switch view to plain text mode 

to

<requestedExecutionLevel level="requireAdministrator" />

<requestedExecutionLevel level="requireAdministrator" />

To copy to clipboard, switch view to plain text mode 

And test if it works fine. But this a weird thing to do. I can't be of more help on just trying to figure things out... But I have a Vista test environment and, if your project isn't a proprietary software (or if it is and you trust me (I know you don't have reasons to do so, but...)), you could send it to me, so i could try to figure out what's wrong. =]

Cheers,
tinsukE

[Gnurou]

I really don't know whats going wrong with your app. It should be able to see and even write (on a virtually stored copy) at the database file.

Yeah, I feel like maybe something else is broken, but as the program works perfectly in XP I see no other culprit than the virtualization mechanism...

But this a weird thing to do. I can't be of more help on just trying to figure things out... But I have a Vista test environment and, if your project isn't a proprietary software (or if it is and you trust me (I know you don't have reasons to do so, but...)), you could send it to me, so i could try to figure out what's wrong. =]

It's nothing secret - my project is open source and available at https://launchpad.net/tagaini-jisho .

However, I'm using scons on top of qmake to build it, and the build process is not feasible on Windows yet (I know, a Windows program that does not compile on Windows, what a weird thing to do!). One of my users agreed to help me figuring this out by running special binaries and sending me the output, so I guess I can do it this way. Thanks anyway for proposing to help - I really appreciate it.

I will keep you informed if I can track this down.

[gfunk]

It's most definitely the virtualization system. I spent considerable pains dealing with it on a previous project. Vista is quite the adversary when trying to do things that it's been designed to discourage (like a centralized database for all users).

[ChrisW67]

This could be Sqlite creating (at least attempting to) temporary files in the same directory as the database you are opening. Rollback and master journal files are possibilities.
http://www.sqlite.org/tempfiles.html

[Gnurou]

This is another possibility, indeed. In that case, I'd be in bigger trouble than expected, since these temporary files are necessary.

But on the other hand, I kind of doubt this: on other platforms, the directory where the databases are contained is not writeable neither, and no problem occur. The real problem seems to be Vista returning a file descriptor that do not corresponds to the requested file.

[Gnurou]

Hello everyone,

Just a follow-up to explain how I solved this problem. After doing some trial-and-error, I realized that the files installed in Program Files had read-write access. Therefore, upon opening these files Vista guessed I could write them and therefore made a copy into the virtual store. The solution was pretty straightforward: ensure the files have read-only attribute when they are installed. I just witnessed that with this attribute, virtual store files are not created anymore.

Thanks to you all for the pointers and tips. I'm really glad this major issue is solved.