[howto] making a windows shell extension with qt open-source edition

[Tilda]

Hello,

I've managed to create a Windows shell extension using qt open-source edition and the freely distributed visualstudioexpress. I'm new to Qt C++ programming and very new to Windows programming. I was told this could make an interesting tutorial, perhaps for the wiki.
As I'm new, there may be much room for improvements, maybe we can even get rid of visualstudioexpress (or at least replace it with some more basic sdk).

This is a work in progress, and you're invited to give feedback, make suggestions or even patches, it would be a team work. :)

[The tutorial follows in the next post, due to the size limit of a forum post]

[Tilda]

In this tutorial, we will make a windows shell extension using Qt and visualstudio express. This extension will add some overlay icons to files in the file explorer. Files whose names contain a 'q' or a 't' will have their icon modified.

http://img6.imageshack.us/img6/8108/screenshot1wt.png (http://img6.imageshack.us/i/screenshot1wt.png/)


Note: perhaps visualstudio express is not even needed, perhaps only some free windows SDK. What is needed from it is its "midl" tool since there is no opensource alternative yet.
Disclaimer: this tutorial is a work in progress.

The ActiveQt OpenGL example was the nearest to this task and helped doing this task. [http://doc.trolltech.com/4.6/activeqt-opengl.html]

Concepts used in the tutorial
=============================

What is a "shell extension"?
----------------------------
A shell extension is a plugin for the windows shell, namely explorer.exe, or open/save file dialogs in any application (e.g. notepad). An extension can for example change the display of icons in the explorer (which we will do in this tutorial), add items in the contextual menu, etc.

A shell extension is a DLL file that needs to be "registered" so it can be enabled and used by the shell.

Registering an extension
------------------------
An extension will start taking effect after it is registered, and separately in apps started _after_ registration. If the extension is just registered, you won't see its effects in explorer yet. If you start notepad after, you will see the extension effects in notepad, but not in explorer.exe. explorer.exe will have to be restarted (or the computer rebooted) to see the effect in explorer.

Symmetrically, unregistration will not disable the extension immediately in all applications. Applications still running while the extension is unregistered will still have the extension loaded until the apps are exited.

This has an importance while developping the extension, as the extension file DLL can't be overwritten while it's in use, unregistering it is not enough.

Here, we will register/unregister extensions using the "regsvr32" command.

To register an extension:
regsvr32 the_extension.dll

To unregister it:
regsvr32 /u the_extension.dll

The /s flag can be added to the regsvr32 command to enable "silent mode", which disables the annoying dialogbox. But we won't use it at first, since it doesn't report registration errors.


32-bits/64-bits compatibility
-----------------------------

An extension, as a native binary DLL, is very sensitive to byte-architecture.
Whereas a 32-bits program can be run on a 64-bits OS, a DLL is more complicated: it has to be be in the same bytes architecture as the program that loads it.
This means that on a 64-bits OS, 2 DLLs will be needed: a 64-bits one for 64-bits programs like explorer.exe, and a 32-bits one for 32-bits programs like many. Remember that your extension can be used in third-party, non-builtin-in-windows programs, for example any program that has an open/save file dialog.


Basic anatomy of a shell extension and COM
------------------------------------------

A shell extension has to provide an API to applications that will use it, through a protocol called "COM". The shell extension will be a COM server, as it exposes its API, and the apps will be COM clients.
A COM server in a DLL is called an "in-process" server.

The API it implements follows some defined "COM interfaces" (interface is the 'I' in "API", it's a kind of abstract class).

The API is implemented in a class, each instance of this class is a "COM object".

Our shell extension, displaying icon overlays, will implement the "IShellIconOverlayIdentifier" interface.

COM objects need a COM factory to be created by the COM clients.

COM interfaces implementations need declaration in a special language called IDL. These IDL declarations will be generated by an "MIDL" tool.


ActiveQt
--------

ActiveQt is a Qt module that will provide us classes for implementing COM servers and building COM factories. It also provides many other features which we won't need here. All classes names from ActiveQt start with "QAx".
We will use its QAxServer submodule.

- http://doc.trolltech.com/4.6/activeqt.html
- http://doc.trolltech.com/4.6/activeqt-server.html

VisualStudio Express
--------------------

In this tutorial, we will use VisualStudioExpress, which is free, for its "midl" tool, that will generate IDL definitions for our shell extension.

- http://www.microsoft.com/express/

GUIDs
-----

GUIDs, Class IDs (CLSID), Interface IDs (IID) are all UUIDs, 128-bits Universally Unique IDentifiers.
They are used to identify every published component of a shell extension and to tell them apart from all shell extensions in the world.
As they are 128-bits, which is big, a randomly generated UUID is generally enough, there is no need of a central registry to check our UUID is not already used.

For example, "60c580d2-41f2-43ed-b5d1-b435d74d1999" is an UUID.

We can use the VisualStudioExpress builtin generator, or use any uuid generator tool. Qt does not provide one, but there are many, for example, python can be used to generate one:
python -c "import uuid; print uuid.uuid4()"



============

Starting
--------
In QtCreator, start a new "C++ library" project, we can call it "shellext_overlay". This project does not need QtGui.

Create a new C++ class (called "ShellOverlay" for example).

The COM object class
--------------------

Our class will be the class for the COM object. So it will have to inherit QObject (first, because Qt needs QObject always be the first), QAxAggregated and the COM interfaces we implement: IShellIconOverlayIdentifier.


class SHELLEXT_OVERLAYSHARED_EXPORT ShellOverlay : public QObject, public QAxAggregated, public IShellIconOverlayIdentifier {


All COM interfaces inherit the IUnknown interface, so we would have to implement it, but Qt provides a sane default implementation for it, which can be included with the QAXAGG_IUNKNOWN macro in our class declaration.

long queryInterface(const QUuid &iid, void**iface);
???

IShellIconOverlayIdentifier interface
+++++++++++++++++++++++++++++++++++++
The IShellIconOverlayIdentifier defines 3 members we will need to implement, here is their declaration


/*! Query information about the overlay icon
\param pwszIconFile output parameter where to put the array of overlay icon (wchar_t **)
\param cchMax size of the pwszIconFile buffer, in characters (not bytes)
\param pIndex output parameter, index of the icon in the pwszIconFile file (e.g. if the file contains multiple icons), starting at 0
\param pdwFlags output parameter, options for the overlay icon
\return S_OK in case of success
*/
STDMETHOD(GetOverlayInfo)(LPWSTR pwszIconFile, int cchMax, int *pIndex, DWORD* pdwFlags);

STDMETHOD(GetPriority)(int* pPriority);

/*! Query if the overlay is present for a particular file
\param pwszPath path of the file to query (wchar_t*)
\param dwAttrib attributes of the file
\return S_OK if the icon has to be overlayed, S_FALSE else
*/
STDMETHOD(IsMemberOf)(LPCWSTR pwszPath,DWORD dwAttrib);

Implementation of the class
+++++++++++++++++++++++++++

long ShellOverlay::queryInterface(const QUuid &iid, void **iface) {
*iface = 0;
if (iid == IID_IShellIconOverlayIdentifier)
*iface = (IShellIconOverlayIdentifier *)this;
else
return E_NOINTERFACE;

AddRef();
return S_OK;
}


Implementation of the interface
+++++++++++++++++++++++++++++++

In this function, we have to return information about our icon. Our icon will be embedded in the DLL file, whose path is QAxFactory::serverFilePath(). It will be the first icon in the DLL file.

STDMETHODIMP ShellOverlay::GetOverlayInfo(LPWSTR pwszIconFile, int cchMax, int *pIndex, DWORD *pdwFlags) {
QString iconPath(QAxFactory::serverFilePath());
if (iconPath.length() > cchMax)
return S_FALSE;

int len = iconPath.toWCharArray(pwszIconFile);
pwszIconFile[len] = L'\0';

*pIndex = 0;
*pdwFlags = ISIOI_ICONFILE | ISIOI_ICONINDEX;

return S_OK;
}

This function is not important in this tutorial.
STDMETHODIMP ShellOverlay::GetPriority(int *pPriority) {
*pPriority = 0;
return S_OK;
}

In this function, we have to tell if a file's icon has to be modified by ours, or not. If the filename contains a 'q' or a 't', we reply it should.
STDMETHODIMP ShellOverlay::IsMemberOf(LPCWSTR pwszPath, DWORD dwAttrib) {
QFileInfo info(QString::fromWCharArray(pwszPath));
QString filename = info.fileName();
if (filename.contains(QChar('q')) || filename.contains(QChar('t')))
return S_OK;
return S_FALSE;
}

- http://msdn.microsoft.com/en-us/library/bb761265%28v=VS.85%29.aspx



To be continued in next post.

[Tilda]

The COM factory
---------------

In "shelloverlay.h", we will declare this class:
class ShellOverlayBinder : public QObject, public QAxBindable {
Q_OBJECT

public:
ShellOverlayBinder(QObject *parent = 0);

QAxAggregated *createAggregate();
};

Add a dllmain.cpp file to the project, which will contain:

#include <qt_windows.h>
#include <ActiveQt>
#include <QAxFactory>
#include "shelloverlay.h"
QT_USE_NAMESPACE
QAXFACTORY_DEFAULT(ShellOverlayBinder,
"{60c580d2-41f2-43ed-b5d1-b435d74d1999}", /* Class ID (CLSID) */
"{21a9e71b-9ae4-4887-8ada-720442394493}", /* Interface ID (IID) */
"{8c996c29-eafa-46ac-a6f9-901951e765b5}", /* event interface ID */
"{a6b1968a-4bbc-4420-9a55-5ce3579f795a}", /* Type Library ID (TLB) */
"{4f7d37e8-b9cb-4e66-a725-f043753b755c}" /* Application ID (AppID) */
)

Every UUID in this file has to be generated. The ones were used on the author's machine and are fine for this tutorial, but if you want to make your own shell extension, you will have to generate new ones yourself.

- http://doc.trolltech.com/4.6/qaxfactory.html

Additional files, embedding the icon
------------------------------------

We will need a .def file, which will declare the exported symbols. The file from <qt>\src\activeqt\control\qaxserver.def will be enough, we can use it verbatim.

We will also need a .rc file, containing this:

1 TYPELIB "qaxserver.rc"
1 ICON DISCARDABLE "qtoverlay.ico"

We will ultimately need an icon file. We will for example use Qt's favicon, modified so it only covers a quarter of the file.

% wget http://qt.nokia.com/favicon.ico
% convert favicon.ico -alpha on -background none -gravity southeast -extent 32x32 qtoverlay.ico

We will put these 3 files at the root of our project dir. You can add also add them to the project in QtCreator.


Build
-----

QMake configuration
+++++++++++++++++++
The qmake .pro file will need some modifications.
Add this:

CONFIG += dll qaxserver qt

qaxserver is used to include some default implementations (for the symbols from the .def file we previously included) ActiveQt provides and which suit our needs.

LIBS += -luser32 -lole32 -loleaut32 -lgdi32 -luuid
DEF_FILE = qaxserver.def
RC_FILE = qaxserver.rc

- http://doc.trolltech.com/4.6/qmake-manual.html

Include VisualStudioExpress tools path
++++++++++++++++++++++++++++++++++++++
TODO this section needs work
Open a "cmd" prompt, exec vcvars32.bat in it, and run qtcreator from there.

Build
+++++
The project should now build fine.


Run
---

Installation
++++++++++++
TODO this section needs work, the registry part should be in DllInstall perhaps in the future

A key with any name must be created in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Explorer\ShellIconOverlayIdentifiers, and its default value should be the bracketed (with '{' and '}' around it) ClassID of our extension (the class id can be found in dllmain.cpp).

Registration and dependencies
+++++++++++++++++++++++++++++
TODO this section needs work

Unfortunately, the DLL has dependencies to other DLL files, that have to be copied in the same directory as our DLL, these are:
- mingwm10.dll (found in <qt>\bin)
- libgcc_s_dw2-1.dll (found in <qt>\bin)
- QtCore4.dll (or QtCored4.dll if the extension is built in "debug")
- QtGui4.dll (or QtGuid4.dll if debug) (TODO this should not be needed)

The extension can now be registered, by typing "regsvr32 shellext_overlay.dll".

If regsvr32 says "the specified module could not be found", it may have some other dependencies. The "Dependency Walker" tool can be used to find them.

- http://www.dependencywalker.com/

Test
++++

Now, we can start notepad (or any app with a file selection dialog), select "open..." and watch our shell extension at work!

http://img215.imageshack.us/img215/2339/screenshot2ie.png (http://img215.imageshack.us/i/screenshot2ie.png/)

[Tilda]

Attaching the project files.

Sorry for the format of the posts, I wrote in semi-restructuredtext, it should be reformatted later.

[caesarxx]

Hi,

Thanks for this post !

I manged to build your project and it registers fine.... but somehow doesn't work.
I added

"
QFile file("/tmp/out.txt");
if (file.open(QIODevice::WriteOnly | QIODevice::Text)) {
QTextStream out(&file);
out << "getoverlayinfo " << "\n";
} else { }
"

to almost every function since i didn't know how to debug the dll file proberly.. but out.txt is emty..
do you have any hints ?

in your project file you named your icon qtdemo.ico not qtoverlay.ico... ?! i created both..
i used your Class IDs... should be fine for testing ?

it's my first qt project and i really don't know where to look....

Thanks,!

[Tilda]

Hello,



I manged to build your project and it registers fine.... but somehow doesn't work.
I added

"
QFile file("/tmp/out.txt");
if (file.open(QIODevice::WriteOnly | QIODevice::Text)) {
QTextStream out(&file);
out << "getoverlayinfo " << "\n";
} else { }
"

to almost every function since i didn't know how to debug the dll file proberly.. but out.txt is emty..
do you have any hints ?

Your .txt file is created by the DLL or not even at all? You also use the "/tmp/out.txt" path on Windows? Does it work?

Are you on a 32bits Windows with a 32bits QtCreator and did you restart explorer (restart explorer means either kill the process/launch again, or close session and restart, or reboot)? If not, it may be the cause if the registration works but the DLL doesn't do its job.
If it's the reason, you can still start a fresh process of a 32bits app (SciTE for example), and go to "File > Open..." see if overlays are present.
This may not be clear enough in this howto, it should be rewritten with better style.




in your project file you named your icon qtdemo.ico not qtoverlay.ico... ?! i created both..

qtdemo.ico is a mistake in the .pro file, it should be qtoverlay.ico but it shouldn't prevent the overlays from being displayed.


i used your Class IDs... should be fine for testing ?

Yes, using the same GUID is OK as both mine and yours are examples on our own machines.


About debugging, I have not tried that yet with but it's a good idea of a thing to look at. On the general principle, you should attach the debugger to a process that uses the DLL (explorer.exe for example) and put breakpoints in your DLL code.

[caesarxx]

Hi,

The TextFile isn't created at all.. i tested the code in another simple programm and "/tmp/out.txt" worked there.. ( even with windows.. )

seems like the dll isn't called at all.... ( or has no rights to access /tmp... who knows :-)


I tried restarting windows, explorer and i started notepad :-)

hmm... still have no idea.. maybe a windows 7 issue....

i try to get a debugger an explorer.exe working..

Thanks !

[Tilda]

Is it a 32bits Windows or 64bits?

I did not try with Windows 7, maybe it requires administrator rights to register the DLL.

[caesarxx]

Its 32 bits and i tried registering with a Administrator cmd line and without... maybe i try windows xp first...

what system do you use ?

[caesarxx]

for future reference :

i did everything a second time on a new windows installation, this time i generated new uuid from http://www.famkruithof.net/uuid/uuidgen
and now it works !!

Thanks for your Help and efford writing this tutorial !!!

[Tilda]

for future reference :

i did everything a second time on a new windows installation, this time i generated new uuid from http://www.famkruithof.net/uuid/uuidgen
and now it works !!

Thanks for your Help and efford writing this tutorial !!!

This is really strange, it can't come from the UUID, but I don't understand what was the problem with your old system :(

Thank you too for trying my tutorial :)
The tutorial still needs work, some things could be done better (like avoiding MS SDK), or explained better.
Don't hesitate to point out things that were unclear, you can even write text if you have inspiration, if the whole becomes good enough, we could post it to wiki as a collaborative work. :)

[caesarxx]

I think it's quite good ! There are only two things i didn't unterstand at first... ( and this was my first qt project ever ! )

1. Where is the icon comming from.. the filename nevers shows anywerer exept the resource file...
2. Where is the tmp folder stuff comming from....

2.-> i found out why it wasn't working on my first try... i don't know why but if i move my project in any other directory than "c:\qt\2010.04\qt" it doesn't work ! if i qmake the projekt the make files are very different than qmake in e.g. c:\projects\. The makefile creates a tmp folder and other stuff but only if qmake is called in the qt directory....... i still can't find out why it has to be in this directroy.. is it you default dir ? where is this saved ? i looked at every file in your example project...

very very stange.... i used cmd, not qtcreator...

i keep you postet if i every find out why... a.. and one other thing...

If anyone want's to use more than one icon :


in dllmain.cpp:

QAXFACTORY_BEGIN(
"{4c52cc51-89f0-48e0-9678-474e3613d9d7}", /* Type Library ID (TLB) */
"{5c52cc51-89f0-48e0-9678-474e3613d9d7}" /* Application ID (AppID) */
)
QAXCLASS(SOSyncedBinder)
QAXCLASS(SOUnsyncedBinder)
QAXFACTORY_END()



and in your class header file for each binded class:

class SOSyncedBinder : public QObject, public QAxBindable {

Q_CLASSINFO("ClassID", "{1955991f-0df0-4563-8a0b-624a227605d6}")
Q_CLASSINFO("InterfaceID", "{2955991f-0df0-4563-8a0b-624a227605d6}")
Q_CLASSINFO("EventsID", "{3955991f-0df0-4563-8a0b-624a227605d6}")

[Tilda]

I think it's quite good ! There are only two things i didn't unterstand at first... ( and this was my first qt project ever ! )

1. Where is the icon comming from.. the filename nevers shows anywerer exept the resource file...
2. Where is the tmp folder stuff comming from....

2.-> i found out why it wasn't working on my first try... i don't know why but if i move my project in any other directory than "c:\qt\2010.04\qt" it doesn't work ! if i qmake the projekt the make files are very different than qmake in e.g. c:\projects\. The makefile creates a tmp folder and other stuff but only if qmake is called in the qt directory....... i still can't find out why it has to be in this directroy.. is it you default dir ? where is this saved ? i looked at every file in your example project...

very very stange.... i used cmd, not qtcreator...

It's weird, I did not put my project anywhere under c:\qt\ but in "my documents", and IIRC I did not put absolute path references.




i keep you postet if i every find out why... a.. and one other thing...

If anyone want's to use more than one icon :


in dllmain.cpp:

QAXFACTORY_BEGIN(
"{4c52cc51-89f0-48e0-9678-474e3613d9d7}", /* Type Library ID (TLB) */
"{5c52cc51-89f0-48e0-9678-474e3613d9d7}" /* Application ID (AppID) */
)
QAXCLASS(SOSyncedBinder)
QAXCLASS(SOUnsyncedBinder)
QAXFACTORY_END()



and in your class header file for each binded class:

class SOSyncedBinder : public QObject, public QAxBindable {

Q_CLASSINFO("ClassID", "{1955991f-0df0-4563-8a0b-624a227605d6}")
Q_CLASSINFO("InterfaceID", "{2955991f-0df0-4563-8a0b-624a227605d6}")
Q_CLASSINFO("EventsID", "{3955991f-0df0-4563-8a0b-624a227605d6}")

Thank you for finding how to use multiple extensions in the same DLL! :)
And also thanks for the feedback!

[tanbmuit]

Error ? I run on QT 2.6.1 . Can you help me ???9086

[tanbmuit]

Icon overlay not show in win 64bit ??? help me

[SHuettner]

Same here. Cannot make it work on Win7 x64 with Visual Studio 2010 and Qt 4.8.4. Any hints?