Plugin design advice needed

[d_stranz]

Most of the applications I write have a common theme, that of processing certain types of scientific data and displaying the results. In nearly all cases, many of the pre-processing steps are the same from app to app, and what makes the apps different is the nature of the samples that were analyzed to produce the data, the details of how the pre-processed data are further processed, and the way in which results are visualized.

There is starting to be a lot of cut and paste style code "reuse". I am thinking that it would be better for me to design a plugin architecture where the common pre-processing and visualization features could be encapsulated into one or more stable plugins. I also like the potential that a plugin framework would give to allow extending the app by simply adding new plugins.

My question involves communication between the framework (that uses the plugins) and the plugins themselves. I can see a choice between two basic options:


Define well-known interfaces for both the framework and plugins, and have the two communicate using these interfaces and their methods.
Define signals and slots for both framework and plugins with minimal dependence on interfaces and methods.

(or perhaps some blend of these two).

Some of the things that each plugin may have are:


Parameters that are used to transform the data
A GUI for editing the parameters
Parameter serializing so the framework store or retrieve them
Commands to add to the framework's menus and / or toolbars
Plugin-specific GUI elements (dock windows, views, etc.)
Status reporting for lengthy operations
A way to interrupt lengthy processing
Results of the processing (new data to be added to the framework's data model)
Results serialization
Result visualization and user interaction with the visuals
etc.


These seem to fall into two categories of functionality:


Configuration-related issues that take place when the plugin is loaded / unloaded
Dynamic issues in which the plugin responds to changes caused by the framework


The Qt examples of plugins (as well as what I have read in books) tends to lean toward the "well-known interfaces" model, with plugins exposing multiple interfaces for various functions. I can see where this would work for exposing features like building menus and editing and serializing parameters.

In my case, some of these plugins might form part of a processing pipeline, where the inputs needed by a plugin may not be available until some earlier plugin has done its work. This implies that a signal / slot mechanism might work better here, where a plugin listens for a signal from the framework that announces availability of required data before it goes to work.

I would like it if the framework knew as little as possible about the nature of the plugins (in particular, where in a pipeline it will act or even if it is part of a pipeline at all).

Does anyone have experience in building this type of a plugin framework, or can anyone point me to a serious example of such a framework?

Should I forget plugins entirely and simply build a library that exposes the common functionality? (Although I really like the appeal of being able to extend the app with new plugins, which you can't do with linked libraries).

Any advice would be appreciated.

[anda_skoa]

Not sure if this will really help you, but QtCreator is really serious about using plugins.
I.e. basically everything in it is a plugin and plugins have dependencies on other plugins and so on.

See http://doc.qt.digia.com/qtcreator-extending/index.html and similar documentation.

Cheers,
_

[d_stranz]

Not sure if this will really help you, but QtCreator is really serious about using plugins.
I.e. basically everything in it is a plugin and plugins have dependencies on other plugins and so on.

Thanks - I wasn't aware that QtCreator was plugin-based. I use MSVC and the Qt Add-in for development, so I am unfamiliar with some of the Qt-native tools. I will check it out. From what I can imagine, I think QtCreator must deal with similar issues.

[lanz]

I use a script and a bunch of modules, that expose their interfaces via qt metacalls.
So the framework is basically loading the modules, loading the script and firing it up. Script then does all the work.
Downside is there is a need in another framework inside script (or writing custom script for every project, that is what I'm doing now).

Also I can think of using qt messaging system to deliver commands and data to plugins. Say plugin consist of interface that produces message handler that is installed into framework's message loop and a method that stores backreference to this loop.
Framework then starts pumping messages (it can emit some kind of "init" message in the beginning).
Plugins then can establish direct connections between themselves if they want.

[wysota]

Bear in mind the difference between a plugin and a shared library. What makes them different is that plugins are optional whereas linked libraries are mandatory for a program to work. If you are going to always load a plugin into the program (simply because it requires its functionality to work) then it is easier (development wise) to make it a library. Added functionality can be spread among libraries and plugins, e.g. you can have a library that defines common interfaces and you can link that library to the main app and some of the (optional) plugins that require those interfaces.

In my previous work we were developing a large application that was composed of a large number of plugins. It had a terrible design because functionality from plugin A depended on plugins B and C and A was required for D, E and F to work, etc. Eventually everything depended (=mandatory dependency) on everything else and we were always deploying all the plugins anyway (even if some functionality was disabled through the config). It didn't really make any sense to implement all those functions as plugins, they could have been regular libs or even everything could have been a monolithic executable. Especially that all the plugins to be loaded had to be listed in the config file (this was the only place plugins were actually useful as you could have prevented some of them from being loaded if you knew (by trial and error) which of them were not required by the functionality you wanted to test/debug -- however it could have been achieved using libraries as well). All in all -- don't implement plugins just because you can. Think whether there are any benefits in your situation compared to using shared libraries.

We also had another application system that was modelled after QtCreator -- i.e. it was composed of many small plugins that "pushed" functionality into a common pool of objects that other plugins (which those initial ones depended on) could pull from the pool throughout a known interface and use. It was a much better design as it really allowed us to have optional functions -- even if some plugin failed to load other plugins could function without it and failing to load a plugin would automatically fail all plugins that depended on it. However in this design the functionality that was related, could be spread among many different plugins. The bright side was that adding new functionality was in total separation of existing functionality.

[d_stranz]

If you are going to always load a plugin into the program (simply because it requires its functionality to work) then it is easier (development wise) to make it a library.

I think I agree with this, although QtCreator has adopted the approach of putting all the required bits into the Core module, which is a mandatory plugin. The QtCreator executable is little more than a plugin loader and manager, with Core the first (and required) plugin loaded. So I suppose if you wanted a reusable plugin-based framework where any kind of functionality could be provided by a Core-like required plugin, this would be a way to do it. I don't know if I need to be so generic, so I will probably move the basic functionality into the main program.


Eventually everything depended (=mandatory dependency) on everything else

That's another worry, but I think by careful segregation of functionality into a set of well-defined plugins, I can hopefully avoid this.


We also had another application system that was modeled after QtCreator -- i.e. it was composed of many small plugins that "pushed" functionality into a common pool of objects that other plugins (which those initial ones depended on) could pull from the pool throughout a known interface and use.

Yes, I like the "object pool" concept. I might get some significant advantage out of the pool and the objects in it signalling that they have been modified, and letting plugins decide if they are interested. This means creating QObject wrappers for my pure C++ computational classes, but that is not hard; more difficult is that many of my collections of object instances are based around boost smart pointers so management of wrapped object lifetimes will be a problem to be solved, probably by creating QObject smart pointers and eliminating QObject parent-child relations. (Or by designing collections such that the objects within them do not need to be wrapped).


The bright side was that adding new functionality was in total separation of existing functionality.

This is the hope, isn't it?

[wysota]

QtCreator has adopted the approach of putting all the required bits into the Core module, which is a mandatory plugin.
The Core plugin is different. It is being loaded by name before other plugins are loaded. It might have been a library instead.


That's another worry, but I think by careful segregation of functionality into a set of well-defined plugins, I can hopefully avoid this.
Sharing my experience (I wasn't responsible for this terrible architecture, nevertheless I couldn't find a way to untie this Gordian Knot without totally rewriting the system) I strongly advise you to decide apriori which functionality goes where and what depends on what and then stick to what you designed. Splitting or merging as you go will lead you exactly where we landed.


This is the hope, isn't it?
Well... it depends. My problem was that I was the only one fully understanding the plugin architecture so I was being asked a lot of questions how to do things with it.

[d_stranz]

I strongly advise you to decide apriori which functionality goes where and what depends on what

I typically do this; by the time I get to actually writing code it almost writes itself, it has been so thoroughly planned out. This helps me later, because I know exactly where everything is and which parts depend on something else when changes must be made. But it frustrates the people who keep asking, "When can you show me something?"


and then stick to what you designed.

And that's the hard part. Fortunately most of my efforts are one-person projects, so it is a bit easier to enforce my own design rules.