Implimentation advice for a large data plotting application.

[Uwe]

A QwtSeriesData<QPointF> object is a bridge between the samples and the curve object, that is responsible for the representation of the samples on the plot canvas.

If you don't want to make a copy of your arrays simply derive from QwtSeriesData<QPointF> and implement the virtual methods getting the values from your integer arrays. Be careful with YourData::copy() - copy the interface only.

Qwt offers also a couple of classes derived from QwtSeriesData, that copy and store the samples ( in case of using Qt containers its not really copied, because those are implicitely shared ). These classes are often easier to use, but in your case you need to convert and copy your samples - f.e. into a QVector<QPointF>.

I guess it helps to look at the implementation of QwtArraySeriesData to understand how to implement your individual type of data class.

Uwe

[mike_the_tv]

Thanks Uwe for the swift reply. I now have something working as you suggested based on a QwtSeriesData<QPointF> which I have added to the simple test program in addition to the cos and sin plots. The class I created takes a pointer to my integer data, very similar to the QwtArraySeriesData class does with the QVector.

Am I correct in assuming that the plot data has to be ether float or double, as the derived class templates on QPointF rather than QPoint?

From what I now understand of the QwtSeriesData classes, when the QwtPlot needs to regenerate the view it retrieves every sample point value from my new data class via the sample() member, is that correct? In terms of large data sets is this approach generally efficient or is it best to consider converting my data into a better format for panning/zooming from my integer array?

If I add zooming and panning to the plot, does the sample() member just get called with the clipped area needed for plotting, for example if zoomed to samples 100-900 out of say 0-1000000, then the sample() member is called with the index between 100-900?

Is it best to fix the axis range or let the plot auto scale it? The reason I ask is that when I commented out the fixed axis range from the simple example to allow my integer only data to be more visible I found that the plot speed slowed down considerably. I did notice while looking at the QwtSeriesData class that the boundingRect() member had a warning about large data sets being slow. The implementation in my quick class knock up just set the QRect to a fixed range, so is it the QwtSyntheticPointData implementation that is slow then? I am likely to have series plot of up to 4 million points, potentially from 4 or more plot series on a single plot.

Am I best using the Qwt 6.x trunk code or would I be better off using a 5.x release? I think the other reason I went for the 6.x line was that I had issues compiling against Qt 4.6.1/2 with VS 2005.

Oh I remember the final question, is there a better source of documentation for these plot classes. I am using the Doxygen manual from sourceforge but as I said earlier I could not find reference to some of the deeper plot data classes. I would rather not post potentially "dumb" questions for forums if its explained in a manual somewhere which I have missed.

Thanks for the help.

[Uwe]

Am I correct in assuming that the plot data has to be ether float or double, as the derived class templates on QPointF rather than QPoint?

No, converting temporary QPointF samples from your integer arrays seems to be the right solution for me. Building an array of QPointF in advance would be a huge waste of memory and won't have any significant effect on the performance.

If I add zooming and panning to the plot, does the sample() member just get called with the clipped area needed for plotting, for example if zoomed to samples 100-900 out of say 0-1000000, then the sample() member is called with the index between 100-900?

No, QwtPlotCurve doesn't know about the internal order of your samples.

If you need to zoom large datasets and want to have a responsive GUI I recommend to implement different level of details inside of your data class. F.e. you could use Douglas Peucker ( QwtWeedingCurveFitter ) or any specific weeding algo to calculate smaller data sets for the different zoom levels. Also activate the QwtPlotCurve::ClipPolygons flag ( assuming you want to display the curve as polygon ). Polygon clipping will reduce the number of points for detailed views, weeding for the others.

Is it best to fix the axis range or let the plot auto scale it? The reason I ask is that when I commented out the fixed axis range from the simple example to allow my integer only data to be more visible I found that the plot speed slowed down considerably. I did notice while looking at the QwtSeriesData class that the boundingRect() member had a warning about large data sets being slow.

Of course you have to implement the boundingRect method of your data class. F.e. calculate the bounding rectangle only when your samples are changing and store it. Then let YourData::boundingRect() return the stored rectangle.

Am I best using the Qwt 6.x trunk code or would I be better off using a 5.x release? I think the other reason I went for the 6.x line was that I had issues compiling against Qt 4.6.1/2 with VS 2005.

As far as I can see you don't need any new feature of Qwt 6.x (maybe the weeding algo) and the compiler issues are solved in the 5.2 SVN branch as well. But there are heavy API changes between Qwt 5-2 and Qwt 6.0. It's up to you to decide if you want to work with a development snapshot or if you prefer to use a stable release having in mind, that you might have to port your code later.

Oh I remember the final question, is there a better source of documentation for these plot classes.

The missing high level documentation is the weakest part of Qwt. This is subject of critics for a long time and I'm aware of this. The only reason why I didn't start to work on it is, that I want to have the Qwt 6.x release first.

Uwe

[mike_the_tv]

No, QwtPlotCurve doesn't know about the internal order of your samples.

If you need to zoom large datasets and want to have a responsive GUI I recommend to implement different level of details inside of your data class. F.e. you could use Douglas Peucker ( QwtWeedingCurveFitter ) or any specific weeding algo to calculate smaller data sets for the different zoom levels. Also activate the QwtPlotCurve::ClipPolygons flag ( assuming you want to display the curve as polygon ). Polygon clipping will reduce the number of points for detailed views, weeding for the others.

If I understand correctly then, I need to add a type of filtering class to my data class to return effectively a lower resolution version of the data it internally stores to the QwtPlot (or QwtPlotCurve not sure which)?

You suggested I looked at the QwtWeedingCurveFitter, however I don't seem to be able to find this, where should I be looking?

I was also wondering if there is a description somewhere as to how the QwtPlotZoomer works with a QwtPlot and its associated QwtPlotCurve classes (and ultimately my data class derived from QwtSeriesData). I have added my data class as an additional plot line to the simple example. I have also now added a zoomer from a previous Qwt 5.2. This all works really nicely. I can rubber band an area and it zooms in, however I was a little surprised when the sample index that my data class was asked for always ranged over the entire data set. I had expected the zoom area to have been the bound on this.

Oddly if you track up the call stack you eventually get to a QwtPlotAbstractSeriesItem::draw() member which calls drawSeries() with a from and to of 0 and -1, which in the drawSeries() member are defined as the index of the first and last point to be plotted. Why is this not the first and last point identified from the zoomer? Is it the QwtPlot that discards data points which fall outsize of its zoom range?

Cheers

[Uwe]

If I understand correctly then, I need to add a type of filtering class to my data class to return effectively a lower resolution version of the data it internally stores to the QwtPlot (or QwtPlotCurve not sure which)?

Inside of your data object. I recommend to connect something to the QwtScaleWidget::scaleDivChanged() signals to activate the best resolution for the current scale ranges.

You suggested I looked at the QwtWeedingCurveFitter, however I don't seem to be able to find this, where should I be looking?

qwt_curve_fitter.[h|cpp] - didn't you write, that you are on trunk ?

I was also wondering if there is a description somewhere as to how the QwtPlotZoomer works with a QwtPlot and its associated QwtPlotCurve classes (and ultimately my data class derived from QwtSeriesData).

The zoomer manipulates the scales - your curves and your data object are completely unrelated. Note that the plot has current scale ranges, but the zoom stack is an internal thing of the zoomer. For rendering the content of the plot canvas it is unimportant if the scales are the result of a zoom operation or not.

... however I was a little surprised when the sample index that my data class was asked for always ranged over the entire data set. I had expected the zoom area to have been the bound on this.

In general the index of a point has nothing to do with its coordinates ( start the realtime example, zoom in and you will understand immediately ) !

I guess in your specific situation the values are ordered, what could be used to optimize painting but using specific characteristics of a series can only be implemented, where these are known: your data object.

Why is this not the first and last point identified from the zoomer?

Again, there are not first and last points according to coordinates. But even coordinates, that are outside of the scale ranges might be important: f.e. in case of QwtPlotCurve::Lines the connecting line might be intersecting the visible area.

Is it the QwtPlot that discards data points which fall outsize of its zoom range

QwtPlotCurve does some optimizations like polygon clipping or discarding points depending on the type of visualization. But it can't do it without knowing the points.

Uwe

[mike_the_tv]

qwt_curve_fitter.[h|cpp] - didn't you write, that you are on trunk ?

Yes I am on the trunk and I searched for QwtWeedingCurveFitter using Visual Studios search in files, but for some reason it found nothing. Now I go and look for the file you say I can find it. Not quite sure what went on there .

In general the index of a point has nothing to do with its coordinates ( start the realtime example, zoom in and you will understand immediately ) !

I guess in your specific situation the values are ordered, what could be used to optimize painting but using specific characteristics of a series can only be implemented, where these are known: your data object.

OK point taken. The realtime example plots random point whos position in the plot area bares no relationship to its position in the data set. I guess I was assuming that a data object based on a QwtSeriesData class may make some assumptions about the ordering or type of data it contained and therefore optimise access at different zoom/crops.

Again, there are not first and last points according to coordinates. But even coordinates, that are outside of the scale ranges might be important: f.e. in case of QwtPlotCurve::Lines the connecting line might be intersecting the visible area.

Again I had not considered this as you would need to know points outside the visible area in order to correctly plot the first and last visible points.

Inside of your data object. I recommend to connect something to the QwtScaleWidget::scaleDivChanged() signals to activate the best resolution for the current scale ranges.
...
The zoomer manipulates the scales - your curves and your data object are completely unrelated. Note that the plot has current scale ranges, but the zoom stack is an internal thing of the zoomer. For rendering the content of the plot canvas it is unimportant if the scales are the result of a zoom operation or not.
...
QwtPlotCurve does some optimizations like polygon clipping or discarding points depending on the type of visualization. But it can't do it without knowing the points.

I am still confused about this and I think it comes from not fully understanding the interaction between the various components within a plot. If you can bare with me on this I would really appreciate a quick summary of how things connect and interact (or please if there is a concise summary document somewhere I have missed point me at it so I don't waste your time .

What I think I understand at the moment is this. If I want to plot some data I need a QwtPlot object, or at least something derived from this. I am not sure at the moment whether deriving from QwtPlot is necessary but most of the examples do.

The to provide my data to the QwtPlot I need one or more QwtPlotCurve objects which I attach to my QwtPlot. My QwtPlotCurve is supplied with data via a QwtSeriesData object, which in my case is a derived data class based on QwtSeriesData that returns QPointF() values for samples specified by the sample index with a call to the sample() member.

From tracing with the debugger I can see that when my QwtPlot wants to plot my data, the sample() member of my derived data class is called for each data point, QwtPlot then does something with this data to cull points outside the current view, but ultimately results in my data appearing on screen.

Where my understanding now breaks down is how zooming and panning (preferably scrolling) fits in with all this. I also don't understand how the weeding out of excess data is going to work, even now I have found the weeding class. I have done similar data plotting in a previous version of my application. In this I used MFC and DirectX. Now in this set up if my plot width, in pixels, was less than the number of data points to display, i.e. we have way more detail than can be physically plotted, my code would step over each display pixel, calculate how many data samples this single pixel was representing, and then calculate a single value for all those data points to display at that pixel location. In my case I did a min/max and plotted a vertical line to indicate the min/max extent on the data.

Now the way QwtPlot works is probably completely different to what I have done previously. I have tried looking at the QwtWeedingCurveFitter() class and attached it to my QwtPlotCurve class via setCurveFitter(), however when I debug the code to work out how things connect and interact, I don;t see any of the weeding class members being called. How and where does this class get called?

I also don't really understand how the zooming works. You had said that the QwtPlotCurves and my data object are unrelated to the zoomer, this just affecting the scales on the QwtPlot I assume. You also said that I should use the QwtScaleWidgets scaleDivChanged() signal to select the best data resolution, however this signal does not seem to supply any parameters about the scale change, so I am not quite sure how I should use this notification.

The sort of effect I am after in the end is similar to your realtime example, except that the data will be available in one go and will be a linear time series plot. The ability to zoom and then pan the plot is what I need to match my original application under MFC/DirectX, however I don't really understand how to achieve this. When I tried to follow what the example was doing I quickly got lost because I think I am missing this fundemental understanding of how the various bits fit together.

Apologies for such a long reply, and many thanks so far for your assistance.

[Uwe]

I am still confused about this and I think it comes from not fully understanding the interaction between the various components within a plot.

Looks like you are trying to understand something, that is not there. Things are much simpler, than you expect:

1) The plot widget knows the transformations and scale ranges and passes them to the internal render engine. But it doesn't know much about the plot items - beside a couple of informations accessible via the QwtPlotItem base class.

2) There are 2 types of plot items: decorators ( like the grid or scales items ) or items representing some type of data. F.e. a curve represents a series of 2D points, a spectrogram represents raster data.

3) Zoomer, panner and other helper objects translate user interactions into scale changes. The plot widget doesn't know these helpers and it is of no importance if the scales were set from a zoomer, the autoscaler or are assigned from your code ( setAxisScale() ).

I have done similar data plotting in a previous version of my application. In this I used MFC and DirectX. Now in this set up if my plot width, in pixels, was less than the number of data points to display, i.e. we have way more detail than can be physically plotted, my code would step over each display pixel, calculate how many data samples this single pixel was representing, and then calculate a single value for all those data points to display at that pixel location. In my case I did a min/max and plotted a vertical line to indicate the min/max extent on the data.

You can do this as well (overloading QwtPlotCurve), but Douglas Peucker has more potential as you don't need the widget geometry. You can resample your points in advance and don't need to slow down resize ( or even worse repaint ) operations.

Now the way QwtPlot works is probably completely different to what I have done previously. I have tried looking at the QwtWeedingCurveFitter() class and attached it to my QwtPlotCurve class via setCurveFitter(),

This way ( you forgot to enable the QwtPlotCurve::Fitted attribute ) the curve resamples the points after they have been translated into widget coordinates each time your curve is painted. Because Douglas Peucker is an expensive algo this will have a negative effect and slow down your repaints.

Instead I recommend to use QwtWeedingCurveFitter to calculate a couple of arrays with less points from your series only once - when you assign your samples. You find your arrays by increasing the tolerance parameter until the number of points is below a certain limit ( f.e 1000 points ). A good step size for the tolerance depends on the bounding rect of your series, but I guess you don't need to have more than 5 arrays.

Then the only thing you need to do is to activate one of your arrays depending on the current scale ranges ( f.e. in a slot connected to scaleDivChanged() ). You can find the current scale divisions by plot->axisScaleDiv(...). Of course you can add some additional optimizations (calculating start/end indices) using the order of your points here too.

When the plot displays large scale ranges your data object will return less points from a much smaller array. If the scale ranges are small your data object will return many points, but most of them will be clipped. In both situation the number of paint operations is heavily reduced and you will see a good performance for painting your plot. ( Implementing a such a data object calculating different level of details internally is on my TODO list. )

Uwe

[darkforce]

When the plot displays large scale ranges your data object will return less points from a much smaller array. If the scale ranges are small your data object will return many points, but most of them will be clipped. In both situation the number of paint operations is heavily reduced and you will see a good performance for painting your plot. ( Implementing a such a data object calculating different level of details internally is on my TODO list. )
Uwe

The implementation of the Douglas-Puecker algorithm in QwtWeedingCurveFitter works perfectly and the painting performance gets much better with large datasets. However, when I scroll the plot (using a scrollzoomer), the algorithm is called unnecessarily even though the level of resolution did not change. This could pose a problem when really large datasets are loaded. Another feature that could be useful is to allow users to control the display of symbols either on the original data points or on the reduced data points. Maybe it is already implemented somewhere, I just did not do any research on that. Thank you for the great job!

[Uwe]

However, when I scroll the plot (using a scrollzoomer), the algorithm is called unnecessarily even though the level of resolution did not change.

It's your code, that decides, when to rebuild the internal datasets. As long as those doesn't use clipping I can't see any reason why you should rebuild them, when the resolution doesn't change significantly.

Another feature that could be useful is to allow users to control the display of symbols either on the original data points or on the reduced data points. Maybe it is already implemented somewhere

You would have to overload QwtPlotCurve::drawSymbols(), but unfortunately it's less comfortable as it sounds, because there is some substance in the implementation of the base class you would have to copy.

Uwe