QwtPlotCurve draw it's full contents without using current zoom information

[Dmitri]

I'm check Qwt 6.0.1.

I have data from ADC with 200 Hz. Next i'm load to QwtPlot data with one hour from file. That's 3600*200 = 720 000 points. Second i'm install QwtPlotZoomer for zooming and draging plot (In future i'm want to use scrollbar navigation). The problem that QwtPlotCurve draw it's full contents without gettings information from current zoom is VERY SLOW.

I'm find this code that always calling before the curve is draw:

qwt_plot_seriesitem.cpp:

void QwtPlotAbstractSeriesItem::draw( QPainter *painter,
        const QwtScaleMap &xMap, const QwtScaleMap &yMap,
        const QRectF &canvasRect ) const
{
    drawSeries( painter, xMap, yMap, canvasRect, 0, -1 );
}

void QwtPlotAbstractSeriesItem::draw( QPainter *painter,
        const QwtScaleMap &xMap, const QwtScaleMap &yMap,
        const QRectF &canvasRect ) const
{
    drawSeries( painter, xMap, yMap, canvasRect, 0, -1 );
}

To copy to clipboard, switch view to plain text mode 

We can see that curve always draw it's full contents (from = 0, to = -1 (size of data)).

Is it possible to workaround this behaviour?

[Uwe]

Using the characteristics of the data is the job of the application code - QwtPlotCurve doesn't know about them.

I recommend to derive from QwtSeriesData<QPointF> ( or one of its derived classed ), where you return only points from the current zoom interval: check the pure virtual methods.

To know the current scale interval do:

connect( plot->axisWidget( QwtPlot::xBottom ), SIGNAL( scaleDivChanged() ), this, updateCurrentZoom() );

connect( plot->axisWidget( QwtPlot::xBottom ), SIGNAL( scaleDivChanged() ), this, updateCurrentZoom() );

To copy to clipboard, switch view to plain text mode 

and something like this:

void YourWhatever::updateCurrentZoom()
{
    const QwtScaleDiv scaleDiv = plot->axisScaleDiv( QwtPlot::xBottom );
    updateCurrentZoom( scaleDiv->interval() );
}

void YourWhatever::updateCurrentZoom()
{
    const QwtScaleDiv scaleDiv = plot->axisScaleDiv( QwtPlot::xBottom );
    updateCurrentZoom( scaleDiv->interval() );
}

To copy to clipboard, switch view to plain text mode 

Uwe

[Dmitri]

Ok, i'm understand. Now it's works faster for smaller amount of points. Of course it's works SLOW for minimum zoom when user try to see all points. I'm also try to workaround this behaviour using QwtWeedingCurveFitter (Douglas/Peuker algorithm), but that's works slower as compared with disabled fitter. After Douglas/Peuker i see smaller amount of points. It looks like the alogrithm of QwtWeedingCurveFitter working more time than direct drawing operation of full points.

[Uwe]

Ok, i'm understand. Now it's works faster for smaller amount of points. Of course it's works SLOW for minimum zoom when user try to see all points

Check the archive of this forum for "levels of detail".

Using Douglas/Peucker you create ( in advance ) f.e 4 different sets of samples. Depending on the current zoom level one of these sets gets activated. Also enable polygon clipping.

Then you have the effect, that clipping ( the algo is fast ) - or/and your algorithm using the sample order - reduces the number of point when zooming deep and weeding for large scale ranges.

Uwe

[mike_the_tv]

Using the characteristics of the data is the job of the application code - QwtPlotCurve doesn't know about them.

I recommend to derive from QwtSeriesData<QPointF> ( or one of its derived classed ), where you return only points from the current zoom interval: check the pure virtual methods.

To know the current scale interval do:

connect( plot->axisWidget( QwtPlot::xBottom ), SIGNAL( scaleDivChanged() ), this, updateCurrentZoom() );

connect( plot->axisWidget( QwtPlot::xBottom ), SIGNAL( scaleDivChanged() ), this, updateCurrentZoom() );

To copy to clipboard, switch view to plain text mode 

and something like this:

void YourWhatever::updateCurrentZoom()
{
    const QwtScaleDiv scaleDiv = plot->axisScaleDiv( QwtPlot::xBottom );
    updateCurrentZoom( scaleDiv->interval() );
}

void YourWhatever::updateCurrentZoom()
{
    const QwtScaleDiv scaleDiv = plot->axisScaleDiv( QwtPlot::xBottom );
    updateCurrentZoom( scaleDiv->interval() );
}

To copy to clipboard, switch view to plain text mode 

Uwe

Hi
I am interested in doing exactly the same optimization on a zoom. My data is also time domain so for a rectangle x, x + n all I really need to do is return samples x, x+1, x+2, ... x + n.

I understand the concept but not the example code given. I currently have a derived AnalogPlotData class based on QwtSeriesData<QPointF> which stores my plot data. Where however to I add the connect() to get notified of the plot scale change and to what class do I add the updateCurrentZoom() function to limit the from/to parameters to drawSeries(). I tried searching the Qwt sources for references to updateCurrentZoom() but found none (Qwt 6.0.1).

Do I need to derive a version of QwtPlotAbstractSeriesItem as well?

Addituonal guided prods in the right direction much appreciated.

David

[Uwe]

It is probably enough to implement:

virtual void AnalogPlotData::setRectOfInterest( const QRectF &rect )
{
   // rect is the visible area in scale coordinates - the scales the corresponding curve is attached to.
 
   ...
}

virtual void AnalogPlotData::setRectOfInterest( const QRectF &rect )
{
   // rect is the visible area in scale coordinates - the scales the corresponding curve is attached to.

   ...
}

To copy to clipboard, switch view to plain text mode 

In Qwt 6.0 it gets called automatically when using Qwt from SVN trunk only when the QwtPlotItem::ScaleInterest flag is set.

HTH,
Uwe

PS: internally QwtSyntheticPointData uses this hook too

[mike_the_tv]

Hi Uwe

It is probably enough to implement:

virtual void AnalogPlotData::setRectOfInterest( const QRectF &rect )
{
   // rect is the visible area in scale coordinates - the scales the corresponding curve is attached to.
 
   ...
}

virtual void AnalogPlotData::setRectOfInterest( const QRectF &rect )
{
   // rect is the visible area in scale coordinates - the scales the corresponding curve is attached to.

   ...
}

To copy to clipboard, switch view to plain text mode 

In Qwt 6.0 it gets called automatically when using Qwt from SVN trunk only when the QwtPlotItem::ScaleInterest flag is set.

Yep, already implemented this, though I was not using it. When I run the code the rectangle parameter passed to the function does indeed define the area selected for zooming. However I don't see how I can use this to limit the from/to range requested by the drawSeries() call. I could use the rectangle to return a NOP sample value, 0.0, for all points outside the rectangle, but would this improve the plotting time as drawSeries() would still call out sample values for indexes 0 to m, even if only x to x+n are the only ones in the rectangle of interest. I could possibly see a speed up if the code around this could then cull the NOP values, is that how it works?

I must be using an older version of Qwt 6 (I think I took the 6.0.1 tag last year some time) as the QwtPlotItem::ScaleInterest is not present in the header. Interestingly however the setRectOfInterest() still gets called. Quite a bit in that header alone has changed, is it worth updating to the trunk?

HTH,
Uwe

PS: internally QwtSyntheticPointData uses this hook too

OK I had a look at the QwtSyntheticPointData use of setRectOfInterest(). If I understand it correctly, the rectangle is used to generate the synthetic data rather than weed out points outside the selected region. Is there not a way given application knowledge of the data set to limit the from/to range?

I am also trying to implement a weeding algorithm to reduce the overall number of points to plot when zoomed out (10,000,000+ sometimes). I have looked at the QwtWeedingCurveFitter as an example and if I understand this correctly, the QwtCurveFitter derived class gets called after the sample data has been converted from sample coordinates to plot coordinates.

In a previous Direct X implementation of our software I found it quite effective to reduce the plot points by taking the min/max of all sample points represented by one plot point (screen pixel) and plot that as a simple vertical line indicating the range of data represented by that one plot point. This I should be able to do from the polygon data passed to the fitCurve() member.

One thing I did notice however was that even when zoomed in, the whole data set if passed to the fitCurve() function. Would it be reasonable at this stage to use the scaleDivChanged() SIGNAL to update the QwtCurveFitter context to enable me to cull all points outside the rectangle of interest? I guess if this is not bad practice I can also use this to cull extra points from the zoomed data as well? Or is the better way to do this to limit the original from/to range?

Thanks

[Uwe]

When I run the code the rectangle parameter passed to the function does indeed define the area selected for zooming. However I don't see how I can use this to limit the from/to range requested by the drawSeries() call.

Store the rectangle and use it in your implementation of:

• virtual size_t AnalogPlotData::size() const;
• virtual QPointF AnalogPlotData::sample( size_t i ) const;

I must be using an older version of Qwt 6 (I think I took the 6.0.1 tag last year some time) as the QwtPlotItem::ScaleInterest is not present in the header. Interestingly however the setRectOfInterest() still gets called.

With Qwt 6.0 setRectOfInterest() is always called - with trunk it is called depending on the flag.

]Quite a bit in that header alone has changed, is it worth updating to the trunk?.

Depends on your requirements, but the version in trunk is probably the version with the most new features ever. For curves you find a couple of performance optimizations and when you are not on X11 ( here you have hardware acceleration ) you might be interested in checking the experimental OpenGL canvas.

]I am also trying to implement a weeding algorithm to reduce the overall number of points to plot when zoomed out (10,000,000+ sometimes). I have looked at the QwtWeedingCurveFitter as an example and if I understand this correctly, the QwtCurveFitter derived class gets called after the sample data has been converted from sample coordinates to plot coordinates.

Douglas Peucker is an expensive operation - better don't assign it as fitter to the curve.

Instead I recommend to build a couple of series with different tolerances ( maybe about 5 ) in advance ( using QwtWeedingCurveFitter directly ) and store them in your data class. Then activate one of them according to setRectOfInterest(). Together with clipping ( fast operation ) this should be what you are looking for.

I always wanted to implement a small demo showing how to implement different levels of detail for a huge curve. When you offer me a generator for curve points that make sense for such a demo you would find a working implementation in SVN.

Uwe

[mike_the_tv]

Store the rectangle and use it in your implementation of:

• virtual size_t AnalogPlotData::size() const;
• virtual QPointF AnalogPlotData::sample( size_t i ) const;

OK, I think I understand now. The AnalogPlotData would store its actual size as well as the rectangle of interest size. When the ROI is smaller, return that value for size() and for sample() return an offset subset of the data based on where the ROI is in the whole data vector, i.e. if the total size is m and the ROI in the x axis is x to x+n, size() would return n and sample(0), (1), (2), ... would return actual data values data[x], data[x+1], data[x+2], ... instead of data[0], data[1], data[2], .... I can then see that the plot routines would lookup the smaller ROI size() to calculate the from/to values and the sample() function would only return data from the ROI.

I would have to modify some of my existing code and add one or two members to the AnalogPlotData class to return the actual data size and the complete sample data as other parts of my code rely on knowing the real total number of samples and being able to access the whole data array, for example to compute an FFT, etc.

With Qwt 6.0 setRectOfInterest() is always called - with trunk it is called depending on the flag.


Depends on your requirements, but the version in trunk is probably the version with the most new features ever. For curves you find a couple of performance optimizations and when you are not on X11 ( here you have hardware acceleration ) you might be interested in checking the experimental OpenGL canvas.

Are any of the performance improvements likely to benefit Windows?

Douglas Peucker is an expensive operation - better don't assign it as fitter to the curve.

Just to see how the curve fitter class worked I did try attaching this directly and it was indeed very slow.

Instead I recommend to build a couple of series with different tolerances ( maybe about 5 ) in advance ( using QwtWeedingCurveFitter directly ) and store them in your data class. Then activate one of them according to setRectOfInterest(). Together with clipping ( fast operation ) this should be what you are looking for.

The only thing that puts me off using this is the time needed to compute the pyramid of scales prior to displaying the data. As the data sets could be very large (I just calculated the theoretical limit of samples returned by our hardware and it was in excess of 256,000,000, 32bit samples) the time taken to initially plot the data, even if I backgrounded calculation of other pyramid scale levels, would be too large. Once calculated the redraw would indeed be very fast as it would be all cached.

I now have a rough implementation of my min/max of all samples represented by one plot pixel and this works very well even when calculated on each redraw and for all data samples. Once you get multiple line plotted for sub-pixel samples, all you really see if the waveform envelope anyway.

What would be really good would be to push this weeding of sample points back into the AnalogPlotData class as this would eliminate the need for weeding a large data set down before plotting and also reduce the memory requirements of the application as the QPolygonF vector constructed in drawLines() from the sample() data would not need to be as big as the data set (x2 as its x and y coordinates), but only as big as the weeded data set (x2). With the potential number of samples I need to support, the amount of host memory used in storing the actual data and any temp copies is getting quite significant. To do this I would need to know the plot canvas size and also get updates when this changes.

I always wanted to implement a small demo showing how to implement different levels of detail for a huge curve. When you offer me a generator for curve points that make sense for such a demo you would find a working implementation in SVN.

Uwe

If I come up with something simple I will pass it on.

[Uwe]

The only thing that puts me off using this is the time needed to compute the pyramid of scales prior to displaying the data.

As you already have the optimization ( according to the ROI ) for the detailled zoom levels in place you can start with a larger tolerance value that should be very successful with weeding out points. How successful depends on the characteristics of your curve, but if it is not noise only you should have a result that is below 10% of the original size. For the following sets ( maybe 1 or 2 more ) you can use the result of the first run.

So there should be no memory problem and performancewise it boils down to the first run only. How long does it take to run the algo for a huge set on your system: more than a second ?

Note that you can split your set into parts and run the algorithm one by one reuniting the results later. This way you can avoid the need of a temporary copy and you can distribute the calculation to different threads. On multicore systems I would expect to have almost a linear effect for each core.

Are any of the performance improvements likely to benefit Windows?

First of all note, that when you have your implentation of the levels of detail I would expect that you are fine with or without optimizations.

Qwt 5.x had a integer based painting engine ( because of Qt3 compatibility ), Qt 6.0 runs on floats. Unfortunatly Qt 4.7 ( raster paint engine ) had a performance issue so that QPainter::drawPolylineF was 3 times slower than QPainter::drawPolyline - for the same values !

So I modified the code in SVN trunk to use integers for paint devices with integer based coordinate systems ( f.e on screen ) and floats for the others ( PDF, SVG ). Having integers ( or rounded floats since Qt 4.8 ) makes it possible to remove duplicates before passing them to Qt. F.e. when drawing a curve with 256,000,000 points to a widget with ~1000 pixels width almost all points will be duplicates.

Here it would also be possible to reduce the memory for the translated points heavily - but this is not done yet ( see qwtToPolylineFiltered in qwt_point_mapper.cpp ). But if you want to avoid a huge temporary buffer you could implement YourCurve::drawSeries calling QwtPlotCurve::drawSeries for intervals of f.e 10000 samples each: from -> from + 10000, from +10000 -> from + 20000 ...

Caching of symbols is another optimization - but this is more important for scatter plots than for your use case.

Using OpenGL is the only way to have hardware accelerated graphics on Windows, but this is more for oscilloscope alike use cases and as QwtPlotGLCanvas is in an experimental state ( f.e no caching yet ) I wouldn't recommend it to you. But you can have a try: setting an OpenGL canvas is one line of code only.

HTH,
Uwe

[mike_the_tv]

As you already have the optimization ( according to the ROI ) for the detailled zoom levels in place you can start with a larger tolerance value that should be very successful with weeding out points. How successful depends on the characteristics of your curve, but if it is not noise only you should have a result that is below 10% of the original size. For the following sets ( maybe 1 or 2 more ) you can use the result of the first run.

So there should be no memory problem and performancewise it boils down to the first run only. How long does it take to run the algo for a huge set on your system: more than a second ?

I tried an experiment with one channel run through the QwtWeedingCurveFitter and the other through the min/max weeding routine I have. With 1,000,000 samples the QwtWeedingCurveFitter with a tolerance of 50 (not sure if that's large or small) took about 20+ seconds to weed the 1,000,000 points to about 1557. The routine I used was less than a second and generated about 1600 points (2 x number of plot pixels). This was at full zoom out so no ROI optimisations at this stage. To be fair this was plotting two waveforms, the time domain, which had the weeders attached, and a frequency domain FFT, which had no weeders attached. However the FFT plot time was the same complexity for both time domain plots and from a visual guess was maybe 1-2 seconds to display.

I think the killer for the QwtWeedingCurveFitter routine is the heavy use of qSqrt(). Also from a visual perspective, because I had used what seemed to be a fairly high tolerance, the plot was not all that accurate to the general profile of the data, there were noticable gaps in the plot envelope where sample peaks had obvously been optimised out but where there was data. The data being plotted is simply a sine wave with a slight amount of modulated noise and phase variation to simulate the hardware variations.

Note that you can split your set into parts and run the algorithm one by one reuniting the results later. This way you can avoid the need of a temporary copy and you can distribute the calculation to different threads. On multicore systems I would expect to have almost a linear effect for each core.

Yeh had thought about doing that but given the speed test I tried, even distributing over the 4 cores in my Intel Core i5 would still not be fast enough for the number of points I need. I gave up timing 10,000,000 samples.

Here it would also be possible to reduce the memory for the translated points heavily - but this is not done yet ( see qwtToPolylineFiltered in qwt_point_mapper.cpp ). But if you want to avoid a huge temporary buffer you could implement YourCurve::drawSeries calling QwtPlotCurve::drawSeries for intervals of f.e 10000 samples each: from -> from + 10000, from +10000 -> from + 20000 ...

If you are overloading the drawSeries() could you also restrict the to/from at this level also to solve the ROI data point reduction?

Using OpenGL is the only way to have hardware accelerated graphics on Windows, but this is more for oscilloscope alike use cases and as QwtPlotGLCanvas is in an experimental state ( f.e no caching yet ) I wouldn't recommend it to you. But you can have a try: setting an OpenGL canvas is one line of code only.

What is the magic code loine for this and does it require the latest trunk code?


Thanks for the help.