Spliting data aquisition/visualisation into threads

[robodude]

Hi,

Im currently working on a program to visualise data from several sensor like accelerometers, gyroscopes, etc.
Because Im not too familiar with QT programming and multithreading I am not too sure about best way to split my programm into threads and how to manage data synchronisation.

My current idea is to put all data acquisition, in a fact reading from the serial port, into seperate thread, lets call it Serial. The second thread is a DataProcessor, which task is to pre-process these data, split it and send accordingly to different QWT plots and to QGLViewer(OpenGl model visualisation).
As far as I know, using widgets(like QwtPlot) in QT is possible only in main thread. Hence the data from DataProcessor have to be send using signals-slot mechanism.

Im afraid that doing it in this way will lead to significantly slowing down the main thread and the user interface.

Because I dont have too much time for testing several solutions I would like to ask you about opinion of presented idea. Please let me know if I am wrong in some point or give some suggestions how can I do somtehing better.

Thanks a lot for any replies!

[amleto]

As long as you manage ownership of data correctly then you can send the data using pointers. Your situation shouldn't necessarily have any problems keeping the ui responsive.

[swamyonline]

Hi,
I am also having similar situation here with me. It would be of great help if you (amleto) put some more light on this:

As long as you manage ownership of data correctly then you can send the data using pointers.

. Thanks in advance.

swamy.

[ChrisW67]

Reading from a serial port, splitting data, and updating widgets can all be done without threads. You do not want the extra hassle of threads if you do not need it.

If you do use threads:
Accessing data that is shared between threads, e.g. through a pointer to a data buffer, requires use of synchronisation mechanisms. You must ensure readers are not reading a data structure that writers may be 'simultaneously' writing to. Failing to do this is a Bad Thing that will eventually cause grief if it does not crash outright. This a general thread programming issue and not Qt specific, though Qt does provide QMutex as a suitable mechanism.

[robodude]

First of all, thank you guys for your opinions!

I think I have to use threads because data from sensors will be coming "continously", measurments will be done with frequency about 500 Hz. Whats more, the data fusion will be done(for ex. Kalmann filter, etc.) In such case wouldnt it be too absorbing for one thread?

Actually, I use such data synchronization as you mentioned, using buffer declared as a static constant size array of double and QSemaphore mechanism as described in producer-consumet in QT documentation but sharing parts of buffer instead of one element at once.

This works well for threds Serial and DataProcessor. My problem is how to send pre-processed data form DataProcessor thread to the main thread to display it in appropriate widgets. Is the only way is using signals-slot mechanism? As far as I know I cant use mutexes/semaphores here if I want to keep GUI responsive?

Thank you a lot!

[amleto]

If there are multiple consumers of your data then signal/slots is the correct mechanism. If there is only one consumer then there is little point in using signal/slot - just use direct method calls or QMetaObject::invokeMethod to jump thread contexts. You do not need to send the actual data through a signal, though!

MyDataStructure data;
 
// set up data
 
emit DataReady(data); // bad

MyDataStructure data;

// set up data

emit DataReady(data); // bad

To copy to clipboard, switch view to plain text mode 

MyDataStructure data;
 
// set up data
 
// finished writing to data - move it to some 'safe for reading' area
MyDataStructure* dataPointerThatIsSafeForSharing = MoveToSharedReadingBuffer(data);
 
emit DataReady(dataPointerThatIsSafeForSharing); // good. signal signature is void DataReady(MyDataStructure const * ptr); so data is read only

MyDataStructure data;

// set up data

// finished writing to data - move it to some 'safe for reading' area
MyDataStructure* dataPointerThatIsSafeForSharing = MoveToSharedReadingBuffer(data);

emit DataReady(dataPointerThatIsSafeForSharing); // good. signal signature is void DataReady(MyDataStructure const * ptr); so data is read only

To copy to clipboard, switch view to plain text mode 

you can optimise this version to only create one MyDataStructure.

[robodude]

I think I see the point.

However could you say somthing more about this:

// finished writing to data - move it to some 'safe for reading' area
MyDataStructure* dataPointerThatIsSafeForSharing = MoveToSharedReadingBuffer(data);

// finished writing to data - move it to some 'safe for reading' area
MyDataStructure* dataPointerThatIsSafeForSharing = MoveToSharedReadingBuffer(data);

To copy to clipboard, switch view to plain text mode 

Im not sure about it. Could you give some simple example what should I do in MoveToSharedReadingBuffer(data)?

[amleto]

class ProcessAndShareData<T>
{
public:
 
  void ProcessData(...);
 
private:
  T const * MoveToSharedReadingBuffer(T const & data)
  {
    T* newData = new T(data); // make a copy constructor if needed...
    // setup newData from data
    processed_data << newData;
 
    return processed_data.back();
  }
 
private:
  QList<T const *> processed_data;
}

class ProcessAndShareData<T>
{
public:

  void ProcessData(...);

private:
  T const * MoveToSharedReadingBuffer(T const & data)
  {
    T* newData = new T(data); // make a copy constructor if needed...
    // setup newData from data
    processed_data << newData;

    return processed_data.back();
  }

private:
  QList<T const *> processed_data;
}

To copy to clipboard, switch view to plain text mode 

As I say, this is a bit inefficient and you can avoid the copying of data with a bit of restructuring.

[wysota]

I would suggest to use a circular buffer protected by two semaphores to avoid overflow and underflow. This usually works very well for quick concurrent access. Of course if you can afford to be put on hold to prevent overload (if you can't a list won't help as sooner or later you'll run out of memory).

[robodude]

Great

Thank you once more. Your answers were very helpful.

---

Added after 7 minutes:

I would suggest to use a circular buffer protected by two semaphores to avoid overflow and underflow. This usually works very well for quick concurrent access. Of course if you can afford to be put on hold to prevent overload (if you can't a list won't help as sooner or later you'll run out of memory).

I do it in exactly same way in comunication between Serial and DataProcessor threads. I thought that it is not correct way in synchronization with the main thread.

[wysota]

I have no idea what you do so it is not possible for me to determine if what you are doing is right or not. A circular buffer protected by two semaphores is a classic non-blocking solution for a producer-consumer problem. I'm assuming it is not what you are doing if you're asking your question (about signals and slots) here.