Model/View framework: streaming data in a QTableView

[yannickt]

Greetings,

I am writing an application where I am receiving a lot of data from the network and have to display it in a QTableView. I can receive hundreds of messages per second and I need the table to remain responsive when data is added or changed. My function that updates the model looks like this:

ModelItemRowTable::const_iterator rowIter = m_modelItemRows.find(symbol.c_str());
	if (m_modelItemRows.end() != rowIter)
	{
		int row = rowIter.value();
		m_modelItems[row].data.Update(quote, fieldUpdateFlags);
		emit dataChanged(index(row, 0), index(row, columnCount()-1));
	}
	else
	{
		emit beginInsertRows(QModelIndex(), rowCount(), rowCount());
		ModelItem item;
		item.symbol = symbol.c_str();
		item.data = quote;
		m_modelItems.push_back(item);
		m_modelItemRows[symbol.c_str()] = m_modelItems.count() - 1;
		emit endInsertRows();
	}

ModelItemRowTable::const_iterator rowIter = m_modelItemRows.find(symbol.c_str());
	if (m_modelItemRows.end() != rowIter)
	{
		int row = rowIter.value();
		m_modelItems[row].data.Update(quote, fieldUpdateFlags);
		emit dataChanged(index(row, 0), index(row, columnCount()-1));
	}
	else
	{
		emit beginInsertRows(QModelIndex(), rowCount(), rowCount());
		ModelItem item;
		item.symbol = symbol.c_str();
		item.data = quote;
		m_modelItems.push_back(item);
		m_modelItemRows[symbol.c_str()] = m_modelItems.count() - 1;
		emit endInsertRows();
	}

To copy to clipboard, switch view to plain text mode 

where m_modelItems is a list of items and m_modelItemRows is a hash table storing the item rows as values for faster item lookup. This is a 3000 x 9 table.

Now data insertion is very fast as it occurs when the model is created, the problem is when data is changed. I receive a lot of messages at a time and I was wondering how to do it without having to emit the dataChanged signal on every update. I have tried queuing the items and updating the range of queued items at regular intervals (e.g. every 100 ms, 250 ms, 1s) but the view is still not responsive enough and data is not updated as fast as I would like (1 to 4 times per second). Or maybe there is something about the dataChanged() signal that I am am missing?

Any ideas?

[wysota]

You don't have to emit the signal on every update. Update data in blocks (for instance group updates for each second) end emit a single signal afterwards.

[yannickt]

I have tried something along those lines already by updating the range of items at regular time intervals. When I add a new item to the list of items in the model I keep track of the top and bottom updated rows. I also have a QTimer that will fire say every second and on timeout will update only the block of items between these top and bottom rows. Specifically:

// This is where I update the model items
	ModelItemRowTable::const_iterator rowIter = m_modelItemRows.find(symbol.c_str());
	if (m_modelItemRows.end() != rowIter)
	{
		int row = rowIter.value();
		m_topUpdatedRow = (std::min)(m_topUpdatedRow, row);
		m_bottomUpdatedRow = (std::max)(m_bottomUpdatedRow, row);
		m_modelItems[row].data.Update(quote, fieldUpdateFlags);
	}
	else
	{
		emit beginInsertRows(QModelIndex(), rowCount(), rowCount());
		ModelItem item;
		item.symbol = symbol.c_str();
		item.data = quote;
		m_modelItems.push_back(item);
		m_modelItemRows[symbol.c_str()] = m_modelItems.count() - 1;
		emit endInsertRows();
	}
 
// This is how I update a block of data in a separate slot that is run whem my timer times out
	emit dataChanged(index(m_topUpdatedRow, 0), index(m_bottomUpdatedRow, columnCount()-1));
	m_topUpdatedRow = (std::numeric_limits<int>::max)();
	m_bottomUpdatedRow = (std::numeric_limits<int>::min)();

// This is where I update the model items
	ModelItemRowTable::const_iterator rowIter = m_modelItemRows.find(symbol.c_str());
	if (m_modelItemRows.end() != rowIter)
	{
		int row = rowIter.value();
		m_topUpdatedRow = (std::min)(m_topUpdatedRow, row);
		m_bottomUpdatedRow = (std::max)(m_bottomUpdatedRow, row);
		m_modelItems[row].data.Update(quote, fieldUpdateFlags);
	}
	else
	{
		emit beginInsertRows(QModelIndex(), rowCount(), rowCount());
		ModelItem item;
		item.symbol = symbol.c_str();
		item.data = quote;
		m_modelItems.push_back(item);
		m_modelItemRows[symbol.c_str()] = m_modelItems.count() - 1;
		emit endInsertRows();
	}

// This is how I update a block of data in a separate slot that is run whem my timer times out
	emit dataChanged(index(m_topUpdatedRow, 0), index(m_bottomUpdatedRow, columnCount()-1));
	m_topUpdatedRow = (std::numeric_limits<int>::max)();
	m_bottomUpdatedRow = (std::numeric_limits<int>::min)();

To copy to clipboard, switch view to plain text mode 

Not much has changed and the GUI slows down to a crawl after the table is populated and the updates start rolling in. Any ideas? I am probably doing something horribly wrong but can't figure it out at the moment.

[wysota]

When exactly is the snippet above executed? I think you still update one row at once...

[yannickt]

I will clarify. The snippets are executed in two separate functions.

void CModel::OnNewQuote(const std::string& symbol, const CQuote& quote, uint fieldUpdateFlags)
{ 
    ModelItemRowTable::const_iterator rowIter = m_modelItemRows.find(symbol.c_str());
    if (m_modelItemRows.end() != rowIter)
    {
        int row = rowIter.value();
        m_topUpdatedRow = (std::min)(m_topUpdatedRow, row);
        m_bottomUpdatedRow = (std::max)(m_bottomUpdatedRow, row);
        m_modelItems[row].data.Update(quote, fieldUpdateFlags);
    }
    else
    {
        emit beginInsertRows(QModelIndex(), rowCount(), rowCount());
        ModelItem item;
        item.symbol = symbol.c_str();
        item.data = quote;
        m_modelItems.push_back(item);
        m_modelItemRows[symbol.c_str()] = m_modelItems.count() - 1;
        emit endInsertRows();
    }
}

void CModel::OnNewQuote(const std::string& symbol, const CQuote& quote, uint fieldUpdateFlags)
{ 
    ModelItemRowTable::const_iterator rowIter = m_modelItemRows.find(symbol.c_str());
    if (m_modelItemRows.end() != rowIter)
    {
        int row = rowIter.value();
        m_topUpdatedRow = (std::min)(m_topUpdatedRow, row);
        m_bottomUpdatedRow = (std::max)(m_bottomUpdatedRow, row);
        m_modelItems[row].data.Update(quote, fieldUpdateFlags);
    }
    else
    {
        emit beginInsertRows(QModelIndex(), rowCount(), rowCount());
        ModelItem item;
        item.symbol = symbol.c_str();
        item.data = quote;
        m_modelItems.push_back(item);
        m_modelItemRows[symbol.c_str()] = m_modelItems.count() - 1;
        emit endInsertRows();
    }
}

To copy to clipboard, switch view to plain text mode 

The above function is a slot connected to a signal that will be emitted by the working thread when new data is received (I use a queued connection here). In my test case the row insertion part takes place only when the model is created as this is the only time I am populating the table. After that I am only updating existing entries in my table. In the above code, m_topUpdatedRow and m_bottomUpdatedRow are used to keep track of the block containing all of the updated rows.

The model owns a QTimer that will time out at 1 second intervals. The timeout signal is connected to a slot that updates the table.

void OnTimeOut()
{
    emit dataChanged(index(m_topUpdatedRow, 0), index(m_bottomUpdatedRow, columnCount()-1));
    m_topUpdatedRow = (std::numeric_limits<int>::max)();
    m_bottomUpdatedRow = (std::numeric_limits<int>::min)();
}

void OnTimeOut()
{
    emit dataChanged(index(m_topUpdatedRow, 0), index(m_bottomUpdatedRow, columnCount()-1));
    m_topUpdatedRow = (std::numeric_limits<int>::max)();
    m_bottomUpdatedRow = (std::numeric_limits<int>::min)();
}

To copy to clipboard, switch view to plain text mode 

The slot resets m_topUpdatedRow and m_bottomUpdatedRow to "degenerate" values to ensure that they will be set to sensible values the next time a new quote is received.

I am still observing the same problems and the table is still slow and unresponsive during updates.

[yannickt]

I finally managed to test a release build today (couldn't do it before due to dependencies out of my control), and the table responds wonderfully in release mode, even if I emit a dataChanged() signal on every update. In debug mode even when I reduce the number of rows to say 50 or even 20 the table wasn't responsive. I did not expect such a huge difference between debug and release in this test.

[wysota]

Well... this is wrong. You can't update the content and emit a signal "some time later". You still add one row at a time, instead of doing it all at once. I meant something like:

// this one executed when new data is available
void XXX::newContentAvailable(const Item &item){
  m_queue.enqueue(item);
}
 
// this one executed once per second
void XXX::performUpdate(){
  int topRow = rowCount();
  int bottomRow = -1;
  while(!m_queue.isEmpty()){
    Item item = m_queue.pop();
    // insert the item in the model
    if(row<topRow) topRow = row;
    if(row>bottomRow) bottomRow = row;
  }
  // emit one signal for all updates
  emit dataChanged(index(0, topRow), index(columnCount()-1, bottomRow));
}

// this one executed when new data is available
void XXX::newContentAvailable(const Item &item){
  m_queue.enqueue(item);
}

// this one executed once per second
void XXX::performUpdate(){
  int topRow = rowCount();
  int bottomRow = -1;
  while(!m_queue.isEmpty()){
    Item item = m_queue.pop();
    // insert the item in the model
    if(row<topRow) topRow = row;
    if(row>bottomRow) bottomRow = row;
  }
  // emit one signal for all updates
  emit dataChanged(index(0, topRow), index(columnCount()-1, bottomRow));
}

To copy to clipboard, switch view to plain text mode