QTcpSocket Chat : Mutiple Client using FD_SET and select()

[cooler123]

I was able to send the message using a single client. Now I want to implement for multiple client. How can I do it under Qt using select() method. I prefer this because I have done it using normal socket without Qt.

My application is using TCP socket which have client and server. I want to make a simple chat program which can 1 client send to server and server send to all the rest of client connected.

[wysota]

But is there a specific question here somewhere? And what does it have to do with Qt? I understand you are using BSD sockets, right?

[cooler123]

I using QTcpSocket. From the SimpleChat Example posted here, it is using QHash to store the client list. Is it possible to use select() method to do it?

[wysota]

If you use Qt sockets, don't use select(). It just won't work. Use signals and slots instead.

[cooler123]

OIC. Thanks for the reply.

[bob2oneil]

I wanted to follow up on thread in this year of 2012 and see if things might have changed, or if there is additional thoughts and wisdom based on
changes that have been made to Qt since the original article posting.

I have to manage up to 10 TCP-IP connections and singular UDP socket
as the highest priority task in my application. I have used the Linux
'select" API in the past successfully, and wanted to leverage this
method for handling 10 Tcp Sockets in either separate threads, or
perhaps a singular thread managing all of them.

My current application implementation avoids using the default Qt
event loop in secondary threads to more clearly control running
thread activity (avoid duty cycle of event loop).

The following code sample illustrates my first implementation using
a QUdpSocket and a collection of QTcpSockets.

It in general works, I can send and receive data with UDP recipients and
TCP-IP servers.

Now here is the downside. The Linux statement will exit in a few
microseconds each time on the file descriptor corresponding to the
TCP-IP socket. This will occur even when no data is available.
As a result, my desire for this thread to pend until TCP-IP data
becomes available does not happen, and this leads to great
inefficiencies in the design.

So the question I have is, is the QTcpSocket class fundamentally
structured so that it can not be simple modified to handle this
one issue? Is there a small tweak that can be made (say a
subclass or socket option) that would be a quick fix to this
singular issue.

Assuming I replace the QTcpSocket class with standard Linux sockets,
what value added features/performance of QTcpSocket would be badly missed
(e.g.buffering).

Wsoyata the sage, what do you think?



timeval tvNow, tvTimeout, tvNextEpoch;
tvNow = TimeUtils::getTime();
tvTimeout = tvNow;
tvNextEpoch = timevalNextEpoch(tvNow, m_epochMicroseconds);

quint32 processingDelayAdjustment = 0;
fd_set readS, exS;
int maxFd = 0;

#define SELECT_TIMING_TEST
#ifdef SELECT_TIMING_TEST
timeval tvStart, tvElapsed;
tvStart = tvNow;
tvElapsed = tvNow;
#endif

// Task loop
while (!m_stopped)
{
// Setup for ::select() statement
FD_ZERO(&readS);
FD_ZERO(&exS);

maxFd = 0;
maxFd = qMax(maxFd, udpSocketDescriptor);

// Add UDP socket to read file descriptor set
FD_SET(udpSocketDescriptor, &readS);
FD_SET(udpSocketDescriptor, &exS);

// Iterate through list of active sockets TCP-IP sockets
for (QMap<int, SockInfo *>::iterator it = m_currentSockets.begin(); it != m_currentSockets.end(); ++it)
{
int socket = it.key();

// Add TCP socket to read and exception file descriptor sets
FD_SET(socket, &readS);
FD_SET(socket, &exS);

maxFd = qMax(maxFd, socket);
}

// Update select statement timeout value to the interval to the start of the next epoch
timeoutUpdate(tvTimeout, tvNextEpoch, processingDelayAdjustment);

// Count of available file descriptors
int count = 0;

// Invoke select if the timeout to the next epoch is non-zero
if (tvTimeout.tv_usec != 0 || tvTimeout.tv_sec != 0)
{
#ifdef SELECT_TIMING_TEST
#define SELECT_TIMING_TEST_LOGLEVEL logDEBUG4
if (FILELog::ReportingLevel() >= SELECT_TIMING_TEST_LOGLEVEL)
{
// Get start time prior to select
TimeUtils::getTime(&tvStart);
msg = QString("Select In: now (%1/%2), timeout (%3/%4), tvNextEpoch (%5/%6)").arg(tvStart.tv_sec).arg(tvStart.tv_usec).arg(tvT imeout.tv_sec).arg(tvTimeout.tv_usec).arg(tvNextEp och.tv_sec).arg(tvNextEpoch.tv_usec);
FILE_LOG(SELECT_TIMING_TEST_LOGLEVEL) << qPrintable(msg);
}
#endif

// Wait for a number of file descriptors to change status or a timeout
count = ::select(maxFd + 1, &readS, NULL, &exS, &tvTimeout);

#ifdef SELECT_TIMING_TEST
if (FILELog::ReportingLevel() >= SELECT_TIMING_TEST_LOGLEVEL)
{
// Get system time after select call
tvNow = TimeUtils::getTime();

// Display elapsed time and other content
tvElapsed = tvNow - tvStart;
msg = QString("Select Out: now (%1/%2), timeout (%3/%4), tvNextEpoch (%5/%6), elapsed (%7/%8), count %9").
arg(tvNow.tv_sec).arg(tvNow.tv_usec).arg(tvTimeout .tv_sec).arg(tvTimeout.tv_usec).
arg(tvNextEpoch.tv_sec).arg(tvNextEpoch.tv_usec).a rg(tvElapsed.tv_sec).arg(tvElapsed.tv_usec).arg(co unt);
FILE_LOG(SELECT_TIMING_TEST_LOGLEVEL) << qPrintable(msg);
}
#endif
}

// Get system time after select call
tvNow = TimeUtils::getTime();

// On epoch boundary, perform epoch activities
if (tvNextEpoch <= tvNow)
{
// Update the time of the next epoch (calculate the time of day to the start of the next epoch boundary)
tvNextEpoch = timevalNextEpoch(tvNow, m_epochMicroseconds);

// Should be aligned to even multiple of epoch on timeout. Calculate how far off it was to make runtime adjustment for the next pass (routine timeoutUpdate())
processingDelayAdjustment = tvNow.tv_usec % m_epochMicroseconds;

if (FILELog::ReportingLevel() >= logDEBUG4)
{
int epochNumber = timevalToEpoch(tvNow.tv_usec, m_epochMicroseconds);

msg = QString("TDMA: Epoch %1 at (%2/%3), timeout (%4/%5), delay adjust (%6)").
arg(epochNumber).arg(tvNow.tv_sec).arg(tvNow.tv_us ec).arg(tvTimeout.tv_sec).arg(tvTimeout.tv_usec).a rg(processingDelayAdjustment);
FILE_LOG(logDEBUG4) << qPrintable(msg);
}

onEpoch();

} // end Epoch activities



// Exit thread when stopped
if (m_stopped)
{
break;
}

// Packets found, process them
if (count > 0)
{

// RadioG-LM TA radio queue status messages
if (FD_ISSET(udpSocketDescriptor, &readS))
{
if (FILELog::ReportingLevel() >= logDEBUG4)
{
msg = QString("TDMA: select() exit with count %1: UDP socket read file descriptor %2 set at (%3/%4)").
arg(count).arg(udpSocketDescriptor).arg(tvNow.tv_s ec).arg(tvNow.tv_usec);
FILE_LOG(logDEBUG4) << qPrintable(msg);
}

static QHostAddress recvFrom;
static QByteArray datagram;

// Assemble a full UDP datagram
do
{
datagram.resize(m_udpSocket->pendingDatagramSize());
m_udpSocket->readDatagram(datagram.data(), datagram.size(), &recvFrom);
} while (m_udpSocket->hasPendingDatagrams());

// Process contents of UDP datagram
processLinkManagerMessage(datagram, recvFrom);
}

// UDP socket exception
if (FD_ISSET(udpSocketDescriptor, &exS))
{
FILE_LOG(logERROR) << "TDMA: select() exception detected for UDP socket: " << udpSocketDescriptor;
}

// Check all TCP-IP LM-QM connections in the collection
for (QMap<int, SockInfo *>::iterator it = m_currentSockets.begin(); it != m_currentSockets.end(); ++it)
{
int socket = it.key();

// Socket read
if (FD_ISSET(socket, &readS))
{
if (FILELog::ReportingLevel() >= logDEBUG4)
{
msg = QString("TDMA: select() exit with count %1: TCP-IP socket read file descriptor %2 set at (%3/%4)").
arg(count).arg(socket).arg(tvNow.tv_sec).arg(tvNow .tv_usec);
FILE_LOG(logDEBUG4) << qPrintable(msg);
}

// QM TCP-IP receive handler
SockInfo *sockInfo = it.value();
if (sockInfo)
{
processSocketMessage(sockInfo);
}
}

}

} // end count > 1

} // end while

[wysota]

Qt uses standard sockets too, so if you're just after calling select manually, you can do that if you really want to, however I don't see how your code would be faster than what Qt does (apart the trick Qt does with adding an extra descriptor to the set).

As for your problem... try detecting WHAT causes select() to return (EINTR maybe?).

[bob2oneil]

Hi Wyosota, thanks for the reply (and to all the past suggestions). The motivation for using the select() method is in part to reduce the thread count to manage 10 active TCP-IP client connections.

I also need to transmit along a fairly constrained time line of say a 100 ms epoch which begins with a burst of UDP traffic, then TCP-IP traffic, following by schedule calculation.

Using the singular select() implementation as the highest priority thread works out well for this requirement. I do not want to have potentially 10 TCP-IP threads as blocking clients to manage and the additional locking
on handling shared data between these threads and the UDP thread.

To answer your observation on why select() exits, I was unable to display all of the souce code because of message limitations, but select is exiting with a count of 1, and the file descriptor set upon exit corresponds to
the file descriptor for the singular TCP-IP socket that is opened. I can observe this under Windows and Linux by inspecting the fd_set file descriptor for read.

When the TCP-IP socket is NOT opened, then select exits according to a 100ms epoch rate that is being calculated. This results in the duty cycle that I desire for this particular thread, which is
intended to suspend as much as possible after completing its activities.

Walking into my application code when select exits, FD_ISSET is true for the singular TCP-IP socket that I am using for testing (this will expand to 10 TCP-IP connections).

When I get to the routines to process TCP-IP messages received, the call to socket->bytesAvailable() is most frequently zero. When I do finally get data, the frames are correctly decoded.

So the symptoms are essentially that select() is exiting on the TCP-IP socket unnecessarily on the read file descriptor for the socket. If this did not occur, it would be perfect solution for
my requirements.

Looking through some of the Qt source code for QTcpSocket, is it based on buffered content, and this is a requirement for the class.

How the Qt implementation some how provides a file descriptor whose attributes cause select() to exit prematurely from my perspective is completely unknown to me. I am stumped on this one, and have
no idea of how to troubleshoot select() further. Note that this behavior is roughly consistent under both Windows and Linux.

There are no select() error conditions upon exit from this API.

While I desire to have the code cross platform, the deployed platform will be Linux exclusively. Still, the Qt framework for sockets provided the platform abstraction on sockets that I desired.

I am certainly willing to code the socket routines using simple BSD sockets, but wanted to do some due diligence before I toss the QTcpSocket implementation in case there was something simple that I
might have missed. I have not viewed any particular socket option that I think would change the behavior.

Assuming I have to go outside Qt to get to the more primitive socket APIs, is there any C++ wrapper available that provides both BSD and Winsock capability for cross platform support that you would
recommend?

I am certainly open to other architectural options, perhaps a separate thread (or threads) based on the event loop and signals/slots.

However, performance of this single networking thread is the essence of my application, which will either succeed or fail based on the implementation. I have to perform activities at a relatively
frequent interval (as quick as possible 25 ms epochs), so speed and no delays are important attributes. That is what made use of select() attractive, and perhaps one could argue that the
more primitive socket APIs might also provide performance benefits.

In a nutshell, the file descriptor returned from QTcpSocket seems to trigger as read active even when this is not the case. I don't know why.

Added after 8 minutes:

Missing source code for select() error handling (based on message size limitations)




} // end count > 1

else
{
switch (count)
{
// Timeout
case 0:
break;

// Error
case -1:

// Ignore a non blocked signal being caught
#ifdef Q_OS_WIN
if (errno != WSAEINTR)
{
FILE_LOG(logERROR) << "TDMA: select() statement error exit: " << WSAGetLastError();
++m_nSelectErrors;

// An invalid file descriptor was given
if (errno == WSAEBADF)
{
clearClosedSocketsFromCollection();
}
}
#else
if (errno != EINTR)
{
// Linux: Select returns -1 on error and sets errno appropriately
SysUtils::handleLastError("TDMA: select() statement error exit");
++m_nSelectErrors;

// An invalid file descriptor was given
if (errno == EBADF)
{
clearClosedSocketsFromCollection();
}
}
#endif
break;

// Unknown select return value
default:
++m_nSelectErrors;
FILE_LOG(logERROR) << "TDMA: select returned " << count;
break;
}

} // end else if (count > 0)

//debugElapsed();

} // end while

[wysota]

The motivation for using the select() method is in part to reduce the thread count to manage 10 active TCP-IP client connections.
Qt can do that with exactly 0 additional threads. I doubt you can do better with a manual select().


To answer your observation on why select() exits, I was unable to display all of the souce code because of message limitations, but select is exiting with a count of 1, and the file descriptor set upon exit corresponds to
the file descriptor for the singular TCP-IP socket that is opened. I can observe this under Windows and Linux by inspecting the fd_set file descriptor for read.
Is there anything to read from the descriptor after the such situation occurs?


When I get to the routines to process TCP-IP messages received, the call to socket->bytesAvailable() is most frequently zero.
How did you implement bytesAvailable() for your socket? As I understand you are not using QTcpSocket... If you are using QTcpSocket without letting Qt process the socket, you can't rely on the infrastructure the class provides. bytesAvailable() will likely return incorrect values. I think in your situation there is really no point in using QTcpSocket if you want to do all the processing yourself anyway. What benefits does it bring?

By the way, your code logic is really strange... Especially the fragment when you discard all UDP datagrams but the last one. You are aware that the last datagram you receive doesn't have to be the last that is sent as UDP can reorder datagrams?

[bob2oneil]

Is there anything to read from the descriptor after the such situation occurs?

No, QTcpSocket->bytesAvailable() returns 0. It is as if the socket file descriptor always indicates that read is available, but it is not. When I do send a message to this socket, it is received correctly.



How did you implement bytesAvailable() for your socket? As I understand you are not using QTcpSocket... If you are using QTcpSocket without letting Qt process the socket, you can't rely on the infrastructure the class provides. bytesAvailable() will likely return incorrect values. I think in your situation there is really no point in using QTcpSocket if you want to do all the processing yourself anyway. What benefits does it bring?

It was not obvious from my original source code (due to size limits), but sockInfo* contains a QTcpSocket as an element from the following code sample for the socket connection:


const int Timeout = 5 * 1000;

// Create socket
QTcpSocket *socket = new QTcpSocket();

// Attempt LM-QM connection
QString serverIP = m_queueManagerCommSettings.at(i)->getIP();
quint16 serverPort = m_queueManagerCommSettings.at(i)->getPort();
socket->connectToHost(QHostAddress(serverIP), serverPort);

// Wait on connection or failure
if (!socket->waitForConnected(Timeout))
{
FILE_LOG(logWARNING) << "TDMA: failed making LM to QM TCP-IP socket connection at ip '" << qPrintable(serverIP) << "' - port " << serverPort << ", error: " << qPrintable(socket->errorString());
delete socket;
++m_nQMConnectErrors;
}
else
{
int socketDescriptor = socket->socketDescriptor();

FILE_LOG(logINFO) << "TDMA: TCP-IP socket (descriptor " << socketDescriptor << ") connection made from LM to QM at ip '" << qPrintable(serverIP) << "' - port " << serverPort;
++m_nQMConnects;

// Add connected socket to collection
m_currentSockets.insert(socketDescriptor, new SockInfo(socket, m_queueManagerCommSettings.at(i)->getTransceiverID(), 0));

// Optimize socket for latency, set to TCP_NODELAY to diable Nagle's algorithm, set to 1 to enable
const QVariant value = 0;
socket->setSocketOption(QAbstractSocket::LowDelayOption, value);

// Set DSCP field of socket to high priority setting
if (!SysUtils::setSocketTypeOfService(socket->socketDescriptor(), dscpNetworkCritical))
{
++m_nErrors;
FILE_LOG(logERROR) << "TDMA: failed to set TCP-IP transmit socket DSCP value with errno - " << qPrintable(SysUtils::getErrorString());
}
}



As far as my UDP processing looking strange, I am not sure I understand. The build loop is just like the content from the Thelin text that you had a hand in, I simply have a do--while rather than a while .. where I assume
since the UDP file read descriptor indicated reception of data, then there was content to read. I can certainly change this. Maybe I am missing your point.

[wysota]

No, QTcpSocket->bytesAvailable() returns 0. It is as if the socket file descriptor always indicates that read is available, but it is not. When I do send a message to this socket, it is received correctly.
I'm not asking what bytesAvailable() returns, I'm asking whether there is actual data to be read. If you're not letting Qt process the socket, I would assume bytesAvailable() returns bogus data. Get the socket descriptor and call ::read() on it.



As far as my UDP processing looking strange, I am not sure I understand. The build loop is just like the content from the Thelin text that you had a hand in, I simply have a do--while rather than a while

I don't think so...

Here is your code:

static QHostAddress recvFrom;
static QByteArray datagram;

// Assemble a full UDP datagram
do
{
datagram.resize(m_udpSocket->pendingDatagramSize());
m_udpSocket->readDatagram(datagram.data(), datagram.size(), &recvFrom);
} while (m_udpSocket->hasPendingDatagrams());

// Process contents of UDP datagram
processLinkManagerMessage(datagram, recvFrom);

You exit the loop when the socket has no more pending datagrams and while you are in the loop you are only reading datagrams from the socket overwriting what was written in the previous iteration. Only when you exit the loop you do some kind of processing. So effectively you are processing the last datagram only and discarding all the previous ones. You are trying to process something even if there are no datagrams available at all. I would assume the loop should look like the following:


static QHostAddress recvFrom;
static QByteArray datagram;

do {
datagram.resize(m_udpSocket->pendingDatagramSize());
m_udpSocket->readDatagram(datagram.data(), datagram.size(), &recvFrom);
// Process contents of UDP datagram
processLinkManagerMessage(datagram, recvFrom);
} while (m_udpSocket->hasPendingDatagrams());

I don't have Johan's book at hand right now so I can't verify what you say but I'd assume the book tells you to process the datagram while being inside the loop.

[bob2oneil]

I'm not asking what bytesAvailable() returns, I'm asking whether there is actual data to be read. If you're not letting Qt process the socket, I would assume bytesAvailable() returns bogus data. Get the socket descriptor and call ::read() on it.

Hi Wytold, good suggestion, I was not sure where you were heading with this one. While I am not executing an event loop or using signals/slots for QTcpSocket, I am using the blocking modes APIs similar to the blocking samples. Nonetheless, when I call ::recv() on the socket file descriptor, this API is able to read the raw socket data, and by reading the data, the file descriptor no longer passes immediately through the select() statement. It would seem that the QTcpSocket.read API, which reads buffered content, does not perform the native socket read analogous to ::recv. The call stack for the QTcpSocket read is QIODevice::read() to QAbstractSocket::readData. Perhaps this proves your original statement that QTcpSocket does not work well with select(). I believe I have a solution using it, but requires bypassing the buffering to get to the raw socket read API. However, having done this, the logical next step is to perhaps use the raw socket APIs directly in the interest of speed and performance. Nonetheless, excellent observation that explains a bit what the observed behavior is.

On the topic of my UDP code, you are correct. Sometimes stuff is right in front of your eyes, and it takes a 2nd pair of eyes to see the obvious.

Thanks again.

[wysota]

I see no benefit in using QTcpSocket for your tcp sockets. Since you are using select(), you are limiting yourself to one platform only, so you might as well use ::socket().

[bob2oneil]

My code is actually still cross platform (select/recv etc are all available under Windows/Winsock)

Where the benefit of using QTcpSocket to establish the connections, is that I will eventually have to use secure sockets for these comm interfaces, and QSslSocket seems like a real good match that will reduce my coding efforts.
The devil well be in the details on this effort.

By using the low level socket APIs I am to some extent shortcutting the buffering required in the base class, and I am using TCP-IP to send small (32 byte) control frames that need to be handled at epoch rates of 100 ms and less.

[wysota]

TCP isn't really a good competitor when it comes to time critical tasks since a temporary network congestion disrupts further communication.

[bob2oneil]

No argument, this protocol was imposed on me, it was not my selection. I am trying to make the most of it, open to ideas. One step is to get closer to the socket APIs, and to set selected socket properties.
The frames are very small but frequent (up to 10 send and receive in 100 ms epochs).

[wysota]

Small frames (or actually segments) is another thing TCP doesn't like. If you want to send those really quick, you have to force a push flag on each segment otherwise TCP will be trying to wait for more data.

[bob2oneil]

I have the DSCP field set to high prioirty and have the TCP_NODELAY option on the socket as well. I am not sure what else I can do. It is important that these small frames are sent without delay (hence the possible need to avoid the QTcpSocket buffering) on both send and receive. I also have UDP traffic being sent with similar timing requirements. If there is a viable chance that the small TCP-IP sends can delay the timely delivery of the more important UDP traffic, I can certainly put all the TCP-IP routine in its own thread. However, the single high prioirty thread with both does offer some advantages, especially when I want to define time line sequences where a UDP burst (20 small frames) is quickly followed by a TCP-IP burst (10 frames) at a 100 ms epoch rate.

Next wrinkle is using SSL/OpenSLL on both protocols/sockets efficiently.

Thanks for the suggestions.

Added after 19 minutes:

Quick read on the PSH flag you mentioned in the TCP-IP header control flags, not sure if these map to the socket option (TCP_NODELAY) or how they may be set programmatically.

Flags (9 bits) (aka Control bits) – contains 9 1-bit flags

PSH (1 bit) – Push function. Asks to push the buffered data to the receiving application.

[wysota]

If I remember correctly at least the TCP stack implementation on Linux sets the push flag when you flush the socket.

Anyway, I don't think it matters how many threads you use and if you use any additional threads at all unless you are trying to handle hundreds of sockets at the same time. You have no influence on what happens over the link which is the real bottleneck in networking problems. And once the traffic increases, congestion will start to happen, you'll get retransmissions and you'll have to say good bye to your time critical design. The more small segments you send, the bigger the chance for congestion as you'll be wasting frames in the link layer (e.g. in Ethernet). And the faster the network, the larger the frames so you lose more and more making yourself more vulnerable to congestion and retransmissions. This design is simply flawed and you should talk to the person who imposed the protocol on you to seriously reconsider. Unless of course you are doing it all in LAN or even in a point-to-point connection. But then you should be using UDP anyway.

[bob2oneil]

Hi Wytold, I will do the socket flush and check Wireshark to see if the push flag bit is being set. Thanks for the input on the protocol, I really can't push back on that decision, we tried months ago suggesting UDP was a better fit, but you know how external decisions can be imposed on a design. I will be cognizant of the loading and retry issues in our testing, with perhaps a "I told you so" prepared to be launched somewhere down the line. I guess if I can find ways of making TCP-IP behave more like UDP, all the better. I really don't want any transmit retries, as the information returned will be stale at the time of the new request, which will be the same request message. There is some notion of the recipient sending out data unsolicited at a perioidic 100 ms epoch rate once a connection is established (the other end is the TCP-IP server), but then the retry issue occurs on their end.

[wysota]

I guess if I can find ways of making TCP-IP behave more like UDP, all the better. I really don't want any transmit retries, as the information returned will be stale at the time of the new request, which will be the same request message.
You have no choice if you are to use TCP. You can receive all the segments but the first one, but until the first one arrives, the receiving application will get nothing. And with bad implementation of TCP the sender will have to retransmit not only this single segment but the remaining ones too.