How to limit TCP writes to particular size and then block untlil the data is read

[ustulation]

{Qt 4.7.0 , VS 2010}

I have a Server written in Qt and a 3rd party client executable. Qt based server uses QTcpServer and QTcpSocket facilities (non-blocking).

Going through the articles on TCP I understand the following: the original implementation of TCP mentioned the negotiable window size to be a 16-bit value, thus maximum being 65535 bytes. But implementations often used the RFC window-scale-extension that allows the sliding window size to be scalable by bit-shifting to yield a maximum of 1 gigabyte. This is implementation defined. This could have resulted in majorly different window sizes on receiver and sender end as the server uses Qt facilities without hardcoding any window size limit. Client 1st asks for all information it can based on the previous messages from the server before handling the new (accumulating) incoming messages. So at some point Server receives. This the server processes and puts it into the sender buffer. Client however is unable to handle the messages at the same pace and it seems that client’s receiver buffer is far smaller (65535 bytes maybe) than sender’s transmit window size. The messages thus get accumulated at sender’s end until the sender’s buffer is full too after which the TCP writes on sender would block. This however does not happen as sender buffer is much larger. Hence this manifests as increase in memory consumption on the sender’s end.

To prevent this from happening, I used Qt’s socket’s waitForBytesWritten() with timeout set to -1 for infinite waiting period. This as I see from the behaviour blocks the thread writing TCP data until the data has actually been sensed by the receiver’s window (which will happen when earlier messages have been processed by the client at application level)

is there a better alternative to this if i want to restrict the memory consumption at server end to say x MB's?

[wysota]

Qt sends the data directly to the tcp implementation on your system. It is responsible for control flow. If you fill the write buffer in your system, the socket will stop receiving data (thus it is important to check whether all data has actually been written). Therefore you should not worry too much about it. If you really want, append all data to be written to some internal buffer, control its size and connect to bytesWritten() signal to put more data into the socket. Then you'll have a sort of internal write buffer that you have total control over. You can either reject accepting more data or (if possible in your particular case) sleep on a wait condition (the latter is similar to waitForBytesWritten).

[ustulation]

ok. I had this problem that client asks for 9-10 MB's of data per message and it sends around 15 messages. It doesn't process data equally fast as Server writes it. So the memory consumption at the server end was peaking to around 200 MB's per client connection. This was seen as a concern and i was asked to reduce it. Merely on using waitForBytesWritten( -1 ) immediately after QTcpSocket::write(..) the memory consumption reduced to 30-35 MB at peak. This behaviour is however evidently only in Windows 7. In Win XP the peak memory consumption reduced from 200 MB's to 100 MB's so this is still viewed as a concern. Do you think it is necessary to control it further as tester is concerned that such memory usage will restrict the number of clients the server can connect to before running out of memory?

What function exists that can actually tell me that the client has done reading the sent data from it's read buffer? Then i'll wait for this and only when it's over will i process the other bufferred messages to be replied to/sent?

[wysota]

The best thing you can do is queue requests. If the client requests 15 messages, send him just one and queue 14 remaining ones until sending the first one is complete. Only then process the next one. This way you'll be using memory for only one message per client. Blocking until sending is complete is not a good idea because your program can't do other things (like serving other clients) at the same time.

[ustulation]

right this is what i think i exactly need. You wrote: "send him just one and queue 14 remaining ones until sending the first one is complete" -- How do i know that the first one is complete? Client does not do it at application level. So is there any Qt function to detect that client has read the message from it's receiving window?

[wysota]

How do i know that the first one is complete?
Connect to bytesWritten() and/or monitor what write() returns.


So is there any Qt function to detect that client has read the message from it's receiving window?
No at there would be no sense in doing that. The way TCP works is that it requires acknowledging every segment it sends and if it doesn't receive proper ACKs from the other side, it retransmits data starting from the last ack'ed octet. Thus all unacknowledged data sits in the write buffer of the sender. If the write buffer is full the socket stops accepting data from the application (so write() will return a value smaller than the size of data you fed it with). Thus implicitly all that means that if the socket stops accepting data, it means the other side can't process it fast enough. Of course you have to bear in mind that the receiver has its own read buffer that is filled with incoming data. If the other app is written with QTcpSocket and you allow the app to process its events, then QTcpSocket will read pending data into its own buffers. By default the buffer is unlimited so as long as the application is responsive, only network congestion and not application processing speed will influence the capacity of the write buffer on the sending side.

[ustulation]

thank you so much. I'll try a few things based on what you said. One last query: is there anything similar to QAbstractSocket::setReadBufferSize( qint64 size ) for WriteBufferSize(..) ? Then i can precisely control how much memory consumption i want befor Socket blocks on further writes.

[wysota]

One last query: is there anything similar to QAbstractSocket::setReadBufferSize( qint64 size ) for WriteBufferSize(..) ?
No, I can't see an equivalent but you can implement it by subclassing QTcpSocket and reimplementing write related virtual methods.


Then i can precisely control how much memory consumption i want befor Socket blocks on further writes.
Not entirely. Remember that the operating system has buffers of its own.