sending encrypted data via mail using smtp

[vermarajeev]

Hello,
I'm writing a mail utility program in my application. I make use of smtp example given in Qt3 (/examples/network/mail).

Description of my application:
The application is very similar to mspaint in windows. The only difference is my application creates data files while mspaint creates image files.

Problem:
The files created by my application can be a plain/encrypted one. I use AES::CBC( Advanced encryption standard in Cipher Block Chaining) algorithm to encryt the files.
I'm able to send both types of files( plain/encrypt ) via mail. The problem is when I send an encrypted file, the recipient is unable to decrypt the file.

Consider below scenario.
PersonA sends an encrypted file to PersonB
PersonB detaches the file
PersonB supplies detached file through decryption functionality (AES::CBC) to get back the plain text but fails.

I went through some of the specifications of SMTP and found that, smtp can only transfer data with ascii value less than 127. Also mentioned in the specification that we have to use base64 alogorithm to encode the text/binary data.
Basically the structure is something like below.

Sending email:
File --> (Encoding base64) --> encoded file

Receiving email:
encoded file --> (Decoding base64) --> File

The file can be text or binary.


Even though I'm using the same structure as discussed above, the recipient is unable to decrypt the file (the decryption of file fails). I wonder why? I have been trying to figure out what is wrong but was unsuccessful. Can somebody please help me?

Thanks in advance.

[marcel]

But there can be any number of reasons.
Have you tried to isolate the encoding (to base 64 and the reverse ) and test it locally?
Also have you tested the encryption and decryption separately?

Regards

[vermarajeev]

Also have you tested the encryption and decryption separately?
Yes the encryption and description works fine separately. On the local machine I can encrypt/decrypt, works fine.
The only problem comes when I send the encrypted file via mail and then try to decrypt the file.

Have you tried to isolate the encoding (to base 64 and the reverse ) and test it locally?
I didnt get you.

[marcel]

I went through some of the specifications of SMTP and found that, smtp can only transfer data with ascii value less than 127. Also mentioned in the specification that we have to use base64 alogorithm to encode the text/binary data.

Don't you do this? Encode all data(encrypted or not) prior to sending to base 64 and decode it upon receiving?

I was under the impression that you did this.

Regards

[vermarajeev]

Don't you do this? Encode all data(encrypted or not) prior to sending to base 64 and decode it upon receiving?

I was under the impression that you did this.

Regards

Encode all data by what?
below is the structure
Sending email:
File --> (Encoding base64) --> encoded file

Receiving email:
encoded file --> (Decoding base64) --> File

File can be plain/encoded
I use AES::CBC to encode the file initially, then encode it again using base64 algorithm then give it to SMTP. I use Transfer encoding scheme as base64 in smtp.
Please be more specific?

[marcel]

I use AES::CBC to encode the file initially, then encode it again using base64 algorithm then give it to SMTP. I use Transfer encoding scheme as base64 in smtp.

This is what I was asking.
If you encode the data to base 64 before sending it.
Also, I asked if you tested this base 64 encoding/decoding locally.
Maybe you don't decode from base64 ok.

I imagine you cannot post the code, but are you sure there aren't any differences between plain files? Not even a byte?
Do you handle well SMTP transparency and the end of data message indicator?
Because a plain file might work even if you miss a byte, but an encoded file won't.

Regards

[vermarajeev]

This is what I was asking.
If you encode the data to base 64 before sending it.
Also, I asked if you tested this base 64 encoding/decoding locally.
Maybe you don't decode from base64 ok.
Regards
The decoding from base64 is taken care by smtp once I use
Content-Transfer-Encoding: base64
after the DATA tag used for smtp.

I imagine you cannot post the code, but are you sure there aren't any differences between plain files? Not even a byte?
Do you handle well SMTP transparency and the end of data message indicator?
Because a plain file might work even if you miss a byte, but an encoded file won't.

I can here is the small sample

void SynsupMail::sendMessage()
{
QString message;
string msg;

QFile f ( textFilename->text() ); //textFilename->text() contains file path. The file can be en
//encrypted using AES::CBC or a plain text
if ( f.open (IO_ReadOnly) )
{
// file opened successfully
QTextStream t ( &f );
// read the contents of the file into message
while( ! t.eof() )
{
message += t.readLine();
message += "\n";
}
f.close();

//encode the data using base64 using crypto++ library for Qt3, in Qt4
//there is a function tobase64().
StringSource (string(message.data()), true, new Base64Encoder(new StringSink(msg))) ;

Smtp *smtp = new Smtp( textFrom->text(), textTo->text(), textSubject->text(),
QString(msg.data()), fileName, );
}
}

All the code in smtp.cpp is same as that of /examples/network/mail except this
else if in function void Smtp::readyRead(). I made this modification to
send the file as an attachment

else if ( state == Body && responseLine[0] == '3' )
{
QString fileName("");
QString version;
version += QString("MIME-Version: 1.0\r\n");

QString Header;
Header += QString("Content-Type: multipart/mixed; ") +
QString("boundary=unique-boundary-1\r\n") ;

QString headerContent;
headerContent += QString("--unique-boundary-1\r\n");
headerContent += QString("Content-Type: application/octet-stream ") + QString("\r\n");

headerContent += QString("Content-Transfer-Encoding: base64\r\n");
headerContent += QString("Content-Disposition: attachment;") +
QString( "filename=" ) + _fileName ;

//t is QTextStream
*t << version;
*t << address;
*t << Header;
*t << headerContent;

*t << message;
QString closeTag;
closeTag += QString("--unique-boundary-1--\r\n");

*t << closeTag << ".\r\n";
state = Quit;
}

These are the things that happen,
Plain text file--->
plaintext file --> Encoding base64 --> Over N/w --> SMTP decodes base64 --> plaintext file to receiver

Encrypted file (encrypted with AES::CBC)--->
Encrypted file --> Encoding base64 --> Over N/w --> SMTP decodes base64 --> Encrypted file

M i doing anything wrong above ?

[marcel]

while( ! t.eof() )
{
message += t.readLine();
message += "\n";
}
I see you add a newline character for each line read from the file.
Why is that? RFC 821 specifies maximum line lengths of 998 bytes.
Do you do this for both encrypted and plain files? Because it is not ok to modify an encrypted file like this. A plain file would work, though.

If you do this for encrypted files, then do you also do you do the reverse process on receiving?

Regards

[wysota]

I suggest you try to base64 encode a plain text file using Qt methods and send it over smtp to see if it is decoded correctly by your mail client. Then you can focus on encrypting the content.

[vermarajeev]

while( ! t.eof() )
{
message += t.readLine();
message += "\n";
}
I see you add a newline character for each line read from the file.
Why is that?
A newline. So mean to say I shouldnt append the new line like above.
Ok I'll try removing the newline "\n" from message and see if it works.

If you do this for encrypted files, then do you also do you do the reverse process on receiving?
No I was not doing that.

Marcel one more doubt, I use
message = QString::fromLatin1( "\n\n" ) + body + "\n";
replace( message, QString::fromLatin1( "\n" ), QString::fromLatin1( "\r\n" ) );
replace( message, QString::fromLatin1( "\r\n.\r\n" ), QString::fromLatin1( "\r\n..\r\n" ) );
where message is the base64 encoded data. Do I need to use replace() here? I think replacing "\n" with "\r\n" might also modify the data. I use the same code as in smtp.cpp given in /examples/network/mail

[vermarajeev]

I suggest you try to base64 encode a plain text file using Qt methods and send it over smtp to see if it is decoded correctly by your mail client. Then you can focus on encrypting the content.

Yes, I have tried it and it works fine for plain text. The only problem is with encrypted file.

[marcel]

Marcel one more doubt, I use
message = QString::fromLatin1( "\n\n" ) + body + "\n";
replace( message, QString::fromLatin1( "\n" ), QString::fromLatin1( "\r\n" ) );
replace( message, QString::fromLatin1( "\r\n.\r\n" ), QString::fromLatin1( "\r\n..\r\n" ) );where message is the base64 encoded data. Do I need to use replace() here? I think replacing "\n" with "\r\n" might also modify the data. I use the same code as in smtp.cpp given in /examples/network/mail

It is not ok to alter the encrypted message in this way, unless you apply the reverse before decrypting it.


replace( message, QString::fromLatin1( "\r\n.\r\n" ), QString::fromLatin1( "\r\n..\r\n" )
It is required.
This is actually the transparency part.
So you do this after encryption but prior to sending?
Then do the reverse after you receive it.

Regards

[vermarajeev]

So you do this after encryption but prior to sending?
Did you mean,
For plainText
plaintext-->base64-->replace() for transparency part -->send

For encrypted file with AES::CBC
encrypted file-->base64-->replace() for transparency part-->send


Then do the reverse after you receive it.
Ok for plain text file there in no problem in reversing the opr, but how do I reverse
for an encrypted file as the data is binary ?:(

[marcel]

Load the encrypted message in a QByteArray and replace the required bytes.

Regards

[wysota]

The order while sending should be:
1. encrypt file
2. encode it with base64
3. divide it into blocks 64 (or 80 or whatever) bytes each
4. concatenate blocks using newlines (\r\n)
5. pass the result into the mail stream adding a base64 header

when receiving a mail:
1. parse headers
2. concatenate lines
3. base64 decode into QByteArray
4. decrypt
5. pass the result into the stream

[vermarajeev]

Load the encrypted message in a QByteArray and replace the required bytes.

Regards

Hey marcel, I'll make changes what you have suggested and let you know the status.
Thanks

[vermarajeev]

Hi marcel, I modified the code as you said. I'm unable to
revert the operation after receiving the file, the contents
gets deleted, Can you please check the code below.


//Sending message------>
void SynsupMail::sendMessage()
{
QByteArray dat;
string msg;
QFile f ( textFilename->text() );
if ( f.open (IO_ReadOnly) )
{
// file opened successfully
dat = f.readAll();
}
else
{
// problems opening the file - emit a warning message
QMessageBox::warning( this, tr("error"),
tr("Error opening file %1").arg( textFilename->text() ) );
enableSubmit();
return;
}
f.close();
StringSource ( string(QString(dat).data()), true, new Base64Encoder(new StringSink(msg)));
Smtp *smtp = new Smtp( textFrom->text(), textTo->text(),
textSubject->text(),
QString(msg.data()), fileName );
}

Smtp::Smtp( const QString &from, const QString &to,
const QString &subject,
const QString &body, const QString &fileName)
{
address = QString::fromLatin1( "From: " ) + from +
QString::fromLatin1( "\nTo: " ) + to +
QString::fromLatin1( "\nSubject: " ) + subject + "\n";

message = QString::fromLatin1( "\n\n" ) + body + "\n";

/*replace( message, QString::fromLatin1( "\n" ),
QString::fromLatin1( "\r\n" ) );*/
replace( message, QString::fromLatin1( "\r\n.\r\n" ),
QString::fromLatin1( "\r\n..\r\n" ) );
replace( address, QString::fromLatin1( "\n" ),
QString::fromLatin1( "\r\n" ) );
replace( address, QString::fromLatin1( "\r\n.\r\n" ),
QString::fromLatin1( "\r\n..\r\n" ) );
}



//Receiving message------>
int SysupClass::openFile( const char* infile )
{
QByteArray dat;
QFile f ( infile );
if ( f.open (IO_ReadOnly) )
{
// file opened successfully
dat = f.readAll();
}
else
{
f.close();
return -1;
}
f.close();

//Here I might not be correct to reverse the operation
//i.e removing \r\n..\r\n
/*QString ss = dat;
replace( ss, QString::fromLatin1( "\r\n..\r\n" ),
QString::fromLatin1( "\r\n.\r\n" ) );*/

QFile f1 ( strcat(infile, ".mod") );
if ( f1.open (IO_WriteOnly|IO_Truncate) )
{
// file opened successfully
f1.writeBlock(ss, ss.length());
}
f1.close();

//decrypt the file with AES::CBC mode
}

wysota--->
I'm unable to understand these points
3. divide it into blocks 64 (or 80 or whatever) bytes each ---- need some more explanation
4. concatenate blocks using newlines (\r\n) ----- ---- need some more explanation
5. pass the result into the mail stream adding a base64 header --------- ---- need some more explanations
If possible please correct me with my code above and tell me how can I achieve the above three points

[vermarajeev]

The order while sending should be:
1. encrypt file
2. encode it with base64
3. divide it into blocks 64 (or 80 or whatever) bytes each
4. concatenate blocks using newlines (\r\n)
5. pass the result into the mail stream adding a base64 header

when receiving a mail:
1. parse headers
2. concatenate lines
3. base64 decode into QByteArray
4. decrypt
5. pass the result into the stream

Did you mean this below code

while sending
int MAX_BLK_SIZE = 64;
string sSrcBlk = dat;
int hManyBlks = sSrcBlk.length()/MAX_BLK_SIZE;
char** pSrcBlk = new char*[hManyBlks + 1];

for( int i=0; i < hManyBlks; ++i )
{
pSrcBlk[i] = new char[MAX_BLK_SIZE + 1];
memcpy(pSrcBlk[i], sSrcBlk.data(), MAX_BLK_SIZE);
}

//append the block
for( int i=0; i < hManyBlks; ++i )
QString blks = QString( pSrcBlk[i] ) + QString("\r\n");

//send blks through mail stream adding base64 header



while receiving
1) parse -- How to do this ?
2) contcatenate lines -- How to do this ?
3) base64 decode into ByteArray -- I can do it
4) decrypt ---- I can do it
5) pass the result into the stream -- I can do it

[wysota]

I'm unable to understand these points
3. divide it into blocks 64 (or 80 or whatever) bytes each ---- need some more explanation
4. concatenate blocks using newlines (\r\n) ----- ---- need some more explanation
5. pass the result into the mail stream adding a base64 header --------- ---- need some more explanations
If possible please correct me with my code above and tell me how can I achieve the above three points

Ad 3. and 4. base64 encoded data is a string of characters. The usual thing to do is to divide the long line into shorter lines.
Ad 5. Make the result part of your outgoing mail embedding it into proper attachment headers.

Something like:
QString base64encoded; // your encoded data here
QStringList slist;
while(base64encoded.length()>64){
slist << base64encoded.left(64);
base64encoded.remove(0, 64);
}
slist << base64encoded; // remainder

QString result = slist.join("\x0a\x0C"); // \r\n


while receiving
1) parse -- How to do this ?
2) contcatenate lines -- How to do this ?
3) base64 decode into ByteArray -- I can do it
4) decrypt ---- I can do it
5) pass the result into the stream -- I can do it

Oh come on... I'm not going to write a mail client for you :) Use your head - think.

[vermarajeev]

Thank you wysota,
It is a long time that I have send the post and till then I have been trying to solve the problem. And the good thing is I'm just near to solution. I'm able to send the encrypted data, and also able to decode it but only partial i.e only half. I think there is a problem using std::string with binary data.

I'm just trying to figure out and let you know if I get.
I'm really sorry to ask for solution without myself going in depth.

Thanks

[wysota]

Why do you use std::string for binary data? It should be safe anyway (contrary to using QString)... You should simply use QByteArray and its data() method if you need to operate on the raw data directly.