Character encoding issues

**caduel** · 15th December 2008, 12:13

no, I'd drop cin here.
try something like

Qt Code:

Switch view

QTextStream in(stdin);
in.setLocale("eucTR");
in >> searchedAuthor;

QTextStream in(stdin);
in.setLocale("eucTR");
in >> searchedAuthor;

To copy to clipboard, switch view to plain text mode

alternatively:

Qt Code:

Switch view

cin >> author;
searchedAuthor = QString::fromAscii(author.c_str()); // after setting the locale
//   ... if (see docs) QTextCodec::setCodecForCStrings() has been set

cin >> author;
searchedAuthor = QString::fromAscii(author.c_str()); // after setting the locale
//   ... if (see docs) QTextCodec::setCodecForCStrings() has been set

To copy to clipboard, switch view to plain text mode

HTH

**gerome69** · 15th December 2008, 12:42

First of all:
Qt always uses unicode for character encoding, so all the turkish, german, chinese letters etc. are represented.

Microsoft windows always uses a country specific encoding,
f.e:
- Codepage 850 in western europe: http://de.wikipedia.org/wiki/Codepage_850
- Codepage 857 for turkish: http://de.wikipedia.org/wiki/Codepage_857

=> So you have to convert the input from your specific encoding to unicode.

Let's have a look at:
http://doc.trolltech.com/4.4/qtextcodec.html
There you can read:
The supported encodings are:
[...]
# IBM 850
# IBM 866
# IBM 874
[...]
# ISO 8859-1 to 10

So, unfortunately your needed encoding "IBM 857" is missing.
I don't know if it works, but tryp ISO 8859-9:
http://de.wikipedia.org/wiki/ISO_8859-9

A simple test program goes like this (look at my comments in the code!):

Qt Code:

Switch view

#include <QtCore>
#include <QtGui>
 
#include <iostream>
using namespace std;
 
int main(int argc, char** argv) {
    QApplication app(argc, argv);
 
    char author[100];
 
    cout<<"Authorname ?\n";
    cin>>author;
 
    // Just for to get an idea of how the character codes are seen internally. 
   //Please enter some of your arbitrary chars. In german I always use "Ã¤Ã¶Ã¼".
    for (int i=0; i<100; i++) {
        int c=author[i];
 
        if (c<0) c+=256;
        printf("%d ", c);
    }
 
    QByteArray encodedString=author;
   // QTextCodec *codec=QTextCodec::codecForName("IBM 850"); // western europe
    // QTextCodec *codec=QTextCodec::codecForName("IBM 850"); // turkish but will not work :-(
    QTextCodec *codec=QTextCodec::codecForName("ISO 8859-9"); // try it
 
    if (!codec) {
        printf("Codec not supported.\n");
 
        return 0;
    }
 
    QString searchedAuthor=codec->toUnicode(encodedString);    
 
    // this message box gives you a validation if the encoding is interpreted correctly.
   // out put on the console does not show you anything, because it does not use unicode   
    QMessageBox::information(NULL, "Ausgabe", searchedAuthor);
 
	return 0;
}

#include <QtCore>
#include <QtGui>

#include <iostream>
using namespace std;

int main(int argc, char** argv) {
    QApplication app(argc, argv);

    char author[100];
    
    cout<<"Authorname ?\n";
    cin>>author;
    
    // Just for to get an idea of how the character codes are seen internally. 
   //Please enter some of your arbitrary chars. In german I always use "Ã¤Ã¶Ã¼".
    for (int i=0; i<100; i++) {
        int c=author[i];

        if (c<0) c+=256;
        printf("%d ", c);
    }

    QByteArray encodedString=author;
   // QTextCodec *codec=QTextCodec::codecForName("IBM 850"); // western europe
    // QTextCodec *codec=QTextCodec::codecForName("IBM 850"); // turkish but will not work :-(
    QTextCodec *codec=QTextCodec::codecForName("ISO 8859-9"); // try it
    
    if (!codec) {
        printf("Codec not supported.\n");
        
        return 0;
    }
    
    QString searchedAuthor=codec->toUnicode(encodedString);    

    // this message box gives you a validation if the encoding is interpreted correctly.
   // out put on the console does not show you anything, because it does not use unicode   
    QMessageBox::information(NULL, "Ausgabe", searchedAuthor);

	return 0;
}

To copy to clipboard, switch view to plain text mode

Have fun, GÃ©rÃ´me

**yagabey** · 15th December 2008, 14:32

I couldnt make "c_str()" function work although I added <cstring>, <string> headers?

Qt Code:

Switch view

searchedAuthor = QString::fromAscii(author.c_str());

searchedAuthor = QString::fromAscii(author.c_str());

To copy to clipboard, switch view to plain text mode

That function returned true..(Codec supported..)

Qt Code:

Switch view

if (!codec) {
        printf("Codec not supported.\n");
        return 0;
    }

if (!codec) {
        printf("Codec not supported.\n");
        return 0;
    }

To copy to clipboard, switch view to plain text mode

I also tried:

Qt Code:

Switch view

QTextCodec *q;
    q = q->codecForName("ISO-8859-9");
    QTextCodec::setCodecForCStrings(q);

QTextCodec *q;
    q = q->codecForName("ISO-8859-9");
    QTextCodec::setCodecForCStrings(q);

To copy to clipboard, switch view to plain text mode

and

Qt Code:

Switch view

QTextCodec *codec=QTextCodec::codecForName("ISO 8859-9");

QTextCodec *codec=QTextCodec::codecForName("ISO 8859-9");

To copy to clipboard, switch view to plain text mode

in the message box, characters are not correct again..(it shows "Ã„Â°" instead of "İ" )

Qt Code:

Switch view

QMessageBox::information(NULL, "Ausgabe", searchedAuthor);

QMessageBox::information(NULL, "Ausgabe", searchedAuthor);

To copy to clipboard, switch view to plain text mode

what else should i do?

**caduel** · 15th December 2008, 14:43

I assumed that author is a std::string; if it is not (maybe it's just a char[20] or so...), just drop it.

**yagabey** · 15th December 2008, 14:47

Qt Code:

Switch view

searchedAuthor = QString::fromAscii(author);

searchedAuthor = QString::fromAscii(author);

To copy to clipboard, switch view to plain text mode

didnt fix the problem

**caduel** · 15th December 2008, 15:08

show us the (complete) code you are using

**gerome69** · 15th December 2008, 15:23

Originally Posted by yagabey

Qt Code:

Switch view

QTextCodec *codec=QTextCodec::codecForName("ISO 8859-9");

QTextCodec *codec=QTextCodec::codecForName("ISO 8859-9");

To copy to clipboard, switch view to plain text mode

in the message box, characters are not correct again..(it shows "Ã„Â°" instead of "İ" )

Qt Code:

Switch view

QMessageBox::information(NULL, "Ausgabe", searchedAuthor);

QMessageBox::information(NULL, "Ausgabe", searchedAuthor);

To copy to clipboard, switch view to plain text mode

what else should i do?

So, the right codec is definetyly Codepage 857, which is not supported by Qt :-(

You have to implement the conversion from CP 857 to ISO-8859-9 yourself.

Just walk throught the byte array and convert chars>127:

Qt Code:

Switch view

for (int i=0; i<auth_len; i++) {
    int c=author[i];
 
    if (c<0) {
        c+=256;
 
        switch(c) {
             // f.e. the g with a "bow" on it
             case 167: author[i]=240; break; // or maybe 240-256=-16, try it, I've got no turkish windows to test it!
            // I think you don't have to do it for all 128 chars, only for the few
            // turkish special chars you need
        }
     }
}

for (int i=0; i<auth_len; i++) {
    int c=author[i];

    if (c<0) {
        c+=256;

        switch(c) {
             // f.e. the g with a "bow" on it
             case 167: author[i]=240; break; // or maybe 240-256=-16, try it, I've got no turkish windows to test it!
            // I think you don't have to do it for all 128 chars, only for the few
            // turkish special chars you need
        }
     }
}

To copy to clipboard, switch view to plain text mode

G.

**yagabey** · 15th December 2008, 21:34

Here is a summary of the code:

Qt Code:

Switch view

#include <QHttp>
#include <QUrl>
#include <QBuffer>
#include <QFile>
#include <QTextStream>
#include <QXmlStreamReader>
#include <QHttp>
#include <QByteArray>
 
 
class ColumnListing 
{
    Q_OBJECT
public:
    ColumnListing();
 
public slots:
    void fetch();
    void readData(const QHttpResponseHeader &);
 
private:
    void parseXml();
 
    QXmlStreamReader xml;
    QString currentTag;
    QString linkString;
    QString titleString;
    QString descriptionString;
    QString authorString;
    QString urltext;
    QString inputNews;
    QString searchedAuthor;
    QFile file;
    QFile *htmlFile;
    QHttp httpInstance;
    int subconnectionId;
    QUrl urlColumnToGo;
 
    QHttp http;
    int connectionId;
 
};

#include <QHttp>
#include <QUrl>
#include <QBuffer>
#include <QFile>
#include <QTextStream>
#include <QXmlStreamReader>
#include <QHttp>
#include <QByteArray>


class ColumnListing 
{
    Q_OBJECT
public:
    ColumnListing();

public slots:
    void fetch();
    void readData(const QHttpResponseHeader &);

private:
    void parseXml();

    QXmlStreamReader xml;
    QString currentTag;
    QString linkString;
    QString titleString;
    QString descriptionString;
    QString authorString;
    QString urltext;
    QString inputNews;
    QString searchedAuthor;
    QFile file;
    QFile *htmlFile;
    QHttp httpInstance;
    int subconnectionId;
    QUrl urlColumnToGo;

    QHttp http;
    int connectionId;

};

To copy to clipboard, switch view to plain text mode

constructor part:

Qt Code:

Switch view

ColumnListing::ColumnListing(QWidget *parent)
    : QWidget(parent)
{
 
    connect(&http, SIGNAL(readyRead(const QHttpResponseHeader &)),
             this, SLOT(readData(const QHttpResponseHeader &)));
 
    char  *input;
    char  *author;
    input = new char[100];
    author = new char[100];
 
    cout<<"Newspaper ?\n";
    cin>>input;
    inputNews=input;//inputNews is global QString
 
    cout<<"Author ?\n";
    cin>>author;
    searchedAuthor=author;//searchedAuthor is global QString
 
    if (inputNews=="milliyet"){
                urltext.append("http://www.milliyet.com.tr/D/rss/rss/RssY.xml?ver=51");
            }
    else if (inputNews=="sabah"){
                urltext.append("http://www.sabah.com.tr/rss/yazarlar.xml");
            }
    else if (inputNews=="radikal"){
                 urltext.append("http://www.radikal.com.tr/radikal_yazar.xml");
             }
/*** More News sites here....***/
 
    file.setFileName("output.txt");//file is global QFile
    if (!file.open(QFile::ReadWrite | QFile::Truncate))
        return;
 
    htmlFile= new QFile("htmloutput.html");//htmlFile is global QFile
    if (!htmlFile->open(QFile::ReadWrite | QFile::Truncate))
        return;
}

ColumnListing::ColumnListing(QWidget *parent)
    : QWidget(parent)
{

    connect(&http, SIGNAL(readyRead(const QHttpResponseHeader &)),
             this, SLOT(readData(const QHttpResponseHeader &)));

    char  *input;
    char  *author;
    input = new char[100];
    author = new char[100];

    cout<<"Newspaper ?\n";
    cin>>input;
    inputNews=input;//inputNews is global QString

    cout<<"Author ?\n";
    cin>>author;
    searchedAuthor=author;//searchedAuthor is global QString

    if (inputNews=="milliyet"){
                urltext.append("http://www.milliyet.com.tr/D/rss/rss/RssY.xml?ver=51");
            }
    else if (inputNews=="sabah"){
                urltext.append("http://www.sabah.com.tr/rss/yazarlar.xml");
            }
    else if (inputNews=="radikal"){
                 urltext.append("http://www.radikal.com.tr/radikal_yazar.xml");
             }
/*** More News sites here....***/

    file.setFileName("output.txt");//file is global QFile
    if (!file.open(QFile::ReadWrite | QFile::Truncate))
        return;
    
    htmlFile= new QFile("htmloutput.html");//htmlFile is global QFile
    if (!htmlFile->open(QFile::ReadWrite | QFile::Truncate))
        return;
}

To copy to clipboard, switch view to plain text mode

fetching:

Qt Code:

Switch view

void ColumnListing::fetch()
{
    xml.clear();
    QUrl url(urltext);
    http.setHost(url.host());
    connectionId = http.get(url.path());
}

void ColumnListing::fetch()
{
    xml.clear();
    QUrl url(urltext);
    http.setHost(url.host());
    connectionId = http.get(url.path());
}

To copy to clipboard, switch view to plain text mode

parsing rs doc:

Qt Code:

Switch view

void ColumnListing::parseXml()
{
    QTextStream inputText(&file);
 
    while (!xml.atEnd()) {
        xml.readNext();
        if (xml.isStartElement()) {
            if (xml.name() == "item")
                linkString = xml.attributes().value("rss:about").toString();
            currentTag = xml.name().toString();
        } else if (xml.isEndElement()) {
            if (xml.name() == "item") {
 
                if(authorString.contains(searchedAuthor,Qt::CaseInsensitive)  ){
                inputText << titleString<< "   "<<linkString <<"   "<< descriptionString << authorString <<"\n";
 
                QUrl url(linkString);
 
                httpInstance.setHost(url.host());
                subconnectionId = http.get(url.path(),htmlFile);//write the author page into html file
            }
 
                titleString.clear();
                linkString.clear();
                descriptionString.clear();
                authorString.clear();
 
          }
 
        } else if (xml.isCharacters() && !xml.isWhitespace()) {
            if (currentTag == "title"){
                titleString += xml.text().toString();
            }
            else if (currentTag == "link"){
                linkString += xml.text().toString();
            }
            else if (currentTag == "description"){
                descriptionString += xml.text().toString();
            }
            else if (currentTag == "dc:creator"){
                authorString += xml.text().toString();
            }
        }
    }
    if (xml.error() && xml.error() != QXmlStreamReader::PrematureEndOfDocumentError) {
        qWarning() << "XML ERROR:" << xml.lineNumber() << ": " << xml.errorString();
        http.abort();
    }
}

void ColumnListing::parseXml()
{
    QTextStream inputText(&file);

    while (!xml.atEnd()) {
        xml.readNext();
        if (xml.isStartElement()) {
            if (xml.name() == "item")
                linkString = xml.attributes().value("rss:about").toString();
            currentTag = xml.name().toString();
        } else if (xml.isEndElement()) {
            if (xml.name() == "item") {
                
                if(authorString.contains(searchedAuthor,Qt::CaseInsensitive)  ){
                inputText << titleString<< "   "<<linkString <<"   "<< descriptionString << authorString <<"\n";

                QUrl url(linkString);

                httpInstance.setHost(url.host());
                subconnectionId = http.get(url.path(),htmlFile);//write the author page into html file
            }

                titleString.clear();
                linkString.clear();
                descriptionString.clear();
                authorString.clear();

          }

        } else if (xml.isCharacters() && !xml.isWhitespace()) {
            if (currentTag == "title"){
                titleString += xml.text().toString();
            }
            else if (currentTag == "link"){
                linkString += xml.text().toString();
            }
            else if (currentTag == "description"){
                descriptionString += xml.text().toString();
            }
            else if (currentTag == "dc:creator"){
                authorString += xml.text().toString();
            }
        }
    }
    if (xml.error() && xml.error() != QXmlStreamReader::PrematureEndOfDocumentError) {
        qWarning() << "XML ERROR:" << xml.lineNumber() << ": " << xml.errorString();
        http.abort();
    }
}

To copy to clipboard, switch view to plain text mode

main.cpp

Qt Code:

Switch view

int main(int argc, char **argv)
{
    QApplication app(argc, argv);
 
    QTextCodec *q;
    q = q->codecForName("ISO-8859-9");
    QTextCodec::setCodecForCStrings(q);
 
    ColumnListing *columnlisting = new ColumnListing;
    columnlisting->fetch();
    return app.exec();
}

int main(int argc, char **argv)
{
    QApplication app(argc, argv);

    QTextCodec *q;
    q = q->codecForName("ISO-8859-9");
    QTextCodec::setCodecForCStrings(q);

    ColumnListing *columnlisting = new ColumnListing;
    columnlisting->fetch();
    return app.exec();
}

To copy to clipboard, switch view to plain text mode

**yagabey** · 15th December 2008, 22:20

Ok at last I made it work:

instead of:

Qt Code:

Switch view

searchedAuthor = QString::fromAscii(author);

searchedAuthor = QString::fromAscii(author);

To copy to clipboard, switch view to plain text mode

I used :

Qt Code:

Switch view

searchedAuthor = QString::fromUtf8(author);

searchedAuthor = QString::fromUtf8(author);

To copy to clipboard, switch view to plain text mode

and now everything works perfectly, thank you all...

Thread: Character encoding issues

Thread Tools

Search Thread

Display

Hybrid View

Re: Character encoding issues

The following user says thank you to caduel for this useful post:

Re: Character encoding issues

Re: Character encoding issues

Re: Character encoding issues

Re: Character encoding issues

Re: Character encoding issues

Re: Character encoding issues

Re: Character encoding issues

Re: Character encoding issues

Bookmarks

Bookmarks

Posting Permissions