ReadAllStandardOutput() is returning empty characters if they are not recognised

[INeedADollar]

I have a QProcess that runs a youtube-dl command to get the title of videos. I read the output of the process when it's finished and if the title of the video has some japanese characters for example, readAllStandardOutput() is not returning them. For example, if I have title "Learn Hiragana *some japanese characters here* (Japanese alphabet)" the readAllStandardOutput() function is returning "Learn Hiragana (Japanese alphabet)".

How can I solve this?

[ChristianEhrlicher]

How do you display the text? Since readAllStandardOutput() returns a QByteArray you have to convert it properly to a QString with QString::fromLocal8Bit()

[INeedADollar]

How do you display the text? Since readAllStandardOutput() returns a QByteArray you have to convert it properly to a QString with QString::fromLocal8Bit()

I tried using QString::fromLocal8Bit() but it's the same because I'm printing the QByteArray as it is.
My code:

mainwindow.h


#ifndef MAINWINDOW_H
#define MAINWINDOW_H

#include <QWidget>
class mainWindow : public QWidget
{
Q_OBJECT
//here are defined some variables
public:
mainWindow(QWidget *parent = 0);
~mainWindow();
void createUI();
void process();
QProcess *process = new QProcess(this);

private slots:
void ReadOutput(int, QProcess::ExitStatus);

};
#endif // MAINWINDOW_H


mainwindow.cpp

#include <QWidget>
#include <QPushButton>
#include <QProcess>
#include <QByteArray>
#include <QTextCodec>
#include <QString>
#include <QDebug>
#include "mainwindow.h"

mainWindow::mainWindow(QWidget *parent) : QWidget(parent)
{
createUI();
connect(process, SIGNAL(finished(int, QProcess::ExitStatus)), this, SLOT(ReadOutput(int, QProcess::ExitStatus)));
}


mainWindow::~mainWindow()
{

}
void mainWindow::createUI(){
//here I create the look of the window
QPushButton buttonsearch = new QPushButton("Start process", this);
buttonsearch->setToolTip("Start process");
buttonsearch->setGeometry(200, 290, 100, 30);
connect(buttonsearch, &QPushButton::clicked, [this]() {process(); });
}

void mainWindow::process(){
process->setProcessChannelMode(QProcess::MergedChannels);
process->start("\"D:\\YTDownloader\\youtube-dl.exe\" -e --no-playlist https://www.youtube.com/watch?v=6V-wwfuxZxw");

void readOutput(int exitCode, QProcess::ExitStatus exitStatus){
qDebug() << exitCode;
qDebug() << exitStatus;
QByteArray a = process->readAllStandardOutput();
qDebug() << a; //here is not showing characters that are not recognised
QTextCodec* utfCodec = QTextCodec::codecForName("UTF-8");
processStdout = utfCodec->toUnicode(a);
qDebug() << processStdout;
}


main.cpp

#include <QApplication>
#include "mainwindow.h"

int main(int argl,char *argv[])
{
QApplication app(argl,argv);

mainWindow *window = new mainWindow();
window->setWindowTitle("Test");
window->setFixedSize(700, 335);
window->show();

return app.exec();
}

[ChristianEhrlicher]

Don't rely on windows' cmd.exe - it can't handle unicode properly. Use a QMessageBox or similar.

[INeedADollar]

Don't rely on windows' cmd.exe - it can't handle unicode properly. Use a QMessageBox or similar.

Sorry but I don't understand what you mean. How to use a QMessageBox to run a QProcess?

[ChristianEhrlicher]

Sorry but I don't understand what you mean. How to use a QMessageBox to run a QProcess?

Where did I told you to do so? I said you should print the return value from readAllStandardOutput() in a QMessageBox (and don't forget to properly encode it into a QString with QString::fromLocal8Bit())

[Lesiok]

What You see in file result.txt when You try this from command line :
D:\YTDownloader\youtube-dl.exe -e --no-playlist https://www.youtube.com/watch?v=6V-wwfuxZxw >>result.txt

[INeedADollar]

Where did I told you to do so? I said you should print the return value from readAllStandardOutput() in a QMessageBox (and don't forget to properly encode it into a QString with QString::fromLocal8Bit())

Ok thank you, but I get the same result.


What You see in file result.txt when You try this from command line :
D:\YTDownloader\youtube-dl.exe -e --no-playlist https://www.youtube.com/watch?v=6V-wwfuxZxw >>result.txt

I get the same result. For example, if I have title "Petric? Moise - ??tia-s b?n??enii mei" in the txt file I get "Petric Moise - tia-s bnenii mei".

[d_stranz]

I get the same result.
Then maybe your "youtube-dl" program itself does not support Unicode.

[INeedADollar]

Then maybe your "youtube-dl" program itself does not support Unicode.

Hmmm, curious because when I run the command in cmd I get the correct title with all characters.

[ChristianEhrlicher]

Can you please show us your code?

[INeedADollar]

Can you please show us your code?

Yes, of course.

mainwindow.h

#ifndef MAINWINDOW_H
#define MAINWINDOW_H

#include <QWidget>
#include <QProcess>
#include <QString>
#include <QLabel>

class mainWindow : public QWidget
{
Q_OBJECT

public:
mainWindow(QWidget *parent = nullptr);
~mainWindow();
void createUI();
void process();
QProcess *process1 = new QProcess(this);
QString processStdout;
QLabel *label;

private slots:
void ReadOutput(int, QProcess::ExitStatus);

};
#endif // MAINWINDOW_H


mainwindow.cpp

#include "mainwindow.h"
#include <QPushButton>
#include <QtDebug>
#include <QTextCodec>

mainWindow::mainWindow(QWidget *parent) : QWidget(parent)
{
createUI();
connect(process1, SIGNAL(finished(int, QProcess::ExitStatus)), this, SLOT(ReadOutput(int, QProcess::ExitStatus)));
}

mainWindow::~mainWindow()
{

}
void mainWindow::createUI(){
label = new QLabel(this);
label->setGeometry(100, 200, 100, 30);

QPushButton *buttonsearch = new QPushButton("Start process", this);
buttonsearch->setToolTip("Start process");
buttonsearch->setGeometry(200, 290, 100, 30);
connect(buttonsearch, &QPushButton::clicked, [this]() {process(); });
}

void mainWindow::process(){
process1->setProcessChannelMode(QProcess::MergedChannels);
process1->start("\"D:\\YTDownloader\\youtube-dl.exe\" -e --no-playlist https://www.youtube.com/watch?v=6V-wwfuxZxw");
}

void mainWindow::ReadOutput(int exitCode, QProcess::ExitStatus exitStatus){
qDebug() << exitCode;
qDebug() << exitStatus;
QByteArray a = process1->readAllStandardOutput();
qDebug() << a;
QTextCodec* utfCodec = QTextCodec::codecForName("UTF-8");
processStdout = utfCodec->toUnicode(a);
qDebug() << processStdout;
label->setText(processStdout);
}


main.cpp

#include <QApplication>
#include "mainwindow.h"

int main(int argl,char *argv[])
{
QApplication app(argl,argv);

mainWindow *window = new mainWindow();
window->setWindowTitle("Test");
window->setFixedSize(700, 335);
window->show();

return app.exec();
}

[Lesiok]

Perhaps the problem is the Windows code page. I did the test with "Polish letters" and everything is OK but the result from youtube-dl is encoded in 1250 (8-bit code page) and not UTF-8. This is the link used for the test: https://www.youtube.com/watch?v=ndG8bM-9CMA
Can you provide the Japanese alphabet link to test?
P.S.
Test with arabic alphabet : https://www.youtube.com/watch?v=7Tc7LWpe7mg. Output generated on Windows is 29 bytes long, on Linux 85 bytes long. In my opinion it's an internal problem of youtube-dl.

[INeedADollar]

Perhaps the problem is the Windows code page. I did the test with "Polish letters" and everything is OK but the result from youtube-dl is encoded in 1250 (8-bit code page) and not UTF-8. This is the link used for the test: https://www.youtube.com/watch?v=ndG8bM-9CMA
Can you provide the Japanese alphabet link to test?
P.S.
Test with arabic alphabet : https://www.youtube.com/watch?v=7Tc7LWpe7mg. Output generated on Windows is 29 bytes long, on Linux 85 bytes long. In my opinion it's an internal problem of youtube-dl.

Link with Japanese Alphabet: https://www.youtube.com/watch?v=Bsfi4XbPE8M. I tried your link both with cmd and then with QProcess. When I tried the command in cmd with arabic alphabet link I got something like this ????? ??????? ?????? _35_ ????? ?????? The Ultimate Ninj, but if I copy paste the output(the title) in a browser I get the correct title.

When I tried with QProcess I got something like this: " _35_ The Ultimate Ninj". Maybe it's a problem with youtube-dl, I don't know.

I attached a photo to see how title looks after I copy-paste it in a browser from cmd.

[Lesiok]

definitely an environmental problem (operating system). On linux in a graphical environment (fonts installed for all alphabets) this result:
13227
On linux in a character environment like this:
13228

[INeedADollar]

definitely an environmental problem (operating system). On linux in a graphical environment (fonts installed for all alphabets) this result:
13227
On linux in a character environment like this:
13228

Ok thank you very much! Thank you very much all that tried to help me!

[ChristianEhrlicher]

You're still outputting the data to the command line instead using a QMessageBox as I said in my second post so it can't work... https://www.qtcentre.org/threads/70446-ReadAllStandardOutput()-is-returning-empty-characters-if-they-are-not-recognised?p=306088#post306088

[anda_skoa]

If it is a problem with the normal output of the Windows version of youtube-dl you could try its JSON output options.

Cheers,
_