Run several external programs sequetialy usig QProcess while reading stdout?

[qlands]

Hi,

I need to run a series of external programs one after the other but getting the output for each program. I have the programs in a QStringList and I can run them in a separate thread sequentially using:

m_process->execute(programName,params);

m_process->execute(programName,params);

To copy to clipboard, switch view to plain text mode 

The problem is that if I wait until finished I get any output until the program finish and one of the programs outputs a progress percentage I need to capture.. But with waiting until finished I get the last percentage (100). However I have no idea how to control the sequential execution of programs without waiting til one is finished.

Here is the code:

gobletthread::gobletthread(QObject * parent):
    QThread(parent)
{
 
}
 
void gobletthread::run()
{
    qDebug() << "Start of Run";
    m_process = new QProcess();
    connect(m_process,SIGNAL(finished(int)),this,SLOT(programExited(int)));
    connect(m_process,SIGNAL(readyReadStandardOutput()),this,SLOT(readData()));
    m_process->setWorkingDirectory("/data/cquiros/projects/ILRI-MARIO/software/nile");
    int pos;
 
    for (pos = 0; pos <= m_programs.count()-1;pos++)
    {
        m_programID = pos;
        //Run the program
        runProgram(m_programs[pos].description,m_programs[pos].name,m_programs[pos].params);
    }
 
    qDebug() << "End of Run";
 
}
 
void gobletthread::runProgram(QString description,QString program, QStringList params)
{
    QSettings settings("ILRI", "NILE");
    QString gobletPath = settings.value("GOBLETPath","").toString();
    QString programName;
    if (!gobletPath.isEmpty())
    {
        programName = gobletPath + "/" +  program;
        qDebug() << "Runing:" << program;
        if (QFile::exists(programName))
        {
            if ((program == "goblet-importdataset") ||
                    (program == "goblet-importshape"))
                catchSubprocess = true; //Both import programs generate progress report in the stdout
            else
                catchSubprocess = false;
            emit processing(description);
                m_process->execute(programName,params); //Executes the program waiting until finish
        }
        else
        {
            qDebug() << program << "Does not exists!!!";
        }
    }
    else
    {
        qDebug() << "Goblet path does not exists";
    }
}
 
void gobletthread::setPrograms(QList<TprogramInfo> programs)
{
    m_programs.append(programs);
}
 
void gobletthread::programExited(int exitCode)
{        
    emit programFinished(m_programID,exitCode);
}
 
void gobletthread::readData()
{   
   if (!catchSubprocess)
   {
        //Intents to capture the total output after the end of execution
        QString data(m_process->readAll());
        if (!data.isEmpty())
            emit readyRead(m_programID,data);
   }
    else
    {
        // Intents to capture the output each time the the program writes in the stdout
        // Here we try to get the percentage processed by the program
        QString data(m_process->readLine(14));
        qDebug() << data;
        data = data.replace("%","");
        data = data.replace("inserted","");
        data = data.simplified();
        int perc;
        perc = data.toInt();
        if (perc > 0)
            emit subprocess(perc);
    }
}

gobletthread::gobletthread(QObject * parent):
    QThread(parent)
{

}

void gobletthread::run()
{
    qDebug() << "Start of Run";
    m_process = new QProcess();
    connect(m_process,SIGNAL(finished(int)),this,SLOT(programExited(int)));
    connect(m_process,SIGNAL(readyReadStandardOutput()),this,SLOT(readData()));
    m_process->setWorkingDirectory("/data/cquiros/projects/ILRI-MARIO/software/nile");
    int pos;

    for (pos = 0; pos <= m_programs.count()-1;pos++)
    {
        m_programID = pos;
        //Run the program
        runProgram(m_programs[pos].description,m_programs[pos].name,m_programs[pos].params);
    }

    qDebug() << "End of Run";

}

void gobletthread::runProgram(QString description,QString program, QStringList params)
{
    QSettings settings("ILRI", "NILE");
    QString gobletPath = settings.value("GOBLETPath","").toString();
    QString programName;
    if (!gobletPath.isEmpty())
    {
        programName = gobletPath + "/" +  program;
        qDebug() << "Runing:" << program;
        if (QFile::exists(programName))
        {
            if ((program == "goblet-importdataset") ||
                    (program == "goblet-importshape"))
                catchSubprocess = true; //Both import programs generate progress report in the stdout
            else
                catchSubprocess = false;
            emit processing(description);
                m_process->execute(programName,params); //Executes the program waiting until finish
        }
        else
        {
            qDebug() << program << "Does not exists!!!";
        }
    }
    else
    {
        qDebug() << "Goblet path does not exists";
    }
}

void gobletthread::setPrograms(QList<TprogramInfo> programs)
{
    m_programs.append(programs);
}

void gobletthread::programExited(int exitCode)
{        
    emit programFinished(m_programID,exitCode);
}

void gobletthread::readData()
{   
   if (!catchSubprocess)
   {
        //Intents to capture the total output after the end of execution
        QString data(m_process->readAll());
        if (!data.isEmpty())
            emit readyRead(m_programID,data);
   }
    else
    {
        // Intents to capture the output each time the the program writes in the stdout
        // Here we try to get the percentage processed by the program
        QString data(m_process->readLine(14));
        qDebug() << data;
        data = data.replace("%","");
        data = data.replace("inserted","");
        data = data.simplified();
        int perc;
        perc = data.toInt();
        if (perc > 0)
            emit subprocess(perc);
    }
}

To copy to clipboard, switch view to plain text mode 

Any idea is much appreciated.

Thanks,
Carlos.

[wysota]

What do you need threads for?

[qlands]

Hi,

I don't know....Is the idea that I had to not hang the GUI with the waiting ... How would it be a better way?

Thanks in advance,
Carlos.

[amleto]

m_process->execute(programName,params); //Executes the program waiting until finish

m_process->execute(programName,params); //Executes the program waiting until finish

To copy to clipboard, switch view to plain text mode 

I don't think that is right. A new process is started - that command does not wait until it is finished

[qlands]

From the documentation:

Execute(const QString & program, const QStringList & arguments)

Starts the program program with the arguments arguments in a new process, waits for it to finish, and then returns the exit code of the process

[wysota]

From the documentation:

Execute(const QString & program, const QStringList & arguments)

Starts the program program with the arguments arguments in a new process, waits for it to finish, and then returns the exit code of the process

It also says it is a static method. Anyway, if you don't want to wait then don't use a function that waits. Use one that doesn't.

[amleto]

Ah, I just read the bit for the 1 argument method

This is an overloaded function.

Starts the program program in a new process. program is a single string of text containing both the program name and its arguments. The arguments are separated by one or more spaces.

Anyway, as wysota says, just use start instead of execute for asynchronous launching

[ChrisW67]

You effectively want to run a shell script while maintaining UI responsiveness. Something along these lines:

1. Implement some sort of queue to contain the list of commands to execute.
2. Take the first item from the queue and QProcess::start() it.
3. Use the QProcess::readyReadStandardOutput () and QProcess::readyReadStandardError() signals to pick up output as it arrives
4. Use the QProcess::finished() signal to:
   
   • Handle any remaining output from the program
   • Detect normal exit and return to 2.
   • Detect error states and do something like stop processing the script

There should be no threads, blocking calls, or unresponsive UI.

[qlands]

ok. I re-implement without a QThread but still readyReadStandardOutput() does not get emmited when a program writes in the stdout.

gobletrunner::gobletrunner(QWidget *parent) :
    QDialog(parent),
    ui(new Ui::gobletrunner)
{
    ui->setupUi(this);
    this->setWindowFlags(Qt::Dialog | Qt::FramelessWindowHint);
 
    //Creates the process and connects it
    m_process = new QProcess(this);
    connect(m_process,SIGNAL(finished(int)),this,SLOT(programFinished(int)));
    connect(m_process,SIGNAL(readyReadStandardOutput()),this,SLOT(readyRead()));
 
    totalPrograms = 0;
    totalProcessed = 0;
}
 
gobletrunner::~gobletrunner()
{
    delete ui;
}
 
void gobletrunner::on_pushButton_clicked()
{
    qDebug() << "Start commands";
    ui->progressBar->setMaximum(totalPrograms);
    if (m_programs.count() > 0)
    {
        processCommand(0); //Process the first program
    }
    else
    {
        close();
    }
}
 
void gobletrunner::processCommand(int pos)
{
    currProgram = pos;
    QSettings settings("ILRI", "NILE");
    QString gobletPath = settings.value("GOBLETPath","").toString();
    QString program;
    if (!gobletPath.isEmpty())
    {
        program = gobletPath + "/" +  m_programs[pos].name;
        qDebug() << "Runing:" << m_programs[pos].name;
        if (QFile::exists(program))
        {
            ui->lblproc->setText(m_programs[pos].description + ". Please wait...");
            m_process->start(program,m_programs[pos].params); //Starts the program
        }
        else
        {
            if (currProgram +1 <= m_programs.count()-1)
            {
                processCommand(currProgram +1);
            }
            else
            {
                commandsFinished();
            }
        }
    }
    else
    {
        if (currProgram +1 <= m_programs.count()-1)
        {
            processCommand(currProgram +1);
        }
        else
        {
            commandsFinished();
        }
    }
}
 
 
void gobletrunner::programFinished(int exitCode)
{
    totalProcessed++;
    ui->progressBar->setValue(totalProcessed);
    m_programs[currProgram].finished = true;
    m_programs[currProgram].exitcode = exitCode;
    QString data(m_process->readAll());
    m_programs[currProgram].output = data;
    if (currProgram +1 <= m_programs.count()-1) //If I have more programs
    {       
        processCommand(currProgram +1);
    }
    else
    {
        commandsFinished();
    }
}
 
void gobletrunner::readyRead()
{    
    qDebug() << "Ready to read";
 
    //Both imports produce a percentage in the stdout
    if ((m_programs[currProgram].name == "goblet-importdataset") ||
        (m_programs[currProgram].name == "goblet-importshape"))
    {
        //Gets the data from the stdout and only grabs the percentage
        QString data(m_process->readLine(14));
        qDebug() << data;
        data = data.replace("%","");
        data = data.replace("inserted","");
        data = data.simplified();
        int perc;
        perc = data.toInt();
        if (perc > 0)
            ui->progressBar_2->setValue(perc);
    }
}
 
void gobletrunner::commandsFinished()
{
    close();
}

gobletrunner::gobletrunner(QWidget *parent) :
    QDialog(parent),
    ui(new Ui::gobletrunner)
{
    ui->setupUi(this);
    this->setWindowFlags(Qt::Dialog | Qt::FramelessWindowHint);
    
    //Creates the process and connects it
    m_process = new QProcess(this);
    connect(m_process,SIGNAL(finished(int)),this,SLOT(programFinished(int)));
    connect(m_process,SIGNAL(readyReadStandardOutput()),this,SLOT(readyRead()));

    totalPrograms = 0;
    totalProcessed = 0;
}

gobletrunner::~gobletrunner()
{
    delete ui;
}

void gobletrunner::on_pushButton_clicked()
{
    qDebug() << "Start commands";
    ui->progressBar->setMaximum(totalPrograms);
    if (m_programs.count() > 0)
    {
        processCommand(0); //Process the first program
    }
    else
    {
        close();
    }
}

void gobletrunner::processCommand(int pos)
{
    currProgram = pos;
    QSettings settings("ILRI", "NILE");
    QString gobletPath = settings.value("GOBLETPath","").toString();
    QString program;
    if (!gobletPath.isEmpty())
    {
        program = gobletPath + "/" +  m_programs[pos].name;
        qDebug() << "Runing:" << m_programs[pos].name;
        if (QFile::exists(program))
        {
            ui->lblproc->setText(m_programs[pos].description + ". Please wait...");
            m_process->start(program,m_programs[pos].params); //Starts the program
        }
        else
        {
            if (currProgram +1 <= m_programs.count()-1)
            {
                processCommand(currProgram +1);
            }
            else
            {
                commandsFinished();
            }
        }
    }
    else
    {
        if (currProgram +1 <= m_programs.count()-1)
        {
            processCommand(currProgram +1);
        }
        else
        {
            commandsFinished();
        }
    }
}


void gobletrunner::programFinished(int exitCode)
{
    totalProcessed++;
    ui->progressBar->setValue(totalProcessed);
    m_programs[currProgram].finished = true;
    m_programs[currProgram].exitcode = exitCode;
    QString data(m_process->readAll());
    m_programs[currProgram].output = data;
    if (currProgram +1 <= m_programs.count()-1) //If I have more programs
    {       
        processCommand(currProgram +1);
    }
    else
    {
        commandsFinished();
    }
}

void gobletrunner::readyRead()
{    
    qDebug() << "Ready to read";
    
    //Both imports produce a percentage in the stdout
    if ((m_programs[currProgram].name == "goblet-importdataset") ||
        (m_programs[currProgram].name == "goblet-importshape"))
    {
        //Gets the data from the stdout and only grabs the percentage
        QString data(m_process->readLine(14));
        qDebug() << data;
        data = data.replace("%","");
        data = data.replace("inserted","");
        data = data.simplified();
        int perc;
        perc = data.toInt();
        if (perc > 0)
            ui->progressBar_2->setValue(perc);
    }
}

void gobletrunner::commandsFinished()
{
    close();
}

To copy to clipboard, switch view to plain text mode 

But still the signal readyReadStandardOutput() does not get emmited when an import program writes a line in the stdout. This is what is written by the import programs:

29 % inserted                                                                                                                                                
56 % inserted                                                                                                                                                
86 % inserted                                                                                                                                                
100 % inserted                                                                                                                                               
 
Upload finished. 2072200 cells imported.                                                                                                                     
Finished in 0 Hours,0 Minutes and 33 Seconds.

29 % inserted                                                                                                                                                
56 % inserted                                                                                                                                                
86 % inserted                                                                                                                                                
100 % inserted                                                                                                                                               
                                                                                                                                                             
Upload finished. 2072200 cells imported.                                                                                                                     
Finished in 0 Hours,0 Minutes and 33 Seconds.

To copy to clipboard, switch view to plain text mode 

However I only get such result when the program finish.

Any idea is much appreciated.

[amleto]

Are you sure the data is on standard out and not error out? Have you tried to read the error channel as well?
Have you checked that the signal/slot connection is made ok?

[qlands]

Hi,

I added:

m_process->setProcessChannelMode(QProcess::MergedChannels) ;

So I merge the channels. The import programs the I am running uses printf for example: printf("%i %% inserted \n",perc);

QT does not give me any warnings when I connect the signal.

Still... the signal readyReadStandardOutput is not emmited when the import program writes to the stdout.

Thanks,
Carlos.

[wysota]

How do you know the signal is not emitted? Why are you calling readLine(14)?

[qlands]

Hi,

Because that signal is connected to the slot readData() line 11. In readData() I have a qDebug() << "Ready to read"; in line 97 so I know that such slot is executed when the signal is emitted. If the import program writes 4 time or more I would expect 4 or more ""Ready to read" messages... But I Just get one "Ready to read" message once the program is finish

I have readLine(14) to I just read 14 bytes.

Thanks.

[wysota]

I have readLine(14) to I just read 14 bytes.

And what do you intend to do with the rest of the line? Also does the slave program do any line buffering of its output? Does your code work with a different slave program?

[qlands]

I don't need the rest of the line. In fact the import programs produce a max of 14 bytes for reporting the progress. But the main problem is that such readline is not executed because the signal is not emmited!

Only two programs report progress. I can try with other linux command programs to see if the signal gets emmited... but the import programs are nothing special.. just write in the stdout using printf()

Thanks,
Carlos.

[amleto]

once you have started a process, does control get back to the event loop before the process finishes?


e: hmmm, looks like it does.

[qlands]

ok. I tried with a another program (a java that generates a lot of stdout) and the signal is emmited properly every time the program writes in the stdout. So there is somethig wrong with the import programs. Both import programs when ran get really busy inserting data into a mysql database but I can see in my terminal that they report the status. Is there anything else that I need to do besides a printf()? like refresing the stdout or something?

[wysota]

I don't need the rest of the line. In fact the import programs produce a max of 14 bytes for reporting the progress.

So read the whole line and then discard whatever you don't need. If you only read 14 bytes then next time you call readLine(14), you'll get next (up to) 14 bytes of the first line. That's probably not what you want.

Is there anything else that I need to do besides a printf()? like refresing the stdout or something?

You need to write the end of line character and/or fflush stdout.

[amleto]

writing '\n' doesn't guarantee a flush, but std::endl does, or so I have read. That must be for cout rather than printf, though. Although why use printf in the first place?

[qlands]

I don't know why but this line

printf("\r%i %% inserted", perc);

printf("\r%i %% inserted", perc);

To copy to clipboard, switch view to plain text mode 

Was working properly the last time I run the program.. Not is does not print in the terminal and I just get the final percentage (100)..... the line gets executed in between heavy inserts to a mysql database... so maybe the stdout is not refreshed in time.

What else can I use besides printf()?

Thanks,

---

Added after 13 minutes:

Ha!!! If I change the printf for a qDebug() the signal is emmited and all works fine!.... very interesting!