30 fps desired but getting 30 frames every ~21 seconds.

[PeterVE]

Hi All and thank you for reading this.

I have a MainWindow with three widgets (QWidget, QDialog and QOpenGLWidget). The QWidget is a navigation bar at the bottom and the QDiaglog is for user input to later serial communitcation. Neither of these are doing anyting right now as it is the early stags of the project. I setup the structure and visual elements with no event handling for now. My main focus is the QOpenGLWidget as this is the view of one of many future cameras. I put in a frame counter in the paintGL () that once it hits 30 frames, give me a duration. Every 30 frames takes roughly 21 seconds. This is very unacceptable.

As background, I have setup the default format as such in main...

QSurfaceFormat format;
    format.setProfile ( QSurfaceFormat::CoreProfile );
    format.setRenderableType ( QSurfaceFormat::OpenGLES );
    format.setSwapBehavior ( QSurfaceFormat::TripleBuffer );
    format.setSwapInterval ( 0 );  // 0 = draw to vsync
    format.setVersion ( 3, 0 );  // imx6 vivante revision 5.0.11
    // must be called before the widget
    // or its parent window gets shown
    QSurfaceFormat::setDefaultFormat ( format );

QSurfaceFormat format;
    format.setProfile ( QSurfaceFormat::CoreProfile );
    format.setRenderableType ( QSurfaceFormat::OpenGLES );
    format.setSwapBehavior ( QSurfaceFormat::TripleBuffer );
    format.setSwapInterval ( 0 );  // 0 = draw to vsync
    format.setVersion ( 3, 0 );  // imx6 vivante revision 5.0.11
    // must be called before the widget
    // or its parent window gets shown
    QSurfaceFormat::setDefaultFormat ( format );

To copy to clipboard, switch view to plain text mode 

I then setup my subclassed mainwindow through my own initialization function to propagate the items before show(). At the end of this, all items at created and initialized. They have their geomotry, everything is connected and we're ready to to be shown. show() then starts the visual loops and off we go. In the QOpenGLWidget, as standard, I follow the standard three overrides. Here is the top of my class in question...

class AVC8000nano : public QOpenGLWidget, protected QOpenGLFunctions
{
    Q_OBJECT
 
    public:
        explicit AVC8000nano ( QWidget *parent = 0 );
        ~AVC8000nano ( void );
 
        APP_STATUS Initialize ( WhichSetup* structWhichSetup );
        void DoWork ( WORKTODO myTask );
        bool SetCameraToViewPort ( const WHICH_CAMERA eCamera );
 
    protected:
        void initializeGL () Q_DECL_OVERRIDE;
        void paintGL () Q_DECL_OVERRIDE;
        void resizeGL ( int width, int height );
        QOpenGLExtraFunctions* m_OGLES3Functions;
        ...

class AVC8000nano : public QOpenGLWidget, protected QOpenGLFunctions
{
    Q_OBJECT

    public:
        explicit AVC8000nano ( QWidget *parent = 0 );
        ~AVC8000nano ( void );

        APP_STATUS Initialize ( WhichSetup* structWhichSetup );
        void DoWork ( WORKTODO myTask );
        bool SetCameraToViewPort ( const WHICH_CAMERA eCamera );

    protected:
        void initializeGL () Q_DECL_OVERRIDE;
        void paintGL () Q_DECL_OVERRIDE;
        void resizeGL ( int width, int height );
        QOpenGLExtraFunctions* m_OGLES3Functions;
        ...

To copy to clipboard, switch view to plain text mode 

Inside of paintGL(), I do very little...

void AVC8000nano::paintGL ()
{
    if ( ( m_bAreWeExitting ) ||
         ( !m_bInitialized ) )
    {
        return;
    }
 
    QOpenGLContext *oglContext = context ();
    if ( !oglContext ) // QOpenGLWidget not yet initialized
    {
        return;
    }
 
    //GLenum enumError;
 
    m_qimage2Paint = m_helperAVC8KV4L2->GrabImage ();
    if ( m_qimage2Paint != NULL )
    {
        QPainter qPainter ( this );
        qPainter.translate ( m_nDeltaX / 2, m_nDeltaY / 2 );
        qPainter.drawImage ( m_rectImage, *m_qimage2Paint );
 
//        if ( ( enumError = m_OGLES3Functions->glGetError () ) != GL_NO_ERROR )
//        {
//            qDebug ( "OpenGL error: %d.", enumError );
//        }
    }
 
 
#ifdef DEBUG_LIVE_TIMING
    m_nFrameGrabbed++;
    if ( m_nFrameGrabbed == 30 )
    {
        gettimeofday ( &m_tv_now, NULL );
        m_tk_Diff = TV_SECONDS ( m_tv_now, m_tv_begin );
        qDebug ( "30 frames took %lld seconds", m_tk_Diff, m_nFrameGrabbed );
        m_nFrameGrabbed = 0;
        gettimeofday ( &m_tv_begin, NULL );
    }
#endif
 
    delete m_qimage2Paint;
 
    // Schedule composition. Note that this will use QueuedConnection, meaning
    // that update() will be invoked on the gui thread.
    QMetaObject::invokeMethod ( this, "update");
    //update ( m_rectWidget );
}

void AVC8000nano::paintGL ()
{
    if ( ( m_bAreWeExitting ) ||
         ( !m_bInitialized ) )
    {
        return;
    }

    QOpenGLContext *oglContext = context ();
    if ( !oglContext ) // QOpenGLWidget not yet initialized
    {
        return;
    }

    //GLenum enumError;

    m_qimage2Paint = m_helperAVC8KV4L2->GrabImage ();
    if ( m_qimage2Paint != NULL )
    {
        QPainter qPainter ( this );
        qPainter.translate ( m_nDeltaX / 2, m_nDeltaY / 2 );
        qPainter.drawImage ( m_rectImage, *m_qimage2Paint );

//        if ( ( enumError = m_OGLES3Functions->glGetError () ) != GL_NO_ERROR )
//        {
//            qDebug ( "OpenGL error: %d.", enumError );
//        }
    }


#ifdef DEBUG_LIVE_TIMING
    m_nFrameGrabbed++;
    if ( m_nFrameGrabbed == 30 )
    {
        gettimeofday ( &m_tv_now, NULL );
        m_tk_Diff = TV_SECONDS ( m_tv_now, m_tv_begin );
        qDebug ( "30 frames took %lld seconds", m_tk_Diff, m_nFrameGrabbed );
        m_nFrameGrabbed = 0;
        gettimeofday ( &m_tv_begin, NULL );
    }
#endif

    delete m_qimage2Paint;

    // Schedule composition. Note that this will use QueuedConnection, meaning
    // that update() will be invoked on the gui thread.
    QMetaObject::invokeMethod ( this, "update");
    //update ( m_rectWidget );
}

To copy to clipboard, switch view to plain text mode 

I have placed inside of my /etc/appcontroller.conf file as well as my /etc/profile (appropriately edited) these lines...

env=QT_QPA_PLATFORM=eglfs
env=QT_QPA_PLATFORM_PLUGIN_PATH=/usr/lib/plugins/platforms
env=QT_QPA_EGLFS_WIDTH=1920
env=QT_QPA_EGLFS_HEIGHT=1080
env=QT_QPA_EGLFS_PHYSICAL_WIDTH=1920
env=QT_QPA_EGLFS_PHYSICAL_HEIGHT=1080
env=QT_QPA_EGLFS_SWAPINTERVAL=0

env=QT_QPA_PLATFORM=eglfs
env=QT_QPA_PLATFORM_PLUGIN_PATH=/usr/lib/plugins/platforms
env=QT_QPA_EGLFS_WIDTH=1920
env=QT_QPA_EGLFS_HEIGHT=1080
env=QT_QPA_EGLFS_PHYSICAL_WIDTH=1920
env=QT_QPA_EGLFS_PHYSICAL_HEIGHT=1080
env=QT_QPA_EGLFS_SWAPINTERVAL=0

To copy to clipboard, switch view to plain text mode 

My main point here is the SWAPINTERVAL matching the applications format.setSwapInterval above. I have tested setting these values to 34, 17, 1 and 0. None of these effect the 30 frames in 21 seconds measurement.

For uboot, I set the video args as such for the kernel. Changing the @60 to @30 again makes no diff.

vidargs=acpi=force mxc_hdmi.only_cea=1 video=mxcfb0:dev=hdmi,1920x1080M@60,if=RGB24 video=mxcfb1ff video=mxcfb2ff video=mxcfb3ff fbmem=32M

So here I am now not sure where to look. Does anyone have any ideas as to where to look next? Thank you for your time.

Cheers,
Pete

[wysota]

What is the point of using OpenGL if the only thing you do is display a 2d image?

How much time does the following call take?

m_qimage2Paint = m_helperAVC8KV4L2->GrabImage ();

m_qimage2Paint = m_helperAVC8KV4L2->GrabImage ();

To copy to clipboard, switch view to plain text mode 

[PeterVE]

Thank you wysota for the reply. The project is early on and it has a requirement to draw text on top of the image as well as image processing algorithms. Also, the target will be handling between 4-12 media streams at the same time and constant and forever from startup to shutdown. This has to be optimized for performance. Even now, I don't feel comfortable using QImage and QPaint. Anyways, your second question is exactly where I went right after posting this. It is in fact that the device is not providing an image as the product advertisements claim. I put the select() call on a loop to examine the issue and it is timing out with a 1s timeout. I am reviewing my port of their example as I type this and so far, I did find one missed line which is the setting of the resolution. I have also contacted the manufacturer's tech support to discuss possible things I have missed or is my expectations wrong. I also found that the adverts state they support RGB565 which is the closest to the default RGB8888 (RGB24) which is the internal of Qt as well as OpenGL. I found a readme in their driver that stated they only support YUYV. So I'm losing confidence in this device right now. I have done format conversions in vertex and fragment shaders as well as added an alpha channel. I'm not lost, but just debating on hardware at this point.

No matter if I messed up or the product is sub-par, this thread is dead. Thank you again for the reply.

[wysota]

Thank you wysota for the reply. The project is early on and it has a requirement to draw text on top of the image as well as image processing algorithms. Also, the target will be handling between 4-12 media streams at the same time and constant and forever from startup to shutdown. This has to be optimized for performance.

That's not a reason to use OpenGL.

Even now, I don't feel comfortable using QImage and QPaint. Anyways, your second question is exactly where I went right after posting this. It is in fact that the device is not providing an image as the product advertisements claim. I put the select() call on a loop to examine the issue and it is timing out with a 1s timeout.

Grabbing frames and converting them to images is usually a very lengthy process. It's much quicker to make the device render content directly where you want it.

[PeterVE]

To be clear, I am using QOpenGL and not currently native OpenGL. The reasoning is because there are proven paths on the imx6 that a gpu loaded image can in fact have zero copies to the screen. They used QOpenGL to do so. Granted, I am not opengl native yet and am working out all of the easier paths right now, but it seems logical to follow what others have done. It is Qt's own website that pushed this info out to us. Also, shaders are (until we end this thread and am convinced otherwise) required. Not sure how you can write shaders and not be OpenGL within Qt. At least with the easiest path of hooks. Image processing as stated above means more than just displaying it to a screen. I am using 2D images to move robotics in 3d space and some of it automated for the user as well as algorithms to perform automated tedious tasks such as counting thousands of objects and sizes and a lot more. All of which I have done on DirectX inside of shaders and not on the CPU because of the floating point abilities. The GPU is more than just for rendering images, but for image processing as well. As a newb to this Qt and linux GPU programming, I am fully open to design options that will work with Qt as the GUI environment though.

As a newb to Qt, I haven't seen and am not knowledgeable on best practices. I am assuming you are referring to QPaintDevice from the Widget itself to render directly to where I want it (?). I cannot see the path as to take a char buff[] directly to QPaintDevice as I believe you may be saying. I would greatly appreciate advise to a better performance path. Thank you.

[wysota]

To be clear, I am using QOpenGL and not currently native OpenGL.

There is no "native" and "non-native" OpenGL. QOpenGL calls such as your drawImage call are translated into regular OpenGL calls.

The reasoning is because there are proven paths on the imx6 that a gpu loaded image can in fact have zero copies to the screen.

Not if you draw QImage which is a purely software entity. Then you have at least two copy operations - first to obtain the data for QImage on the CPU and then to make a texture usable for a GPU out of it.

Not sure how you can write shaders and not be OpenGL within Qt.

OpenGL is an API. You can use shaders to render to an offscreen surface (i.e. an FBO) which can then be converted to a QImage and used without OpenGL. So having an OpenGL based widget is not a requirement. But that's irrelevant here. What's relevant is you are not leveraging any of OpenGL capabilities in the code you posted in this thread. A lot of conversions are made and that is why you have such poor performance.

Image processing as stated above means more than just displaying it to a screen. I am using 2D images to move robotics in 3d space and some of it automated for the user as well as algorithms to perform automated tedious tasks such as counting thousands of objects and sizes and a lot more. All of which I have done on DirectX inside of shaders and not on the CPU because of the floating point abilities. The GPU is more than just for rendering images, but for image processing as well. As a newb to this Qt and linux GPU programming, I am fully open to design options that will work with Qt as the GUI environment though.

This does not require you to use OpenGL widget for rendering the result to screen. You can of course have an OpenGL widget to render a texture you had earlier rendered to using shader computation but certainly not the way you have used in the code above.

I have successfully used QtQuick with I.MX6, so assuming you can implement a QtQuick item that takes an existing GPU buffer and treats it as a texture for the item, that would give the best performance.

[PeterVE]

The qopengl i did is saved in another folder. This thread is not the full project. Yup, this post doesn't show the usage of opengl but on purpose. The full blown usage is not a part of the test/post. Simple tests show building blocks work. This flushed out that the camera hardware/driver is sub-par and won't work for us. Anyways, QtQuick seems to be your suggestion and I will look into it. Thanks for your time.

[wysota]

This flushed out that the camera hardware/driver is sub-par and won't work for us.

How did you test that? Because the code posted in this thread is not an adequate test.

Anyways, QtQuick seems to be your suggestion and I will look into it.

I said it might work for you, if you put some effort of building efficient custom items. I'm not suggesting you should use it, because I don't know the whole picture of what you are doing.

[alexpaul]

That's not a reason to use OpenGL.