Passing some Unicode characters to Linux Qt applications

[PCXT]

Hello

My configuration:
- Debian 10 Buster, Qt 5.11.3 (from distribution packages), X11-based desktop.

I have a problem with Qt (probably with Qt) and I just don't know where to start looking for its culprit. First of all, I decided to make a small program in which I need to insert characters to other applications. Generally, the window is like on-screen keyboard and pressing a button will cause character being sent to currently open application.
The code I have for this action is a known approach, XLib and Xtest to feed re-defined keycodes:

void MainWindow::push_char(char character)
{
    Display * disp;              //current X11 display
    disp=XOpenDisplay(nullptr);  //TODO: fail gracefully when abused in console.
    KeySym * xkeysyms = nullptr;
    int temporaryKeyCode=0;
    int keycodes_start=0;
    int keycodes_end=0;
    int keysyms_per_keycode=0;
    XDisplayKeycodes(disp,&keycodes_start,&keycodes_end); //obtain keycode range for current display.
    xkeysyms=XGetKeyboardMapping(disp,keycodes_start,keycodes_end-keycodes_start,&keysyms_per_keycode); //get keysyms start-end
 
    for (int i=keycodes_start; i<=keycodes_end; i++) //Iterate through keycode list to obtain free one
    {
        bool empty_keycode=true;
        for (int j=0;  j<keysyms_per_keycode; j++) //check all keysyms to be zero
        {
             int keysym_index = (i - keycodes_start) * keysyms_per_keycode + j; //obtain keysym position
             if(xkeysyms[keysym_index] != 0) //if the key is not empty...
             {
                empty_keycode=false;            //nope... next one
                break;
             }
        }
        if (empty_keycode)              //use this code for our character.
        {
            temporaryKeyCode=i;
            break;
        }
    }                    
    XFree(xkeysyms);   //this finding and freeing must be done every time, not at the program boot-up,
    XFlush(disp);         //as user may change X settings on the fly, including key tables.
    usleep(100*1000);   //0.1 delay
 
    //Prepare for character send
    KeySym sym = XStringToKeysym("U01F92A"); //Uxxxx links unknown UTF char to key even when it's not in any definition. As in xkbcommon-keysyms.h
  //  KeySym sym = XStringToKeysym("U0142"); //THIS WORKS., the above does not in all applications.
 
    if (sym == NoSymbol) //This should rarely happen.
    {
        sym=XStringToKeysym("x"); //have anything here or throw error.
    }
 
    KeySym symlist[2]={sym, sym};                     //create a keysym list for shifted and unshifted chr
    XChangeKeyboardMapping(disp,temporaryKeyCode,2,symlist,1);  //assign the list to the mapping
    XFlush(disp);
    KeyCode code=temporaryKeyCode;                              //create code out of temporary
 
    XTestFakeKeyEvent(disp, code, true, 0); //Press...
    XFlush(disp);
    usleep(90 * 1000);
    XTestFakeKeyEvent(disp, code, false, 0); //Release...
    XFlush(disp);
 
    symlist[0]=0;
    symlist[1]=0;
    XChangeKeyboardMapping(disp,temporaryKeyCode,2,symlist,1);  //revert the mapping
    usleep(90 * 1000);
 
    XCloseDisplay(disp);    //End routine
    return;
}

void MainWindow::push_char(char character)
{
    Display * disp;              //current X11 display
    disp=XOpenDisplay(nullptr);  //TODO: fail gracefully when abused in console.
    KeySym * xkeysyms = nullptr;
    int temporaryKeyCode=0;
    int keycodes_start=0;
    int keycodes_end=0;
    int keysyms_per_keycode=0;
    XDisplayKeycodes(disp,&keycodes_start,&keycodes_end); //obtain keycode range for current display.
    xkeysyms=XGetKeyboardMapping(disp,keycodes_start,keycodes_end-keycodes_start,&keysyms_per_keycode); //get keysyms start-end

    for (int i=keycodes_start; i<=keycodes_end; i++) //Iterate through keycode list to obtain free one
    {
        bool empty_keycode=true;
        for (int j=0;  j<keysyms_per_keycode; j++) //check all keysyms to be zero
        {
             int keysym_index = (i - keycodes_start) * keysyms_per_keycode + j; //obtain keysym position
             if(xkeysyms[keysym_index] != 0) //if the key is not empty...
             {
                empty_keycode=false;            //nope... next one
                break;
             }
        }
        if (empty_keycode)              //use this code for our character.
        {
            temporaryKeyCode=i;
            break;
        }
    }                    
    XFree(xkeysyms);   //this finding and freeing must be done every time, not at the program boot-up,
    XFlush(disp);         //as user may change X settings on the fly, including key tables.
    usleep(100*1000);   //0.1 delay

    //Prepare for character send
    KeySym sym = XStringToKeysym("U01F92A"); //Uxxxx links unknown UTF char to key even when it's not in any definition. As in xkbcommon-keysyms.h
  //  KeySym sym = XStringToKeysym("U0142"); //THIS WORKS., the above does not in all applications.

    if (sym == NoSymbol) //This should rarely happen.
    {
        sym=XStringToKeysym("x"); //have anything here or throw error.
    }

    KeySym symlist[2]={sym, sym};                     //create a keysym list for shifted and unshifted chr
    XChangeKeyboardMapping(disp,temporaryKeyCode,2,symlist,1);  //assign the list to the mapping
    XFlush(disp);
    KeyCode code=temporaryKeyCode;                              //create code out of temporary

    XTestFakeKeyEvent(disp, code, true, 0); //Press...
    XFlush(disp);
    usleep(90 * 1000);
    XTestFakeKeyEvent(disp, code, false, 0); //Release...
    XFlush(disp);

    symlist[0]=0;
    symlist[1]=0;
    XChangeKeyboardMapping(disp,temporaryKeyCode,2,symlist,1);  //revert the mapping
    usleep(90 * 1000);

    XCloseDisplay(disp);    //End routine
    return;
}

To copy to clipboard, switch view to plain text mode 

In the code, I have two test lines for making a keysym of UTF character: First one makes it from "U0142", second is "U01F92A" (or U1F92A - an emoji which, at least in my system, shows a smiley).
The routine (currently argument is not processed) works, but not for all characters and not for all applications. Things I observed:
- When characters are like Uxxxx, 2 hex bytes after U, it's OK for all applications I tested: Based on GTK, Qt, Qt Creator itself too, WxWidgets and even an old ?Qt3-based? TDE.
- When characters are longer, like Uxxxxxx - 3 hex bytes after U, it works for GTK, WxWidgets, TDE, but not Qt applications, even not in Qt Creator itself.

This is a strange behavior and for me it looks like there may be some problem with library configuration in my set-up?
When instead of my code I use xdotool (xdotool type "?") I have a similar behavior, that's why I think about misconfiguration.

Thanks in advance.
MW

[d_stranz]

In the code you posted, I don't see any references to Qt at all, other than that this method is in a MainWindow class. Where is the Qt connection?

[PCXT]

There are two things:
1. I'm doing it from Qt application, but it has smaller meaning.
2. It cannot insert character to any Qt5 application - even to itself. That's why I'm thinking about Qt misconfiguration.
The last character I can pass, if U+FFFD, all bigger code points fail (I think the last is U+FFFF, but it's not character by standard).

[d_stranz]

Have your read the Qt documentation concerning Unicode? Your problem may be in how you are handling the characters either as input to QString or as output to a Qt-based widget for display.

There are references to "16-bit Unicode", which could be the problem.

[PCXT]

Thanks for the information about Unicode in Qt! I found that another, not related problem I had was caused by casting the characters wrong and now I can process these characters well inside my application (as QStrings), so I worked it around using X selection buffer (lower characters are typed as above, higher - copied through the buffer, good that I can stuff ctrl-ins there).

The problem seems to be not especially in my application's code (which currently is a single form and a button calling the routine), but it may be in Qt configuration in my system, as any X11 "keyboard-simulating" program doing it the similar way as shown in my first post cannot send higher Unicode characters specifically to Qt5-based applications (?or sends and Qt applications can't process them?), even to programs which are from repository, not compiled by me. That's why I suspect some define has been switched wrong during Qt build (I have Qt from Debian repository, so that's why I'm a bit dubious about its build configuration, I had a similar problem before with Qt4's QSerialPort).

So my short reproduction using Linux, xdotool and two applications: In the terminal:

sleep 1; xdotool type "?"

During one second of delay, I switch to another application.
So, it types the smiley into Wx or GTK applications, but not into Qt5 applications, like Qt Creator.

But when I put a lower UTF character, let's try with the last one (replacement sign, U+FFFD):

sleep 1; xdotool type "?"

It is typed into application in both cases.

EDIT:
P.S.
OK, for some reason the emoji I pasted here got replaced with ?, so it may be here or in the forum. The first ? I put in xdotool was U+1F60A character, the second ? was U+FFFD character.