[source level parsing] read the source code, analyze and solve the problem that scrcpy cannot input Chinese normally

Time:2020-11-13

In the era of mobile Internet, mobile phones can do more and more things, but if you want to make work more efficient, mouse and keyboard are still essential. However, many software (named alisystem) do not provide the corresponding desktop version, and are not compatible with Android simulator based on x86 architecture, which makes us use projection software to operate mobile phones on computers. Scrcpy is one of the most distinctive projection software. As an open source software, it has excellent performance and rich functions, but this software has some problems in Chinese input. This article will introduce how to make scrcpy input Chinese normally, so that this very easy to use projection software becomes better.
This article was originally published in the unnamed station, from the author’s own synchronization to Zhihu, reprint please indicate the original author’s blog address or this link, thank you!

At the time of writing this article, the latest version of scrcpy is 1.14. There are still some problems described below. When you read this article, maybe scrcpy has solved this problem. Therefore, the content of this paper is only for reference and technical sharing.

Problem recurrence

Scrcpy only relies onadb shell screencapandadb shell inputThe software for equipment control has better performance, which benefits from its system architecture

[source level parsing] read the source code, analyze and solve the problem that scrcpy cannot input Chinese normally

The server runs on Android side every time scrcpy is started. The collected images are encoded by mediacodec API and transmitted to PC through socket using multithreading. The PC uses ffmpeg and sdl2 to decode and display the picture in real time. The server is developed by Java, and the client is developed by C. The specific technical details can refer to the official documents and will not be repeated here.

To get to the point, scrcpy has always had a huge problem in Unicode text input. Since a long time ago, users reported that they could not input characters other than ASCII (for example,, the mobile phone does not display after inputting text on the PC, and the terminal reports an error, as shown in Figure 1). However, the author has recently officially added support for ASCII characters, but it has not been incorporated into the master branch (see ා 1426).

[source level parsing] read the source code, analyze and solve the problem that scrcpy cannot input Chinese normally

Figure 1. Unable to input Chinese characters normally and prompt that characters cannot be inserted

The author tries to pull the code base and build the branch d613b10efcdf0d1cf76e30871e136ba0ff444e6e according to the development document.

After construction, we will find that the problem is slightly improved, but the letters in the input process are also passed into scrcpy, as shown in Figure 2

[source level parsing] read the source code, analyze and solve the problem that scrcpy cannot input Chinese normally

Figure 2. I want to input “test”, but in the process of inputceshi It’s also entered into Android

Because it is not yet released features, there is no relevant information for reference, we can only read the source code to find the reason.

0x02 problem analysis

First of all, let’s take a look at the SDL process for handling user input (and other events)

  1. useSDL_WaitEvent(&event)Gets the events in the event queue
  2. according toevent->typeDistinguish different events

The event type here can refer to the official document of sdl2: SDL_ Event。

Following this idea, we can quickly find the relevant code for scrcpy to handle events: APP / SRC / scrcpy. C

static enum event_result
handle_event(SDL_Event *event, bool control) {
    switch (event->type) {
        case EVENT_STREAM_STOPPED:
            LOGD("Video stream stopped");
            return EVENT_RESULT_STOPPED_BY_EOS;
        case SDL_QUIT:
            LOGD("User requested to quit");
            return EVENT_RESULT_STOPPED_BY_USER;
        case EVENT_NEW_FRAME:
            if (!screen.has_frame) {
                screen.has_frame = true;
                // this is the very first frame, show the window
                screen_show_window(&screen);
            }
            if (!screen_update_frame(&screen, &video_buffer)) {
                return EVENT_RESULT_CONTINUE;
            }
            break;
        case SDL_WINDOWEVENT:
            screen_handle_window_event(&screen, &event->window);
            break;
        case SDL_TEXTINPUT:
            if (!control) {
                break;
            }
            input_manager_process_text_input(&input_manager, &event->text);
            break;
        case SDL_KEYDOWN:
        case SDL_KEYUP:
            // some key events do not interact with the device, so process the
            // event even if control is disabled
            input_manager_process_key(&input_manager, &event->key, control);
            break;
...

Combined with code and SDL documentation, you can findSDL_KEYDOWNandSDK_KEYUPEvents are passed toinput_manager_process_key()Function, these two events are only for keystrokes, not for input methods; andSDL_TEXTINPUTEvents are passed to theinput_manager_process_text_input()Function, this event handles the text sent after the input method confirms the input.

Here we can use the debugging tool (or universal printf) to understand the process of event transmission in the input process of “test” above. The sequence of events is as follows:

2020-06-17 17:16:49.831 scrcpy[57759:5396589] INFO: KEYINPUT # c
2020-06-17 17:16:49.919 scrcpy [57759:5396589] Info: keyinput ා C lift
2020-06-17 17:16:50.731 scrcpy[57759:5396589] INFO: KEYINPUT # e
2020-06-17 17:16:50.840 scrcpy [57759:5396589] Info: keyinput ා e lift
2020-06-17 17:16:51.341 scrcpy[57759:5396589] INFO: KEYINPUT # s
2020-06-17 17:16:51.440 scrcpy [57759:5396589] Info: keyinput ා s lift
2020-06-17 17:16:51.657 scrcpy[57759:5396589] INFO: KEYINPUT # h
2020-06-17 17:16:51.719 scrcpy [57759:5396589] Info: keyinput ා h lift
2020-06-17 17:16:51.933 scrcpy[57759:5396589] INFO: KEYINPUT # i
2020-06-17 17:16:52.041 scrcpy [57759:5396589] Info: keyinput ා I lift
2020-06-17 17:16:52.408 scrcpy [57759:5396589] Info: keyinput ා space
2020-06-17 17:16:52.408 scrcpy [57759:5396589] Info: textinput ා test
2020-06-17 17:16:52.519 scrcpy [57759:5396589] Info: keyinput ා space raised

As you can see, except for the last lineTEXTINPUTEvents, others areKEYDOWNorKEYUPevent. So the question is, how can we block out these unnecessary events?

Read oninput_manager_process_key()Function suminput_manager_process_text_input()Function, we will find that they all judge “a certain special case” and discard some inputs under special circumstances

// input_manager_process_key()
...
struct control_msg msg;
// convert_ input_ Only when key() returns true can the character be inserted. But why does it return false?
if (convert_input_key(event, &msg, im->prefer_text)) {
    if (!controller_push_msg(controller, &msg)) {
        LOGW("Could not request 'inject keycode'");
    }
}
...

// input_manager_process_text_input()
...
void
input_manager_process_text_input(struct input_manager *im,
                                 const SDL_TextInputEvent *event) {
    //Why do you want to choose_ How about early return when text is false?
    if (!im->prefer_text) {
        char c = event->text[0];
        if (isalpha(c) || c == ' ') {
            assert(event->text[1] == '
// input_manager_process_key()
...
struct control_msg msg;
// convert_ input_ Only when key() returns true can the character be inserted. But why does it return false?
if (convert_input_key(event, &msg, im->prefer_text)) {
if (!controller_push_msg(controller, &msg)) {
LOGW("Could not request 'inject keycode'");
}
}
...
// input_manager_process_text_input()
...
void
input_manager_process_text_input(struct input_manager *im,
const SDL_TextInputEvent *event) {
//Why do you want to choose_ How about early return when text is false?
if (!im->prefer_text) {
char c = event->text[0];
if (isalpha(c) || c == ' ') {
assert(event->text[1] == '\0');
// letters and space are handled as raw key event
return;
}
}
...
'); // letters and space are handled as raw key event return; } } ...

yes! Both of these suspicious places are related toprefer_textIf this parameter is true or false, it will control the behavior of scrcpy to process key input and text input in a diametrically opposite way

  1. ifprefer_textFalse:

    • TEXTINPUTIf the character length obtained by the event is 1, the event is discarded directly
    • KEYDOWNorKEYUPNot affected
  2. ifprefer_textTrue:

    • TEXTINPUTEvents are not affected
    • KEYDOWNorKEYUPIf the character corresponding to the event is a letter or a space, the event is discarded directly

It should be noted that according to the official SDL documents and our actual tests, when inputting English (without input method), pressing a key will trigger three events:

  1. KEYDOWNevent
  2. TEXTINPUTevent
  3. KEYUPevent

That is to say, if notprefer_textParameter control, press a key, we will get two keys. Keep flipping through the code and we’ll find outprefer_textThe default value of the parameter is false, which satisfies the first case above.

Combined with the test of Chinese input above, we can find thatKEYDOWNandKEYUPEvent inprefer_textIf the default value is false, it will be directly input to the Android side; if we want to block these two events, we must ensure thatprefer_textIt’s true.

What’s more, I believe readers have found some clues here?

0x03 problem solving

Continue to read the source code, you will findprefer_textThe parameter comes from the--prefer-textOption, which is described in the document as follows:

Text injection preference
There are two kinds of events generated when typing text:

key events, signaling that a key is pressed or released;
text events, signaling that a text has been entered.
By default, letters are injected using key events, so that the keyboard behaves as expected in games (typically for WASD keys).

But this may cause issues. If you encounter such a problem, you can avoid it by:

scrcpy --prefer-text

(but this will break keyboard behavior in games)

The explanation of this parameter is very abstract. It only describes the behavior of this parameter and the consequences under special circumstances, and does not introduce the general purpose of this parameter. It is no wonder that the author did not find out at the beginning.

Here we use the branch d613b10efcdf0d1cf76e30871e136ba0ff444e6e mentioned above to rebuild and carry it at startup--prefer-textParameters:

[source level parsing] read the source code, analyze and solve the problem that scrcpy cannot input Chinese normally

At this time, the problem no longer appears!

0x04 follow up

In view of the fact that the documents of scrcpy are too brief and not easy to be noticed, the author submitted an issue () to the author to explain the situation, hoping to attract the author’s attention and improve the relevant documents (or consider enabling this option by default?). If the author intends, I also hope to be able to write Chinese documents for it, so that other users with the same needs can get started quickly. After all, in the domestic Internet environment, too many software only have mobile versions, and there are countless Chinese users who have this demand. However, few people can read the source code and read this article by chance.

Perhaps when readers read this article, the software version has been higher than 1.14, and the related problems have been solved. However, the author still hopes to provide readers with a solution to the problem through this article

Read more source code!