AoE: A Fast Integrating AI Terminal Running Environment SDK

Time:2019-10-9

Background

1.1 What is AoE

AoE (AI on Edge) is an open source terminal-side AI integrated runtime environment (IRE). Based on the design principle of “stability, ease of use, security”, it helps developers deploy deep learning algorithms of different frameworks to terminals for efficient execution. The Github address is https://github.com/didi/aoe.

There are two reasons why an AI terminal integration runtime framework should be built:

Firstly, with the rapid development of AI technology, many inference frameworks have emerged in the last two years, which not only bring more choices to developers, but also increase the cost of deploying AI to terminals.

Secondly, the process of directly accessing AI through reasoning framework is complicated, which involves dynamic library access, resource loading, pre-processing, post-processing, resource release, model upgrade, and how to ensure stability.

At present, AoE SDK has been applied on OCR of drip bank card. To understand the relationship between AoE and reasoning framework, host App more clearly, we can understand it through the following business integration schematic diagram.

AoE: A Fast Integrating AI Terminal Running Environment SDK

1.2 Terminals Inference Framework

Below are eight mainstream reasoning frameworks (ranked in no particular order) for terminal operation.

AoE: A Fast Integrating AI Terminal Running Environment SDK

1.3 How AoE supports various reasoning frameworks

Essentially, no matter what reasoning framework is, it must contain the following five processes. The abstraction of these reasoning processes is the basis for AoE to support various reasoning frameworks.

At present, AoE implements the support of two reasoning frameworks, NCNN and TensorFlow Lite. Taking these two reasoning frameworks as examples, the forms of five reasoning processes in their respective reasoning frameworks are illustrated.

AoE: A Fast Integrating AI Terminal Running Environment SDK

1.4 Which platforms AoE supports

At present, AoE’s open source runtime environment SDK includes Android and iOS platforms. In addition, the Linux platform runtime environment SDK is under intensive development and is expected to meet with you formally by the end of September.

II. Working Principle

2.1 Processing process of abstract reasoning framework

As mentioned earlier, different reasoning frameworks contain common processes, which are initial enabling, pre-processing, executing reasoning, post-processing and releasing resources. For AoE integrated operating environment, the most basic is abstract reasoning operation. By relying on inverted design, business only relies on the upper abstraction of AoE, and does not care about the access implementation of specific reasoning framework.The greatest advantage of this design is that developers can add new reasoning framework at any time without modifying the framework implementation, so that business development and AoE SDK development are completely decoupled.

In AoE SDK, this abstraction is Interpreter Component (used to process model initialization, executing reasoning and releasing resources) and Convertor (used to process model input pre-processing and model output post-processing). Interpreter Component is implemented as follows:

/**
 * Model Translation Component
 */
interface InterpreterComponent<TInput, TOutput> extends Component {
    /**
     * Initialization, reasoning framework loading model resources
     *
     *@ Param context context, with service binding
     *@ Param Model Options Model Configuration List
     *@ Return Inference Framework Loading
     */
    boolean init(@NonNull Context context, @NonNull List<AoeModelOption> modelOptions);
 
    /**
     * Perform reasoning operations
     *
     *@ Param input service input data
     *@ return service output data
     */
    @Nullable
    TOutput run(@NonNull TInput input);
 
    /**
     * Releasing resources
     */
    void release();
 
    /**
     * Whether the model is loaded correctly or not
     *
     *@ return true, model loaded correctly
     */
    boolean isReady();
}

Convertor is implemented as follows:

interface Convertor<TInput, TOutput, TModelInput, TModelOutput> {
    /**
     * Data preprocessing, converting input data into model input data
     *
     *@ Param input service input data
     *@ return model input data
     */
    @Nullable
    TModelInput preProcess(@NonNull TInput input);
 
    /**
     * Data post-processing, converting model output data into business output data
     *
     *@ Param model Output model output data
     * @return
     */
    @Nullable
    TOutput postProcess(@Nullable TModelOutput modelOutput);
}

2.2 Stability Guarantee

As we all know, an important issue in Android platform development is model adaptation, especially in scenarios involving a large number of Native operations. The problem of model adaptation is particularly important. Once the application crashes on a certain model, the experience damage is enormous. Data show that due to performance problems, mobile App loses 5% of its active users every day. Sixty percent of these lost users choose to remain silent and no longer use the application. Thirty percent of users change to competitors. The remaining users will uninstall the application directly. Therefore, guaranteeing the availability of the main App process at any time is the most basic and important thing for a mobile application with a huge user base. Considering the AI reasoning process, it is inevitable that a large number of operations will occur in the Native process, not only reasoning operations, but also some operations of pre-processing and resource recovery are more prone to compatibility problems.To this end, the AoE runtime environment SDK has developed a mechanism for independent processes on Android platform, allowing Native operations to run in independent processes, while ensuring the stability of reasoning (accidental collapse will not affect subsequent reasoning operations) and the stability of the main process (the main process will not collapse at any time).

There are three main parts in the implementation process: registration independent process, exception rebinding process and cross-process communication optimization.

The first part is to register the independent process and add a RemoteService component in Manifest. The code is as follows:

<application>
    <service
        android:name=".AoeProcessService"
        android:exported="false"
        android:process=":aoeProcessor" />
 
</application>

In the second part, the exception rebinds the independent process. If the RemoteService terminates during reasoning, the “bindService ()” method is executed and the RemoteService is restarted.

@Override
public Object run(@NonNull Object input) {
    if (isServiceRunning()) {
        ... (Code ellipsis)// Perform reasoning
    } else {
        BindService (); // Restart the independent process
    }
    return null;
}

The third part is the optimization of cross-process communication, because independent processes inevitably involve cross-process communication. The biggest problem in cross-process communication is time-consuming loss. Here, two factors cause time-consuming loss:

Transmission time-consuming
Serialization/deserialization time-consuming

Compared with the transmission time using binder mechanism, serialization/deserialization accounts for 90% of the total communication time. Thus, the optimization of serialization/deserialization is the focus of cross-process communication optimization.
Compared with the current mainstream serialization/deserialization tools, the AoE integrated running environment uses kryo library for serialization/deserialization.The following is a comparison of the results. Data refer to oschina’s article “Performance comparison test results of various Java serialization libraries”.

3. Examples of MNIST integration

3.1 Inheritance of TensorFlowLiteInterpreter

When we want to access a new model, we must first determine which reasoning framework the model runs on, and then inherit the implementation of Interpreter Component of the reasoning framework to complete the specific business process. MNIST is a model running on the TF Lite framework, so,We implement the Interpreter abstract class of TF Lite of AoE, convert the input data into the input of the model, and then read the data needed by the business from the output of the model.Initialization, reasoning execution and resource recovery follow the default implementation of TensorFlowLite Interpreter.

public class MnistInterpreter extends TensorFlowLiteInterpreter<float[], Integer, float[], float[][]> {
 
    @Nullable
    @Override
    public float[] preProcess(@NonNull float[] input) {
        return input;
    }
 
    @Nullable
    @Override
    public Integer postProcess(@Nullable float[][] modelOutput) {
        if (modelOutput != null && modelOutput.length == 1) {
            for (int i = 0; i < modelOutput[0].length; i++) {
                if (Float.compare(modelOutput[0][i], 1f) == 0) {
                    return i;
                }
            }
        }
        return null;
    }
}

3.2 Runtime Environment Configuration

The second step to access MNIST is to configure the reasoning framework type and model related parameters. The code is as follows:

mClient = new AoeClient(requireContext(), "mnist",
        new AoeClient.Options()
                .setInterpreter(MnistInterpreter.class)/*
                .useRemoteService(false)*/,
        "mnist");

3.3 Inference Execution

The following are the implementation of MINST’s initial enabling reasoning framework, reasoning execution and resource recovery:

// Initial Reasoning Framework
int resultCode = mClient.init();
// reasoning execution
Object result = mClient.process(mSketchModel.getPixelData());
if (result instanceof Integer) {
    int num = (int) result;
    Log.d(TAG, "num: " + num);
    mResultTextView.setText((num == -1) ? "Not recognized." : String.valueOf(num));
}
// Resource recovery
if (mClient != null) {
    mClient.release();
}

IV. Join us

Helping AI land on the terminal, the open source AoE integrated operating environment is the first step we have taken! In the future, it is our unremitting goal to provide more reasoning framework support and more valuable features for terminal developers. If you are interested in this project, if you have any idea about the running environment of terminal AI, and if you have any questions when using it, you are cordially invited to join us.

GitHub address:

AoE: A Fast Integrating AI Terminal Running Environment SDK
Every Star of yours is our greatest affirmation:)

QQ Exchange Group:

AoE: A Fast Integrating AI Terminal Running Environment SDK
Every question you ask will help us become better selves:)
QQ group number: 815254379

At the same time, you are also welcome to pay attention to the drip technology public number, we will release the latest open source information and technology dry goods in time!

Drip Technology Public No.

AoE: A Fast Integrating AI Terminal Running Environment SDK