onnxruntime-inference-examples use qwen error

[godguy23]

I use the demo which is onnx-runtime/onnxruntime-inference-examples/mobile/examples/phi-3/android , but I use the qwen model which is https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct/tree/main to instead of the phi-3. Then I push the model_uint8.onnx file to the android phone. When I run the process，it occur some error below:

01-10 18:05:47.205 1957 2157 D AwareLog: iawared: WorkingsetPauseCollect ai.onnxruntime.genai.demo/.MainActivity
01-10 18:05:48.120 24819 24961 W System.err: ai.onnxruntime.genai.GenAIException: Failed to create Tokenizer.
01-10 18:05:48.120 24819 24961 W System.err: at ai.onnxruntime.genai.SimpleGenAI.generate(SimpleGenAI.java:103)
01-10 18:05:48.120 24819 24961 W System.err: at ai.onnxruntime.genai.demo.MainActivity$2$1.run(MainActivity.java:229)
01-10 18:05:48.120 24819 24961 W System.err: at java.lang.Thread.run(Thread.java:933)
01-10 18:05:48.120 24819 24961 W System.err: Caused by: ai.onnxruntime.genai.GenAIException: Token generation loop failed.
01-10 18:05:48.120 24819 24961 W System.err: at ai.onnxruntime.genai.SimpleGenAI.generate(SimpleGenAI.java:97)
01-10 18:05:48.120 24819 24961 W System.err: ... 2 more
01-10 18:05:48.120 24819 24961 W System.err: Caused by: ai.onnxruntime.genai.GenAIException: Got invalid dimensions for input: past_key_values.0.key for the following indices
01-10 18:05:48.120 24819 24961 W System.err: index: 3 Got: 14 Expected: 64
01-10 18:05:48.120 24819 24961 W System.err: Please fix either the inputs/outputs or the model.
01-10 18:05:48.120 24819 24961 W System.err: at ai.onnxruntime.genai.Generator.appendTokenSequences(Native Method)
01-10 18:05:48.120 24819 24961 W System.err: at ai.onnxruntime.genai.Generator.appendTokenSequences(Generator.java:93)
01-10 18:05:48.120 24819 24961 W System.err: at ai.onnxruntime.genai.SimpleGenAI.generate(SimpleGenAI.java:91)
01-10 18:05:48.120 24819 24961 W System.err: ... 2 more
01-10 18:05:48.120 24819 24961 D onnxActivity: GenAIException..

Here is the code :
`try {
Log.d(TAG, "start generator result");
SimpleGenAI generator = new SimpleGenAI(getExternalFilesDir(null).getPath());

                        GeneratorParams params = generator.createGeneratorParams();
                        params.setSearchOption("max_length", 20);
                        ByteBuffer data = ByteBuffer.allocateDirect(4 * Float.BYTES).order(ByteOrder.nativeOrder());
                        FloatBuffer floatBuffer = data.asFloatBuffer();
                        floatBuffer.put(new float[] {1.0f, 2.0f, 3.0f, 4.0f});
                        Tensor tensor = new Tensor(data, new long[]{64, 64}, Tensor.ElementType.uint8);
                        params.setInput(promptQuestion, tensor);
                        String result =
                                generator.generate(params, promptQuestion, null);
                        Log.d(TAG, "Result: " + result);
                        runOnUiThread(() -> {
                            generatedTV.setText(result);
                            sendMsgIB.setEnabled(true);
                            sendMsgIB.setAlpha(1.0f);
                        });

                    } catch (GenAIException e) {
                        e.printStackTrace();
                        Log.d(TAG, "GenAIException..");
                    }`

I don't know how to fix this error. Could anyone help me to fix this error? Thanks a lot.

[edgchen1]

onnxruntime-genai takes files other than the .onnx file as input. For example, one can provide this directory as input to onnxruntime-genai for phi3:
https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx/tree/main/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4

Simply replacing the .onnx model with another one is not likely to work unless the models are compatible.

01-10 18:05:48.120 24819 24961 W System.err: Caused by: ai.onnxruntime.genai.GenAIException: Got invalid dimensions for input: past_key_values.0.key for the following indices
01-10 18:05:48.120 24819 24961 W System.err: index: 3 Got: 14 Expected: 64
01-10 18:05:48.120 24819 24961 W System.err: Please fix either the inputs/outputs or the model.

It seems like there is at least some incompatibility with this model input.

If you want to run Qwen, you can try getting a version of it that can be used with onnxruntime-genai. There's a helper script here that might be useful:
https://github.com/microsoft/onnxruntime-genai/blob/main/src/python/py/models/builder.py

The readme says that it supports Qwen. If you have questions about it, please start a discussion or file an issue in the onnxruntime-genai repo.