Query regarding support of Executorch for ARM Ethos-U65 backend

[vikasbalaga]

Hi,

I have started working with Executorch and in this section of launching Executorch on ARM Ethos-U, I have observed that only Ethos-U55 and Ethos-U85 are mentioned.
Also in the model conversion script and setup utilities, the supported targets only contain Ethos-U55 and Ethos-U85 variants

But I am trying to work with a Ethos-U65 based system, so does that mean Executorch only supports the above mentioned variants or does it also support Ethos-U65?

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

[kimishpatel]

@zingo @Erik-Lundell @digantdesai can any of you answer?

[freddan80]

Hi @vikasbalaga , thx for your interest in Executorch and Ethos-U 🥇 Ethos-U65 is supported with Executorch as well, but we haven't given it too much love, yet. A couple of reasons; 1) there's no FVP to test it on 2) the AOT flow is very similar to Ethos-U55. Ethos-U65 is supported conceptually, it just needs some plumbing. For example the list you mention (model conversion script) and ArmCompileSpecBuilder. There's are some more places as well. If you are happy to give it a go, we can support you. Just push a PR and tag us (@digantdesai @freddan80 @per @zingo @oscarandersson8218 ). We'll give Ethos-U65 more attention medium term future.

The runtime flow is slightly different. Ethos-U65 sits on an "ML island" (Cortex-M + Ethos-U subsystem, embedded) as part of a larger system (Cortex-A, rich OS). That means Executorch runtime should be on the ML island, and your application calling into Executorch runtime needs to communicate somehow with the Cortex-A system. That mean of communication could build on e.g. ethos-u-linux-driver-stack. Some (not too big) modifications will probably be needed for Executorch workloads.

Hope this helps 👍

[vikasbalaga]

@freddan80 and others, thanks for your quick response.

The runtime flow is slightly different. Ethos-U65 sits on an "ML island" (Cortex-M + Ethos-U subsystem, embedded) as part of a larger system (Cortex-A, rich OS)...

Yes in my case it is (Cortex-A55, OS) and (Cortex-M33 + Ethos-U65) ML island and also I have a hardware setup available, so I don't need FVP

There's are some more places as well. If you are happy to give it a go, we can support you

Yes, I am interested in trying it. I could find the following places, which require modification :

• model conversion script
• ArmCompileSpecBuilder
• run (It looks like some config is being done, not sure what it will be for Ethos-U65)

So, could you help me in finding other modifications that are required?

Also, (I think this is a naive question), will this Executorch implementation work for my CPU Cortex-M33?

[freddan80]

So, could you help me in finding other modifications that are required?

I'd start with those and debug from there. The important thing is that the call to vela argument looks right. (we can help checking that)

run (It looks like some config is being done, not sure what it will be for Ethos-U65)

The run.sh script will AOT compile, build and run an inference using the FVP. In your case you'd probably be happy just generating a .pte (with arm_aot_compile.py) AOT, then use build_executorch_runner.sh to build modified under the hood to your application code, linker script and startup code (rather than the Corstone-FVP's) to produce an .elf. Or perhaps, maybe it's even better to modify the cmake build to adapt to your setup.

Note that you'd want to use a 'vela.ini' file that fits your system config, and provide that to build_executorch_runner.sh. Probably you have such a file already with you dev board SDK?

[vikasbalaga]

@freddan80 ,

I have modified the arm_aot_compile.py and I think I am able to generate *.pte model for Ethos U65 backend. The configuration details I picked based on my hardware type. (I have forked the repo and committed my changes in a private branch for your reference)

then use build_executorch_runner.sh to build modified under the hood to your application code....

I tried modifying the build_executorch_runner.sh for my system (Cortex-M33 CPU and Ethos U65 NPU) but here I am observing cmake errors

CMake Error at CMakeLists.txt:408 (message):
  Unsupported SYSTEM_CONFIG: Ethos_U65_High_End

I tried to debug it, but I couldn't understand how to update the "NPU timing adapters" as per Ethos U65 requirements. Also, it looks like we need to specify a TARGET_BOARD which corresponds to Corstone-300/320 but since in this case there is no simulator, how can I set that macro?

[freddan80]

I have forked the repo and committed my changes in a private branch for your reference

Would it be possible to share the changes?

I assigned this to @AdrianLundell. He'll help you.

[vikasbalaga]

This points to the private branch I created by forking the repo.

Thanks!

[AdrianLundell]

Hi, nice work so far!

The examples/arm/executor_runner-code and related CMake-scripts used to built it should be viewed as an example to get you started when building your own application. The build_executorch_runner.sh script and all flags containing target specific info in the runtime flow is there to make this example convenient to run and to help our testing, rather than being an official API.

For example, the timing adapters and related macros TARGET_BOARD, SYSTEM_CONFIG and MEMORY_MODE which you mention are only relevant for the simulators so to answer you question there you can ignore those completely. The relevant parts of this CMakeScript is the linking of the libraries and the converting of the .pte to a header-file, with that done you can approach this as writing for any other application for u65.

The simulator is of course very useful when developing so if you have not done so, I would suggest to start testing your model and executor_runner on u55 using the Corstone-300 target, and move to u65 when you have that working.

[vikasbalaga]

I would suggest to start testing your model and executor_runner on u55 using the Corstone-300 target

I have tried performing inference on Corstone-300 FVP by following the steps mentioned here. With this I am able to perform inference on the FVP (I have tried simple model with "ADD" operation)

I [executorch:arm_perf_monitor.cpp:133] NPU Inferences : 1
I [executorch:arm_perf_monitor.cpp:134] Profiler report, CPU cycles per operator:
I [executorch:arm_perf_monitor.cpp:138] ethos-u : cycle_cnt : 0 cycles
I [executorch:arm_perf_monitor.cpp:145] Operator(s) total: 0 CPU cycles
I [executorch:arm_perf_monitor.cpp:151] Inference runtime: 2585 CPU cycles total
I [executorch:arm_perf_monitor.cpp:153] NOTE: CPU cycle values and ratio calculations require FPGA and identical CPU/NPU frequency
I [executorch:arm_perf_monitor.cpp:162] Inference CPU ratio: 90.10 %
I [executorch:arm_perf_monitor.cpp:166] Inference NPU ratio: 9.90 %
I [executorch:arm_perf_monitor.cpp:175] cpu_wait_for_npu_cntr : 256 CPU cycles
I [executorch:arm_perf_monitor.cpp:180] Ethos-U PMU report:
I [executorch:arm_perf_monitor.cpp:181] ethosu_pmu_cycle_cntr : 411
I [executorch:arm_perf_monitor.cpp:184] ethosu_pmu_cntr0 : 6
I [executorch:arm_perf_monitor.cpp:184] ethosu_pmu_cntr1 : 43
I [executorch:arm_perf_monitor.cpp:184] ethosu_pmu_cntr2 : 3
I [executorch:arm_perf_monitor.cpp:184] ethosu_pmu_cntr3 : 634
I [executorch:arm_perf_monitor.cpp:187] Ethos-U PMU Events:[ETHOSU_PMU_AXI0_RD_DATA_BEAT_RECEIVED, ETHOSU_PMU_AXI1_RD_DATA_BEAT_RECEIVED, ETHOSU_PMU_AXI0_WR_DATA_BEAT_WRITTEN, ETHOSU_PMU_NPU_IDLE]
I [executorch:arm_executor_runner.cpp:630] model_pte_program_size:     2032 bytes.
I [executorch:arm_executor_runner.cpp:631] model_pte_loaded_size:      2032 bytes.
I [executorch:arm_executor_runner.cpp:645] method_allocator_used:     308 / 62914560  free: 62914252 ( used: 0 % )
I [executorch:arm_executor_runner.cpp:652] method_allocator_planned:  64 bytes
I [executorch:arm_executor_runner.cpp:654] method_allocator_loaded:   220 bytes
I [executorch:arm_executor_runner.cpp:655] method_allocator_input:    24 bytes
I [executorch:arm_executor_runner.cpp:656] method_allocator_executor: 0 bytes
I [executorch:arm_executor_runner.cpp:659] temp_allocator_used:       0 / 1048576 free: 1048576 ( used: 0 % )
I [executorch:arm_executor_runner.cpp:675] Model executed successfully.
I [executorch:arm_executor_runner.cpp:679] 1 outputs:
Output[0][0]: (int) 2
Output[0][1]: (int) 2
Output[0][2]: (int) 2
Output[0][3]: (int) 2
Output[0][4]: (int) 2

However, I have observed that the arm_executorch_runner application is (~62 MB) which is huge (Am I missing something here?)

[zingo]

Nope that is correct, the example just allocates a 60MB buffer so we can test/used large models out of the box as the FVP can use quite much memory.

See

#if !defined(ET_ARM_BAREMETAL_METHOD_ALLOCATOR_POOL_SIZE)
#define ET_ARM_BAREMETAL_METHOD_ALLOCATOR_POOL_SIZE (60 * 1024 * 1024)
#endif
const size_t method_allocation_pool_size =
    ET_ARM_BAREMETAL_METHOD_ALLOCATOR_POOL_SIZE;
unsigned char __attribute__((
    section("input_data_sec"),
    aligned(16))) method_allocation_pool[method_allocation_pool_size];

You can either just change the code or set ET_ARM_BAREMETAL_METHOD_ALLOCATOR_POOL_SIZE from cmake as a workaround.

We hope/plan to look into making the handling of this area better, and not ending up in the elf and such, but right now it is working like this. It's just a bad left over from when we "forked" from the examples/devtools/example_runner/example_runner.cpp :)

[vikasbalaga]

The examples/arm/executor_runner-code and related CMake-scripts used to built it should be viewed as an example to get you started when building your own application...

@AdrianLundell, I tried to build my own application by adding CMake-scripts but I have hit a road block in providing custom linker script. I almost spent 2 weeks with not much progress :(

So, I gave it up and then tried a second approach, where I will try to integrate Executorch libraries into a "working firmware application" that is available for my board (which comes with its own linker script).
With this approach, atleast I can see the application is launching but I have received a following error message from Executorch runtime

E [executorch:method.cpp:748] Missing operator: [0] aten::add.out
E [executorch:method.cpp:989] There are 1 instructions don't have corresponding operator registered.See logs for details

(I have taken the arm_executor_runner as a reference for my application and using a sample pte model with only "add" operation)

It looks like there is some operator registry in which all the ops need to be registered, but I am not sure how it works.
So, can I please get some insights into this "operator registration"?

Also, when I tried the examples, it looks like "add" is mapped to ethos-u delegate, so when I try to run it without delegate option, even on simulator I observed similar errors

Thanks!