ArmSoM
← All stories

RKNN or a CPU Runtime? Choosing an Inference Path for Your ARM SBC

CPU inference is a useful baseline even when an RKNN path is available. It helps show whether conversion changes accuracy and whether acceleration improves the application’s total response time. The largest gain in an isolated kernel may not be the largest gain for the user.

ArmSoM Sige5

ArmSoM Sige5

Use the CPU path as a reference

Run the same inputs through the original model and the converted version. Compare outputs before comparing time. Include preprocessing, postprocessing and data movement in the application measurement. If the model contains unsupported operations, record where they execute and how much time they add.

Evaluate conversion as an engineering step

Check the current RKNN target support and model example. Review quantization and operator behavior, then compare outputs against the reference. A lower inference time is useful only if quality remains acceptable and data movement does not consume the saving. Keep conversion settings with the model artifact.

Choose a maintainable release

For Sige5 or Sige7, pin compatible toolkit, runtime and driver versions. Decide how a model update will be converted, tested and rolled back. If a small CPU model already meets the requirement, additional acceleration may not be worth the integration work. If the CPU is overloaded by concurrent services, an NPU path may free resources even when raw throughput is not the primary objective.

ArmSoM Sige7

ArmSoM Sige7

Match the choice to your requirements

Requirement or situation

What to evaluate

Decision detail

CPU baseline

Reference accuracy and timing

Simplifies initial diagnosis.

RKNN trial

Conversion and runtime validation

Measure total application benefit.

Release

Versioned model and rollback

Repeat after updates.

Example design

Keep a CPU implementation as a correctness reference

A CPU path can be valuable during development even when the deployed application uses the NPU. Run the same input through both and compare outputs within the application’s tolerance. If acceleration changes the result, inspect preprocessing, precision and unsupported operations before optimizing speed. The preferred production path is the one that meets accuracy and latency requirements with an update and recovery process the team can maintain.

Hardware and documentation

Sige5 · RK3576

RK3576, four Cortex-A72 and four Cortex-A53 cores, 6 TOPS NPU, dual Gigabit Ethernet and M.2 PCIe 2.1 x1.

Hardware documentation · Product details

Sige7 · RK3588

RK3588, 6 TOPS NPU, dual 2.5GbE and M.2 PCIe 3.0 x4. Check the selected eMMC and wireless configuration.

Hardware documentation · Product details

Frequently asked questions

Should every model use the NPU?

No. Unsupported models and already adequate CPU workloads may not benefit from conversion.

Product information and support

Explore Sige5

Contact ArmSoM

Sources

Written by ArmSoM using the documentation below. Example designs are proposals, not measured customer installations.

Related reading