CPU inference is a useful baseline even when an RKNN path is available. It helps show whether conversion changes accuracy and whether acceleration improves the application’s total response time. The largest gain in an isolated kernel may not be the largest gain for the user.

ArmSoM Sige5
Use the CPU path as a reference
Run the same inputs through the original model and the converted version. Compare outputs before comparing time. Include preprocessing, postprocessing and data movement in the application measurement. If the model contains unsupported operations, record where they execute and how much time they add.
Evaluate conversion as an engineering step
Check the current RKNN target support and model example. Review quantization and operator behavior, then compare outputs against the reference. A lower inference time is useful only if quality remains acceptable and data movement does not consume the saving. Keep conversion settings with the model artifact.
Choose a maintainable release
For Sige5 or Sige7, pin compatible toolkit, runtime and driver versions. Decide how a model update will be converted, tested and rolled back. If a small CPU model already meets the requirement, additional acceleration may not be worth the integration work. If the CPU is overloaded by concurrent services, an NPU path may free resources even when raw throughput is not the primary objective.

ArmSoM Sige7
Match the choice to your requirements
Requirement or situation | What to evaluate | Decision detail |
CPU baseline | Reference accuracy and timing | Simplifies initial diagnosis. |
RKNN trial | Conversion and runtime validation | Measure total application benefit. |
Release | Versioned model and rollback | Repeat after updates. |
Example design
Keep a CPU implementation as a correctness reference
A CPU path can be valuable during development even when the deployed application uses the NPU. Run the same input through both and compare outputs within the application’s tolerance. If acceleration changes the result, inspect preprocessing, precision and unsupported operations before optimizing speed. The preferred production path is the one that meets accuracy and latency requirements with an update and recovery process the team can maintain.
Hardware and documentation
Sige5 · RK3576
RK3576, four Cortex-A72 and four Cortex-A53 cores, 6 TOPS NPU, dual Gigabit Ethernet and M.2 PCIe 2.1 x1.
Hardware documentation · Product details
Sige7 · RK3588
RK3588, 6 TOPS NPU, dual 2.5GbE and M.2 PCIe 3.0 x4. Check the selected eMMC and wireless configuration.
Hardware documentation · Product details
Frequently asked questions
Should every model use the NPU?
No. Unsupported models and already adequate CPU workloads may not benefit from conversion.
Product information and support
Sources
Written by ArmSoM using the documentation below. Example designs are proposals, not measured customer installations.
