ArmSoM
← All stories

How to Benchmark RK3588 and RK3576 NPU Performance Fairly

NPU timing is easiest to interpret when it describes a repeatable task. A single inference measurement leaves out loading, data conversion and queueing. Report those costs separately, along with the accuracy reached by the tested model.

ArmSoM Sige7s

ArmSoM Sige7s

Define the measurement boundary

State whether timing begins at a captured frame, a prepared tensor or the inference call. Include warm-up behavior and repeat the run on a representative dataset. Record model, precision, runtime, input size and concurrent workloads. Keep latency variation visible rather than presenting the best observed sample as normal performance.

Measure at more than one boundary

Inference-only time is useful for diagnosing an accelerator, while end-to-end time includes capture, data conversion, processing and output. Report both where possible. Throughput and latency answer different questions: a batched system can process many frames while taking longer to return any one result. Include tail latency and dropped frames in interactive applications.

Run long enough to reveal constraints

Test the intended enclosure, cooling and supply. Observe temperature, clock changes, memory use and service stability. Repeat trials and retain raw logs. Present Sige5, Sige7s and Sige7 results as measurements of those tested systems, not permanent rankings for every image and model.

ArmSoM Sige5

ArmSoM Sige5

Match the choice to your requirements

Requirement or situation

What to evaluate

Decision detail

Quality

Task accuracy after conversion

Reject incomparable outputs.

Timing

Inference and complete pipeline

State the measurement boundary.

Conditions

Cooling, image and power

Make repetition possible.

Example design

Publish a benchmark that another developer can repeat

A useful NPU result identifies the board, image, converter, runtime, model, precision, input shape and cooling. Include repeated runs and separate cold loading from steady-state execution. If the test excludes acquisition or postprocessing, say so beside the result. Report accuracy alongside latency, because a faster converted model that fails the application’s quality requirement is not an improvement. Preserve the raw measurements rather than only the best chart value.

Keep the model and accuracy comparable

Record architecture, model checksum, input dimensions, batch size, precision and conversion settings. Include the output interpretation and thresholds. A smaller input or a different model can make a platform appear faster while solving a different problem.

Measure accuracy on the same representative evaluation set. Decide the minimum acceptable quality before comparing speed. For a detection application, inspect difficult scenes and class-specific behavior as well as an aggregate metric.

Rockchip's Model Zoo is useful for finding supported reference workloads. Treat any upstream performance table as a result under its stated conditions, not a guarantee for your board and application.

Separate four timing measurements

Measurement

What it answers

Inference latency

How long does one inference take under the defined runtime conditions?

Throughput

How many completed inputs are processed per second?

End-to-end latency

How old is an input when its useful result becomes available?

Startup time

How long until the system can process its first valid input?

Do not invert average latency to report throughput unless the execution model makes that valid. As a simple arithmetic example, a strictly serial 20 ms operation has an ideal upper rate of 50 operations per second before other work. Concurrent queues and pipeline stages change that relationship.

Measure end-to-end delay with timestamps that have a known relationship. Camera exposure timestamps, network arrival and application timestamps may come from different clocks; define what your measurement actually includes.

Use repeatable timing boundaries

Warm up the system before a steady-state run, but report cold-start behavior separately when it matters. Wait for asynchronous work to complete before stopping an inference timer. Otherwise the measurement may capture only submission overhead.

Use a monotonic clock and collect a distribution rather than one sample. Report a central value and a tail measure such as the 95th percentile, with the number of samples and run duration. Keep dropped or failed inputs visible instead of silently excluding them.

Measure power at a stated boundary

If energy matters, state where power is measured: board input, complete device input or a particular rail. Include the workload duration and baseline behavior. An input measurement that includes an SSD and fan cannot be compared directly with a processor-only number.

For battery-powered equipment, consider the duty cycle. Startup, idle periods, communication and sleep behavior can matter more than the peak inference interval. Use the product's operating pattern to estimate energy use.

Additional references

Hardware and documentation

Sige5 · RK3576

RK3576, four Cortex-A72 and four Cortex-A53 cores, 6 TOPS NPU, dual Gigabit Ethernet and M.2 PCIe 2.1 x1.

Hardware documentation · Product details

Sige7s · RK3588S

RK3588S, 6 TOPS NPU, Gigabit Ethernet and removable eMMC. Dimensions: 85 × 56 × 17 mm. Disconnect power before replacing eMMC.

Hardware documentation · Product details

Sige7 · RK3588

RK3588, 6 TOPS NPU, dual 2.5GbE and M.2 PCIe 3.0 x4. Check the selected eMMC and wireless configuration.

Hardware documentation · Product details

Frequently asked questions

Should I publish only the best run?

No. Report repeated results and explain variation.

Can vendor benchmark numbers be mixed with our own?

Only with clearly separated sources and conditions; they should not appear to be one controlled test.

Product information and support

Explore Sige5

Contact ArmSoM

Sources

Written by ArmSoM using the documentation below. Example designs are proposals, not measured customer installations.

Related reading