ArmSoM
← All stories

Running LLMs on RK3588: Models, Memory and Deployment

Running a local language model on RK3588 involves model format, runtime support and memory as well as the processor. A short text-generation demonstration is a starting point. A useful evaluation also measures prompt handling, sustained output and memory use.

ArmSoM AIM7

ArmSoM AIM7

Choose a model the software can actually load

Check the supported model architecture, quantization and runtime version before downloading large weights. Record the model identifier and conversion steps. Measure startup, prompt processing and generation separately with representative input lengths. Keep the tokenizer and model configuration together so a later change can be distinguished from a hardware difference.

Budget more than weights

Model weights are only part of memory use. Context state, temporary buffers, the operating system and other services also need space. Longer prompts and multiple sessions can change peak consumption. Select a memory configuration based on the full application and reserve headroom rather than treating all advertised RAM as available to the model.

Measure the experience users will notice

Record prompt length, output length, quantization, context limit and cooling. Measure time to first token separately from generation rate, and repeat with the actual retrieval or tool workflow. Inspect output quality after conversion. A fast short demonstration should not be advertised as sustained performance for every model or context size.

ArmSoM Sige7

ArmSoM Sige7

Match the choice to your requirements

Requirement or situation

What to evaluate

Decision detail

Model compatibility

Architecture and export path

Confirm with the current toolkit.

Memory

Weights plus context and services

Measure peak use.

Interaction

First token and sustained generation

Test representative prompts.

Example design

Measure an interactive language-model workload

A chat application needs more than a single token-per-second number. Define a realistic prompt, expected response length, context size and concurrency. Measure the wait before the first useful output separately from subsequent generation. Include model loading and memory pressure if the application switches models. A demonstration with a short prompt cannot establish the experience of a longer document-analysis session. No throughput result is claimed here without a reproducible run.

Hardware and documentation

Sige7 · RK3588

RK3588, 6 TOPS NPU, dual 2.5GbE and M.2 PCIe 3.0 x4. Check the selected eMMC and wireless configuration.

Hardware documentation · Product details

AIM7 · RK3588

RK3588 module with a 6 TOPS NPU and 260-pin SO-DIMM connector. ArmSoM has tested a Jetson Nano B01 carrier; this does not establish compatibility with all Jetson carriers or NVIDIA software.

Hardware documentation · Product details

Frequently asked questions

Can I promise a token rate from RK3588 specifications?

No. It needs a named model, runtime, context and reproducible measurement.

Does larger RAM guarantee a faster answer?

It can prevent memory pressure, but compute, memory bandwidth and runtime behavior still determine speed.

Product information and support

Explore Sige7

Contact ArmSoM

Sources

Written by ArmSoM using the documentation below. Example designs are proposals, not measured customer installations.

Related reading