[QNN] Hope to know how to support `v68` arch soc with QNN?

[ecccccsgo]

Hello,

I was work with some old platform like SA8295 with htp version v68. In this repo and the aihub repo, When i use w4a16 recipe to quantize the model like smolvlm/qwen, it was error with:

[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.linear.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]:  <E> [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]:  <E> QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]:  <E> Failed to validate op aten_native_layer_norm_default_24 with error 0xc26

But in some paper like AutoNeural, they use the ViT: W8A16, language model:W4A16 (page 10, in the table2).

it seems the v68 arch support this type of ops. I hope you can give me some infomation about how I can use w4a16 in llm/lvm to support larger model. :)

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin

[ecccccsgo]

seems to find some info to do some experiments: #15410

[shewu-quic]

Hi @ecccccsgo,
According to your error log, the issue is caused by a layer_norm op validation failure.
By default, we use 16a16w for 16a layer_norm to maintain accuracy. However, as stated in the QNN documentation, 16a16w requires at least arch V73.
If you need to run this on a V68 device, you can try the following adjustment.

Patch to run Qwen 2.5 0.5b on V68 (click to expand)

diff --git a/backends/qualcomm/quantizer/custom_annotation.py b/backends/qualcomm/quantizer/custom_annotation.py
index 9558aeced2..aef4afeadb 100644
--- a/backends/qualcomm/quantizer/custom_annotation.py
+++ b/backends/qualcomm/quantizer/custom_annotation.py
@@ -10,6 +10,7 @@ import torch
 from executorch.backends.qualcomm.quantizer.quantizer import (
     get_16a8w_qnn_ptq_config,
     get_16a8w_qnn_qat_config,
+    get_16a4w_qnn_ptq_config,
     get_8a8w_qnn_ptq_config,
     get_8a8w_qnn_qat_config,
     get_ptq_per_channel_quant_config,
@@ -499,3 +500,31 @@ def get_custom_quant_ios_dtype(
         u.target for u in list(node.users.keys())
     ] + [node.target]:
         return sharding_dtype
+
+
+def custom_annotation_16a4w_layer_norm(gm):
+    use_16a4w_config = get_16a4w_qnn_ptq_config()
+    use_16a4w_config.weight = QuantizationSpec(
+        dtype=torch.uint8,
+        quant_min=0,
+        quant_max=15,
+        qscheme=torch.per_tensor_symmetric,
+        ch_axis=0,
+        observer_or_fake_quant_ctr=use_16a4w_config.weight.observer_or_fake_quant_ctr,
+    )
+    for node in gm.graph.nodes:
+        if node.target != torch.ops.aten.layer_norm.default:
+            continue
+        act_node = node.args[0]
+        weight_node = node.args[2]
+        bias_node = None
+        input_qspec_map = {act_node: use_16a4w_config.input_activation, weight_node: use_16a4w_config.weight}
+        if len(node.args) > 2:
+            bias_node = node.args[3]
+            input_qspec_map[bias_node] = use_16a4w_config.bias
+        
+        node.meta[Q_ANNOTATION_KEY] = QuantizationAnnotation(
+            input_qspec_map=input_qspec_map,
+            output_qspec=use_16a4w_config.output_activation,
+            _annotated=True,
+        )
diff --git a/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py b/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py
index ca06a89142..fdeffa79d6 100644
--- a/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py
+++ b/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py
@@ -7,7 +7,7 @@
 from typing import Optional
 
 import torch
-from executorch.backends.qualcomm.quantizer.custom_annotation import annotate_kv_8bit
+from executorch.backends.qualcomm.quantizer.custom_annotation import annotate_kv_8bit, custom_annotate_matmul_16a8w, custom_annotation_16a4w_layer_norm
 from executorch.backends.qualcomm.quantizer.quant_recipe import (
     QuantGranularity,
     QuantRecipe,
@@ -485,16 +485,17 @@ class Qwen2_5_0_5BQuantRecipe(StaticLLMQuantRecipe):
             act_observer=MinMaxObserver,
             granularity=QuantGranularity.PER_TENSOR,
             verbose=verbose,
-        ).add_node_target(
-            {
-                torch.ops.aten.conv2d.default,
-            },
-            QuantDtype.use_16a4w_block,
-            False,
-            act_observer=MinMaxObserver,
-            granularity=QuantGranularity.PER_BLOCK,
-            extra_kwargs={"block_size": (1, 16, 1, 1)},
+        # ).add_node_target(
+        #     {
+        #         torch.ops.aten.conv2d.default,
+        #     },
+        #     QuantDtype.use_16a4w_block,
+        #     False,
+        #     act_observer=MinMaxObserver,
+        #     granularity=QuantGranularity.PER_BLOCK,
+        #     extra_kwargs={"block_size": (1, 16, 1, 1)},
         )
+        self.recipe.custom_quant_annotations.extend([custom_annotation_16a4w_layer_norm,custom_annotate_matmul_16a8w, annotate_kv_8bit])
 
 
 class Qwen2_5_1_5BQuantRecipe(StaticLLMQuantRecipe):
``

[cccclai]

@shewu-quic thanks for the detailed patch! Feel like it was asked more than once, is it possible to expand the llm quantization recipe by SoC?

[shewu-quic]

Yes, it’s a good idea to begin expanding the LLM quantization recipe by SOC. Butt I think we’ve already enabled validation in the quantizer, and I’m currently looking into whether it’s possible to automatically update the config if it’s not supported by the given SOC.

[ecccccsgo]

@shewu-quic Sorry for delay, I actually hope to test vlm on SA8295, like SmolVLM. I use your patch like:

class SmolVLMQuantRecipe(StaticLLMQuantRecipe):
    default_quant_dtype = QuantDtype.use_16a4w

    def __init__(self, verbose: bool = False):
        super().__init__()

        self.recipe = QuantRecipe(
            self.default_quant_dtype,
            False,
            act_observer=MinMaxObserver,
            granularity=QuantGranularity.PER_TENSOR,
            verbose=verbose,
        # ).add_node_target(
        #     {
        #         torch.ops.aten.conv2d.default,
        #     },
        #     QuantDtype.use_16a8w,
        #     False,
        #     act_observer=MinMaxObserver,
        #     granularity=QuantGranularity.PER_CHANNEL,
        )
        self.recipe.custom_quant_annotations.extend([custom_annotation_16a4w_layer_norm, custom_annotate_matmul_16a8w, annotate_kv_8bit])

and

from torchao.quantization.pt2e.quantizer import (
    annotate_input_qspec_map,
    annotate_output_qspec,
    QuantizationAnnotation,
    SharedQuantizationSpec,
    QuantizationSpec
)

def custom_annotation_16a4w_layer_norm(gm):
    use_16a4w_config = get_16a4w_qnn_ptq_config()
    use_16a4w_config.weight = QuantizationSpec(
        dtype=torch.uint8,
        quant_min=0,
        quant_max=15,
        qscheme=torch.per_tensor_symmetric,
        ch_axis=0,
        observer_or_fake_quant_ctr=use_16a4w_config.weight.observer_or_fake_quant_ctr,
    )
    for node in gm.graph.nodes:
        if node.target != torch.ops.aten.layer_norm.default:
            continue
        act_node = node.args[0]
        weight_node = node.args[2]
        bias_node = None
        input_qspec_map = {act_node: use_16a4w_config.input_activation, weight_node: use_16a4w_config.weight}
        if len(node.args) > 2:
            bias_node = node.args[3]
            input_qspec_map[bias_node] = use_16a4w_config.bias
        
        node.meta[Q_ANNOTATION_KEY] = QuantizationAnnotation(
            input_qspec_map=input_qspec_map,
            output_qspec=use_16a4w_config.output_activation,
            _annotated=True,
        )

1. I tried qwen2.5-0.5b and it works fine.
2. Then, i hope to run smolvlm, I ran the following cmd:

export BASE_DIR=$EXECUTORCH_ROOT/output_dir/smolvlm-8295
export PYTHONUNBUFFERED=1
mkdir -p $BASE_DIR
CUDA_VISIBLE_DEVICES=   python examples/qualcomm/oss_scripts/llama/llama.py -a $BASE_DIR/ -b build-android -m  SA8295 --decoder_model smolvlm_500m_instruct --model_mode hybrid --prefill_ar_len 16 --max_seq_len 1024 --compile_only --prompt "Can you describe this image?" --image_path "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" > $BASE_DIR/llama_inference.log 2>&1

it came out two types error with:
the ops error (may mean the revision not works):

WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten._softmax.default | True
[QNN Partitioner Op Support]: aten.mul.Tensor | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]:  <E> [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]:  <E> QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]:  <E> Failed to validate op aten_matmul_default_4 with error 0xc26

[WARNING] [Qnn ExecuTorch]: Qnn Backend op validation failed with error: 3110
[QNN Partitioner Op Support]: aten.matmul.default | False
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.linear.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.linear.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.linear.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]:  <E> [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]:  <E> QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]:  <E> Failed to validate op aten_native_layer_norm_default_4 with error 0xc26

and the InsertIOQDQ error, and i find some issuses about it and i will work to fix it:

[INFO 2026-03-19 08:07:30,797 base_component.py:56] MultiModalManager::compile completed in 33.63414478302002s
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/passes/infra/pass_manager.py", line 277, in __call__
    res = fn(module)
          ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/passes/infra/pass_base.py", line 47, in __call__
    res = self.call(graph_module)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/backends/qualcomm/_passes/insert_io_qdq.py", line 151, in call
    self._insert(graph_module)
  File "/workspace/executorch/backends/qualcomm/_passes/insert_io_qdq.py", line 147, in _insert
    self.q_dq_map[n.meta[QCOM_QUANT_ATTRS][QCOM_ENCODING]],
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: <EdgeOpOverload: quantized_decomposed.dequantize_per_tensor.default>: schema = quantized_decomposed::dequantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensor

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 717, in main
    export_llama(args)
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 649, in export_llama
    compile(
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 180, in compile
    multi_modal_mgr.compile(compile_specs=compile_specs, pte_filenames=pte_filenames)
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/wrappers/base_component.py", line 63, in wrapper
    func(cls, *args, **kwargs)
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/wrappers/llm_wrappers.py", line 1020, in compile
    self.process(compile_request)
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/wrappers/llm_wrappers.py", line 1002, in process
    Processor.process(self, request)
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/wrappers/base_component.py", line 155, in process
    return self._next_handler.process(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/wrappers/base_component.py", line 183, in process
    getattr(self, request.method_name)(request)
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/wrappers/llm_wrappers.py", line 905, in compile
    edge_prog_mgr = to_edge_transform_and_lower_to_qnn(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/backends/qualcomm/utils/utils.py", line 449, in to_edge_transform_and_lower_to_qnn
    return to_edge_transform_and_lower(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/exir/program/_program.py", line 116, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/exir/program/_program.py", line 1405, in to_edge_transform_and_lower
    edge_manager = edge_manager.to_backend(method_to_partitioner)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/exir/program/_program.py", line 116, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/exir/program/_program.py", line 1707, in to_backend
    new_edge_programs = to_backend(method_to_programs_and_partitioners)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/functools.py", line 912, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/exir/backend/backend_api.py", line 762, in _
    lower_all_submodules_to_backend(
  File "/workspace/executorch/exir/backend/backend_api.py", line 591, in lower_all_submodules_to_backend
    backend_name_to_subclass[backend_id].preprocess_multimethod(
  File "/workspace/executorch/backends/qualcomm/qnn_preprocess.py", line 176, in preprocess_multimethod
    py_op_wrappers = QnnBackend._build_op_wrappers(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/backends/qualcomm/qnn_preprocess.py", line 56, in _build_op_wrappers
    graph_module = QnnPassManager().transform_for_preprocess_pipeline(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/executorch/backends/qualcomm/_passes/qnn_pass_manager.py", line 269, in transform_for_preprocess_pipeline
    self._transform(exported_program.graph_module)
  File "/workspace/executorch/backends/qualcomm/_passes/qnn_pass_manager.py", line 138, in _transform
    return self(graph_module).graph_module
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/passes/infra/pass_manager.py", line 303, in __call__
    raise Exception(msg) from e  # noqa: TRY002
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: An error occurred when running the 'InsertIOQDQ' pass after the following passes: ['FoldQDQ', 'InsertRequantize']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 728, in <module>
    main()
  File "/workspace/executorch/examples/qualcomm/oss_scripts/llama/llama.py", line 723, in main
    raise Exception(e)
Exception: An error occurred when running the 'InsertIOQDQ' pass after the following passes: ['FoldQDQ', 'InsertRequantize']

cc @cccclai @DannyYuyang-quic

[ecccccsgo]

BTW, expand the llm quantization recipe by SoC is a great idea.

[haowhsu-quic]

Hi, since the image encoder is quite sensitive, we're using 16bit activation for the matmul as well. Currently there is no custom annotation for it but you could set use_fp16 to true for the image encoder in this line. SA8295 should have fp16 support.