LayerInfo doesn't support fp8 and int4_awq dtype?

[youki-sada]

I built LLaMA 3B w4a8_awq by blow command.

$ trtllm-build --checkpoint_dir ./trt_models/llama3.2-3b-hf_w4a8KVfp8 --output_dir /trt_engines/tllm_llama3.2_3b_w4a8KVfp8_prof --gemm_plugin float16 --max_batch_size 1 --max_input_len 2048 --profiling_verbosity detailed
$ du -sh /trt_engines/*
6.8G    /trt_engines/tllm_llama3.2_3b_fp16_prof
2.9G    /trt_engines/tllm_llama3.2_3b_w4a8KVfp8_prof

and then, export layer information.

linfo = runner.session.engine_inspector.get_engine_information(trt.LayerInformationFormat.JSON)
with open("layerinfo.json","w") as f: f.write(linfo)

However, there is no Int4 description for WeightOnlyGroupwiseQuantMatmul weights.

$ grep weights layerinfo.json|tail -n 10
  "weights": {"Type": "Half", "Count": 196608},
  "weights": {"Type": "Half", "Count": 3072},
  "weights": {"Type": "Float", "Count": 1},
  "weights": {"Type": "Half", "Count": 6291456},
  "weights": {"Type": "Half", "Count": 196608},
  "weights": {"Type": "Half", "Count": 8192},
  "weights": {"Type": "Float", "Count": 1},
  "weights": {"Type": "Half", "Count": 6291456},
  "weights": {"Type": "Half", "Count": 196608},
  "weights": {"Type": "Half", "Count": 394002432},

layerinfo.json

[youki-sada]

I guess it because the quantized weight is packed into fp16. Is there any way to get quant_algo for WeightOnlyGroupwiseQuantMatmul?

[nv-guomingz]

@Barry-Delaney would u please take a look this issue?

[Barry-Delaney]

@youki-sada QuantAlgo is stored in config.json of the engine directory, and this is the way we recommended for getting those info.
If you want to infer QuantAlgo from the json file you provided, the only information are parameters' shapes and data types.
For now, we have W8A16_GPTQ/W4A16_GPTQ/W4A16_AWQ/W4A8_AWQ.

• If any parameters inside the layer has "Format/Datatype": "Float", then QuantAlgo == QuantAlgo.W4A8_AWQ.
• Else if any parameters has vector shape, i.e., prequant_scaling_factor, then QuantAlgo == QuantAlgo.W4A16_AWQ.
• Else the model is using GPTQ, as the shape of weights is [*, out_features // elem_in_FP16], whether the model is using INT4 or INT8 can be inferred from scale.shape[1] / weights.shape[1].

To be noted, this method may become invalid in the future if any of the parameters' definition changes.

[youki-sada]

@Barry-Delaney Thank you, quant_cfg.json was helpful for me. I am currently working on mixed precision. According to the quant_cfg.json, kv_cache_quant_algo is confusing since there is both "FP8" and null value. Is kv_cache_quant_algo of transformer.layers.0.attention.qkv higher priority or model-level kv_cache_quant_algo?

{
    "version": "0.15.0",
    "quant_algo": "MIXED_PRECISION",
    "kv_cache_quant_algo": "FP8",
    "quantized_layers": {
        "transformer.layers.0.attention.qkv": {
            "quant_algo": "W4A8_AWQ",
            "kv_cache_quant_algo": null,
            "group_size": 128,
            "smoothquant_val": 0.5,
            "clamp_val": null,
            "use_meta_recipe": false,
            "has_zero_point": false,
            "pre_quant_scale": true
        },