Just some testing
Pin down what is needed between 50s,40s,30s cards
Translate info for use in TensorRT fp4 converting / onnx
Anyone that has made fp4 onnx before with NVIDIA’s Model Optimizer PTQ help in the Community tab is welcomed.
Currently working on correct Quantize/Dequantize nodes for onnx export to build fp4 RT engine for 50s cards.
BL Klein 9B NVFP4 Variants — txt_attn dtype notes
bl_Klein9b-nvfp4.safetensors
txt_attnweights: NVFP4
bl_Klein9b-nvfp4_BF16.safetensors
txt_attnweights: BF16