[CPU] add arithmetic_mode impl for bf16_emitters

[liubo-intel]

Details:

• add arithmetic_mode impl for bf16_emitters to fix the 'inf' out issue when input data is out of rang [bf16_min,bf16_max] during f32->bf16, which may lead to 'UNK' outputs in some LLM cases

Tickets:

• CVS-157254

[liubo-intel]

Hi, @chenhu-wang and @dmitry-gorokhov : could you please help review this pr ? thanks

[chenhu-wang]

@liubo-intel , You use fixed "saturation" mode on avx512_core_bf16 ISA, but on some ISA like avx2, fixed "truncation" mode is used. This is not aligned.
We have arithmetic_mode mode_ in jit_store_emitter. jit_uni_vcvtneps2bf16 is used in store f32 to bf16 case, where the mode info is not respect there. I think we should take mode_(truncation or saturation) into jit_uni_vcvtneps2bf16 as a field and convert according to the mode. We can set the mode "saturation" as default.

[chenhu-wang]

@liubo-intel The positive overflow data is 0x7F7FXXXX. X could be any value. The convert should do smt as following function:

bf16 fp32_to_bf16(float data) {
    if (data is overflow) {
        if (saturation mode)
            return bf16_max;    // bf16_max is 0x7F7F
        else // truncation mode
            return 0x7F7F;       // drop 16 lower bit, directly
    } else {
        return vcvtneps2bf16(data);
    }
}

As we can see the saturation mode and truncation mode return same result for overflowed data. I think we can remove the explicit mode for simplicity, extend mode when it is really needed explicitly in future.
For the vectorized version in implementation, we can saturation the f32 overflow data 0x7F7FXXXX to f32 data 0x7F7F0000. Then ne round the vector along with another data. As the round bit is 0 for 0x7F7F0000, it is still 0x7F7F0000 after ne round. This keep result correct. So what we need is like a pre process to saturate the overflowed input. The remaining vcvtneps2bf16 have no change. The negative overflow is the same logic.
What do you think?

[liubo-intel]

@liubo-intel The positive overflow data is 0x7F7FXXXX. X could be any value. The convert should do smt as following function:

bf16 fp32_to_bf16(float data) {
    if (data is overflow) {
        if (saturation mode)
            return bf16_max;    // bf16_max is 0x7F7F
        else // truncation mode
            return 0x7F7F;       // drop 16 lower bit, directly
    } else {
        return vcvtneps2bf16(data);
    }
}

As we can see the saturation mode and truncation mode return same result for overflowed data. I think we can remove the explicit mode for simplicity, extend mode when it is really needed explicitly in future. For the vectorized version in implementation, we can saturation the f32 overflow data 0x7F7FXXXX to f32 data 0x7F7F0000. Then ne round the vector along with another data. As the round bit is 0 for 0x7F7F0000, it is still 0x7F7F0000 after ne round. This keep result correct. So what we need is like a pre process to saturate the overflowed input. The remaining vcvtneps2bf16 have no change. The negative overflow is the same logic. What do you think?

Hi, @chenhu-wang : as we synced offline, 'vcvtneps2bf16' is not used for truncation mode because this instruction output of overflow data is 'inf' instead of truncation values.
And I have tried to use 'saturate_input()' function for both mode(saturation and truncation), and keep the following process the same as previous before this pr in order to keep use 'vcvtneps2bf16' instruction for bf16 platform truncation mode, but this method will make the "src/frontends/onnx/tests/tests_python/test_backend." CI fail, I think it means this method not align with onnx truncation impl. so it seems we need to keep both saturation and truncation impl in order to keep the correct outputs.
for the performance concern, maybe let's consider it after perf tests?

[chenhu-wang]

Hi, @chenhu-wang : as we synced offline, 'vcvtneps2bf16' is not used for truncation mode because this instruction output of overflow data is 'inf' instead of truncation values. And I have tried to use 'saturate_input()' function for both mode(saturation and truncation), and keep the following process the same as previous before this pr in order to keep use 'vcvtneps2bf16' instruction for bf16 platform truncation mode, but this method will make the "src/frontends/onnx/tests/tests_python/test_backend." CI fail, I think it means this method not align with onnx truncation impl. so it seems we need to keep both saturation and truncation impl in order to keep the correct outputs. for the performance concern, maybe let's consider it after perf tests?

The input of vcvtneps2bf16 is bf16_max for overflow data after saturation. vcvtneps2bf16 to bf16_max is still bf16_max, not clear why inf is generated. How onnx truncation impl work, could you please elaborate more detail?

[liubo-intel]

Hi, @chenhu-wang : as we synced offline, 'vcvtneps2bf16' is not used for truncation mode because this instruction output of overflow data is 'inf' instead of truncation values. And I have tried to use 'saturate_input()' function for both mode(saturation and truncation), and keep the following process the same as previous before this pr in order to keep use 'vcvtneps2bf16' instruction for bf16 platform truncation mode, but this method will make the "src/frontends/onnx/tests/tests_python/test_backend." CI fail, I think it means this method not align with onnx truncation impl. so it seems we need to keep both saturation and truncation impl in order to keep the correct outputs. for the performance concern, maybe let's consider it after perf tests?

The input of vcvtneps2bf16 is bf16_max for overflow data after saturation. vcvtneps2bf16 to bf16_max is still bf16_max, not clear why inf is generated. How onnx truncation impl work, could you please elaborate more detail?

Hi, @chenhu-wang : the failed CI case is an onnx backend testcase(the same as shown in commit f0b3f10 CI failure of this pr). I'm not sure whether is it worthy to investigate deeper this onnx backend impl or involve onnx team related colleagues, if the impact of current method on performance is within an acceptable range. Hi, @dmitry-gorokhov what's your suggestions about this?

[liubo-intel]

Hi, @chenhu-wang and @dmitry-gorokhov : I think I know the reason why onnx backend test fails if we use 'saturation' method(function) for truncation mode. this 'saturation' function will change the original (f32)input ['nan, -nan, inf, -inf'] values which is not expected.
so it seems we should also handle these special values in 'saturation' method(function) besides normal overflow values.

[chenhu-wang]

Could you please double check we can remove h->vfixupimmps(aux, in, table_val("selector"), 0); here. Lines above have possibility to spoil the inf/nan bit again.

[liubo-intel]

done

[chenhu-wang]

We can consider reuse Vmm "in" with Vmm "clamped". Vmm "in" will store clamped data and save one aux. clamp does not break "keep_source_intact" as I understand.

[liubo-intel]

saturate_input(in, in, "bf16_min", "bf16_max"); will make the wrong outputs, maybe because the temp data will need this original 'in' data?

[chenhu-wang]

My point is:

saturate_input(clamped, in, "bf16_min", "bf16_max");
h->uni_vmovups(in, clamped);

clamped is just a aux vec, do not hold for long. Just a taste.

[yuxu42]

Hi @dmitry-gorokhov Could you please take a review? Thanks!

[liubo-intel]

Hi, @chenhu-wang and @dmitry-gorokhov : since I found truncation impl instructions may affect the model's infer time in certain situations, how about we limit these truncation impl to eltwise Constant inputs only?(commit:d5c00a2) . From my understanding, overflow cases during f32->bf16 are mostly from input constant values(models are trained in f32 precision). Instead, activation overflow values could(or to be) handled by runtime process kernels.