Add transformer optimization for pre layer normalization?

[npuichigo]

The optimization tool for bert or transformer now only searches node path of post layer normalization pattern. Request to add support for pre layer norm.

[tianleiwu]

It needs to change https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/fusion_attention.py to add some new patterns in matching.

I have not encountered such a model with pre layer norm yet. Most models from huggingface transformers does not have such pattern. I know that some other frameworks could support this.

Contributions are welcome for this if anyone has done such optimization.

[Rishi-Dave]

Hi, I'd like to take this on! I'm planning to add pre-layer normalization pattern matching to fusion_attention.py with corresponding tests. Will open a PR when ready.