Connect the controller in switch mode (its name will now be "pro controller"). All the buttons will be recognized.

You'll find the whole procedure here: https://retropie.org.uk/forum/topic/21481/8bitdo-m30-bluetooth-controller-issue

For the layout, I put B, A, R2, as ABC and Y, X, L2 as XYZ (this way, Z, C and the shoulder buttons don't overlap). It works like a charm!