My problem is solved !

The following thread is French but it describes the exact same problem I described here.

https://forum.recalbox.com/topic/21653/pi4-recalbox-7-x-mupen64plus-bug-enregistrement-du-mapping-dans-retroarch/65?_=1605037947077

There's a very simple howto at the end of the thread sent by @fanou - which is exactly the type of solution I was looking for - a simple way to configure your button layout for either one specific emulator or even for a specific game.

Basically it tells you to create the file ../share/roms/n64/.retroarch.cfg, containing the following line :

input_autodetect_enable = true

Then

Start your game Enter the retroarch menu (Hotkey+B while in-game) Go to "Buttons" Reconfigure your D-Pad to : Up = Y- Down = Y+ Left = X- Right = X+ Save remap to Core (here's where you can choose to save only for your game) Go back to and quit game Start game

Solved.