My problem is solved !
The following thread is French but it describes the exact same problem I described here.
There's a very simple howto at the end of the thread sent by @fanou - which is exactly the type of solution I was looking for - a simple way to configure your button layout for either one specific emulator or even for a specific game.
Basically it tells you to create the file ../share/roms/n64/.retroarch.cfg, containing the following line :input_autodetect_enable = true
ThenStart your game Enter the retroarch menu (Hotkey+B while in-game) Go to "Buttons" Reconfigure your D-Pad to : Up = Y- Down = Y+ Left = X- Right = X+ Save remap to Core (here's where you can choose to save only for your game) Go back to and quit game Start game