My problem is solved !
The following thread is French but it describes the exact same problem I described here.
There's a very simple howto at the end of the thread sent by @fanou - which is exactly the type of solution I was looking for - a simple way to configure your button layout for either one specific emulator or even for a specific game.
Basically it tells you to create the file ../share/roms/n64/.retroarch.cfg, containing the following line :
input_autodetect_enable = true
Then
- Start your game
- Enter the retroarch menu (Hotkey+B while in-game)
- Go to "Buttons"
- Reconfigure your D-Pad to :
- Up = Y-
- Down = Y+
- Left = X-
- Right = X+
- Save remap to Core (here's where you can choose to save only for your game)
- Go back to and quit game
- Start game
Solved.
