My problem is solved !
The following thread is French but it describes the exact same problem I described here.
There's a very simple howto at the end of the thread sent by @fanou - which is exactly the type of solution I was looking for - a simple way to configure your button layout for either one specific emulator or even for a specific game.
Basically it tells you to create the file ../share/roms/n64/.retroarch.cfg, containing the following line :
input_autodetect_enable = true
Then
- Start your game
 - Enter the retroarch menu (Hotkey+B while in-game)
 - Go to "Buttons"
 - Reconfigure your D-Pad to :
- Up = Y-
 - Down = Y+
 - Left = X-
 - Right = X+
 
 - Save remap to Core (here's where you can choose to save only for your game)
 - Go back to and quit game
 - Start game
 
Solved.