N64 emulator is configured to display a screen with a 640x480 resolution (native n64 resolution)
So you must use one of these video modes (DMT 4 HDMI,CEA 1 HDMI).
If your screen is not compatible with one of these video modes, please check the recalbox's wiki.
n64.videomode=DMT 9 HDMI ((( a changer)))
If you are using a CRT screen, please change the setting above with this one :