I have the same N64 issue with 17.12.02 using the Pi3 (no overclock) and the internal Bluetooth controller.

FYI, it took me a while to get my SZMY-POWER clone controllers to work.

I had the driver initially set to Official and they would pair, but no buttons would work. So reset the controllers and cleared the Bluetooth controllers from Recalbox. I then tried the Bluez driver and again they would pair, but no buttons world work. I then changed the driver back to Official, rebooted my Pi3, turned the controller on, it connected, but no buttons working again. I then connected the controller(s) via usb again, waited 20 seconds, disconnected (like I was pairing them again, but I had not reset them or cleared the Bluetooth controllers from Recalbox) and they are now working over Bluetooth (with a little bit of lag, but still playable ).

For piece of mind, I created a new SD Card with clean Recalbox 17.12.02, and went through the process again. Same issue. Official would not work initially, neither would any of the other drivers. I had to set to Bluez, pair via usb, change to official, pair again and everything works.