I have successfully done so.
However, I'm a little confused on what you are doing. Are you installing a 64bit on a x86 system? Because that won't work.
Also, you have you checked this page? https://github.com/recalbox/recalbox-os/wiki/PC-compatibility-(EN)
Do you have the required bioses in the proper folder? You can check the web GUI by going to the system's IP address on another system and seeing if you have the right bioses for N64 (I don't know what they are off the top of my head)