Ok, I figured it out, in case anyone needs to know... some of the stuff in this build just baffles my mind.

Anyway you must first go into the Retroarch menu (hotkey+B)
Then go into the Main Menu > Settings > Configuration
When in the configuration select "ON" for "Save Configuration on Quit"
(why this is not set to on from the start is beyond me)

Now go back to your quick Menu and then down to Controls.
Select Port 1 Controls - set up all your buttons (this method of setting up the controls is very unintuitive. It is based on what you set up from the very start when locking in all your controls for the OS. You will have to use that as a reference to try and map your buttons).

Exit from the Control menu and scroll up to where it says "Save Core Remap File"
(don't forget to set up controller two if you have one. You will need to set up controller two's settings from controller one. So it's not like MAME in that regard).

Now you can go back into your game to test and make sure you set up all the button figs properly. If it's all good, exit the game and next time you go back to that emulator your button config will be set up properly still.

Is there a good tutorial to teach you how to change the controls to your personal setup in the different emulators? I've followed the guide here: https://recalbox.gitbook.io/documentation/tutorials/controllers/configuring-custom-buttons/how-to-customize-controller-mapping

But it does not do a very good job of explaining things thew noobs. I was able to change the controlling and save the file, but I can't figure out how to use the terminal to allow me to have those saved settings take effect after you've left the game. I don't understand why you can't just save the settings and why you have to go in and modify another config file for it to work.

@zing I'm not able to test it out, but I do believe this should work for the console emulators (so thank you), but for MAME each individual game (on PC anyway) is able to take a custom input for that game, and that game only. Usually, you have to press TAB and it will let you modify the buttons for that specific game, I'm not sure why that has been disabled for Recalbox. Then again, they probably have it hidden somehow.

I'm needing to change the controller mapping for different emulators within recalbox. For mame, I need to change some of the games controlling and some of the console emulators, as they don't line up with my available buttons. How can I get in and change those specific controls?

On the other side of the question, I'm not using a control pad that has enough buttons to do the regular button config from the GUI. How do I skip over buttons that I don't plan on using. On other Raspberry Pi setups, I just have to hold a button down and it will skip over those options, that doesn't seem to work with Recalbox.