In case anyone reads this, this is my update.
So what I ended up doing, worked for me. I ended up making individual configs per emulator. What I found was in every config, the default mappings for keyboard controls, as well as recent gamepad configs were embedded in the config, due to the way the previous single config setup did things. I deleted all button mappings except for the keyboard controls, as well mappings for hotkeys. Then I created an autoconfig per controller, adding the hotkey assignments in that config. This made issues I had previously using my arcade controller, which has fewer buttons, null...as now I can set up my own hotkey config that makes sense for that controller.
This also fixed the emulator deciding which controller it wanted as first player, because it doesnt matter, if I wanted to pick another controller as p1, it will autoremap appropriately.
My only real hurdle was how to name the config file, to work with each emulator. I had to guesse and that took a while, as some like the different mame/libretro cores need different name configs.
Now when someone plugs in a new controller that the system doesn't know, I just set it up, and save autoconfig.
