In case anyone reads this, this is my update.
So what I ended up doing, worked for me. I ended up making individual configs per emulator. What I found was in every config, the default mappings for keyboard controls, as well as recent gamepad configs were embedded in the config, due to the way the previous single config setup did things. I deleted all button mappings except for the keyboard controls, as well mappings for hotkeys. Then I created an autoconfig per controller, adding the hotkey assignments in that config. This made issues I had previously using my arcade controller, which has fewer buttons, null...as now I can set up my own hotkey config that makes sense for that controller.
This also fixed the emulator deciding which controller it wanted as first player, because it doesnt matter, if I wanted to pick another controller as p1, it will autoremap appropriately.

My only real hurdle was how to name the config file, to work with each emulator. I had to guesse and that took a while, as some like the different mame/libretro cores need different name configs.
Now when someone plugs in a new controller that the system doesn't know, I just set it up, and save autoconfig.

When playing mame roms, games vary so much in size that setting an overlay to a fixed size makes some game look stretched. I was looking for an option to just have a full screen image behind the game that's playing, to fill black space.

Lot of good info here, thanks for the responses. What is the role of autoconfig in retroarch gui settings menu?
In relation to the config'ing the fightstick scenario I had previously mentioned: I plugged it in, added buttons in menu area, not rgui, according to standard ps3 fightstick layout. Went into psx and went to street fighter alpha button config. Rgui had the r2 button as the start button (which in layout is punch 3). Thus I had to configure in rgui/settings/input, the correct button layout. I'd love to just use front-end setup but emulator doesn't always agree.

Asked before, can't find answer. I had seen others with similar issue, I'm hoping this has been addressed.
Configured controls for all emulators with controller A (Xbox afterglow) first initially in menu, then in RGUI. Plugging in any other gamepad while Xbox (A) is plugged in, causes me to have to go into rgui and change, player 1 input back to the xbox controller (which has proper hotkey and buttons set up for that controller) because new gamepad seems to have taken priority (p1), but with the Xbox mappings. So let's say I go through this and configure (A) to player 1, and new gamepad to 2 I'm lucky if it doesn't still make itself player 1 as well, god help me I enter a game with only the xbox one plugged in and have to go plug second controller in and fix (A)
I had asked previously, thinking the actual USB port had something to do with controller priority but got no answer. I've since taped each gamepads name to side of pi to keep from swapping them. It doesn't however seem to matter. I had asked if priority is given to the controller that starts the game. As well, why doesn't it have the buttons correctly set up for it if it does end up getting player 1 priority.
Please help whoever can and thank you guys for your work.

Summarize:

1. What determines priority of gamepad to p1, p2.....
2. Why would another controller keep taking player 1's spot but not use its own button configuration.
3. Why doesn't it seem like the controllers retain their individual configuration, rather than whatever emulator says player one should be, irregardless of which controller is actually plugged in.
4. Any chance I can tweak this, or an option to have whoever starts the game be player 1? (Have a fight stick that I'd occasionally like to use but not have to reconfigure rgui every time to make it p1)
   Re: Multiple controller issues

Yeah there's really not an easy fix for this. I spent days cleaning (culling) my Roms for this issue. Clrmamepro can give you lists that you can then make batch delete files with (or move). Romlister is I believe what I used, as I was able to make plain text lists including removal of roms like trackball, hanafuda, quiz... Another trick if you have on your computer the emulators, is to go through their favorites, fighting, 4 button.....lists, and make those batch files that move or delete the roms (ask the Google how its not hard).
It should go without saying that you use a split set if you didn't already know, otherwise you can't remove the individual builds from the roms. In case you were unaware. Good luck, it just takes some tlc.

So i have been trying to make my own custom configurations per core and it has been going ok...ish. No matter what I set for a config file for "nes.configfile=/recalbox/share/system/.config......." it always edits recalbox.conf nes entry to use "..../mednafen_supergrafx_libretro.cfg". I don't know why or what would dictate that. I have gone through my custom configs and nowhere does it override what I set the NES config to use. If I do it in RGUI (save New), after game reload it still uses mednafens. Isn't the purpose of the .conf to be the global override, or am I missing something. As I configure i'm going to check to see if other cores do this too. the problem is going to be that i'm going to setup all my settings, and the core is just going decide to use it's own config at some point.

@Substring Four button controller, six button controller, 8 button fighting stick. Playing a four button controller on sega ms, or 32x, or playing mame arcade games with these controllers, makes for very different setups. Then add to that, the emulator doesn't have a pre defined config for every controller out there (my afterglow xbox, as well as my MCCthulhu fighting stick doesn't work out of the box). Then you run into Doom port mappings. That's why these controllers need setup in each emulator because there's no blanket config that reconciles all the issues. So given these issues, what do you suggest is the right way to do this?
Per my original question, is there an option to have a controller config, specific to my controller(s) that I can set up with mappings? How does the emulator knkw which controller to allow hotkey button usage, because in regards to that whatever i 'had' set to p1, if disconnected, retains that right and I have to reconnect to change that.

So this sounds like something that happened to my friend. He had 2 of the same gamepad the problem was that when we went to input and changed the input to another device, the name didn't change therefore we weren't sure that it was even reading both seperately. We fixed it by taking what we considered p1, moving the device name in the emulators input config to the far left, and configured our buttons. Now when we went to set up player 2 we started far left then clicked one to the right. The name didn't change but it did in fact pick the other controller. We then set up accordingly.

My response I'm sorry won't be helpful, I'm hoping you may have stumbled across the answer.. I deleted my n64 setup after dealing with this problem for a while. Even after I edited the config with root access and terminal, the config always reverted to its defaults. I hope someone helps us.

Need advice, maybe I'm missing something, if so sorry. I'll try to quick summarize. Spent time setting up two controllers for every emu, my Xbox in p1, Logitech for p2. Got another controller, that is a six button, unplugging Xbox now using 6 button. Now have to remap all buttons in each emu, as well hotkeys. If I want to get into OSD or make another controller p1 (as well now remap all) I have to plug Xbox one in, go to input, change it to the other controller (as soon as I do I lose control on Xbox controller and hope I didn't hit wrong direction) and remap. Now let's say I bring this to someone else's house, use their controllers, set up in EmuSta. Now go into the emulator. The emulator buttons are configured for my controllers (to which I hope new controller has similar enough button codes that I can even get into the OSD ).
Is this really how it has to be, am I missing a per controller config option? It seems logical to me that if I start a game with a controller, that controller should be player 1 and as well have the ability to get into the OSD regardless of wether the core thinks that controller is player 1 or not. It should as well retain it's own mapping so my six button controller works as I had it set, as well my other controllers. As it seems to me, currently the input has to be set up per core, with a dedicated p1 controller mapped and if I say boot into a game with one controller unplugged (or swapped), the mappings are now wrong wrong.

Sorry for the novel. It may seem like a lot of info to say "is there an option to have each controller configured with it's own core specific setup, independent of wether it's player one or two" but I want to try to be as clear as possible. Thanks for any advice.