@thedude-0
Update: I ended up swapping the encoder boards from the working set to the non-working set, kept everything else the same (buttons, joystick, ect). Issue has gone away, so it was the encoder boards.
I now have the working set on the Raspberry Pi 4, and the non-working set on another control board that's connected to a PC running Recalbox. It doesn't have any issues on the PC, so its something with those particular boards and the Pi.
I'd still like to know what the issue could be. Here's the joysticks I use:
Non-working set: https://www.amazon.com/dp/B07QRRLCKD?psc=1&ref=ppx_yo2ov_dt_b_product_details
Working set: https://www.amazon.com/dp/B07JFXQSM5?psc=1&ref=ppx_yo2ov_dt_b_product_details

@eroxx
Hello
There is a file located in hidden folder :
es_input.cfg

May be you can copy it and try if it helps.

@Zing thank you so much for your time and help. I've ended up going for a new install again. I cannot say for sure what caused the issue, however after a clean install everything went back to normal.

John