Update: I ended up swapping the encoder boards from the working set to the non-working set, kept everything else the same (buttons, joystick, ect). Issue has gone away, so it was the encoder boards.
I now have the working set on the Raspberry Pi 4, and the non-working set on another control board that's connected to a PC running Recalbox. It doesn't have any issues on the PC, so its something with those particular boards and the Pi.
I'd still like to know what the issue could be. Here's the joysticks I use:
Non-working set: https://www.amazon.com/dp/B07QRRLCKD?psc=1&ref=ppx_yo2ov_dt_b_product_details
Working set: https://www.amazon.com/dp/B07JFXQSM5?psc=1&ref=ppx_yo2ov_dt_b_product_details