I figured it out.
My specific issue was because I'm using the Retroflag GPi case. The case's screen requires a patch to be installed to use Retropie. I forgot about it because the Recalbox has an install package specifically for the GPi case (which is super nice) and it just works with the screen/buttons. I've re-flashed Recalbox a few times and I forgot to manually reinstall the patch when switching back to RetroPie.