I'm failling to pair my XBOX one controller with a RPI4 as well. Using Recalbox 7.2 (version 8 does not work for me, black screen, seems not to boot). The controller shows up during BT devices discovery but the pairing fails.
I used retropie with xpadneo driver before, which worked well.