I didn't get if you are also having problems with your xbox controller or not. But back to the ps3, you can't activate the xbox driver and expect the ps3 driver to work. That is clearly stated on recalbox.conf. Have you tried deactivating the xbox driver, and activating (the right) ps3 driver?
Just to be clear... don't except both activated drivers to coexist.
If that doesn't help you... buying a compatible controller is always a good idea 😉
