I also had problems with the PSP emulator on RPi3, and I've seen other reports about that, the "solution" I found was:
- If you're using a 1080p theme, you need to change to 720p
- In addition, in the folder "\share\system.emulationstation", in the es_settings.cfg file, find the line "<int name =" MaxVRAM "value =" 200 "/>", and change the value from 200 to 80, getting : <int name = "MaxVRAM" value = "80" />
- Go to the folder \share\system, in the file recalbox.conf, find the line "dreamcast.videomode =" and add just below it the line: "psp.videomode = default" (without quotation marks, obvious)
With this, the PSP emulator is able to work most of the time, and if the game does not enter immediately, just wait a bit, and try again, the game will start.
I know some people do not like this solution, but that's what solved for me.