Thank you for your help.
As I said I tested a few settings with xrandr the other day and ran into the problems described above.
Today I wanted to proceed and started out with setting the external display settings in recalbox.conf. After reboot everything works now.
I wonder why the behavior was different when setting the same modes on my TV with xrandr.
Anyhow, works perfectly now! Now I can finally dive into some oldschool Gaming 
