Hello,
Simple overclock of the cpu is not enough to handle the dreamcast.
With some custom overclock settings I managed to run Virtua Tennis 2 around 90-95% of its full speed with very few minor glitchs. So the game is playable and enjoyable.
A lot of games are close to work at full speed, if only the pi3 have had a bit more cpu speed.
Every pi are different, so if it crashes or displays a black screen, you must lower some values (see the range in my config.txt below).
WARNING: you REALLY need a good cooling setup (fan + heatsinks) if you don't want to grill your pi.
My settings works for me and my pi3 can stay on for hours, BUT I can not be held responsible if your pi goes down.
In config.txt of the sdcard (plug it in your computer), here are my settings:
kernel=zImage
#old extrem default values
#force_turbo=0
#arm_freq=1375
#core_freq=500
#sdram_freq=575
#over_voltage=4
#over_voltage_sdram_p=6
#over_voltage_sdram_i=4
#over_voltage_sdram_c=4
#gpu_freq=500
#sdram_schmoo=0x02000020
#my custom overclock: cpu / gpu (core & gpu freq should be equals) / sdram / voltage
#arm_freq (1400-1450)
arm_freq=1425
#core_freq & gpu_freq (500-575)
core_freq=550
gpu_freq=550
#v3d_freq (520-530) or more ? under 500 is not enough for the dreamcast
v3d_freq=530
h264_freq=333
gpu_mem=256
#sdram_freq (500-580)
sdram_freq=575
sdram_schmoo=0x02000020
#over_voltage (4-6)
over_voltage=6
force_turbo=0
over_voltage_sdram_p=6
over_voltage_sdram_i=4
over_voltage_sdram_c=4
More significant values are cpu, core_freq and v3d_freq.
Hope this help.
If anyone has a more powerful & stable setup, please share 
