Simple overclock of the cpu is not enough to handle the dreamcast.
With some custom overclock settings I managed to run Virtua Tennis 2 around 90-95% of its full speed with very few minor glitchs. So the game is playable and enjoyable.
A lot of games are close to work at full speed, if only the pi3 have had a bit more cpu speed.
Every pi are different, so if it crashes or displays a black screen, you must lower some values (see the range in my config.txt below).
WARNING: you REALLY need a good cooling setup (fan + heatsinks) if you don't want to grill your pi.
My settings works for me and my pi3 can stay on for hours, BUT I can not be held responsible if your pi goes down.
In config.txt of the sdcard (plug it in your computer), here are my settings:
#old extrem default values
#my custom overclock: cpu / gpu (core & gpu freq should be equals) / sdram / voltage
#core_freq & gpu_freq (500-575)
#v3d_freq (520-530) or more ? under 500 is not enough for the dreamcast
More significant values are cpu, core_freq and v3d_freq.
Hope this help.
If anyone has a more powerful & stable setup, please share