Hi there, I had the same problem as you. I couldn't get the Pi3 composite output working with overscan for Recalbox. I've spent quite a while figuring this out on my Pi3 but I think I've got it.
It may be contradictory in some places, but please set your config.txt in the following way, making sure that each line is in this exact order:
[pi3]
# Raise the first security limit up to 70 instead of 60 for pi3
temp_soft_limit=70
enable_tvout=1
#NTSC-480i
sdtv_mode=0
#NTSC-240p
#sdtv_mode=16
#PAL-576i
#sdtv_mode=2
#PAL-256p
#sdtv_mode=18
#4:3
sdtv_aspect=1
#16:9
#sdtv_aspect=3
disable_overscan=1 #this is on purpose
hdmi_force_hotplug=0
hdmi_ignore_hotplug=1
dtoverlay=vc4-fkms-v3d #fkms not kms
overscan_top=32
overscan_bottom=32
overscan_left=32
overscan_right=32
include crt/recalbox-crt-config.txt
Uncomment (delete #) from the start of the line for the resolution and mode you need. It is also important that you have 'fkms' instead of 'kms'. You will likely need to adjust the overscan values to the desired amounts after testing a few times. For me (Sony Trinitron) I used -14, -5, 13, 25 respectively. You can also use overscan_scale=0 instead of the four individual values and see if that's better for you.
Additionally, you have to edit the file cmdline.txt. The entry is a single line and must stay that way. At the end of the existing text, add:
video=Composite-1:720x480@60ie
for NTSC-480i.
For NTSC-240p you will need to keep 720x480 but the end should say @60e (no i means no interlacing). Setting 720x240p
Likewise, for PAL-576i, it's 720x576@50ie. For PAL-256p it's 720x576@50e.
I hope this helps you and anyone else with this issue.
P.S. I think updating Recalbox will reset all this. Might be better to put it into recalbox-crt-config.txt but I'll cross that bridge when I get to it.
