Yeah, you should try to use RCA cable like here I guess. I think there's a problem of bandwidth : a 800*600 CRT screen have 480k pixel. A pixel use 24 bits of information.
If you want 30 images per second that is 480000 (all pixels) * 24 (bit per pixel) * 30 (images per secodns) = 345 600 000 bits per second aka 345.6 Mbps.
This is meaning you need at least a category 6a copper pair cable.
I'm making assomptions, but if it's just a cheap chineese copper pair, then you will have LOTS of white noise, because of physical stuff I don't want to bother you with.
You would have white noise in the image and in the sound (white noise in image is your classic CRT "snow", and white noise in sound is the sound from that snow screen).
In the image, it's a pixel or another that won't be displayed for 1/30 of a second, you just would notice a bad image stability/quality. But for sound you would hear that very clearly, because it's a 1D signal. This would be "either signal is ?wtf? either it's ok", instead of "just THAT pixel for a 1/30 of a second", in burst.
(sound on top is just about +1Mbps (+192Mops, but I'm really not sure here), this means 6a is still ok. 6a was made for video anyway.)
I cant edit my post above... so edited2: I just configured everything in the gui using mouse and keyboard then saved configuration. Then I moved saved configuration to the main folder and renamed it as default one and then I ran genUAE script for all my games and it now works great :)