@Duske I can't be sure but I have two official Raspberry Pi 4 power supplies and experienced the problem on both. I also tried using my own SATA>USB adapter without any problem. Again, I can't be sure, but from what I tried, the most likely is that there is some 'kinky' stuff with this Retroflag adapter.
Isn't running everything from the SSD too much of a trouble if you have to start your Recalbox from scratch? How do you flash it from an img file?
@Adrian-Nawrocki Yes, like I said couple of hours ago, switching USB3 to USB2 solved the problem which is quite surprising though.