@Zing Thanks for the reply but I think I managed to got it to work now.
I installed the newest stable bootloader instead of a beta one which made the pi boot fast in 50% of the boots, then I set the console parameter in the cmdline.txt to tty1 to see if I can notice a pattern and saw that there is some error with the /dev directory ("udevadm settle failed") every time it boots slow and that the SSD is already in use by another process or something like that.
Next I disabled WIFI and Updates and removed the "fastboot" parameter from the "cmdline.txt" and now it's working.
So I'm not sure if it was the bootloader or the fastboot or a combination of everything but it boots in roughly 20 seconds now.
I'm not using a case and my power supply is the official one for the pi4 btw.
