Even on 4.1 Xbox one gamepads only works on USB.
But Microsoft modifies the gamepad's firmware each time he products a new gamepad generation... Even on the same console...
So new Xbox one gamepads aren't supported (even on USB) on 4.0.x recalbox version.
On 4.1 I b*mped the driver, so your gamepads (new and old) should be supported on 4.1 (if Microsoft didn't release a new gamepad with a new firmware version...)