URLSessionWebSocketTask sometimes only connects after some retries - swift

Question

I don't have a consistent reproduction path. I am creating a plain URLSessionWebSocketTask and try to connect to my server running on localhost.
Sometimes it won't connect instant, but after a few tries it succeed. I don't get any connecting logging on the server side when the error loggings appear in the app.
These are the logs:
[tcp] tcp_input [C9.1.1:2] flags=[R.] seq=0, ack=2805414308, win=0 state=SYN_SENT rcv_nxt=0, snd_una=2805414307
[tcp] tcp_input [C9.1.1:2] flags=[R.] seq=0, ack=2805414308, win=0 state=SYN_SENT rcv_nxt=0, snd_una=2805414307
Connection 9: received failure notification
Connection 9: failed to connect 1:61, reason -1
Connection 9: encountered error(1:61)
Task <F324C4E7-092F-479B-98C2-5FFE835A8C6A>.<10> HTTP load failed, 0/0 bytes (error code: -1004 [1:61])
Task <F324C4E7-092F-479B-98C2-5FFE835A8C6A>.<10> finished with error [-1004] Error Domain=NSURLErrorDomain Code=-1004 "Could not connect to the server." UserInfo={NSErrorFailingURLStringKey=http://mac-studio.local:3012/, NSErrorFailingURLKey=http://mac-studio.local:3012/, _NSURLErrorRelatedURLSessionTaskErrorKey=(
"LocalWebSocketTask <F324C4E7-092F-479B-98C2-5FFE835A8C6A>.<10>"
), _NSURLErrorFailingURLSessionTaskErrorKey=LocalWebSocketTask <F324C4E7-092F-479B-98C2-5FFE835A8C6A>.<10>, NSLocalizedDescription=Could not connect to the server.}
[tcp] tcp_input [C10.1.1:2] flags=[R.] seq=0, ack=836765050, win=0 state=SYN_SENT rcv_nxt=0, snd_una=836765049
[tcp] tcp_input [C10.1.1:2] flags=[R.] seq=0, ack=836765050, win=0 state=SYN_SENT rcv_nxt=0, snd_una=836765049
It retries every second to connect. After the 10th attempt or so, it is connected and remains connected without any problems. Sometimes I do not have any problem and it connect instant.
Is there any property I can use to make sure it will try to connect somewhat longer?