Winsock2: How to unblock from a blocking 'connect' call?

[ElusiveTau]

I'm developing a C++ app that uses a company "Socket" library (call it XSocket). XSocket uses Winsock2 underneath the hood and the API only allows use of a blocking socket. I don't want to discuss modifying XSocket; there's no technical reason why it can't be done, red tape prevents me from doing so.

Rather, I would like to discuss how to work with blocking sockets. The app, in Model-View-Controller terms, consists of a GUI thread, and a model/controller thread. I have the model thread spawn a child thread to manage blocking socket communication with an external system (so the model thread can continue to do other work).

With regards to the child thread, one immediate issue is how to force execution to return from a blocking "connect" call.

Winsock2 documentation [1] mentions that while a blocking socket is connecting, it's put in an alertable wait state that can be interrupted using an Asynchronous Procedure Call (APC). I've successfully used APC to interrupt a blocking recv call and can probably use it again to interrupt connect. However, I was wondering if using APCs is the preferred technique since it seems hacky. I guess the only other option is to have the model (parent) thread terminate the child thread but this seems like a nuclear option.

APC doesn't seem to be a portable solution. How do folks use blocking sockets in Linux using BSD sockets? Do they typically spawn a thread to handle blocking calls? How do they interrupt that call?

Edited #1: This question has been answered here. There appears to be a race condition resulting in a pending/queued APC that wasn't triggered to exit the connect call.

I believe I also read somewhere that the BSD-sockets approach in linux is to use select().

Lots of related articles here [3]. Specifically on I/O strategy selection [4].

[1] MSDN: winsock2
https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-connect

[2] How to exit a blocking connect() call in Windows?
https://stackoverflow.com/questions/42440230/how-to-exit-a-blocking-connect-call-in-windows

[3] Winsock Programmer's FAQ
https://tangentsoft.com/wskfaq/

[4] Which I/O Strategy Should I Use
https://tangentsoft.com/wskfaq/articles/io-strategies.html

[kigar64551]

If you create a blocking socket, which is the default, then well... operations like connect(), send() and recv() are going to block the calling thread. If you want to avoid the "main" thread to get blocked, you could move the call to a separate "worker" thread, but then there still is no clean way to "abort" the operation from another thread. What you can do is setting up a time-out for these operations, so that the calling thread won't be blocked forever. Specifically, you may want to set SO_RCVTIMEO and SO_SNDTIMEO via the setsockopt() function.

According to the manpage:
https://man7.org/linux/man-pages/man7/socket.7.html

Specify the receiving or sending timeouts until reporting an error. The argument is a struct timeval. If an input or output function blocks for this period of time, and data has been sent or received, the return value of that function will be the amount of data transferred; if no data has been transferred and the timeout has been reached, then -1 is returned with errno set to EAGAIN or EWOULDBLOCK, or EINPROGRESS (for connect(2)) just as if the socket was specified to be nonblocking. If the timeout is set to zero (the default), then the operation will never timeout. Timeouts only have effect for system calls that perform socket I/O (e.g., accept(2), connect(2), read(2), recvmsg(2), send(2), sendmsg(2)) [...]

Your other option is to use non-blocking sockets, e.g., by passing the SOCK_NONBLOCK flag to socket() when creating the socket, or by using the fcntl() function with the O_NONBLOCK flag to switch an existing socket into non-blocking mode. This way, operations like connect(), send() and recv() will not block at all, but rather return immediately with an error code like EAGAIN or EWOULDBLOCK (or EINPROGRESS).

Instead of repeatedly re-trying the operation on the non-blocking socket in a loop (which is very inefficient), you would then use the poll() (or epoll) mechanism to figure out when a socket becomes ready for reading and/or writing:
https://man7.org/linux/man-pages/man2/poll.2.html

Note that poll() allows you to "monitor" multiple sockets at the same time, so you can react as soon as any of them becomes ready. Also, the poll() function has an explicit time-out parameter, so you can be sure it will return after this amount of time.

[ElusiveTau]

I've been googling around a bit and have come to the same conclusion - there is no clean way to abort from blocking (synchronous) calls. There's a floating hacky solution out there, to call closesocket, on blocking calls but the Microsoft Winsock2 document warns against doing this (specifically not to call closesocket concurrently with any other blocking winsocket2 calls).

I'm currently investigating the option to use nonblocking sockets.

[kigar64551]

I think if you want to use blocking sockets but still want to avoid “freezing” the main thread, you should use a separate worker thread to handle the blocking operations and use some sort of asynchronous communication between the worker thread and the main thread.

Essentially, calling MyConnection::connect() from the main thread will spawn the worker thread (which will then perform the actual connect() in the background) and return immediately. The main thread will later be informed that the connection has succeeded (or failed) via some sort of asynchronous “message” (e.g., using Window Messages or Qt Signals&Slots). Either that, or use some sort of polling mechanism, e.g., MyConnection::get_state(), or even a (potentially) blocking MyConnection::await_connected().

Similarly, calling MyConnection::cancel() from the main thread requests cancellation, but does not necessarily cancel the connection right away. Again, the main thread will be informed via an asynchronous “message” when the connection has actually been cancelled.

This is very similar to how TerminateProcess() requests a process to be terminated, but does not terminate the process right away. Instead, after a successful TerminateProcess() you still have to call the blocking WaitForSingleObject() (can use optional timeout) or poll the status via GetExitCodeProcess(), if you want to be sure that the process has actually terminated.

[ElusiveTau]

I've decided not to use my company's library, which just wraps Winsock2's blocking sockets calls. The main reason is that even if I moved the blocking calls to a worker thread, the worker thread would be blocked and the only way to unblock it would be to terminate the thread. setsockopt could be use to implement a timeout on blocking send/recv but there'd be no way to unblock from connect().

I utilize my company's message queue library to handle asynchronous communication between threads. Indeed, I've arranged for the main thread to call Worker::Connect(CallbackMessage* cb), which inserts a "connect request" into the worker thread's message queue, and returns almost immediately. When the worker thread processes the connect request from its message queue, it performs a truly non-blocking connect() call [1]. When connecting succeeds, it pushes CallbackMessage onto the main thread's message queue to notify main thread of successful connection. If non-blocking connect() fails (e.g., "connection refused"), the worker thread waits 1 second, and re-inserts the connect request back onto its message queue. Of course, I could elect to implement some other retry strategy (e.g., only retry N times instead of indefinitely). For my use case, I want indefinite retries until user decides to abort.

The interesting thing about using non-blocking sockets is that WSAEventSelect can be used to associate network events (FD_CONNECT and FD_CLOSE) with a non-blocking socket object and a "WSA" Event Object [2] (which is no different from a regular Event Object [2]; the WSA Event Object is atomically signaled when the network events you specified occurs). This WSA Event Object, along with any additional "regular" Event Objects can be used with WSAWaitForMultipleEvents [3] to block until any one (or optionally all) of the events are signaled. A "regular" event object ("Abort Event Object") can be defined to indicate that the user wants to abort the connect operation. Therefore, while WSAWaitForMultipleEvents suspends the worker thread waiting for a network event (FD_CONNECT), it can also be woken up if the main thread signals the Abort Event Object. I've tested this technique and it's working.

I believe the same technique can be used to implement abortable send/receive since WSAEventSelect can be used to create an event object that subscribes to FD_WRITE and FD_READ [5].

[1] MSDN: ioctlsocket (using FIONBIO command)
https://learn.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-ioctlsocket

[2]
MSDN: Event Object (Synchronization)
https://learn.microsoft.com/en-us/windows/win32/sync/event-objects

MSDN: WSACreateEvent
https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsacreateevent

[3] MSDN: WSAWaitForMultipleEvents
https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsawaitformultipleevents

[4] MSDN: WSAEnumNetworkEvents
https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsaenumnetworkevents

[5] MSDN: Notification of Network Events
https://learn.microsoft.com/en-us/windows/win32/winsock/notification-of-network-events-2