Cygwin: open: Fix deadlock for opening fifo both side in a process

Currently, opening both side of fifo in a process hangs if the
read side is opened first. The following test case exhibit the
hang while it works in linux.

  #include <unistd.h>
  #include <pthread.h>
  #include <sys/stat.h>
  #include <fcntl.h>

  #define fifo1 "/tmp/fifo-test"

  void *thr1(void *)
  {
    int fd;
    usleep(100000);
    fd = open(fifo1, O_WRONLY);
    write(fd, "A", 1);
    usleep(100000);
    close(fd);
    return NULL;
  }

  int main()
  {
    int fd;
    pthread_t th;
    char c;
    mkfifo(fifo1, 0600);
    pthread_create(&th, NULL, thr1, NULL);
    fd = open(fifo1, O_RDONLY);
    pthread_join(th, NULL);
    read(fd, &c, 1);
    write(1, &c, 1);
    close(fd);
    unlink(fifo1);
    return 0;
  }

The mechanism of hang is as follows. The main thread tries to open
the fifo for reading, but fhandler_fifo::open blocks until it detects
that someone is opening the fifo for writing. The other thread wants
to do that, but it never gets to the point of calling fhandler_fifo::
open because it is stuck waiting for the lock on cygheap->fdtab.

To fix this, this patch delays the construction of the cygheap_fdnew
object fd until after fhandler_fifo::open has been called.

Fixes: df63bd4 ("* cygheap.h (cygheap_fdmanip): New class: simplifies locking and retrieval of fds from cygheap->fdtab.")
Reviewd-by: Ken Brown <[email protected]>
Signed-off-by: Takashi Yano <[email protected]>
(cherry picked from commit cec8a66)

Author: tyan0

winsup/cygwin/release/3.6.2

@@ -22,3 +22,6 @@ Fixes:
 
 - Fix conserver-config error.
   https://cygwin.com/pipermail/cygwin/2025-April/258086.html
+
+- Fix deadlock for opening both side of a fifo in a process.
+  Addresses: https://cygwin.com/pipermail/cygwin/2025-May/258138.html

winsup/cygwin/syscalls.cc

@@ -1472,11 +1472,6 @@ open (const char *unix_path, int flags, ...)
       mode = va_arg (ap, mode_t);
       va_end (ap);
 
-      cygheap_fdnew fd;
-
-      if (fd < 0)
-	__leave;		/* errno already set */
-
       /* When O_PATH is specified in flags, flag bits other than O_CLOEXEC,
 	 O_DIRECTORY, and O_NOFOLLOW are ignored. */
       if (flags & O_PATH)
@@ -1577,6 +1572,12 @@ open (const char *unix_path, int flags, ...)
       if ((flags & O_TMPFILE) && !fh->pc.isremote ())
 	try_to_bin (fh->pc, fh->get_handle (), DELETE,
 		    FILE_OPEN_FOR_BACKUP_INTENT);
+
+      cygheap_fdnew fd;
+
+      if (fd < 0)
+	__leave;		/* errno already set */
+
       fd = fh;
       if (fd <= 2)
 	set_std_handle (fd);