Why does the execution time of each goroutine increase as the number of goroutines increases？

[lddlww]

i found when increase client concurrency to read a some object from s3 like minio，the time consumed by each coroutine will increase exponentially， and i profiling it and found the issue is stuck when the server returns data to the client and there are no bottlenecks on either the server or the client side（CPU，MEMORY，DISK，NETWORK）

i test it and reproduce by follow step：

test server code：

package main

import (
    "bytes"
    "fmt"
    "math/rand"
    "net"
    "net/http"
    _ "net/http/pprof"
    "time"
    "io"
)

const dataSize = 32 * 1024 * 1024 // 32 MiB

func main() {
    go func() {
        fmt.Println("pprof listening on :6060")
        http.ListenAndServe("10.58.3.23:6060", nil)
    }()

    ln, err := net.Listen("tcp", "10.58.3.23:9090")
    if err != nil {
        panic(err)
    }
    fmt.Println("Server listening on :9090")

    data := make([]byte, dataSize)
    rand.Read(data)

    for {
        conn, err := ln.Accept()
        if err != nil {
            fmt.Println("Accept error:", err)
            continue
        }
        go handleConn(conn, data)
    }
}

func handleConn(conn net.Conn, data []byte) {
    defer conn.Close()

    fmt.Println("Client connected:", conn.RemoteAddr())
    start := time.Now()

    r := bytes.NewReader(data)
    _, err := io.Copy(conn, r)
    if err != nil {
        fmt.Println("Write error:", err)
        return
    }

    duration := time.Since(start)
    fmt.Printf("Sent %d bytes to %v, duration: %v\n", dataSize, conn.RemoteAddr(), duration)
}

test client code:

package main

import (
    "fmt"
    "io"
    "net"
    "sync"
    "time"
)

const concurrency = 4 
func main() {
    var wg sync.WaitGroup

    for i := 0; i < concurrency; i++ {
        wg.Add(1)
        go func(id int) {
            defer wg.Done()
            start := time.Now() 

            conn, err := net.Dial("tcp", "10.58.3.23:9090")
            if err != nil {
                fmt.Printf("[Worker %d] Dial error: %v\n", id, err)
                return
            }
            defer conn.Close()
	    buf := make([]byte, 1*1024*1024) // 1 MiB buffer
            total := 0
            for {
                n, err := conn.Read(buf)
                total += n
                if err == io.EOF {
                    break
                }
                if err != nil {
                    fmt.Printf("[Worker %d] Read error: %v\n", id, err)
                    return
                }
            }
            duration := time.Since(start)
            fmt.Printf("[Worker %d] Finished reading %d bytes, duration: %v\n", id, total, duration)
        }(i + 1)
    }

    wg.Wait()
    fmt.Println("All goroutines completed")
}

when increase client concurrency , the time consumption of a single coroutine increases ， from 59ms（1 concurrency）->110ms（2 concurrency）->200ms（4 concurrency）

what's wrong with it ， does anything ideas？

[Xuyuchao-juice]

There are some directions:

1. Try to increase the TCP receive buffer by using setsockopt.
2. Try to increase the client read buffer size: from 1M to 4M or larger