Skip to content

Broken pipe (Write failed) failures when mixed load of multipart_write, read, list, metadata, delete S3 operations using og-1.13.2 with concurrency > 6  #39

Description

@garyguo

[Issue]
I use og-1.13.2 on Linux box with AlmaLinux release 8.9. I do try to test mixed load of multipart_write, read, list, metadata, delete S3 operations using og-1.13.2. But, I find if concurrency > 6, I hit the following errors constantly like the followings.

2024-04-01 12:52:24.769 INFO  [client-3]  - I/O exception (java.net.SocketException) caught when processing request to {}->http://cxo-s3-cluster39:8443: Broken pipe (Write failed)
2024-04-01 12:52:24.769 INFO  [client-3]  - Retrying request to {}->http://cxo-s3-cluster39:8443
2024-04-01 12:52:26.819 INFO  [client-2]  - I/O exception (java.net.SocketException) caught when processing request to {}->http://cxo-s3-cluster39:8443: Broken pipe (Write failed)
2024-04-01 12:52:26.819 INFO  [client-2]  - Retrying request to {}->http://cxo-s3-cluster39:8443  

But, I don't see "Broken pipe (Write failed)" issue at all when I test the mixed load of write, read, list, metadata, delete S3 operations using og-1.13.2 with concurrency 24 even 48.

[Questions]

1. Is it the right behavior or not for mixed load of multipart_write, read, list, metadata, delete S3 operations using og? 
2. Is there any way to avoid "Broken pipe (Write failed)" issue? 

[Reference mixed load profile of og used]

[root# cat mpu_04_01_2024_100req_6t_wl_og.json
{
  "host": {
    "selection": "roundrobin",
    "choices": ["<S3 storage endpoint IP>:Port"]
  },
  "api": "s3",
  "uri_root": "/",
  "container": {
    "selection": "roundrobin",
    "prefix": "bkt-pl10a-",
    "min_suffix": 0,
    "max_suffix": 31
  },
  "multipart_write": {
     "upload": {
       "part_size": {
         "choices": [
           {
             "choice": 5242880,
	     "weight": 1
	   }
         ],
         "selection": "roundrobin"
         #"selection": "random"
      },
      "parts_per_session": {
        "choices": [],
        "selection": "roundrobin"
        #"selection": "random"
      },
      "target_sessions": 1
    },
    "weight": 100
  },
  "read": {
    "weight": 50,
    "object": {
      "selection": "roundrobin"
    }
  },
  "list": {
    "weight": 20,
    "parameters": {
      "list-type": 2
    },
    "object": {
     "selection": "random"
    }
  },
  "metadata":{
    "weight": 10,
    "object": {
      "selection": "roundrobin"
    }
  },
  "delete": {
    "weight": 50,
    "object": {
      "selection": "roundrobin"
    }
  },
  "filesize": {
    "selection": "random",
    "choices": [
      {
        "choice": {
          "distribution": "uniform",
          "average": 64,
          "average_unit": "mb",
          "spread": 5,
          "spread_unit": "mb"
      },
      "weight": 7.0
    },
    {
      "choice": {
        "distribution": "normal",
        "average": 32,
        "average_unit": "mb"
      },
      "weight": 1.0
    },
    {
      "choice": {
        "distribution": "normal",
        "average": 100,
        "average_unit": "mb"
      },
      "weight": 1.0
    }
    ]
  },
  "concurrency": {
    "type": "threads",
    "count": 6
   },
  "authentication": {
    "type": "awsv4",
    "username": "testuser",
    "password": "testuser"
  },
  "stopping_conditions": {
     "operations": 100
  },
  "client": {
    "connect_timeout": 30000,
    "so_timeout": 30000,
   "retry_count": 40
  }
}

Thank you very much.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions