[QA][k8s] test fails when deleting user

[nirajacharya2]

JIRA: OCISDEV-689

for some reason test fails when deleting user. find why it fails and fix it

    Examples:
      | space-name | file-content                                                                                         | state    | remaining     | used | quota         |
      | Quota1%    | 1                                                                                                    | normal   | 99            | 1    | 100           |
      │
      ╳  Failed to delete user 'Alice'
      ╳  Failed asserting that 500 matches expected 204.
      │
      └─ @AfterScenario # FeatureContext::afterScenario()
      | Quota75%   | 123456789 123456789 123456789 123456789 123456789 123456789 123456789 12345                          | normal   | 25            | 75   | 100           |
        Failed step: Given user "Alice" has been created with default attributes
        Provisioning::userHasBeenCreated cannot create user 'Alice' using Graph API.
        Response:{"error":{"code":"nameAlreadyExists","innererror":{"date":"2026-02-12T00:37:16Z","request-id":"apiSpaces\/quota.feature:69-14"},"message":"a user with that name already exists"}}
        Failed asserting that 409 matches expected 201.

[nirajacharya2]

it happens when deleting user when sharing service is not running

{
    "level": "error",
    "service": "graph",
    "request-id": "apiSharingNgLinkShareManagement/writableShareEnforcePassword.feature:107-82",
    "message": "gateway could not delete shares of StorageSpace:error listing shares: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.42.0.10:9150: connect: no route to host\"",
    "code": 15,
    "time": "2026-02-12T09:55:26Z",
    "message": "could not disable home space: unexpected status code"
}

[nirajacharya2]

/healthz and /readyz both return 200 but this error still persist

{
"level": "error",
"service": "graph",
"request-id": "apiSharingNgLinkShareManagement/writableShareEnforcePassword.feature:107-82",
"message": "gateway could not delete shares of StorageSpace:error listing shares: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.42.0.10:9150: connect: no route to host"",
"code": 15,
"time": "2026-02-12T09:55:26Z",
"message": "could not disable home space: unexpected status code"
}

[2403905]

It is because the DeleteStorageSpace() function is performed RemoveShare
The sharing service must be runnig.

[nirajacharya2]

It is because the DeleteStorageSpace() function is performed RemoveShare The sharing service must be runnig.

we do multiple checks to see if services are running

1. check if pod state is 'Running'
2. check /healthz and /readyz both return 200.

and we still get this error. what else can we do to avoid this?

[2403905]

There are a lot of attempts to run a sharing service in a Kubernetes
The restart happened 125 times
https://drone.owncloud.com/owncloud/ocis/51403/44/2

ocis          0s          Normal    Created                             pod/sharing-77d5fb4b7b-9k6cv                                     Created container sharing
ocis          0s          Normal    Started                             pod/sharing-77d5fb4b7b-9k6cv                                     Started container sharing
ocis          0s          Normal    Killing                             pod/sharing-7d7f97944f-8hswd                                     Stopping container sharing
ocis          0s          Normal    SuccessfulDelete                    replicaset/sharing-7d7f97944f                                    Deleted pod: sharing-7d7f97944f-8hswd
ocis          0s          Normal    SuccessfulCreate                    replicaset/sharing-7d47555748                                    Created pod: sharing-7d47555748-zj2r9
ocis          0s          Normal    Scheduled                           pod/sharing-7d47555748-zj2r9                                     Successfully assigned ocis/sharing-7d47555748-zj2r9 to k3d-drone-server-0
ocis          0s          Normal    Pulled                              pod/sharing-7d47555748-zj2r9                                     Container image "owncloud/ocis-rolling:master" already present on machine
ocis          0s          Normal    Created                             pod/sharing-7d47555748-zj2r9                                     Created container sharing
ocis          0s          Normal    Started                             pod/sharing-7d47555748-zj2r9                                     Started container sharing
ocis          0s          Normal    SuccessfulDelete                    replicaset/sharing-77d5fb4b7b                                    Deleted pod: sharing-77d5fb4b7b-9k6cv
ocis          0s          Normal    Killing                             pod/sharing-77d5fb4b7b-9k6cv                                     Stopping container sharing

[saw-jan]

Sometimes, the restarted service grpc is not reachable even when /healthz and /readyz returns OK.

{
  "error": {
    "code":"generalException",
    "innererror": {
      "date":"2026-03-16T10:39:17Z",
      "request-id":"apiSharingNgLinkShareManagement/passwordPolicy.feature:90-94"
    },
    "message":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.42.0.57:9150: connect: no route to host\""
  }
}

[saw-jan]

Getting this error while deleting user even after the wrapper says the sharing service (:9150) is healthy and ready:

2026/03/17 09:36:07 [ociswrapper] [sharing] Waiting for service to be ready...
2026/03/17 09:36:10 [ociswrapper] [sharing] gRPC service is ready. Pod: sharing-644bc8b89c-wwfs9
2026/03/17 09:36:13 [ociswrapper] [sharing] Service is healthy and ready. Pod: sharing-644bc8b89c-wwfs9

2026:09:36:13 {"level":"error","service":"storage-shares","pkg":"rgrpc","traceid":"3345f7a37be6d1bcaf68a413be083680","user-agent":"GuzzleHttp/7","from":"tcp://10.42.0.38:41570","uri":"/cs3.storage.provider.v1beta1.SpacesAPI/ListStorageSpaces","start":"17/Mar/2026:09:36:13 +0000","end":"17/Mar/2026:09:36:13 +0000","time_ns":146020,"code":"Unavailable","time":"2026-03-17T09:36:13Z","message":"sharesstorageprovider: error calling ListReceivedSharesRequest: sharesstorageprovider: error calling ListReceivedSharesRequest: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.42.0.144:9150: connect: connection refused\""}
2026:09:36:13 {"level":"error","service":"storage-shares","pkg":"rgrpc","traceid":"ebd7a0981deb8fbbe0af0e49bb884670","user-agent":"GuzzleHttp/7","from":"tcp://10.42.0.38:41570","uri":"/cs3.storage.provider.v1beta1.SpacesAPI/ListStorageSpaces","start":"17/Mar/2026:09:36:13 +0000","end":"17/Mar/2026:09:36:13 +0000","time_ns":131167,"code":"Unavailable","time":"2026-03-17T09:36:13Z","message":"sharesstorageprovider: error calling ListReceivedSharesRequest: sharesstorageprovider: error calling ListReceivedSharesRequest: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.42.0.144:9150: connect: connection refused\""}
2026:09:36:14 {"level":"error","service":"storage-shares","pkg":"rgrpc","traceid":"ea46dd240a72f4f77b4ee37c3e968e02","user-agent":"grpc-go/1.79.1","from":"tcp://10.42.0.38:41570","uri":"/cs3.storage.provider.v1beta1.SpacesAPI/ListStorageSpaces","start":"17/Mar/2026:09:36:14 +0000","end":"17/Mar/2026:09:36:14 +0000","time_ns":156535,"code":"Unavailable","time":"2026-03-17T09:36:14Z","message":"sharesstorageprovider: error calling ListReceivedSharesRequest: sharesstorageprovider: error calling ListReceivedSharesRequest: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.42.0.144:9150: connect: connection refused\""}
2026:09:36:14 {"level":"error","service":"graph","request-id":"apiSharingNgLinkShareManagement/passwordPolicy.feature:38-48","message":"gateway could not delete shares of StorageSpace:gateway: error calling ListShares: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.42.0.144:9150: connect: connection refused\"","code":15,"time":"2026-03-17T09:36:14Z","message":"could not disable home space: unexpected status code"}