test_globus failing in Unified RC for Compy

[forsyth2]

test_globus fails in Unified RC for Compy:

$ python -m unittest tests/test_globus.py
****************************************
testLsGlobus
****************************************
*** Setup ***
Creating files.
*** Adding files to HPSS ***
+ zstash create --cache=zstash --hpss=globus://ddb59aef-6d04-11e5-ba46-22000b92c6ec/~/zstash_test/ zstash_test

The test hangs at this point. In Globus, the following appear on the "Activity" page:

[zstash_test index](https://app.globus.org/activity/cafae164-6a95-11ed-8fd2-e9cb7c15c7d2)
transfer canceled at <10% completed

and

[zstash_test 000000](https://app.globus.org/activity/c90507e0-6a95-11ed-8fd2-e9cb7c15c7d2)
transfer canceled at <10% completed

[forsyth2]

Case 1: Above description

If I terminate the tasks on the Globus page, the test concludes with:

======================================================================
FAIL: testLs (tests.test_globus.TestGlobus)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/qfs/people/fors729/zstash/tests/test_globus.py", line 183, in testLs
    self.helperLsGlobus(
  File "/qfs/people/fors729/zstash/tests/test_globus.py", line 170, in helperLsGlobus
    self.create(use_hpss, zstash_path, cache=self.cache)
  File "/qfs/people/fors729/zstash/tests/base.py", line 287, in create
    self.check_strings(cmd, output + err, expected_present, expected_absent)
  File "/qfs/people/fors729/zstash/tests/base.py", line 182, in check_strings
    self.stop(error_message)
  File "/qfs/people/fors729/zstash/tests/base.py", line 140, in stop
    self.fail(error_message)
AssertionError: Command=`zstash create --cache=zstash --hpss=globus://ddb59aef-6d04-11e5-ba46-22000b92c6ec/~/zstash_test/ zstash_test`. Errors=['This was not supposed to be found, but was: ERROR.']

----------------------------------------------------------------------
Ran 1 test in 76.179s

FAILED (failures=1)

Case 2: Running outside zstash tests

I tried running zstash create --cache=zstash --hpss=globus://ddb59aef-6d04-11e5-ba46-22000b92c6ec/~/compy_zstash test_dir directly. Here, the same errors occur on the Globus page, but lines like the following print immediately. (In the above case, they printed after terminating the tasks on the Globus page).

INFO: TransferClient.task_wait(45ca89dc-6b5c-11ed-b0bd-bfe7e7197080, 20, 20)
INFO: TransferClient.get_task(45ca89dc-6b5c-11ed-b0bd-bfe7e7197080, ...)

Those lines continue and once I terminate the tasks on the Globus page I get ERROR: Transfer FAILED.

Case 3: Running on previous Unified

I tried running python -m unittest tests/test_globus.py with e3sm_unified_1.7.1_nompi rather than e3sm_unified_1.8.0rc5_nompi. Identical to Case 1. The test hangs and once I terminate the tasks on the Globus page, I get the test failure.

[forsyth2]

I notice in https://github.com/E3SM-Project/zstash/blob/main/zstash/globus.py that we have:

regex_endpoint_map = {
    r"theta.*\.alcf\.anl\.gov": "08925f04-569f-11e7-bef8-22000b9a448b",
    r"blueslogin.*\.lcrc\.anl\.gov": "61f9954c-a4fa-11ea-8f07-0a21f750d19b",
    r"chr.*\.lcrc\.anl\.gov": "61f9954c-a4fa-11ea-8f07-0a21f750d19b",
    r"cori.*\.nersc\.gov": "9d6d99eb-6d04-11e5-ba46-22000b92c6ec",
    r"perlmutter.*\.nersc\.gov": "6bdc7956-fc0f-4ad2-989c-7aa5ee643a79",  # If this doesn't work, use cori
}

Similarly, in https://github.com/E3SM-Project/zstash/blob/main/tests/test_globus.py we have:

regex_endpoint_map = {
    r"theta.*\.alcf\.anl\.gov": "08925f04-569f-11e7-bef8-22000b9a448b",
    r"blueslogin.*\.lcrc\.anl\.gov": "61f9954c-a4fa-11ea-8f07-0a21f750d19b",
    r"chr.*\.lcrc\.anl\.gov": "61f9954c-a4fa-11ea-8f07-0a21f750d19b",
    r"cori.*\.nersc\.gov": "9d6d99eb-6d04-11e5-ba46-22000b92c6ec",
    r"perlmutter.*\.nersc\.gov": "6bdc7956-fc0f-4ad2-989c-7aa5ee643a79",  # If this doesn't work, use cori
}

Neither of these include Compy.

[forsyth2]

Unfortunately, even if I add r"compy-*\.pnl\.gov": "68fbd2fa-83d7-11e9-8e63-029d279f7e24", to both of those dictionaries, it doesn't change case 1.

[lukaszlacinski]

Can you check transfer events in the Globus Web App? The destination endpoint used in the test/test_globus.pt is the "Globus Tutorial Endpoint 1" that has a very small disk quota. Maybe the test transfer gets "disk quota exceeded" error and Globus tries to resume the transfer after a 5-minute delay.

[forsyth2]

I'm not seeing a "disk quota exceeded" error. The event log for "zstash_test index" from my 11/23 run shows "file not found" followed by:

{
  "context": [
    {
      "operation": "Directory List / File Scan",
      "path": "/qfs/people/fors729/zstash/zstash_test/zstash/index.db"
    }
  ],
  "error": {
    "body": "550-GlobusError: v=1 c=PATH_NOT_FOUND%0D%0A550-GridFTP-Errno: 2%0D%0A550-GridFTP-Reason: System error in stat%0D%0A550-GridFTP-Error-String: No such file or directory%0D%0A550 End.%0D%0A",
    "code": 550,
    "endpoint": "pic#compy-dtn (68fbd2fa-83d7-11e9-8e63-029d279f7e24)",
    "server": "gsiftp://compy-dtn.pnl.gov:2811",
    "type": "FTPServerError"
  }
}

[lukaszlacinski]

When you log into Compy can you see that file (/qfs/people/fors729/zstash/zstash_test/zstash/index.db)? Can you see that file when you access Compy via the Globus endpoint (pic#compy-dtn (68fbd2fa-83d7-11e9-8e63-029d279f7e24)) in the Globus Web App?

[forsyth2]

can you see that file (/qfs/people/fors729/zstash/zstash_test/zstash/index.db)?

Yes, while the test is running. (When the test closes out, it deletes that directory. So, it wasn't there when I checked, but after rerunning the test today, it showed up).

Can you see that file when you access Compy via the Globus endpoint (pic#compy-dtn (68fbd2fa-83d7-11e9-8e63-029d279f7e24)) in the Globus Web App?

No, under File Manager for /qfs/people/, I get "This folder is empty."

Would running from a log-in node on Compy cause this issue? (I'm not sure how to ssh in to the Compy DTN).

[lukaszlacinski]

Maybe the home directories has be unmounted and not mounted back on the DTN nodes by a mistake. Or the home directories are not supposed to be visible by in the Globus Web App and have not been mounted on purpose. Maybe only project or job directories are accessible. Then the test has to be run in those directories. (Unfortunately, I do not have access to Compy).

[forsyth2]

Ah, that does indeed appear to be the issue. The test passes if I run it from /compyfs/dtn/fors729/zstash (still on a log-in node). I suppose that's sufficient. Thanks!