New girder_worker release?

[craig-willis]

Running into problems locally that I'm not certain are related, but noticed that two new versions of the girder_worker package on pypi were released yesterday https://pypi.org/project/girder-worker/#history

[Xarthisius]

What kind of problems?

[craig-willis]

I'm still trying to pinpoint, but the original symptom was the following trace in my task:

[2019-03-09 14:17:24,030: INFO/ForkPoolWorker-1] GIRDER_API_URL http://10.0.0.5:8080/api/v1
[2019-03-09 14:17:24,034: ERROR/ForkPoolWorker-1] Task gwvolman.tasks.build_tale_image[ba7a66ba-a058-4095-9733-fd8a5a3c0355] raised unexpected: ValueError("Error accessing Girder: 'NoneType' object has no attribute 'get'",)
Traceback (most recent call last):
  File "/gwvolman/gwvolman/tasks.py", line 292, in build_tale_image
    user = self.girder_client.get('/user/me')
AttributeError: 'NoneType' object has no attribute 'get'

I traced this back to (I think):

https://github.com/girder/girder_worker/blob/7a590f6f67230e2f98e8acecd313f00d76bdbf00/girder_worker/app.py#L140

But I can construct GirderClient in the running container without a problem:

>>> from girder_client import GirderClient
>>> gc = GirderClient(apiUrl='http://10.0.0.5:8080/api/v1')
>>> print (gc.get('/user/me'))
None

I removed all images on my dev instance and just redeployed (make dev) and am seeing

VersionMismatch: Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6

Which led me to celery/celery#5369.

Stumped

[craig-willis]

Unpinning redis in gwvolman requirements.txt resolves the VersionMismatch error, but I'm back to the original error

[Xarthisius]

That girder/girder_worker#311 looks like a possible breaking change, since we're still using girder 2.5.0 and worker plugin is a part of it. I'd pin the version of girder-worker to 0.5.0 for now.

[craig-willis]

Thanks again for your help. In the end my problem was user error. The error I saw in the gwvolman logs were related to an error that I wasn't paying attention to in the girder logs. I didn't realize that there could be a stack trace in girder for my task.

Did you run into the redis error at all? With redis:latest in deploy-dev, I had to unpin redis in the gwvolman requirements.txt due to the issue above. I expect it would be best to pin the redis image to something other than latest in deployment.

[Xarthisius]

I didn't run into problem with redis because I forgot to pull new image... Agreed about pinning redis image.