offline bundling crash

[kfox1111]

Pacifica CLI Tool version

[Version of the pacifica software where you are encountering the issue]

$ docker run -it --rm pacifica/cli --version
pacifica-cli: error: unrecognized arguments: --version

Pacifica Core Software versions

NA

Platform Details

Docker

Scenario:

Bundling a file offline does not work but instead crashes.

Steps to Reproduce:

docker run -it --rm pacifica/cli upload --local-save retry.tar --do-not-upload --dry-run README

Expected Result:

A bundle to be produced

Actual Result:

$ docker run -it --rm pacifica/cli upload --local-save retry.tar --do-not-upload --dry-run README
Generating New Configuration.

Endpoints are an HTTP URL that looks similar to a website but
are designed for an uploader to interact with.

What are the endpoint URLs for the following...

Upload URL (http://ingestfrontend:8066/upload): Upload Status URL (http://ingestfrontend:8066/get_state): Upload Policy URL (http://policyserver:8181/uploader): Upload Validation URL (http://policyserver:8181/ingest): Download URL (http://cartdfrontend:8081): Download Policy URL (http://policyserver:8181/status/transactions/by_id): 
CA certificate bundle is the path to your certificate authority bundle.

Use this if you have a custom site SSL Certificate for your Site.

Valid values:
- True: verify the SSL server certificiate using system bundle
- False: do not verify the SSL server certificate (not recommended)
- a/path/to/a/cacert/bundle: custom path to the server certificate

CA Certificate Bundle (True): 
There are three kinds of authentication types supported.

- clientssl - This is where you have an SSL client key and cert
- basic     - This is a username and password
- None      - Do not perform any authentication

Authentication Type (None): WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2d46c214a8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /uploader
WARNING:urllib3.connectionpool:Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2d46c1afd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /uploader
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2d46c1aac8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /uploader
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2d46c1a780>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /uploader
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2d46c2ef60>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /uploader
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pacifica/uploader/metadata/policyquery.py", line 63, in set_user
    self.user_id = int(user)
ValueError: invalid literal for int() with base 10: 'root'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 183, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f2d46c43208>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    **response_kw)
  [Previous line repeated 2 more times]
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='policyserver', port=8181): Max retries exceeded with url: /uploader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2d46c43208>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pacifica-cli", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/pacifica/cli/__main__.py", line 144, in main
    args.func(args, config_data)
  File "/usr/local/lib/python3.6/site-packages/pacifica/cli/methods.py", line 178, in upload
    md_update = query(args, interface_data)
  File "/usr/local/lib/python3.6/site-packages/pacifica/cli/methods.py", line 192, in query
    return query_main(md_update, args)
  File "/usr/local/lib/python3.6/site-packages/pacifica/cli/query.py", line 177, in query_main
    query_obj = find_leaf_node(md_update)
  File "/usr/local/lib/python3.6/site-packages/pacifica/cli/query.py", line 30, in find_leaf_node
    md_update.update_parents(query_obj.metaID)
  File "/usr/local/lib/python3.6/site-packages/pacifica/uploader/metadata/metaupdate.py", line 69, in update_parents
    query_results=self.query_results(meta_id))
  File "/usr/local/lib/python3.6/site-packages/pacifica/uploader/metadata/metaupdate.py", line 50, in query_results
    auth=self._auth
  File "/usr/local/lib/python3.6/site-packages/pacifica/uploader/metadata/policyquery.py", line 117, in __init__
    self.set_user(user)
  File "/usr/local/lib/python3.6/site-packages/pacifica/uploader/metadata/policyquery.py", line 72, in set_user
    self.user_id = id_check.get_results()[0]['_id']
  File "/usr/local/lib/python3.6/site-packages/pacifica/uploader/metadata/policyquery.py", line 147, in get_results
    self._uploader_url, headers=headers, data=self.tojson(), **self._auth)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='policyserver', port=8181): Max retries exceeded with url: /uploader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2d46c43208>: Failed to establish a new connection: [Errno -2] Name or service not known',))

[dmlb2000]

@kfox1111 Offline bundling isn't really a thing, I'm not sure what would be gained by an offline bundle as the metadata wouldn't be resolvable. Querying more metadata wouldn't be possible either.

What's the difference between an offline bundle and simply taring up the files you've specified with any local configuration?

[kfox1111]

It use to be, but maybe not anymore, that you could bundle up something that you were going to upload, but not upload it. Say, you had an instrument that you wanted to generate the upload from that had no connectivity, take it to another machine, then finish the upload with it. Is that no longer possible? Sure you could add an extra tar/untar step in the process, but capturing the metadata wouldn't necessarily happen at the best time in that case. It might be best handled at the instrument.
It might not be possible to validate the metadata at bundle time, but upload time should be able to take care of it?

[dmlb2000]

@kfox1111 Okay I think I get what you are talking about now. A --offline option would be there and not resolve any of the metadata provided. All that would be done is take what's given on the command line put that into the metadata.txt. Then perform the local retry bundling options.