How to use it?

[chairy11]

Hi, thank you very much to share this project.

I have tried to install catalyst on my MacOS by anaconda or pip but failed.

I want to try to use your docker image, but how to use it?

I tried to pull image but failed. Please, help me.

$docker image pull engima-catalyst-docker

Using default tag: latest
Error response from daemon: pull access denied for engima-catalyst-docker, repository does not exist or may require 'docker login'

[pythonfortinero]

is easy git clone https://github.com/TheCraftsmen/engima-catalyst-docker.git in any path and inside de repo dir in your machine. use docker-compose up

[chairy11]

Thank you very much .

Sorry, I am new to docker.

After running(not finished yet):

$ git clone https://github.com/TheCraftsmen/engima-catalyst-docker.git

$ cd engima-catalyst-docker

$ docker-compose up

And then ? How could I use catalyst as it installed in my own computer?

I don't understand. What should I do the run catalyst?

Do you mean that I can put my strategy file in the file< /engima-catalyst-docker>, just as your <buy_and_hodl.py> and <mean_reversion_simple.py> , and then run

$ catalyst ingest-exchange -x binance -f minute -i btc_usdt --show-progress

$ python buy_and_hodl.py

[pythonfortinero]

yes, but you need to use docker-compose run bot <:your command>. by default when you make docker-compose up start a jupyter-notebook instance in the 8000 port and you can set your strategy in a notebook file like for example this https://enigma.co/catalyst/beginner-tutorial.html#running-algorithms

[chairy11]

Thank you so much. It's midnight in China now, I have to go sleep. I will try tomorrow.

Thank you again for such a great share! You are so nice!

[chairy11]

Hi, @TheCraftsmen , following your advice, I try to write code in a jupyter-notebook, but meet something wrong at the beginning:

import os
import talib as ta
import numpy as np
import pandas as pd
from collections import deque


---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-b95f33768063> in <module>
      1 import os
----> 2 import talib as ta
      3 import numpy as np
      4 import pandas as pd
      5 from collections import deque

ModuleNotFoundError: No module named 'talib'

As "engima catalyst docker image with TA-lib installed", how can I import talib?

[pythonfortinero]

Hi, is installed. if you try inside a docker-compose run bot python im sure than you can import talib. In the dockerfile you can can see the installation.

[chairy11]

even I run your example <engima-catalyst-docker/first-example.ipynb>, it throws error:

UsageError: Cell magic `%%catalyst` not found.

 docker-compose run bot catalyst --version

 catalyst, version 0.5.21

but this works.

, it throws error:

UsageError: Cell magic `%%catalyst` not found.

 docker-compose run bot catalyst --version

 catalyst, version 0.5.21

but this works.

[pythonfortinero]

okey when you make "docker-compose up". if you see the log the in the terminal you can see the url for access to the jupyter-notebook inside the docker container in this instance you can import all

Attaching to catalyst_bot_1
bot_1 | [I 03:05:42.911 NotebookApp] Serving notebooks from local directory: /code
bot_1 | [I 03:05:42.912 NotebookApp] The Jupyter Notebook is running at:
bot_1 | [I 03:05:42.912 NotebookApp] http://(b0d1ea2d301a or 127.0.0.1):8000/?token=b7b6a256b52915611fa358c9e09afdcc72d2fb7e508a0822
bot_1 | [I 03:05:42.913 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
bot_1 | [W 03:05:42.922 NotebookApp] No web browser found: could not locate runnable browser.
bot_1 | [C 03:05:42.923 NotebookApp]
bot_1 |
bot_1 | To access the notebook, open this file in a browser:
bot_1 | file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
bot_1 | Or copy and paste one of these URLs:
bot_1 | http://(b0d1ea2d301a or 127.0.0.1):8000/?token=b7b6a256b52915611fa358c9e09afdcc72d2fb7e508a0822

[chairy11]

Oh, I see, it's the first time I use jupyter notebook, so I open the jupyter by anaconda, it's wrong.

Now I use the http://localhost:8000/notebooks/first-example.ipynb, and it can run now.

But I meet another problem.

In China, many website such as Binance, Facebook, Google are not allowed to visit.
For visit Binance, I have to set proxy in my terminal like
export http_proxy=http://127.0.0.1:1087;export https_proxy=http://127.0.0.1:1087;
but it doesn't work now.

I tried to write it in the jupter

!export http_proxy=http://127.0.0.1:1087;export https_proxy=http://127.0.0.1:1087;
!catalyst ingest-exchange -x binance -f minute -i eos_usdt

but still get error

Trying to ingest exchange bundle binance...
[2019-07-09 03:49:02.654597] WARNING: CCXT: unable to fetch markets binance: binance GET https://api.binance.com/api/v1/exchangeInfo HTTPSConnectionPool(host='api.binance.com', port=443): Max retries exceeded with url: /api/v1/exchangeInfo (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f28cd590940>, 'Connection to api.binance.com timed out. (connect timeout=10)'))
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
socket.timeout: timed out


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/catalyst", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/catalyst/__main__.py", line 857, in ingest_exchange
    csv=csv
  File "/usr/local/lib/python3.6/site-packages/catalyst/exchange/exchange_bundle.py", line 811, in ingest
    self.exchange = get_exchange(self.exchange_name)
  File "/usr/local/lib/python3.6/site-packages/catalyst/exchange/utils/factory.py", line 43, in get_exchange
    exchange.init()
  File "/usr/local/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 136, in init
    raise ExchangeRequestError(error=e)
catalyst.exchange.exchange_errors.ExchangeRequestError: Request failed: binance GET https://api.binance.com/api/v1/exchangeInfo HTTPSConnectionPool(host='api.binance.com', port=443): Max retries exceeded with url: /api/v1/exchangeInfo (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f28cd590940>, 'Connection to api.binance.com timed out. (connect timeout=10)'))

Do you have any idea for this problem?

[pythonfortinero]

maybe this can help you https://docs.docker.com/network/proxy/

[chairy11]

@TheCraftsmen
Thank you very much.
I have tried for the docker proxy, but not work.
I have to give up to use it in my own computer, maybe sometime I can use catalyst in a HongKong server.

Thank you so much again.

[pythonfortinero]

if you can go yo binance.com using the https_proxy env trick in your machine. Im sure that you need some extra configuration in the dockerfile. if you can access in a browser dont give up.

[chairy11]

It's too difficult for me.

I have open a Shadowsocks proxy, and I can visit Google and Binance website by browser.

And I set the command in the mac terminal:

export http_proxy=http://127.0.0.1:1087;export https_proxy=http://127.0.0.1:1087;

I check the IP , got to know it has already set proxy right. And I can even curl google

curl cip.cc
IP	: xxx.xxx.xxx.xxx
Address: American (something like that in Chinese)


curl http://www.google.com
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script nonce="VISxuUjbmXROUydJEj+4Tg==">(function(){window.google={
...

but when I tried

 docker-compose run bot catalyst ingest-exchange -x binance -i btc_usdt

Traceback (most recent call last):
  File "/usr/local/bin/catalyst", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/catalyst/__main__.py", line 857, in ingest_exchange
    csv=csv
  File "/usr/local/lib/python3.6/site-packages/catalyst/exchange/exchange_bundle.py", line 811, in ingest
    self.exchange = get_exchange(self.exchange_name)
  File "/usr/local/lib/python3.6/site-packages/catalyst/exchange/utils/factory.py", line 43, in get_exchange
    exchange.init()
  File "/usr/local/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 136, in init
    raise ExchangeRequestError(error=e)
catalyst.exchange.exchange_errors.ExchangeRequestError: Request failed: binance GET https://api.binance.com/api/v1/exchangeInfo HTTPSConnectionPool(host='api.binance.com', port=443): Max retries exceeded with url: /api/v1/exchangeInfo (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f95372d26d8>, 'Connection to api.binance.com timed out. (connect timeout=10)'))

Do you know what happen to catalyst in anaconda?
I use to installed catalyst in another macbook successfully and easily by anaconda conda env create -f python3.6-environment.yml
But recently when I tried to install by anaconda, it always told me

catalyst: command not found 

I tried for many times uninstall and install anaconda3 and catalyst, but still failed.

So frustrated! I even don't know why. The installation used to be so easy. It's that cause by anaconda version? There's no catalyst file in <anaconda3/envs/catalyst/bin>.

[chairy11]

Hi, @TheCraftsmen , I have set the global proxy successfully and finally,

docker-compose run bot curl cip.c

IP	: 173.xxx.xxx.xxx
地址	: 美国  宾夕法尼亚州  威尔克斯－巴里 (USA)
运营商	: volumxxx.com

数据二	: 美国 | 宾夕法尼亚州拉克万纳县克拉克斯萨密特自治区VolumeDrive股份有限公司

数据三	: 美国宾夕法尼亚

URL	: http://www.cip.cc/173.xxx.xxx.xxx

but

$ docker-compose run bot catalyst ingest-exchange -x binance -i eth_usdt

Trying to ingest exchange bundle binance...
[2019-07-11 06:36:18.822564] WARNING: CCXT: unable to fetch markets binance: binance GET https://api.binance.com/api/v1/exchangeInfo HTTPSConnectionPool(host='api.binance.com', port=443): Read timed out. (read timeout=10)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
socket.timeout: _ssl.c:835: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 349, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.binance.com', port=443): Read timed out. (read timeout=10)

catalyst.exchange.exchange_errors.ExchangeRequestError: Request failed: binance GET https://api.binance.com/api/v1/exchangeInfo HTTPSConnectionPool(host='api.binance.com', port=443): Read timed out. (read timeout=10)

I follow the load_markets is throwing an "Max retries exceeded" exception #2522, and

$ pip install pyopenssl

Requirement already satisfied: pyopenssl in /Users/yebiying/anaconda3/lib/python3.7/site-packages (19.0.0)
Requirement already satisfied: cryptography>=2.3 in /Users/yebiying/anaconda3/lib/python3.7/site-packages (from pyopenssl) (2.6.1)
Requirement already satisfied: six>=1.5.2 in /Users/yebiying/anaconda3/lib/python3.7/site-packages (from pyopenssl) (1.12.0)
Requirement already satisfied: asn1crypto>=0.21.0 in /Users/yebiying/anaconda3/lib/python3.7/site-packages (from cryptography>=2.3->pyopenssl) (0.24.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /Users/yebiying/anaconda3/lib/python3.7/site-packages (from cryptography>=2.3->pyopenssl) (1.12.2)
Requirement already satisfied: pycparser in /Users/yebiying/anaconda3/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyopenssl) (2.19)

but not work.

Any way, I think the error is nothing to do with the docker. Just let you know the progress.

Thank you for your help and encourage again!