Merge pull request #1970 from Drakkar-Software/dev

Master merge

Author: GuillaumeDSM

.github/workflows/docker.yml

@@ -121,7 +121,7 @@ jobs:
         if: github.event_name != 'push'
         run: |
           echo Start OctoBot docker with branche_name tentacles package
-          docker run -id -e TENTACLES_PACKAGES_SOURCE=dev -e TENTACLES_URL_TAG=${GITHUB_HEAD_REF////_} -e DISABLE_SENTRY=True --name ${{ env.CONTAINER_NAME }} ${{ env.IMAGE }}:${{ env.LATEST }}
+          docker run -id -e TENTACLES_REPOSITORY=dev-tentacles -e TENTACLES_URL_TAG=${GITHUB_HEAD_REF////_} -e DISABLE_SENTRY=True --name ${{ env.CONTAINER_NAME }} ${{ env.IMAGE }}:${{ env.LATEST }}
           sleep ${{ env.CHECK_TENTACLE_CONTAINER_TIME }}
           if docker logs ${{ env.CONTAINER_NAME }} | grep "octobot_tentacles_manager.api.util.tentacles_management Failed to download file at url :" ; then
               docker rm -f ${{ env.CONTAINER_NAME }}

CHANGELOG.md

@@ -6,6 +6,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 *It is strongly advised to perform an update of your tentacles after updating OctoBot. (start.py tentacles --install --all)*
 
+## [0.4.4] - 2022-06-01
+### Added
+- [Exchanges] Future trading engine
+- [TradingModes] 
+  - Scripted trading mode bases
+  - Copy trader bases
+
+### Fixed
+- [WebInterface] Security issue
+
 ## [0.4.3] - 2021-11-23
 ### Added
 - [Trading Modes] Add buy volume parameters

README.md

@@ -1,18 +1,18 @@
-# OctoBot [0.4.3](https://github.com/Drakkar-Software/OctoBot/tree/dev/CHANGELOG.md)
-[![PyPI](https://img.shields.io/pypi/v/OctoBot.svg)](https://pypi.python.org/pypi/OctoBot/)
+# OctoBot [0.4.4](https://octobot.click/gh-changelog)
+[![PyPI](https://img.shields.io/pypi/v/OctoBot.svg)](https://octobot.click/gh-pypi)
 [![Codacy Badge](https://api.codacy.com/project/badge/Grade/e07fb190156d4efb8e7d07aaa5eff2e1)](https://app.codacy.com/gh/Drakkar-Software/OctoBot?utm_source=github.com&utm_medium=referral&utm_content=Drakkar-Software/OctoBot&utm_campaign=Badge_Grade_Dashboard)[![Downloads](https://pepy.tech/badge/octobot/month)](https://pepy.tech/project/octobot)
-[![Dockerhub](https://img.shields.io/docker/pulls/drakkarsoftware/octobot.svg)](https://hub.docker.com/r/drakkarsoftware/octobot)
+[![Dockerhub](https://img.shields.io/docker/pulls/drakkarsoftware/octobot.svg)](https://octobot.click/gh-dockerhub)
 [![Coverage Status](https://coveralls.io/repos/github/Drakkar-Software/OctoBot/badge.svg?branch=dev)](https://coveralls.io/github/Drakkar-Software/OctoBot?branch=dev)
 [![OctoBot-CI](https://github.com/Drakkar-Software/OctoBot/workflows/OctoBot-CI/badge.svg)](https://github.com/Drakkar-Software/OctoBot/actions)
 [![Build Status](https://cloud.drone.io/api/badges/Drakkar-Software/OctoBot/status.svg)](https://cloud.drone.io/Drakkar-Software/OctoBot)
-[![UptimeRobot](https://img.shields.io/uptimerobot/ratio/30/m786447893-903b482e5158c8b6483760e8)](https://status.octobot.online/)
+[![UptimeRobot](https://img.shields.io/uptimerobot/ratio/30/m786447893-903b482e5158c8b6483760e8)](https://octobot.click/gh-status)
 
 #### Octobot Community
 [![Active OctoBot](https://img.shields.io/badge/dynamic/json.svg?&url=https://octobotmetrics.herokuapp.com/metrics/community/count/0/-1/0&query=$.total&color=green&label=OctoBots%20online%20this%20month)]()
-[![Telegram Chat](https://img.shields.io/badge/telegram-chat-green.svg)](https://t.me/joinchat/F9cyfxV97ZOaXQ47H5dRWw)
-[![Discord](https://img.shields.io/discord/530629985661222912.svg?logo=discord)](https://discord.gg/vHkcb8W)
+[![Telegram Chat](https://img.shields.io/badge/telegram-chat-green.svg)](https://octobot.click/gh-telegram)
+[![Discord](https://img.shields.io/discord/530629985661222912.svg?logo=discord)](https://octobot.click/gh-discord)
 [![Telegram News](https://img.shields.io/badge/telegram-news-blue.svg)](https://t.me/OctoBot_Project)
-[![Twitter](https://img.shields.io/twitter/follow/DrakkarsOctobot.svg?label=Follow&style=social)](https://twitter.com/DrakkarsOctobot)
+[![Twitter](https://img.shields.io/twitter/follow/DrakkarsOctobot.svg?label=Follow&style=social)](https://octobot.click/gh-twitter)
 <p align="center">
 <img src="../assets/octopus.svg" alt="Octobot Logo" height="400" width="400">
 </p>
@@ -28,11 +28,11 @@ This repository contains all the features of the bot (trading tools, evaluation
 
 To install OctoBot with its tentacles, just use the [latest release for your system](https://github.com/Drakkar-Software/OctoBot/releases/latest) and your OctoBot is ready ! 
 
-Find the answers to the most common questions in [our FAQ](https://octobot.info/usage/frequently-asked-questions-faq).
+Find the answers to the most common questions in [our FAQ](https://www.octobot.info/usage/frequently-asked-questions-faq).
 
 ## Your Octobot
 <a href="https://www.octobot.online/guides/#telegram"><img src="../assets/telegram-interface.png" height="414" alt="Telegram interface"></a>
-[![Twitter Interface](../assets/twitter-interface.png)](https://octobot.info/interfaces/twitter-interface)
+[![Twitter Interface](../assets/twitter-interface.png)](https://www.octobot.info/interfaces/twitter-interface)
 
 OctoBot is highly customizable using its configuration and tentacles system. 
 You can build your own bot using the infinite [configuration](https://www.octobot.online/guides/#trading_modes) possibilities such as 
@@ -41,7 +41,7 @@ You can build your own bot using the infinite [configuration](https://www.octobo
 OctoBot is **AI ready**: Python being the main language for OctoBot, it's easy to integrate machine-learning libraries such as [Tensorflow](https://github.com/tensorflow/tensorflow) or
 any other lib and take advantage of all the available data and create a very powerful trading strategy. 
 
-Octobot's main feature is **evolution** : you can [install](https://octobot.info/advanced_usage/tentacle-manager), 
+Octobot's main feature is **evolution** : you can [install](https://www.octobot.info/advanced_usage/tentacle-manager), 
 [modify](https://developer.octobot.info/guides/customize-your-octobot) and even [create](https://developer.octobot.info/guides/developer-guide) any tentacle you want to build your ideal cryptocurrency trading robot. You can even share your OctoBot evolutions!
 
 ## Hardware requirements
@@ -52,13 +52,13 @@ Octobot's main feature is **evolution** : you can [install](https://octobot.info
 ## Installation
 OctoBot's installation is **very simple**... because **very documented** ! See the [installation guides](https://www.octobot.online/guides/#installation) for more info.
 
-#### [With executable](https://octobot.info/installation/with-binary)
+#### [With executable](https://www.octobot.info/installation/with-binary)
 Follow the [2 steps installation guide](https://www.octobot.online/executable_installation/) 
 
 In short:
 - Use the latest release on the [release page](https://github.com/Drakkar-Software/OctoBot/releases/latest)
 
-#### [With Docker](https://octobot.info/installation/with-docker)
+#### [With Docker](https://www.octobot.info/installation/with-docker)
 Follow the [docker installation guide](https://www.octobot.online/docker_installation/) 
 
 In short :
@@ -73,15 +73,15 @@ docker-compose up -d
 ```
 And then open [https://octobot.localhost](https://octobot.localhost).
 
-#### [With pip](https://octobot.info/installation/with-pip)
+#### [With pip](https://octobot.click/gh-pip-install)
 
 In short :
 ```
 pip install OctoBot>=0.4.1
 Octobot
 ```
 
-#### [With python sources](https://octobot.info/installation/with-python-only)
+#### [With python sources](https://octobot.click/gh-python-install)
 Follow the [python installation guide](https://www.octobot.online/python_installation/) 
 
 In short :
@@ -93,30 +93,30 @@ python3 start.py
 ```
 
 #### One click deployment
-Follow the [Digital Ocean installation guide](https://www.octobot.online/digital_ocean_installation/) 
+Follow the [Digital Ocean installation guide](https://octobot.click/gh-do-install) 
 
 In short :
 
-[![Deploy to DO](https://mp-assets1.sfo2.digitaloceanspaces.com/deploy-to-do/do-btn-blue.svg)](https://cloud.digitalocean.com/apps/new?repo=https://github.com/Drakkar-Software/OctoBot/tree/master&refcode=40c9737100b1)
+[![Deploy to DO](https://mp-assets1.sfo2.digitaloceanspaces.com/deploy-to-do/do-btn-blue.svg)](https://octobot.click/gh-do-deploy)
 
 - Get 60-day free Digital Ocean hosting by registering with [OctoBot referral link](https://m.do.co/c/40c9737100b1).
 
-[![Develop on Okteto](https://okteto.com/develop-okteto.svg)](https://cloud.okteto.com/deploy?repository=https://github.com/Drakkar-Software/OctoBot&branch=master)
+[![Develop on Okteto](https://okteto.com/develop-okteto.svg)](https://octobot.click/gh-okteto-deploy)
 
 - Free 24-hour demo repeatable indefinitely on Okteto simply using your Github account
 
 ## Exchanges
-[![Binance](../assets/binance-logo.png)](https://accounts.binance.com/en/register?ref=135007948)
-[![Binance](../assets/ftx-logo.png)](https://ftx.com/#a=2525542)
-[![Binance](../assets/okex-logo.png)](https://www.okex.com/join/9403477)
-[![Binance](../assets/gateio-logo.png)](https://www.gate.io/signup/6206289)
-[![Binance](../assets/huobi-logo.png)](https://www.huobi.com/en-us/topic/double-reward/?invite_code=nwcu2223)
-[![Bitmax](../assets/ascendex-logo.png)](https://ascendex.com/en-us/register?inviteCode=ULWCFWL6G)
+[![Binance](../assets/binance-logo.png)](https://octobot.click/gh-binance)
+[![Binance](../assets/ftx-logo.png)](https://octobot.click/gh-ftx)
+[![Binance](../assets/okex-logo.png)](https://octobot.click/gh-okex)
+[![Binance](../assets/gateio-logo.png)](https://octobot.click/gh-gateio)
+[![Binance](../assets/huobi-logo.png)](https://octobot.click/gh-huobi)
+[![Bitmax](../assets/ascendex-logo.png)](https://octobot.click/gh-ascendex)
 [![Coinbase](../assets/coinbasepro-logo.png)](https://pro.coinbase.com)
 [![Kucoin](../assets/kucoin-logo.png)](https://www.kucoin.com)
 [![Bitmex](../assets/bitmex-logo.png)](https://bitmex.com)
 
-Octobot supports many [exchanges](https://exchanges.octobot.info/) thanks to the [ccxt library](https://github.com/ccxt/ccxt). 
+Octobot supports many [exchanges](https://octobot.click/gh-exchanges) thanks to the [ccxt library](https://github.com/ccxt/ccxt). 
 To activate trading on an exchange, just configure OctoBot with your api keys as described [on the exchange documentation](https://www.octobot.online/guides/#exchanges).
 
 ## Disclaimer
@@ -132,7 +132,7 @@ Do not hesitate to read the source code and understand the mechanism of this bot
 ## License
 GNU General Public License v3.0 or later.
 
-See [LICENSE](https://github.com/Drakkar-Software/OctoBot/blob/master/LICENSE) to see the full text.
+See [LICENSE](https://octobot.click/gh-license) to see the full text.
 
 ## Sponsors
 <table>

dev_requirements.txt

@@ -1,6 +1,6 @@
 pytest>=6.1.1
 pytest-cov
-pytest-asyncio==0.14.0
+pytest-asyncio==0.18.3
 pytest-timeout
 
 mock>=4.0.1

octobot/api/__init__.py

@@ -33,17 +33,27 @@
     join_independent_backtesting,
     initialize_independent_backtesting_config,
     stop_independent_backtesting,
+    join_independent_backtesting_stop,
     get_independent_backtesting_report,
 )
 from octobot.api.strategy_optimizer import (
     create_strategy_optimizer,
+    create_design_strategy_optimizer,
     find_optimal_configuration,
+    initialize_design_strategy_optimizer,
+    update_strategy_optimizer_total_runs,
+    generate_and_save_strategy_optimizer_runs,
+    resume_design_strategy_optimizer,
+    cancel_strategy_optimizer,
     print_optimizer_report,
     get_optimizer_report,
     get_optimizer_results,
     get_optimizer_overall_progress,
+    get_design_strategy_optimizer_queue,
+    update_design_strategy_optimizer_queue,
     is_optimizer_in_progress,
     is_optimizer_computing,
+    is_optimizer_finished,
     get_optimizer_errors_description,
     get_optimizer_current_test_suite_progress,
     get_optimizer_strategy,
@@ -72,15 +82,25 @@
     "join_independent_backtesting",
     "initialize_independent_backtesting_config",
     "stop_independent_backtesting",
+    "join_independent_backtesting_stop",
     "get_independent_backtesting_report",
     "create_strategy_optimizer",
+    "create_design_strategy_optimizer",
     "find_optimal_configuration",
+    "initialize_design_strategy_optimizer",
+    "update_strategy_optimizer_total_runs",
+    "generate_and_save_strategy_optimizer_runs",
+    "resume_design_strategy_optimizer",
+    "cancel_strategy_optimizer",
     "print_optimizer_report",
     "get_optimizer_report",
     "get_optimizer_results",
     "get_optimizer_overall_progress",
+    "get_design_strategy_optimizer_queue",
+    "update_design_strategy_optimizer_queue",
     "is_optimizer_in_progress",
     "is_optimizer_computing",
+    "is_optimizer_finished",
     "get_optimizer_errors_description",
     "get_optimizer_current_test_suite_progress",
     "get_optimizer_strategy",

octobot/api/backtesting.py

@@ -24,13 +24,17 @@ def create_independent_backtesting(config,
                                    join_backtesting_timeout=constants.BACKTESTING_DEFAULT_JOIN_TIMEOUT,
                                    run_on_common_part_only=True,
                                    start_timestamp=None,
-                                   end_timestamp=None) -> backtesting.IndependentBacktesting:
+                                   end_timestamp=None,
+                                   enable_logs=True,
+                                   stop_when_finished=False) -> backtesting.IndependentBacktesting:
     return backtesting.IndependentBacktesting(config, tentacles_setup_config, data_files,
                                               data_file_path,
                                               run_on_common_part_only=run_on_common_part_only,
                                               join_backtesting_timeout=join_backtesting_timeout,
                                               start_timestamp=start_timestamp,
-                                              end_timestamp=end_timestamp)
+                                              end_timestamp=end_timestamp,
+                                              enable_logs=enable_logs,
+                                              stop_when_finished=stop_when_finished)
 
 
 async def initialize_and_run_independent_backtesting(independent_backtesting, log_errors=True) -> None:
@@ -49,6 +53,10 @@ async def stop_independent_backtesting(independent_backtesting, memory_check=Fal
     await independent_backtesting.stop(memory_check=memory_check, should_raise=should_raise)
 
 
+async def join_independent_backtesting_stop(independent_backtesting, timeout=None):
+    await independent_backtesting.join_stop_event(timeout=timeout)
+
+
 def check_independent_backtesting_remaining_objects(independent_backtesting) -> None:
     independent_backtesting.octobot_backtesting.check_remaining_objects()
 
@@ -66,7 +74,7 @@ def get_independent_backtesting_progress(independent_backtesting) -> float:
 
 
 def is_independent_backtesting_finished(independent_backtesting) -> bool:
-    return independent_backtesting.get_progress() == 1.0
+    return independent_backtesting.has_finished()
 
 
 def is_independent_backtesting_stopped(independent_backtesting) -> bool:

octobot/api/strategy_optimizer.py

@@ -14,16 +14,55 @@
 #  You should have received a copy of the GNU General Public
 #  License along with OctoBot. If not, see <https://www.gnu.org/licenses/>.
 from octobot.strategy_optimizer.strategy_optimizer import StrategyOptimizer
+from octobot.strategy_optimizer.strategy_design_optimizer import StrategyDesignOptimizer
 
 
 def create_strategy_optimizer(config, tentacles_setup_config, strategy_name) -> StrategyOptimizer:
     return StrategyOptimizer(config, tentacles_setup_config, strategy_name)
 
 
+def create_design_strategy_optimizer(trading_mode, config=None, tentacles_setup_config=None,
+                                     optimizer_config=None) -> StrategyDesignOptimizer:
+    return StrategyDesignOptimizer(trading_mode, config, tentacles_setup_config, optimizer_config)
+
+
+async def initialize_design_strategy_optimizer(strategy_optimizer, is_resuming, is_computing=False):
+    strategy_optimizer.is_computing = is_computing
+    return await strategy_optimizer.initialize(is_resuming)
+
+
+async def update_strategy_optimizer_total_runs(optimizer, runs):
+    if optimizer.empty_the_queue:
+        optimizer.total_nb_runs += len(runs)
+
+
+async def generate_and_save_strategy_optimizer_runs(trading_mode, tentacles_setup_config,
+                                                    optimizer_config, optimizer_id) -> list:
+    optimizer = StrategyDesignOptimizer(trading_mode, None, tentacles_setup_config,
+                                        optimizer_config, optimizer_id)
+    return await optimizer.generate_and_save_run()
+
+
+async def resume_design_strategy_optimizer(optimizer, data_files, randomly_chose_runs, start_timestamp, end_timestamp,
+                                           required_idle_cores, notify_when_complete=False, optimizer_ids=None):
+    empty_the_queue = optimizer_ids is None  # continue till the queue is empty if no optimizer id is specified
+    optimizer_ids = optimizer_ids or await optimizer.get_queued_optimizer_ids()
+    return await optimizer.resume(data_files, optimizer_ids, randomly_chose_runs,
+                                  start_timestamp=start_timestamp,
+                                  end_timestamp=end_timestamp,
+                                  empty_the_queue=empty_the_queue,
+                                  required_idle_cores=required_idle_cores,
+                                  notify_when_complete=notify_when_complete)
+
+
 def find_optimal_configuration(strategy_optimizer, TAs=None, time_frames=None, risks=None) -> None:
     strategy_optimizer.find_optimal_configuration(TAs=TAs, time_frames=time_frames, risks=risks)
 
 
+def cancel_strategy_optimizer(strategy_optimizer):
+    strategy_optimizer.cancel()
+
+
 def print_optimizer_report(strategy_optimizer) -> None:
     strategy_optimizer.print_report()
 
@@ -36,18 +75,30 @@ def get_optimizer_results(strategy_optimizer) -> list:
     return strategy_optimizer.run_results
 
 
-def get_optimizer_overall_progress(strategy_optimizer) -> int:
-    return strategy_optimizer.get_overall_progress()
+async def get_optimizer_overall_progress(strategy_optimizer) -> (int, float):
+    return await strategy_optimizer.get_overall_progress()
+
+
+async def get_design_strategy_optimizer_queue(trading_mode) -> list:
+    return await StrategyDesignOptimizer.get_run_queue(trading_mode)
 
 
-def is_optimizer_in_progress(strategy_optimizer) -> bool:
-    return strategy_optimizer.is_in_progress()
+async def update_design_strategy_optimizer_queue(trading_mode, updated_queue) -> list:
+    return await StrategyDesignOptimizer.update_run_queue(trading_mode, updated_queue)
+
+
+async def is_optimizer_in_progress(strategy_optimizer) -> bool:
+    return await strategy_optimizer.is_in_progress()
 
 
 def is_optimizer_computing(strategy_optimizer) -> bool:
     return strategy_optimizer.is_computing
 
 
+def is_optimizer_finished(strategy_optimizer) -> bool:
+    return strategy_optimizer.is_finished
+
+
 def get_optimizer_errors_description(strategy_optimizer) -> str:
     return strategy_optimizer.get_errors_description()
 

octobot/backtesting/independent_backtesting.pxd

@@ -20,6 +20,8 @@ cimport octobot.backtesting.octobot_backtesting as octobot_backtesting
 
 cdef class IndependentBacktesting:
     cdef list forced_time_frames
+    cdef object optimizer_id
+    cdef object backtesting_id
 
     cdef public dict octobot_origin_config
     cdef public dict backtesting_config
@@ -28,6 +30,8 @@ cdef class IndependentBacktesting:
 
     cdef object logger
     cdef object join_backtesting_timeout
+    cdef bint stop_when_finished
+    cdef public bint enable_logs
 
     cdef public str data_file_path
     cdef public dict symbols_to_create_exchange_classes
@@ -37,10 +41,13 @@ cdef class IndependentBacktesting:
     cdef public bint stopped
 
     cdef public object post_backtesting_task
+    cdef public object previous_log_level
+    cdef public object stopped_event
 
     cdef public octobot_backtesting.OctoBotBacktesting octobot_backtesting
 
     cpdef bint is_in_progress(self)
+    cpdef bint has_finished(self)
     cpdef double get_progress(self)
     cpdef void log_report(self)