Merge pull request #35 from psemdel/dev

Dev

README.md

@@ -25,9 +25,6 @@ There are many similar tools out there, that have similar purposes that are cert
 # Get started
 For installation see [installation guide](https://github.com/psemdel/py-trading-bot/blob/main/docs/installation_guide.md), alternatively you can start the bot in Docker, see the [guide](https://github.com/psemdel/py-trading-bot/blob/main/docs/docker_readme.md).
 
-# Branches
-The main branches contain many strategies that can be used right away. However, the amount of code can make it difficult to understand how the bot works. For this reason a branch "minimum" is available. It contains only the minimum infrastructure to make run the code with usual functionalities, the strategy implementation is then left to you.
-
 # Deployment
 Deployment of the bot on external machine has not been achieved yet for several reasons:
 

docs/backtesting.md

@@ -12,7 +12,7 @@ Advantage of backtesting from downloaded files:
 1. You don't have to download the data again and again
 2. They don't change. It makes comparison between strategies easier
 
-Additionally, you can backtest with recent prices, for instance last 3 years, in the notebooks live.ipynb. It includes also the function scan_presel_all() that will calculate the return of each preselection strategies for each exchanges obtained during the past x days (90 by default). If you have the impression that a strategy does not perform so well right now, it will confirm your doubt or not. The list of strategies tested is defined in trading_bot/settings.py in:
+Additionally, you can backtest with recent prices, for instance last 3 years, in the notebooks live.ipynb. It includes also the function scan_presel_all() that will calculate the return of each preselection strategies for each stock exchanges obtained during the past x days (90 by default). If you have the impression that a strategy does not perform so well right now, it will confirm your doubt or not. The list of strategies tested is defined in trading_bot/settings.py in:
 
     "STRATEGIES_TO_SCAN":["PreselVol","PreselRealMadrid","PreselRetard","PreselRetardMacro","PreselDivergence",
           "PreselDivergenceBlocked","PreselWQ7","PreselWQ31","PreselWQ53","PreselWQ54"],    

docs/installation_guide.md

@@ -19,7 +19,7 @@ Note: it is clearly imaginable to use the bot with the basic version of vbt, but
 It will install some other dependencies needed for the bot, for instance: pandas, numpy, python-telegram-bot and TA-Lib. 
 
 # Django
-Afterwards, you need to install [Django](https://www.djangoproject.com/) on one side and [Redis](https://redis.io/) (or equivalent) with [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html) for the worker. ib_insync is a library to communicate with interactive brokers. They are defined in requirements.txt
+Afterwards, you need to install [Django](https://www.djangoproject.com/) on one side and [Redis](https://redis.io/) (or equivalent) with [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html) for the worker. ib_insync is a library to communicate with interactive brokers. They are defined in requirements.txt. Note that you can comment out the keras installation, if you don't want to use the machine learning functions.
 
     sh
     pip install -r requirements.txt
@@ -133,7 +133,7 @@ When starting the bot, the message appearing will relate to Django, Redis, Teleg
      
 If everything goes well, your Telegram should display a message "I'm back online". If not, type "/start" in Telegram, it will add your chat_id to the list.
 
-Note the admin panel from Django that allows creating finance products.
+Note: the admin panel from Django that allows creating finance products.
 
 
 

docs/machine_learning.md

@@ -0,0 +1,96 @@
+# Machine learning
+The bot is perfectly able to use a machine learning algorithm to generate entries and exits signals. An examplary algorithm is provided in ml.py and the corresponding jupyter notebook ml.ipynb. 
+
+In a similar way to strat.py first the algorithm is able to preprocess the price, with RSI, STOCH... for example. If preprocessing is set to false, the price alone is used as an input. After this signal is scaled and used for the training of model or the prediction. Presently, MLP and LSTM are implemented.
+
+The preprocessing functions used are defined with features_name. The list is provided in the section below.
+
+Keras and SKlearn are used, but it could be replaced by any other library. Only condition for the model to work with the bot is to generate entries and exits signals.
+
+## Preselection
+The model provided select the stock with the highest expected maximum in the next 10 days in the provided data set. Additionnaly, a mechanism is implemented to limit the number of orders by defining a threshold: if the estimation for the stock with the present maximum is not sufficiently above the previous candidate stock estimation, then we keep the old one.
+
+# List of features
+When a "_ent" is in the name, it means it is an entry signal, "_ex" means exit signal, without those suffix, the value is taken directly. This kind of values seem to work better than entry/exit signals with machine learning. The suffix "_RANK" means the rank of the stock compared to the other stock of the dataset for this parameter.
+
+- MA_ent, MA_ex (period 5/15)
+- KAMA_ent, KAMA_ex (on extrema of KAMA function)
+- MFI (threshold 20/80)
+- STOCH, STOCH_ent, STOCH_ex (threshold 20/80)
+- WILLR
+- SUPERTREND_ent, SUPERTREND_ex 
+- BBANDS_BANDWIDTH, BBANDS_ent, BBANDS_ex
+- RSI, RSI20_ent, RSI20_ex (threshold 20/80), RSI30_ent, RSI30_ex (threshold 30/70)
+- ULTOSC
+- GROW30 (increasing in value during past 30 days), GROW30_RANK
+- GROW30_MA (same but on the moving average signal), GROW30_MA_RANK
+- GROW30_DEMA (same but on the DEMA signal), GROW30_DEMA_RANK
+- GROW50 (increasing in value during past 30 days), GROW50_RANK
+- GROW50_MA (same but on the moving average signal), GROW50_MA_RANK
+- GROW50_DEMA (same but on the DEMA signal), GROW50_DEMA_RANK
+- GROW200 (increasing in value during past 30 days), GROW200_RANK
+- GROW200_MA (same but on the moving average signal), GROW200_MA_RANK
+- GROW200_DEMA (same but on the DEMA signal), GROW200_DEMA_RANK
+- OBV
+- AD
+- MACD
+- HIST (from MACD)
+- DIVERGENCE (see predefined strategy documentation, compare the stock price variation to the index price variation)
+- NATR
+- HT_TRENDMODE
+- MACRO_TREND (see trend_calculation documentation)
+- PU_RESISTANCE (try to evaluate the distance from last resistance)
+- PU_SUPPORT (try to evaluate the distance from last support)
+
+And all following patterns. They are defined in constants.py for explanation refer to [talib documentation](https://github.com/TA-Lib/ta-lib-python/tree/master/docs/func_groups). Bear can be used only for exits, bull for entries:            
+
+BEAR_PATTERNS:
+- CDLLONGLINE
+- CDLENGULFING
+- CDLCLOSINGMARUBOZU
+- CDLBELTHOLD
+- CDLHIKKAKE
+- CDLRISEFALL3METHODS
+- CDL3LINESTRIKE
+- CDLBREAKAWAY
+- CDLABANDONEDBABY
+- CDLEVENINGSTAR
+- CDLSEPARATINGLINES
+- CDLEVENINGDOJISTAR
+- CDL3BLACKCROWS
+- CDLDARKCLOUDCOVER
+- CDLMARUBOZU
+- CDLHIKKAKEMOD
+- CDLMORNINGSTAR
+- CDLUNIQUE3RIVER
+- CDLXSIDEGAP3METHODS
+- CDLCOUNTERATTACK
+- CDL3INSIDE
+- CDLMORNINGDOJISTAR
+- CDLBREAKAWAY
+
+BULL_PATTERNS:
+- CDLKICKINGBYLENGTH
+- CDLKICKING
+- CDLMARUBOZU
+- CDLCLOSINGMARUBOZU
+- CDL3WHITESOLDIERS
+- CDLLONGLINE
+- CDLENGULFING
+- CDLDRAGONFLYDOJI
+- CDLTAKURI
+- CDLMORNINGDOJISTAR
+- CDLMORNINGSTAR
+- CDLHANGINGMAN
+- CDL3INSIDE
+- CDLKICKINGBYLENGTH_INV
+- CDLKICKING_INV
+- CDLINVERTEDHAMMER
+- CDLPIERCING
+- CDLHIKKAKEMOD
+- CDLSTICKSANDWICH
+- CDLTRISTAR
+- CDL3LINESTRIKE
+- CDLDARKCLOUDCOVER
+- CDLINNECK
+- CDL3BLACKCROWS   

docs/optimization.md

@@ -3,11 +3,13 @@ Optimization algorithms are important to determine the best set of parameters or
 
 The optimization scripts provided must be started by opt_starter.py (in the root). Several scripts are provided, but obviously they are only examples (no guarantee here!). The only thing that matters here is the resulting strategy, the optimization algorithm is just a mean.
 
-In opt_main, the set will be split in a learning and a test set. By comparing the results on both set, you can verify how reliable is your strategy. By default this split is made on the time (`split_learn_train=time`). You can also split on the symbols (`split_learn_train=symbol`).
+In opt_main, the set will be split in a learning and a test set. By comparing the results on both sets, you can verify how reliable is your strategy. By default this split is made on the time (`split_learn_train=time`). You can also split on the symbols (`split_learn_train=symbol`).
 
-Most of the algorithm will make evolve the strategy array until it cannot be improved anymore. The starting array can be defined by setting `predefined=False`, the array is then given in `a_simple` if no trend is considered or in `a_bull`, `a_bear` and `a_uncertain`. If no starting array is defined, a random one will be generated. The process is performed several time in a row. The number of iterations is defined by the parameter `loops`. To be sure that your optimization met one of the best possible combination, it is recommended to perform many loops (it is a typical local extremum vs global extremum issue). 
+Most of the algorithm will make evolve the strategy array until it cannot be improved anymore. The starting array can be defined by setting `strat_arr=` a strategy array. If no starting array is defined, a random one will be generated, the consideration of the trend depends on the `nb_macro_modes`, if you don't want to consider the trend put 1 otherwise 3. The process is performed several time in a row. The number of iterations is defined by the parameter `loops`. To be sure that your optimization met one of the best possible combination, it is recommended to perform many loops (it is a typical local extremum vs global extremum issue). 
 
-Results are written in opt/output/strat.txt.
+Please note that by default the fees during the optimization are not zero. Keep it in mind, when reviewing your results.
+
+Results are written in opt/output/strat.txt. Additionally, a dataframe is created in opt/tested_arrs/. It gathers all calculate combination and allows an easy postprocessing to select the strategy that fit your need. The jupyter notebook tested_arrs_reader.ipynb is a starting point for this selection.
 
 ## opt_strat
 Opt_strat aims at optimizing a strategy for one stock on a given set of stocks.
@@ -19,14 +21,4 @@ Try to find out the best stop loss threshold for a given strategy.
 Optimize the parameters for the trend calculation in order to have maximum return during the bull trend and minimum return during the bear trend.
 
 ## Correlation
-The following optimization algorithms try to group some stocks together depending on their price correlation and to optimize a strategy for each of those groups.
-
-### opt_symbols
-The initial group to be splited is given as a list of symbols.
-
-### opt_cor
-The initial group of stocks is defined in constants.py.
-
-### opt_by_part and opt_by_part_rec
-The stocks are not grouped depending on their correlation but on their performance on a given strategy. The best are packed together, the rest is  re-optimized.
-
+To display correlation between symbol prices, you can use the jupyter notebook corr.ipynb. You can then perform filtering for some symbols with pandas from the dataframe generated by opt_strat.

docs/orders.md

@@ -40,6 +40,9 @@ In the admin panel, the strategy field `Perform order` must be checked (=True).
 ### Stock exchange level
 In the admin panel, the stock exchange field `Perform order` must be checked (=True).
 
+The strategies that you want to performed for each stock exchange must be selected there.
+Note: keep in mind that the index for a stock exchange may be listed in another stock exchange. For instance, the DAX is listed by EUREX and not by XETRA.
+
 ### Stock level
 Additionally, a black list of stocks can be configured in trading_bot/settings.py with IB_STOCK_NO_PERMISSION. It covers especially complex products, where permission is difficult to get.
 

docs/start_bot.md

@@ -37,8 +37,8 @@ It must be performed after the Celery server is launched, which itself requires
 
 Note: Don't start several times the bot! Telegram does not support it. If it happens, close all instances of py-trading-bot before starting it again.
 
-##Kubernetes folger
-The Kubernetes manifests, in the folger with the same name, create the DB, Redis and Django servers and start them. The commands are the same as those described above.
+##Kubernetes folder
+The Kubernetes manifests, in the folder with the same name, create the DB, Redis and Django servers and start them. The commands are the same as those described above.
 
 ##Closing the bot
 To stop the bot, close the terminal where you opened it. If you started it in the background (not recommended), close the process.

docs/stock_exchange.md

@@ -20,7 +20,8 @@ Following inputs can be defined:
 - strategies_in_use: select the strategies you want to use for this stock exchange, before closing. Overriden by those at sector level if presel_at_sector_level is true!
 - strategies_in_use_intraday: select the strategies you want to use for this stock exchange, during the day. Overriden by those at sector level if presel_at_sector_level is true!
 - presel_at_sector_level: if true, the strategy will be performed at sector level. Is true only for NYSE, as there are too many stocks in the S&P 500. You may want to use the same strategy on 5 bundles of 100 stocks instead of 1 strategy for 500 stocks.
-- main_index: select the index related to this stock exchange 
+- main_index: select the index related to this stock exchange. It is for instance used for the determination of the trend or for calculation of the divergence.
+- calc_report: should the report be calculated, useful to deactivate report about ETF stock exchanges
 
 # Writing a report for the new stock exchange
 The scheduler will generate automatically a report for all stock exchanges where at least a strategy is in use, 15 min before its closing. There is nothing to do.