The purpose of this software is to fully automate the simultaneous collection of historical and live financial data from FXCM, then store these data in a database ready for backtesting or live execution.
The development has been carried out on an ESXi server, however this will run on any hypervisor and most hardware.
Specification:
2x CPUs
4GB RAM
100GB SSD/HDD (for testing hard drive is fine)
Total drive usage for the database (excluding OS etc) from 59 instruments, is 41GB as of 29/04/2017
Operating System:
Ubuntu Server 16.04 (without GUI) for testing, however this will run on most Linux OS with/without GUI.
-
First install all dependencies in the requiements.txt and Python 2.7
-
Install MariaDB 10.x
https://mariadb.org/download/ -
Setup MariaDB to allow the user 'sec_master' to access the database with read and write permissions.
$ mysql -u root -pmysql> CREATE USER 'sec_master'@'localhost' IDENTIFIED BY 'password';
mysql> GRANT ALL PRIVILEGES ON '*.*' TO 'sec_master'@'localhost';
mysql> FLUSH PRIVILEGES;
mysql> set global max_connections = 1000;(optional)
$ sudo service mysql restart -
Download forexconnect and follow instructions.
https://github.com/JamesKBowler/python-forexconnect -
Download this repository and place in a convenient location
-
Create a logs folder in the root directory
$ mkdir ~/fxcmminer/fxcmminer_v1.0/logs -
Set the system time zone to America/New_York, this is important as all data on FXCM servers are stored in America/New_York time zone.
$ sudo timedatectl set-timezone America/New_York -
To start the process just execute:
$ python ~/fxcmminer/fxcmminer_v1.0/engine.py
If you need assistance setting this up or find any bugs, please report using the Issue section.
-
Prioritize queue so that lower time frame data is written to database before higher time frame.
-
Improve logging
-
Add auto offer removal to the fxscout
-
Clean up code!
The process of collecting data is started by executing the engine.py script. This in turn will start the fxscout.py, who's primary job is to scout FXCM for currently tradable instruments (also know as 'offers'). Once the Scout has found 'offers' available, it will contact FXCM for the .xml catalogue and make a local copy to be accessed later by fxcmminer. If FXCM add another 'offer' the Scout will then make a new local copy of the catalogue.
The scout will continue checking FXCM for the entire duration whilst the program is running, and will only place an 'OFFER' event in the events queue on system startup or if a new 'offer' is added in the future.
If an 'OFFER' event is placed in the queue, the Engine class will pass the event over to the DatabaseManager located in db_manager.py. DatabaseManager will compare its local database with the offer. If the database already exists the creation is skipped, if not the corresponding database and tables for the following time frames will be created.
{GBP/USD : ['M1','W1','D1','H8', 'H4', 'H2', 'H1','m30', 'm15', 'm5', 'm1']}
The schema is one database per offer as this will provide plenty of space for future expansion.
After a database check or creation has been carried out, a 'DBReady' event is placed into the queue, which is then passed over to the HistoricalCollector class located in historical.py . The HistoricalCollector asks the DatabaseManager for the lastest date in the database, if this is a new offer or first time system startup, the DatabaseManager will return a date from the .xml catalogue. If the catalog does not have a corresponding date an artificial low date of 2007-01-01 00:00:00 is returned. Now the HistoricalCollector has a starting point, it will begin to call FXCM's API and collect data. Once data is returned, a 'HISTDATA' event is created and placed in the queue, which in turn will be passed to the DatabaseManager and written to the database. After all historical data has been collected for the offer, a 'LIVEREADY' event is placed into the queue and the HistoricalCollector process will exit.
The apscheduler will fire off at market invertals such as 1 minute, 5 minutes etc. On each fire a 'GETLIVE' event is placed into the queue.
On receipt of the 'LIVEREADY' event, LiveDataMiner located in live.py, will update its current list of 'LIVEREADY' offers, and is now waiting for a 'GETLIVE' event from TimeKeeper. On receipt of an 'GETLIVE' event, LiveDataMiner will loop through the list of live ready offers for the corrsponding 'GETLIVE' event time_frame. Once data is collected a 'LIVEDATA' event is placed into the queue for the processing by the DatabaseManager and written to the database.
TimeDelta will provide a range in minutes for each time frame, that will not exceed to 300 bars (data points). This is because the maximum bars return for any one API call to FXCM is 300. Using the information from TimeDelta, DateRange will provide a date block which is used when calling the API. At the moment this is the only way I could exclude calling for data at the weekends whilst FXCM is closed, and due to the nature of the foreign exchange, holidays are not equal in all countries.
There are two queues one for historical data and one for live data.
Copyright (c) 2017 James K Bowler
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Trading foreign exchange on margin carries a high level of risk, and may not be suitable for all investors. Past performance is not indicative of future results. The high degree of leverage can work against you as well as for you. Before deciding to invest in foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with foreign exchange trading, and seek advice from an independent financial advisor if you have any doubts.