Modify Dev_Strat to add technical indicators

[ryanle88]

@Kismuz First of all, thank you so much for creating this awesome gym. It is probably the best gym for trading that I could find on the web at the moment.

As I have read all the threads from other issues and also run the A3C Random notebook that you provided, the reward did not converge over time. I saw you have mentioned that as of now, you have not found any good solution that could lead to the convergence. I have spent sometime with the package, especially with the notebooks examples, and just thought maybe if we provided more information to the agent like technical indicators or the like, it would help improve the performance. I have tried to modify the Dev_Strat_4_6 by adding two simple moving average (50, 100) but keep getting the State observation shape/range mismatch, which I believe is due to the NAN problem since there is no data for the second indicator for the first 100 records. I could fix the problem by set the skip_frame to 100 but this is not a very data-efficient solution. Could you please help instruct me how to tackle this problem? Thank you in advance.

[Kismuz]

@ryanle88 ,

not found any good solution that could lead to the convergence.

I'd say no solution with "acceptable generalisation properties"; at the time present convergence on train data has been improved, see:
https://github.com/Kismuz/btgym/blob/master/examples/unreal_stacked_lstm_strat_4_11.ipynb
as an example.

if we provided more information to the agent like technical indicators or the like, it would help improve the performance

You absolutely correct that finding valuable signal features is critical to agent's performance and is an one open direction for future work.

In brief, it is essential to provide agent with stationary (in terms of data), normalised and noise-free signal features with [possibly] some level of abstraction. It is also a question of finding balance between hard-coded, data-independent features (like SMA or other indicators) and learnable, data-dependent features (agent's convolution encoder).

An example above, providing agent with just simple SMA-type of features greatly improves train data convergence ability. Adding other types of indicator may also be beneficial.

An important complimentary direction, closely related to generalisation ability, is searching for disentangled hidden state representations, such as in beta-VAE approach in DARLA algorithm: https://arxiv.org/pdf/1707.08475.pdf

modify the Dev_Strat_4_6 by adding two simple moving average (50, 100) but keep getting the State observation shape/range mismatch

Take a look at Dev_Strat_4_11 class, it does exactly that:

    def set_datalines(self):
        self.data.sma_16 = btind.SimpleMovingAverage(self.datas[0], period=16)
        self.data.sma_32 = btind.SimpleMovingAverage(self.datas[0], period=32)
        self.data.sma_64 = btind.SimpleMovingAverage(self.datas[0], period=64)
        self.data.sma_128 = btind.SimpleMovingAverage(self.datas[0], period=128)
        self.data.sma_256 = btind.SimpleMovingAverage(self.datas[0], period=256)
        self.data.sma_512 = btind.SimpleMovingAverage(self.datas[0], period=512)

        self.data.dim_sma = btind.SimpleMovingAverage(
            self.datas[0],
            period=(512 + self.time_dim)
        )
        self.data.dim_sma.plotinfo.plot = False

... it purely backtrader-specific:

• to add any other indicator you need to declare data-line via __init__() or, conventionally, via set_datalines() method.

• note the self.data.dim_sma service data-line with period equal to biggest indicator burn-in period, BTGym requires it to be set correctly.

And that's all: bt.Cerebro() engine takes care of computing indicators with burn-in periods, and only after that strategy execution starts, refer to backtrader documentation for details.

Also refer to get_market_state(), get_state() methods:

• note that signal fed to agent is gradient of features taken along concatenation dimension, not time one.
• signal is than amplified and passed through tanh to fit in [-1,1].
• no data-dependent normalisation is performed.

[ryanle88]

Thank you for your quick response @Kismuz . My next step is to figure out how to replace the FOREX data with my own, which I believe you have mentioned in other issues, so I am just gonna read those. Really appreciate your help. Keep up the good work!!

[ryanle88]

@Kismuz
I have spent sometime trying to figure out how to incorporating my own data file with extra columns besides OHLC instead of the FOREX that you used by reading other threads, especially #8 . I replaced:

MyDataset = BTgymDataset(
filename=data_m1_6_month,
start_weekdays={0, 1, 2, 3, 4, 5, 6},
episode_duration={'days': 1, 'hours': 23, 'minutes': 40}, # note: 2day-long episode
start_00=False,
time_gap={'hours': 10},
)

by:

class ExtraPandasDirectData(btfeeds.PandasDirectData):
lines = ('width',)
params = (
('width', 2),
)

datafields = btfeeds.PandasData.datafields + (['width'])

class ExtraLinesDataset(BTgymDataset):

def to_btfeed(self):
    """
    Overrides default method to add custom datalines.
    Performs BTgymDataset-->bt.feed conversion.
    Returns bt.datafeed instance.
    """
    try:
        assert not self.data.empty
        btfeed = ExtraPandasDirectData(
            dataname=self.data,
            timeframe=self.timeframe,
            datetime=self.datetime,
            open=self.open,
            high=self.high,
            low=self.low,
            close=self.close,
            volume=self.volume,
            my_id_1=6,  # Same lines
            my_id_2=7,
        )
        btfeed.numrecords = self.data.shape[0]
        return btfeed

    except:
        msg = 'BTgymDataset instance holds no data. Hint: forgot to call .read_csv()?'
        self.log.error(msg)
        raise AssertionError(msg)

params = dict(
# CSV to Pandas params.
sep=';',
header=0,
index_col=0,
parse_dates=True,
names=['open', 'high', 'low', 'close', 'volume', 'my_id_1', 'my_id_2'],

# Pandas to BT.feeds params:
timeframe=1,  # 1 minute.
datetime=0,
open=1,
high=2,
low=3,
close=4,
volume=5,
openinterest=-1,
my_id_1=6,
my_id_2=7,

# Random-sampling params:
# .....omitted for brevity....

)

MyDataset = ExtraLinesDataset(
filename=data_m1_6_month,
start_weekdays={0, 1, 2, 3, 4, 5, 6},
episode_duration={'days': 1, 'hours': 23, 'minutes': 40}, # note: 2day-long episode
start_00=False,
time_gap={'hours': 10},
**params)

However, when I took a look in the derivative.py file where BTgymDataset located, I saw that you mentioned **kwarg is deprecated, so what passed as **params did not have any impact on reading the new file.

So my question is: what is your plan in streamlining the process of importing new data files? I could keep debugging and modify your files to make it work but this would cause me trouble everytime I tried to pull update request from your github.

Could you provide me some directions on how to tackle this issue?

Also, just FYI, I am very interested in learning more about what you are doing with the btgym, so if you have sometime to mentor me about this package, I can pay back by my time to work on whatever task that you have in mind for the btgym package.

Thank you.

[Kismuz]

@ryanle88 ,

what is your plan in streamlining the process of importing new data files?

There is generic data iterator class BTGymBaseData providing core functionality for:

1. loading data from sources (read_csv())
2. shaping various distributions over data provided
3. sampling and returning child data iterators from above-mentioned distributions (_sample_interval(), _sample_random() etc.)
4. providing environment simulation engine with stream of data in its native format (to_btfeed())
5. some service functions (describe(), reset() )

Pp. 2 and 3 are implementations of data abstractions outlined here and related to my attempts to properly formulate algo-trading problem in terms of learning and meta-learning in changing POMDP environments. All of them internally relies on pandas data frames methods.
P.1 is pure input pipeline question, closely tied with backtrader as later serves as core environment simulation engine.
To implement new input pipe (either it another CSV, or direct pandas frames or whatever) one should subclass generic BTGymBaseData and provide it with extended loading/parsing functionality, such as implement .load_pandas() or load_xls() or load_yahoo_finance() methods so data provided are to be converted into pandas data frames and stored in MyDataset.data attribute, leaving along all data sampling and iterating methods (if there is no need to bring another distribution or sampling method).
Same is true for designing live feeds, though later obviously should be subclass if SequentialDataIterator.

...derivative.py file where BTgymDataset located, ... **kwarg is deprecated so what passed as **params did not have any impact on reading the new file.

Yes, because BTgymDataset is particularly demo "end-user" class which actually not supposed to be further worked with, but may serve as template. To implement your input pipe with similar to BTgymDataset functionality (which is actually simplified) it is better take base class, copy/paste BTgymDataset methods you want, throw away deprecation related staff and everything else you don't need, add your parser setup and name it BTgymDatasetStocksWithNews or whatever you like.

It is an open question how to categorise and grow such an hierarchy of instruments and source-specific classes. Think It is sensible to follow proven backtrader data input hierarchy for such [possibly nested] classes. Suggestions and contributions are welcome.

Due to questions repeatedly asked I see it is time to make some extended roadmap and outline of open work directions; and maybe related papers shortlist. Will work in it.

For now:

https://kismuz.github.io/btgym/intro.html#environment-engine-description

https://kismuz.github.io/btgym/intro.html#a3c-framework-description

[ryanle88]

Thank you @Kismuz !

[ryanle88]

@Kismuz
I finally got the whole thing to run by create a new file for BTGymBaseData and modify the to_btfeed of this file and then have BTgymDatasetWithStocks inherit from it since I could not figure out why to_btfeed is the only function that is executed at the parent level even though I overrode it in the child level in BTgymDatasetWithStocks class. Really appreciate your help.

[nisarahamedk]

@Kismuz
I finally got the whole thing to run by create a new file for BTGymBaseData and modify the to_btfeed of this file and then have BTgymDatasetWithStocks inherit from it since I could not figure out why to_btfeed is the only function that is executed at the parent level even though I overrode it in the child level in BTgymDatasetWithStocks class. Really appreciate your help.

@ryanle88 Could you please elaborate on this method you took to use additional features in the CSV? I am stuck with a similar problem and I am not able to wrap my head around how to solve this. Appreciate your help.

@Kismuz Considering this is an old thread, Has there been any new changes related to this particular problem of being able to use additional features/indicators straight out from the CSV file? Or this is the recommended approach? subclassing BTGymBaseData? Appreciate your insights as well.

Wonderful framework by the way. Keep up the awsome work.