Add dynamic features

Author: ClementPerroud

docs/source/customization.rst

@@ -59,3 +59,11 @@ The ``.add_metric`` method takes 2 parameters :
 
  
 
+.. note::
+
+  If you want to use your metrics to feed a custom logger, to visualize data or to track performance, you can access to results with ``env.get_metrics()`` **at the end of an episode**. In this case, it returns :
+
+ .. code-block:: python
+ 
+  { "Market Return" :  "25.30%", "Portfolio Return" : "45.24%", "Position Changes" : 28417, "Episode Lenght" : 33087 }
+ 

docs/source/environment_desc.rst

@@ -78,8 +78,8 @@ Observation Space
 
 The observation space is an np.array containing:
 
-* The row of your DataFrame columns containing ``features`` in their name, at a given step.
-* The current position of the environment to allow self-awareness for the agent. You can disable it by setting ``include_position_in_features`` to ``False``.
+* The row of your DataFrame columns containing ``features`` in their name, at a given step : the **static features**
+* The **dynamic features ** (by default, the last position taken by the agent, and the current real position).
 
 .. code-block:: python
 

docs/source/features.rst

@@ -0,0 +1,74 @@
+Features
+========
+
+As seen previously in the tutorial. We can easily create features that will be returned as observation at each time step.
+This type of feature is called a **static feature** as it is computed once, at the very beggining of the DataFrame processing.
+
+.. hint::
+
+    **But what if you want to use a feature that we can not pre-compute ?**
+
+In this case, you will use a **dynamic feature** that will be compute at each step. 
+
+Create static features
+----------------------
+
+.. code-block:: python
+
+  # df is a DataFrame with columns : "open", "high", "low", "close", "Volume USD"
+  
+  # Create the feature : ( close[t] - close[t-1] )/ close[t-1]
+  df["feature_close"] = df["close"].pct_change() 
+  
+  # Create the feature : open[t] / close[t]
+  df["feature_open"] = df["open"]/df["close"]
+  
+  # Create the feature : high[t] / close[t]
+  df["feature_high"] = df["high"]/df["close"]
+  
+  # Create the feature : low[t] / close[t]
+  df["feature_low"] = df["low"]/df["close"]
+  
+   # Create the feature : volume[t] / max(*volume[t-7*24:t+1])
+  df["feature_volume"] = df["Volume USD"] / df["Volume USD"].rolling(7*24).max()
+  
+  df.dropna(inplace= True) # Clean again !
+  # Eatch step, the environment will return 5 static inputs  : "feature_close", "feature_open", "feature_high", "feature_low", "feature_volume"
+
+  env = gym.make('TradingEnv',
+    df = df,
+    ....
+  )
+
+
+.. important::
+
+  The environment will recognize as inputs every column that contains the keyword '**feature**' in its name.
+
+Create dynamic features
+-----------------------
+
+A **dynamic feature** is computed at each step. Be careful, dynamic features are *much less efficient* in terms of computing time than static features.
+
+.. important::
+
+    What is presented below is the default configuration of the dynamic features.
+
+.. code-block:: python
+
+    def dynamic_feature_last_position_taken(history):
+        return history['position', -1]
+
+    def dynamic_feature_real_position(history):
+        return history['real_position', -1]
+  
+    env = gym.make(
+        "TradingEnv",
+        df = df,
+        dynamic_feature_functions = [dynamic_feature_last_position_taken, dynamic_feature_real_position],
+        ...
+    )
+
+At each step, the environment will compute and add these 2 features at the end of the *observation*.
+
+

docs/source/history.rst

@@ -21,7 +21,8 @@ It was made to make everything easier :
    'step': 33091, #Step = t.
    'date': numpy.datetime64('2022-03-01T00:00:00.000000000'), #Date at step t, datetime.
    'position_index': 2, #Index of the position at step t among your position list.
-   'position': 1, #Portfolio position at step t.
+   'position': 1, # Last position taken by the agent.
+   'real_position': 1.09848, # Real portfolio position  = (asset owned - asset borrowed - asset interests) * current price / portfolio valuation
    'reward': 0.0028838985262525257, #Reward at step t.
    
    # DataFrame info : Every column (except features) of your initial DataFrame preceded by 'data_'

docs/source/index.rst

@@ -88,6 +88,7 @@ Contents
 
    rl_tutorial
    customization
+   features
    multi_datasets
    vectorize_env
 

docs/source/rl_tutorial.rst

@@ -89,7 +89,7 @@ Create your features
 --------------------
 
 Your RL-agent will need inputs. It is your job to make sure it has everything it needs. 
-**The environment will recognize as inputs every column that contains the keyword 'feature' in its name.**
+
 
 .. code-block:: python
 
@@ -112,10 +112,15 @@ Your RL-agent will need inputs. It is your job to make sure it has everything it
   
   df.dropna(inplace= True) # Clean again !
   # Eatch step, the environment will return 5 inputs  : "feature_close", "feature_open", "feature_high", "feature_low", "feature_volume"
-  
+
+.. important::
+
+  The environment will recognize as inputs every column that contains the keyword '**feature**' in its name.
+
+
 .. note::
 
-  By default, the env will always add the **position reached** at the end of all your custom features. Indeed, in Reinforcement Learning, I find it really useful for the agent to know its current position. To disable this, you need to set the ``include_position_in_features`` parameter of the environment to ``False``.
+  By default, the env will always add the 2 dynamics features. More informations in the **Feature** page.
 
 
 Create your first environment

examples/example_environnement.py

@@ -50,6 +50,6 @@ def reward_function(history):
 while not done and not truncated:
     action = env.action_space.sample()
     observation, reward, done, truncated, info = env.step(action)
-
+    print(observation)
 # Save for render
-env.save_for_render()
+# env.save_for_render()