ESN with multiple reservoirs and only one readout

[jhklarcher]

Hello! Thanks for the amazing package!

I'm trying to create a ESN that has multiple reservoirs, but only one readout, and I'm having trouble figuring out how to get it running.

The scheme I'm trying to use is something like the following:

                 u[n]
                  |
                  |
         _________o_________
        |                   |
  _ _ _ | _ _ _       _ _ _ | _ _ _ 
|      (i)     |    |      (i)     |
|Input-to-Node1|    |Input-to-Node2|
| _ _ _ _ _ _ _|    | _ _ _ _ _ _ _|
        |r1'[n]             | r2'[n]
  _ _ _ | _ _ _       _ _ _ | _ _ _
|     (th)     |    |     (th)     |
| Node-to-Node1|    | Node-to-Node2|
| _ _ _ _ _ _ _|    | _ _ _ _ _ _ _|
        |r1[n]              | r2[n]
        |_____         _____|
              |       |
            _ | _ _ _ | _  
          |               |
          |Node-to-Output |
          | _ _ _ _ _ _ _ |
                  |
              y[n]|

As I understand, using a ESNRegressor at the end would produce another NodeToNode reservoir, just as in the examples, is that correct?

If so, is there any way to have only one readout with multiple reservoirs?

If I train the network using a ELMRegressor at the end (since it doesn't require a NodeToNode), would it produce the correct results from ESN with reservoir and states, since I use node to node in the middle?

Something like so:

import numpy as np
from pyrcn.base.blocks import InputToNode, NodeToNode
from pyrcn.extreme_learning_machine import ELMRegressor
from sklearn.pipeline import Pipeline, FeatureUnion

X = np.sin(np.linspace(0, 10 * np.pi, 3000)).reshape(-1, 1)

X, Y =  X[:-1, :], X[1:, :]

test_size = 1000

X_train, X_test = X[:-test_size, :], X[-test_size:, :]
Y_train, Y_test = Y[:-test_size, :], Y[-test_size:, :]

print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)

reservoirs = [
    ("path1", Pipeline([
        ("i2n1", InputToNode()),
        ("n2n1", NodeToNode())
    ])),
    ("path2", Pipeline([
        ("i2n2", InputToNode()),
        ("n2n2", NodeToNode())
    ]))
]

reservoirs = FeatureUnion(reservoirs)
esn = ELMRegressor(input_to_node=reservoirs)

esn.fit(X_train, Y_train)

Y_pred = esn.predict(X_test)

I can only imagine that using esn = ESNRegressor(input_to_node=reservoirs) would create another NodeToNode, just like in the examples, and I don't want that.

If you can give any help or guidance it would be much appreciated. :)

[renierts]

Hi @jhklarcher
thank you for this very good question. Your proposed solution with the ELMRegressor is correct.

Since I recently stumbled about something similar, I already noticed by myself that the ESNRegressor is not yet as flexible as I would like it to be. Thus, I am working on the issue. However, it will definitely take a couple of weeks.