Liquid State Machine

[Dola47]

Hi @jeshraghian ,

Have you ever used Liquid State Machine (LSM) in one of your previous works? If so, do you find it interesting? As a lot of people have already used it and kept saying that it is quite advantageous.

So, the main idea of the LSM can be summed up from the above figure from https://ieeexplore.ieee.org/document/8852472.

• The liquid (point 1 in the above figure) is a reservoir of Spiking Neurons connected recursively. We should be able to define some neurons to be excitatory and inhibitory. In a more advanced way, we should even be able to define the EE, IE, II, EI (excitatory to excitatory), (inhibitory too inhibitory) connectivity. However, all the weights inside of reservoir or liquid should be fixed and none of the weights should be trained.
• The state vector is a vector of spikes depending on the number of time steps we are going to use. Simply it is the output of the liquid.
• The only weights to be trained in the LSM are the weights from the state to the output layer.
• The main advantage of such a network is that you need only a few neurons inside the liquid and a very small number of neurons to be trained, and you overcome then the vanishing gradient in the case of the RNNs.

The question is, is that doable using snnTorch?

As far as I see we need:

• A recursive spiking neurons, and then we need a function to create a reservoir of them. Where it should take an input size and hidden size(number of neurons inside the liquid), and return the reservoir or liquid where all the neurons are recursively interconnected inside it, is that doable?
• We need to prevent all the weights inside the reservoir from being trained, is that possible?
• Is it possible to define the connectivity between the different neuron types? (I think this can be skipped for now)

Looking forward to your response.

And really thanks for always being supportive.

[jeshraghian]

Hey @Dola47,

LSMs are definitely an interesting idea.
Including the specific components of LSMs are worthwhile; it's on the roadmap to include excitatory & inhibitory spiking neurons, and recurrent LIF neurons.

Preventing weights from being updated is quite straightforward via PyTorch already, though I'm not sure if that's the best way to implement it:
E.g., https://discuss.pytorch.org/t/how-can-i-disable-all-layers-gradient-expect-the-last-layer-in-pytorch/53043/7

Defining the graph connectivity in the reservoir - EchoTorch does a great job in that space already: https://github.com/nschaetti/EchoTorch

It'd be worth looking to see if EchoTorch + SnnTorch could work together here.
If you, or anybody else, are up for looking into taking the lead on any of these issues, I'd be happy to support it.

In the meantime, I'll keep you updated on E/I neurons & R-SNNs.