Is the SNN taking the temporal dimension into consideration?

[Dola47]

Dear @jeshraghian ,

I would like to know if the SNNs are really learning something from the temporal dimension or not. As most frequently the ANNs outperform the SNNs even if I only calculate the accuracy from the last frame of my temporal dimension. This should be weird because the SNNs should remember some information from the temporal domain while the ANNs should not!!

Thanks and looking really very quickly for your response.

[jeshraghian]

Hey @Dola47 ,

Your architecture is quite deep compared to most of those reported in SNN papers.
While vanishing gradients have been addressed in non-spiking networks thanks to good parameter initialization strategies, and batch norm etc., we still don't have equivalent methods that translate perfectly to SNNs.
Until the research community find answers to that, deep ANN/CNN structures will likely outperform SNNs regardless of temporal learning.

I recommend either:
A) doing a parameter sweep to find better values for your surrogate gradient, threshold, architecture, etc.
B) Shrinking your network
C) Adding explicit recurrent connections
D) Reading papers in the field to see what types of networks are more commonly used (there are some references in the tutorials)

Good luck.

[Dola47]

Hi @jeshraghian ,

Really thanks for your quick response it's much appreciated.

So, If I got it correctly there is no problem with my above SNN model implementation and it should be functioning correctly.

A) What do you mean by surrogate gradient, do you mean the slope of surrogate gradient? If so can you briefly clarify what is actually the slope in the Surrogate gradient?

B) Actually for the network and architecture I made a lot of trials, and I have simpler architectures where also the ANN outperforms the SNN.

C) Can it be added to my current architecture. If so can you please illustrate to me how that is can be achieved by a simple coding example?

D) I Will have a look.

E) Do we have any chat channel for the Simulator in Discord for example, where we can get in contact with you in case of urgent stuff?

Really thanks and looking forward to your further responses.

[jeshraghian]

E) Do we have any chat channel for the Simulator in Discord for example, where we can get in contact with you in case of urgent stuff?

As much as I'd love to be available for urgent stuff, unfortunately I have a day job and can't be on-call for 24/7 homework help 😂

[Dola47]

B) This one's hard to answer. You could have either proven a fundamental flaw with SNNs, or that the temporal dimension of your data may just be tough for a simple SNN to learn from. Things I'd test would be to set the decay rate alpha and beta to be learnable (similar to a recurrent neural net).

Actually, SNNs work better than ANNs but after a high number of training epochs, and it exceeds it slightly also on a network with very small computations. When the number of computations increases the ANNs exceed it. I do not get it what do you mean by setting the alpha and beta to be learnable? The only thing I thought of is to do hyper-parameter optimization on them.

[jeshraghian]

When the number of computations increases the ANNs exceed it.

Yeah, so my guess is due to the vanishing gradient/dead neuron issue I explained earlier. Deep SNNs are tough to train.

I do not get it what do you mean by setting the alpha and beta to be learnable? The only thing I thought of is to do hyper-parameter optimization on them.

A parameter sweep is fine, though if you have 100 spiking neurons in a layer, you have a couple of options.
-you can set a global alpha/beta, or
-you can individually set alpha/beta for each of the 100 neurons

Obviously doing the second is really tough via hyperparameter optimization, as you are currently doing.
So setting them to learnable values might help attain a more meaningful temporal dependence; nn.Parameter(beta).
I haven't experimented with this much, though.

[Dola47]

So setting them to learnable values might help attain a more meaningful temporal dependence; nn.Parameter(beta).
I haven't experimented with this much, though.

Ahhhhh, yes looks quite logical. But I am not aware of how to implement them to be learnable. I would be glad if you can provide a small snippet of how can I modify the code above to make them learnable?

[jeshraghian]

@Huizerd very kindly just implemented learnable time constants into the latest merge on GitHub!

Instead of passing beta and alpha directly as an argument, in the older version of snnTorch you're currently using, you should pass it into nn.Parameter(alpha) first.

If alpha and beta are just scalar tensors (i.e., single valued), then that will implement one single decay rate for all neurons.

Otherwise, you can assign independent decay rates to each neuron by sizing alpha and beta appropriately.

[Dola47]

Away from our previous discussion here are some of my findings from my current work that I would like you to have a look at:

1- SNNs exceed the accuracy of ANNs on networks with very low computations and after having a sufficient number of time steps or temporal dimensions.

2- ANNs will perform better on networks with a high number of computations aside from the number of steps you have.

3- As a conclusion from 1, I see that of course there should be some temporal information learned within the SNNs but it is not really very clear until we have a sufficient number of steps.

Questions:

1- If for complex networks ANNs is better and if we thought about digital implementation we have an extra number of computations in the SNNs that need to be done inside of Spiking Neuron, what is the benefit of SNNs in such an approach?

2- If only the temporal information to be learned from the SNNS is to be noticed after a large number of steps, why do not we go directly to GRUs or RNNs, or any model that has some kind of memory?

I am looking forward to your thoughts.

Thanks Jason :)

[jeshraghian]

This is great insight - thanks a lot for sharing!

1- SNNs exceed the accuracy of ANNs on networks with very low computations and after having a sufficient number of time steps or temporal dimensions.

This is quite interesting. One thing to be mindful of is that each time step adds another layer of computation, so SNNs might require more computations than you are considering. However, the advantage is that those computations are much easier to do (i.e., because spikes are just 1s and 0s). Now I understand why you were asking about # of computations / TOPS / GOPS earlier on.

2- ANNs will perform better on networks with a high number of computations aside from the number of steps you have.

Yeah - this sounds right. Rather than number of computations, I would guess that the # of layers is the more important feature here.
I think that finding ways to avoid vanishing gradient problems might help with this issue in SNNs.

1- If for complex networks ANNs is better and if we thought about digital implementation we have an extra number of computations in the SNNs that need to be done inside of Spiking Neuron, what is the benefit of SNNs in such an approach?

This is a good question. I have an online tutorial that explains why spikes are advantageous (at least, in terms of energy when run on dedicated hardware): https://www.youtube.com/watch?v=O2-mT291ygg&ab_channel=ICONS

Obviously, GPUs are not designed to take advantage of spikes, so much of what we do now is just exploratory research.

But keep in mind, your conclusion is only for one specific method of training SNNs (i.e., backprop + gradient approximations).
ANN-to-SNN conversion comes very close to ANN performance, but using spikes. So the training process is not very efficient, but for inference-only, then you can achieve very close accuracy but with only using 1's and 0's for inter-layer communication on neuromorphic hardware.

2- If only the temporal information to be learned from the SNNS is to be noticed after a large number of steps, why do not we go directly to GRUs or RNNs, or any model that has some kind of memory?

I think the tutorial on youtube covers this! The potential energy savings from spikes are the main reason. We are still not completely sure if spikes are technically better for computation. That's why I made snnTorch - to hopefully find a way to make them practical :)

[Dola47]

Thanks a lot for this discussion and your answers, they look interesting I will go over them Tomorrow and hopefully come back with some nice questions :).

One thing I might add for your comment here:

2- ANNs will perform better on networks with a high number of computations aside from the number of steps you have.
Yeah - this sounds right. Rather than number of computations, I would guess that the # of layers is the more important feature here.

From my findings, that is actually not completely true. I already had some Spiking-CNNs with 4 or 5 convolution layers. However, I made the strides quite big and applied Max pooling to decrease the number of computations and I still can find out that the SNNs can exceed the ANNs by 1 or 1.5% of testing accuracy. So I think ANNs vs SNNs is mainly related to the number of computations in the first place.

[jeshraghian]

So I think ANNs vs SNNs is mainly related to the number of computations in the first place.

This is super fascinating. I'd love to hear more. Keep me posted if you publish or post your findings anywhere!

[Dola47]

Hi @jeshraghian ,

Should not I be able after the latest merge to set learn_beta and learn_alpha to True, and automatically the code internally will instantiate self.alpha = nn.Parameter(alpha)? What happens for me after updating to the latest version of snnTorch, that I see that the learn_beta and learn_alpha are not accessible yet.

Sorry, but I do not get it completely how to set alpha and beta to be trainable now.

Thanks.

UPDATE:

I just checked that the pull request has not been merged yet, sorry!! But, it will be totally helpful if you can as well give a short example of how it can be implemented :)

[Dola47]

@Huizerd Why large values for alpha and beta will not be able to capture faster dynamics? So, if we have a movement done all over the 100 timesteps and you still remember most of the 100 timesteps you should be able to detect the dynamics done within these timesteps?
Can you clarify this a little a bit?

Also, if I set the initial value to 0.99 for example, can not it be the case that will it learns it shifts dramatically for this big value and have a very small value, and then the neurons will not be able to take into consideration the old timesteps?

Moreover,

(torch.randn(layer_size) * 0.1 + 0.99).clamp(0, 1) . I usually set the Synaptic neuron after one of my conv layers, I think you mean by the layer_size in such a case the number of output channels of the conv layer if I am correct?

[Huizerd]

@Huizerd Why large values for alpha and beta will not be able to capture faster dynamics? So, if we have a movement done all over the 100 timesteps and you still remember most of the 100 timesteps you should be able to detect the dynamics done within these timesteps?
Can you clarify this a little a bit?

Fast dynamics is synonymous with fast forgetting, so it's either long memory and slow dynamics, or short memory and fast dynamics. You can't have both with the same neuron.

Also, if I set the initial value to 0.99 for example, can not it be the case that will it learns it shifts dramatically for this big value and have a very small value, and then the neurons will not be able to take into consideration the old timesteps?

Yes, that can happen, but it will take lots of learning. It can even be so that the value is so far off that no learning will happen.

Moreover,

(torch.randn(layer_size) * 0.1 + 0.99).clamp(0, 1) . I usually set the Synaptic neuron after one of my conv layers, I think you mean by the layer_size in such a case the number of output channels of the conv layer if I am correct?

For convolutions, I think you can do either (1, 1, out_channels) or (height, width, out_channels), depending on whether you want a neuron per pixel or not. Consequence of the latter would be however that you cannot deal with images of different shapes.

[Dola47]

Thanks a lot, @Huizerd for this illustration.

I am currently testing it, and it looks that very small numbers working correctly with me. But, I am just setting the beta and alpha to a certain value and setting the learn_beta and learn_alpha to True

(torch.randn(layer_size) * 0.1 + 0.99).clamp(0, 1)

My question was what you mean with layer_size here is the output layer size, so it is the number of output channels in case we are talking about CNNs.

Moreover, would it really be different if we set the initial values to certain values instead of a distribution?

Really thanks.

[Huizerd]

My question was what you mean with layer_size here is the output layer size, so it is the number of output channels in case we are talking about CNNs.

As I said, layer_size can be the number of output channels, or a tuple of channels and pixels.

Moreover, would it really be different if we set the initial values to certain values instead of a distribution?

Well, yes, if the distribution is wide enough?

[Dola47]

@Huizerd Just a quick note, I tried really to use the learn of alphas and betas over and over, and usually I have a severe accuracy decrease when I apply them, I think they shift quietly away from the initial values.

[Dola47]

@jeshraghian Just a follow-up question to our previous discussion. If we agree that we will have an increase in the number of computations in the spiking neuron compared to the classical neuron, from a digital implementation perspective the SNNs will not be more energy efficient than the ANN version of it. It will be energy efficient only if we considered the analog implementation. Then, I think the only benefit of having the SNN in the digital domain is to get better accuracy than the ANN in a fewer number of epochs, do you agree?

[jeshraghian]

I get what you mean, although you're making several assumptions that might not hold true in practice. The main one is that the number of operations is the only thing that matters, though this is not the whole picture.

• Rather than comparing ANN to SNN, it is more fair to compare RNN to SNN, in which case the number of operations will be closer.

• Though let's assume we are comparing ANN to SNNs. The number of operations (OPs) is not the only relevant metric. OPs/sec and OPS/watt are very important too. When you communicate activations between layers, hardware dedicated to SNNs only ever needs to transmit 1/0, whereas ANN might have to work with int8 or flt32. Thats 8x / 32x more data being communicated, being stored in registers etc. So even if SNNs have, say, 2x more operations, then that can definitely be offset by using single-bit activations. Of course, membrane potential / hidden state variables are not 1s and 0s, but keep in mind that those values are not communicated between layers. So there are definite savings there.

• Beyond data communication between layers, you also need to account for memory accesses. Spikes are just 1s and 0s. So if you train your neurons in a way that promotes sparsity, then every time your activation is 0, then you don't need to waste time or energy on reading out the weight it is scaled by. And even if it is just 1, then the multiply operation in a conventional ANN (which comes with carry-delay propagation / critical paths) is just swapped with a memory read-out (X*W=W). So not only can you reduce memory accesses, but you reduce the computational complexity by eliminating the multiply step.

We performed a comparison of energy/latency/accuracy in this paper on a variety of SNN/ANN hardware here: https://ieeexplore.ieee.org/abstract/document/9250613/

And I explain some of this in a high-level in this tutorial from the ICONS conference: https://youtu.be/O2-mT291ygg

[Dola47]

@jeshraghian Thanks a lot for this comment :)

[Dola47]

@jeshraghian I can see your points. However, I still wonder because if you have digital neurons you have to update the membrane potential over time to model the leaking. And, when you calculate the neuron on digital hardware you will only have a number of arithmetic units for all neurons. You have a memory containing the state variable of the neurons and a calculation unit that fetches this data and modifies it. Should not all of that be considered as disadvantages from the digital hardware perspective?

[jeshraghian]

If you know that you're updating your state at every time step, a well designed system won't fetch from SRAM/main memory. That is quite inefficient. Instead, do all processing on the data across time steps, storing intermediate results in local registers, and that saves you from much of the challenges you just described. You should check out the literature on systolic arrays.

Also strongly recommend you check out the paper I linked in the previous comment.
Even though an ANN does not have to track hidden states, the evidence is quite strong that the efficiency benefits outweigh the disadvantage of tracking recurrent states. In my opinion, I'd say the biggest challenges we face with SNNs right now are algorithmic.

[Dola47]

@jeshraghian A side from our discussion please inform me of how should I cite the snnTorch simulator?

[jeshraghian]

Of course! We have an associated paper just about to be uploaded to arXiv this week. Once that's online in a few days, you can cite the following. I will update the identifier once it's online:

J. K. Eshraghian, M. Ward, X. Wang, G. Lenz, G. Dwivedi, M. Bennamoun, D. S. Jeong, W. D. Lu, Training Spiking Neural Networks Using Lessons From Deep Learning, arXiv, Sep. 2021.

[Dola47]

@jeshraghian I think I can use the following citation, as you already mentioned your paper is not already published
Eshraghian, J., 2021. GitHub - jeshraghian/snntorch: Deep and online learning with spiking neural networks in Python. [online] GitHub. Available at: <https://github.com/jeshraghian/snntorch>.

[jeshraghian]

We finally got the associated paper with snnTorch online!
The readme has the corresponding bibtex entry.

https://arxiv.org/abs/2109.12894

[Dola47]

Congrats Jason!!

I will have a look at it as soon as possible.

Hope it comes a little earlier, I already cited snnTorch as a GitHub repository as I mentioned earlier, on work submitted a few weeks ago.

However, I will cite then any of my further work using the citation form of the paper :).

[Dola47]

@jeshraghian Sorry for frequently raising questions, but I have a few urgent questions.

I am aware that we are approximating the non-differentiability of the spikes by the gradient of the surrogate gradient function in the backward pass, but can you clarify more what actually the slope of the surrogate gradient means? Is it like how much we are approximating the true gradient of for example sigmoid function? If so how does it mean if we set high or low values for the slope of the surrogate gradient?
Moreover, the slope looks like it is completely ineffective. I am modifying it as:
spike_grad = surrogate.FastSigmoid.apply
surrogate.fast_sigmoid(slope=100)
And even changed their orders but looks like changing the value of the slope is not effective anywhere in the code?

Thanks.

[Dola47]

Thanks a lot for your reply.

You should use spike_grad=surrogate.fast_sigmoid(slope=100) instead.

Should also we do surrogate.FastSigmoid.apply after this step, or it is not needed as we already modified the slope?

So, If I got it correctly, we can either do surrogate.FastSigmoid.apply then we will have the default slope of 25. Or do surrogate.fast_sigmoid(slope=100) without apply if we modified the slope. Is that correct?

[jeshraghian]

Your understanding is spot on; no need to apply the function anymore if you use fast_sigmoid().
The docstrings of the surrogate method should have examples of both approaches.

[Huizerd]

To add to your point about the surrogate gradient: the slope defines how "wide" the surrogate gradient is around your threshold. For the fast sigmoid, if you set the slope low (like 10), the surrogate gradient is quite wide, meaning that even if your voltage is far from the threshold, there will be a gradient (so even in the case of no spike). The higher you set the slope, the narrower the surrogate gradient becomes, meaning that, in the extreme, there will only be gradient if the voltage exactly matches the threshold.

[Dola47]

@Huizerd That was really an awesome explanation. Thanks a lot :)

[jeshraghian]

@Huizerd great explanation! :)

[Dola47]

Hi @jeshraghian ,

Is it expected to have any kind of ResonanteFire neuron added soon? If not do we have any current type of neuron that resembles it?

Thanks.

[jeshraghian]

Never heard of it - can you describe how it works?

[Dola47]

Here is a popular publication about it:
https://www.sciencedirect.com/science/article/pii/S0893608001000788

It should be similar to the leaky-integrate and fire, however, it spikes depending on relative frequencies it can detect from the input signal.

It is very helpful for continuous-wave signals.

[jeshraghian]

Of course - Izhikevich's neuron!
I currently plan on adding 2 neurons at the moment: a linearly decaying version of Leaky, and this one.
We can add these as issues in case someone manages to get to it first.
I would guess backprop might be challenging with this neuron as it is more complex, but still worth including.

[Dola47]

Hi @jeshraghian ,

Just a follow-up question related to our previous temporal dimension discussion.

I started to get the things mixed up and would like you to make it clearer for me.

I have my input dimension in the shape of [frames, chirps, samples], where the frames represent the time dimension.

For that, I sent one frame at a time to my network, so my forward pass looks like

def forward(self, x):
        num_steps = x.shape[1]
        batch_size = x.shape[0]

        syn1, mem1 = self.lif1.init_synaptic()
        syn2, mem2 = self.lif2.init_synaptic()
        syn3, mem3 = self.lif3.init_synaptic()
        syn4, mem4 = self.lif4.init_synaptic()

        spk4_rec = []
        mem4_rec = []

        for step in range(num_steps):
            cur1 = self.bn1(F.max_pool2d(self.conv1(x[:, step]), 2))
            spk1, syn1, mem1 = self.lif1(cur1, syn1, mem1)

            cur2 = self.bn2(F.max_pool2d(self.conv2(spk1), 2))
            spk2, syn2, mem2 = self.lif2(cur2, syn2, mem2)

            cur3 = self.bn3(F.max_pool2d(self.conv3(spk2), 2))
            spk3, syn3, mem3 = self.lif3(cur3, syn3, mem3)

            # Flatten for the FC Layer
            spk3 = spk3.view(batch_size, -1)
            cur4 = self.fc1(spk3)
            spk4, syn4, mem4 = self.lif4(cur4, syn4, mem4)

            spk4_rec.append(spk4)
            mem4_rec.append(mem4)

        return torch.stack(mem4_rec, dim=0), torch.stack(spk4_rec, dim=0)

Where the num_stepsare actually my frames .

Following the above implementation (which is basically being used in most of the tutorials), I think then we are doing:

1- Synchronous implementation of the neurons, as we send one frame at a time in sequence, and I think by default spiking neurons should be Asynchronous? I think the same can be thought of, if we created encoding of a number from MNIST and sent its time_steps one at a time to the network it will also be Synchronous and not Asynchronous?

2- I checked the source code of the Snyaptic neuron, and I found that we are not weighting the spikes, we are just doing syn = alpha*syn + input_, so the neurons themselves do not have any learnable weights or biases, and the only thing that weights the spikes are the weights of the kernel of the Conv layers?

3- We do not have really integration in each num_step, over the spikes or the synaptic current but we just apply the 2 equation for each num_steps, that really should be very simple for the digital hardware implementation, only 2 equations only of adds and multiplication, am I right?

4- Also, I am sending my float data represented in [chirps, samples] to be convolved and then based on the Synaptic neuron, are the spikes generated can be called rate-encoded or temporal-encoded spikes, or not both are incorrect as we will produce spikes when we pass the threshold and as we have not done any kind of encoding beforehand.

5- Finally, his current implementation of the neurons makes the SNN the same as BNN (binary-neural-networks) except that they do not have synaptic current and membrane voltages?

Thanks a lot and looking forward to your response?

[jeshraghian]

May I ask also where all of that is implemented in the source code

That derivation is showing how those two equations are derived.
Though is this from the old version of the tutorial? I recently updated & split them up to hopefully make them easier to follow.

[Dola47]

Dude that paper is my own words 😂

Hahahahahahahahahahahah, totally aware of that :D :D.

That derivation is showing how those two equations are derived.

So the thing is what I got if I just followed the equation written in the source code, then the integration over the input spikes and the integration over the membrane voltages should be taken into consideration, right?

I am totally dead right now, maybe I will try to go over the paper Tomorrow and Tutorial 2 again and re-think :).

But, as a final consideration, as far as I can see there is no weighting being done to the spikes themselves from layer to layer except for the weighting in the neural network layers, correct?

[jeshraghian]

Yeah man rest up, have another shot with the updated version of tutorial 2, and let me if anything is unclear.
The new tutorials try to explain the source code as well, so that might be useful for you.

So the thing is what I got if I just followed the equation written in the source code, then the integration over the input spikes and the integration over the membrane voltages should be taken into consideration, right?

Integration: I[t+1] = I[t] + WX[t]
Leakage: I[t+1] = a * I[t] , given 0<a<1

Separate note-
Synaptic will integrate the input (with leakage), and then membrane potential integrates it again - effectively extending the time constant.
Leaky skips that first step, such that membrane potential integrates the input directly. Not as biologically accurate, but simple.

But, as a final consideration, as far as I can see there is no weighting being done to the spikes themselves from layer to layer except for the weighting in the neural network layers, correct?

Exactly yep. Treat Pytorch's nn layers as the synapses (weighting), and treat snntorch's LIF neurons as... well, LIF neurons.

[Dola47]

Hi @jeshraghian ,

I have gone over Tutorial 2 again, and that is what I have understood so far are:

• The general solution of the LIF can be summarized as:
  [1]
  and it is exactly the same as
  [2]

So, following equation [1] we should be by default integrating over the input and decaying as well in case of the absence of the input spikes.

In the source code equation [1] should be exactly the same as
[3].

And what you meant by

wrt t there's integration.

That is because for example equation [1] should be run every time step then simply this run over time steps resemble the integration. However, if we only go over one single time step where we apply equation [1] only once, then we have only added and multiplications and that refers to your quote here

wrt U[t-1] (as a recursive expression) - multiply/adds/comparisons only.

If all of my above thoughts is correct then for the Synaptic Spike Neuron the general solution equations should be:
[4]
and
[5]

Equations [4] and [5] should by default also contain the integration and decaying processes. I think also for the digital hardware implementation these are the only 2 equations we should care of implementing each time step to have a working version of the Synaptic Spike Neuron over hardware.

Please, correct me if I am wrong.

If all of my thoughts are correct, I recommend that tutorial 4 be formatted to remove this equation from it
because it will just make the reader think that there will be a weighting from the input current, and it is not even written in the source code, and stick to equation [4] instead.

Finally, I believe that if my thoughts as well are correct, that it should be written precisely that equations [1] and [2] are exactly the same in tutorial 2.

Thanks and looking forward to your response.

[Dola47]

Hi @jeshraghian ,

Any further response? In bad need of your answers before Thursday :).

[Dola47]

Hi @Huizerd ,

Is it right to summarize the Surrogate gradient idea by stating that It is an idea where the gradient of the undifferentiable spikes is replaced by the gradient of another differentiable function in the backward pass.?

Thanks.

[Huizerd]

Hi @Dola47,

That's correct! The surrogate gradient idea actually builds on the "straight-through estimator" (STE) introduced by Hinton in his 2012 Coursera lectures on deep learning, and described in more detail Bengio et al. arXiv 2013. There, they use the STE to allow gradient propagation in binary neural networks, where the binarization function is non-differentiable. For surrogate gradients in SNNs this idea is extended to not just copy the incoming gradient in the backwards pass, but to multiply it with some shape (like the sigmoid derivative, or a piecewise-linear, etc.).

[Dola47]

Hi @jeshraghian ,

Do we consider the refractory period in our neurons?

Moreover, is it expected to have any kind of recurrent-LIF implemented soon?

Thanks.

[jeshraghian]

Refractory periods haven't been implemented here, though they would be useful. Let's add that as a feature in the issues.

Recurrent LIF neurons are also on the to-do list.
They are quite easy to hard-code by simply concatenating the output spike with the input.