algorithm - Extremely small or NaN values appear in training neural network

ID : 8933

viewed : 136

Tags : algorithmhaskellneural-networkbackpropagationalgorithm

Top 5 Answer for algorithm - Extremely small or NaN values appear in training neural network

vote vote


Do you know about "vanishing" and "exploding" gradients in backpropagation? I'm not too familiar with Haskell so I can't easily see what exactly your backprop is doing, but it does look like you are using a logistic curve as your activation function.

If you look at the plot of this function you'll see that the gradient of this function is nearly 0 at the ends (as input values get very large or very small, the slope of the curve is almost flat), so multiplying or dividing by this during backpropagation will result in a very big or very small number. Doing this repeatedly as you pass through multiple layers causes the activations to approach zero or infinity. Since backprop updates your weights by doing this during training, you end up with a lot of zeros or infinities in your network.

Solution: there are loads of methods out there that you can search for to solve the vanishing gradient problem, but one easy thing to try is to change the type of activation function you are using to a non-saturating one. ReLU is a popular choice as it mitigates this particular problem (but might introduce others).

vote vote


You can use Google directly.

How about this?

"your_string_to_search" site:: "your_string_to_search" site:: 
vote vote


Today I was trying to look for an exact match of filter class in files named logback.xml in any repo on Github. And I came up with the following query which did the job.

"filter class" in:file filename:logback.xml 

To enable exact matches with quotes you need to follow your search with the "in:file" modifier. The matches are not quite exact, the word "class" will have to follow the word "filter", but it seems there can be 0 or more spaces or symbols characters between the two words.

vote vote


Adding to @mrgloom's answer, if you're looking for code in a specific programming language in Github using Google you could do something like this in Google's search bar:

  • state the specific string you're looking for using the "intext:" search operator
  • add the programming language you're interested in, using the "ext:" operator (i.e. "ext:py", "ext:R", "ext:rb", etc.)
  • search in all public repos in Github using the "site:" operator mrgloom mentioned.


intext:"%% 2 == 0" ext:R 

Google Results from the example

vote vote


If your package is in debian, you can use their code search, which supports regular expressions:

Top 3 video Explaining algorithm - Extremely small or NaN values appear in training neural network