faq.md

FAQs

Q: If the weight of a conv layer is zero, the gradient will also be zero, and the network will not learn anything. Why "zero convolution" works?

A: This is wrong. Let us consider a very simple

$$y=wx+b$$

and we have

$$\partial y/\partial w=x,\partial y/\partial x=w,\partial y/\partial b=1$$

and if $w=0$ and $x\ne 0$, then

$$\partial y/\partial w\ne 0,\partial y/\partial x=0,\partial y/\partial b\ne 0$$

which means as long as $x\ne 0$, one gradient descent iteration will make $w$ non-zero. Then

$$\partial y/\partial x\ne 0$$

so that the zero convolutions will progressively become a common conv layer with non-zero weights.