Which of the following activation functions are prone to vanishing gradient problems?

Question

Options

• AReLU
• BSoftplus
• CTanh
• DSigmoid

• A
  17% (8)
• B
  7% (3)
• C
  76% (35)

Explanation

Tanh and Sigmoid both saturate - they squash inputs into bounded ranges ((-1, 1) and (0, 1) respectively), and their derivatives approach zero at the extremes. During backpropagation, repeatedly multiplying these tiny derivatives across many layers causes gradients to shrink exponentially, effectively stopping learning in early layers. ReLU is not prone to this because its derivative is exactly 1 for any positive input, allowing gradients to flow through unchanged. Softplus (a smooth approximation of ReLU) has a derivative equal to the sigmoid function, but because its output grows unboundedly for large inputs, it avoids the hard saturation that plagues Sigmoid and Tanh in practice.

Memory tip: Saturation causes starvation. If an activation function has a hard ceiling and floor on its output, the gradient "starves" as it tries to pass backward through those flat regions - Sigmoid and Tanh both have this property, while ReLU and Softplus do not.

No community discussion yet for this question.