When training the network, we often encounter many problems. For the problem of gradient disappearance, we can alleviate the problem by choosing which of the following functions to use?

Question

Options

• ASoftsign function
• BRelu function
• Ctanh function
• DSigmoid function

• A
  16% (10)
• B
  70% (43)
• C
  3% (2)
• D
  10% (6)

Explanation

ReLU (Rectified Linear Unit) directly addresses gradient vanishing because its derivative for positive inputs is a constant 1, meaning gradients pass through unchanged during backpropagation rather than being multiplied by a small fraction at each layer. Sigmoid and tanh both suffer from saturation: their outputs are squashed into narrow ranges (0 to 1 and -1 to 1 respectively), so their derivatives approach zero for large or small inputs, causing gradients to shrink exponentially as they travel back through layers. Softsign shares this saturation behavior and similarly allows gradients to diminish, just more slowly than sigmoid or tanh. ReLU avoids this by being unbounded on the positive side, preventing the repeated multiplication of tiny values that kills the gradient signal.

Memory tip: Sigmoid, tanh, and Softsign all "squeeze" their outputs into a bounded range -- squeezing outputs means squeezing gradients. ReLU does not squeeze on the positive side, so gradients stay alive.

No community discussion yet for this question.