The Vanishing Gradient Problem

While neural networks are sometimes intimidating structures, the mechanism for making them work is surprisingly simple: stochastic gradient descent. For each of the parameters in our network (such as weights or biases), all we have to do is calculate the derivative of the loss with respect to the parameter, and nudge it a little bit in the opposite direction.

Disappearing gradients

Stochastic gradient descent seems simple enough, but in many networks we might begin to notice something odd: the weights closer to the end of the network change a lot more than those at the beginning. And the deeper the network, the less and less the beginning layers change. This is problematic, because our weights are initialized randomly. If they're barely moving, they're never going to reach the right values, or it'll take them years.

(read more)
Throwing (and Thinking) Like a Girl

Throwing like a girl is a serious matter. Empirical evidence shows that men significantly outperform women when it comes to throwing velocity and speed (by 2.18 and 1.98 standard deviations, respectively). You can probably easily recognize the signs of a "girly" throw. Its perpetrator faces the target, rather than rotating her hips and shoulders to be perpendicular to the target. She may step forward with the wrong foot (the foot on the same side as her throwing hand), or just use her forearm rather than her back and shoulders.

And what about thinking like a girl? There's lots of evidence suggesting that's not a compliment either. When looking at school-age children, studies have consistently found that more boys than girls fall in the the top percentiles of standardized test scores such as AP Calculus, the math section of the SAT, or the quantitative part of the GRE. Over the past 20 years, there have consistently been double the number of boys than girls scoring in the top 5% of high school math assessments.

At first glance, throwing and thinking are not similar. Thinking is deliberate and calculated, throwing seems instinctual and raw. "I don't really think about the process,'' a baseball-player friend of mine once shrugged, when I demanded to know how he managed to throw the ball so fast and accurately. "I just decide where to throw, and my body just does the rest.''

Let's dig a little deeper, though.

(read more)
A New Paint Job

I've been fortunate enough to cross paths with many mountains over the past year:

Mountains are all very different. Some are formed by the tremendous energy of tectonic plates crashing together or skidding against each other. Sometimes hot magma from deep within the Earth bubbles upwards, cooling and hardening into smooth granite before it cracks the surface. Others are created by the forceful rush of rivers carving deep valleys and leaving mountains protruding high between them.

(read more)

© 2021 Harini Suresh. All rights reserved.