Unsupervised Learning at the Rework Deep Learning Summit

It was a treat to see these three – Yoshua Bengio, Yann Lecun, and Geoffrey Hinton – for an afternoon. Easily the best three consecutive hours I’ve ever seen at a conference.

They remarked that Canada continues to invest in primary research. And this is a strength. Much of the exploratory work these three executed in the 80’s, 90’s and naughties was foundational to industrial applications which came after.

Much of reinforcement and deep learning has moved on into industrial application. For the three grandfathers of deep learning, all of these algorithms and methods move into the realm of solved problems.

For those of us in industry, there remains a lot of work to realize the benefits of deep learning. Data pipelines don’t build themselves. People just don’t educate themselves. Policies don’t just self-update (yet). There’s so much to be done industrially.

For those in academia, there remains a lot to do. As Hinton previously stated, he doesn’t think that human brains learn the same way that computers do. And there’s a lot of truth to that. A baby doesn’t absorb seventy millions of rows to figure out the word for water. A baby engages with its environment and experiments with cause and effect to update itself. A baby is able to produce saliency from a narrow set of data. It’s as though the baby is able to move through a bottleneck and form new experiments to attempt much more rapidly.

Moreover, the brain has to adapt to the primate body it’s riding. There are eyes to focus. There are vocal cords to figure out how operate. A neck to operate. A tongue to move. Hands to figure out how to work. And then a lot of that stuff has to be put together to walk, and then walk while talking.

A baby uses five senses to change its environment. She tosses a cup from the high chair and watches the mess that ensues. She hits a plastic surface to listen to the sound it makes. She jams a triangle shaped block into a triangle shaped hole to see if the triangle fits into the slot. It’s wonderfully iterative. And she herself has no idea that she’s linearizing a function to make sense of a given phenomenon like the tendency of heavy things to fall towards the floor in the bedroom, but not in the bathtub. As far as she’s concerned, gradient descent is what happens when you toss a ball or refuse to eat anything green.

What happens when a machine starts to use their senses and can change their environment? What happens when a machine starts to infer about its environment, without a human to, in essence, supervise its every move?

It becomes unsupervised.

To date, we’re using simulations. Researchers generate a simulated environment and then enable the machine to sense that environment and ultimately change that simulated environment. And there’s progress. There’s such a long way to go.

I’m excited for those challenge and what lies ahead, both industrially in grappling with the applications, socially in arming humans to adjust to the present, and for the researchers up against the challenges of unsupervised learning.