Last week we had the pleasure of welcoming Alex Korbonits to speak at Idibon about Deep Learning. Practically speaking, Alex gave us the low-down on the different tools that people are using to do Deep Learning. Inspirationally speaking, he showed us how computers imagine the Seattle skyline and how they would write if you taught them only James Joyce.
So let’s start with the ooooh and end with the hooooww.
Deep Learning, like other forms of machine learning, is about finding patterns in data. The “depth” of Deep Learning is that it involves a bunch of layers that feed into each other, each layer extracts higher-level features until the final layer where a decision is made.
One of the ways to figure out what’s happening in all those layers is to ask the computer to exaggerate what it’s finding at a given layer. That’s how you get stuff like the doge below—because Google’s training information contains so many dogs and faces, it “sees” dogs and eyes all over the place. (For more information check out Google’s blog here.)
Source: http://bit.ly/1DJoX4t
On the literature front, Alex loves James Joyce, so he wanted to see what Deep Learning would do if he gave it Ulysses and said, “Write me something.” You can read about how he did this (so you can do it yourself…I am) on his blog here. You’ll need to read this as poetry if you’re going to enjoy it.
Bloom works. Quick! Pollyman. An a lot it was seeming, mide, says, up and the rare borns at
Leopolters! Cilleynan’s face. Childs hell my milk by their
doubt in thy last, unhall sit attracted with source
The door of Kildan
and the followed their stowabout over that of three constant
trousantly Vinisis Henry Doysed and let up to a man with hands in surresses afraid quarts to here over
someware as cup to a whie yellow accept thicks answer to me.
As Alex notes, all that he gave the computer was Ulysses. It didn’t know English at all, yet it’s able to make up words that are fairly English-like and it even gets some grammar right-ish–notice the prepositions.
Meanwhile, Andrej Karpathy feeds in Tolstoy and gets:
Pierre aking his soul came to the packs and drove up his father-in-law women.
(Now to synthesize! Read Gretchen McCulloch’s Grammar of Doge.)
Time to get practical. Here are the Deep Learning tools that Alex reviewed for us:
- Torch: This is what Alex actually used for his James Joyce project; one of its benefits is a large number of packages so you don’t have to start from zero. Additionally, it is increasingly the tool of choice used for doing deep learning research.
- Caffe: Like Torch, has a large amount of work already done that you can build off of, one of the easier to use. One of its main strengths is its Model Zoo, where many reference models are already built and pretrained, so if you’re chomping at the bit, you don’t have to wait weeks to train a larger model such as AlexNet.
- Theano: This is probably the most sophisticated tool out there but people tend to find it pretty complicated. However, lots of popular Pythonic projects are being created on top of it and used in places such as Kaggle competitions (check out Keras, PyLearn, and Lasagne).
- GraphLab Create: From a company called Dato, this is also one of the easier tools to use if you’re just getting started. Given a dataset, their toolkit will pick a “sane default” network topology so that you don’t have to build one from scratch.
Since layers are a big part of Deep Learning, we’ll conclude with a picture of Alex presenting to us followed by what two different layers are seeing in the image—the first layer is seeing contours of shapes, while the second is hallucinating all kinds of different more abstract shapes.
Corpus linguistics 