Transfer learning

In short

Transfer learning is training a model on one task and then reusing what it learned on a different task or a different data distribution. The reuse can help, and it can also make the new task worse.

Topic: How models are trained
Level: Some background helps
First described: Pan and Yang published this definition in their 2010 survey of an existing idea¹. The phrase is older: Yosinski and colleagues cite Caruana (1995) and Bengio (2011) for transfer learning².

Pan and Yang define transfer learning as improving the predictor on a target domain and task by using knowledge from a source domain and task, in the case where the source domain or the source task differs from the target¹. A domain is a pair: a feature space, and a probability distribution over that space¹. Two domains differ if the features differ, as when two document collections use different languages, or if the distributions over those features differ¹.

It is not multi-task learning. Pan and Yang say multi-task learning trains the tasks together, while transfer learning cares most about the target, so the source and target roles are not symmetric¹. It can also fail. When the source and target are unrelated, a blunt transfer may not work and, in the worst case, can hurt performance on the target. They call that negative transfer¹.

For deep networks, Yosinski and colleagues describe the usual approach. Train a base network, copy its first n layers into a target network, randomly initialise the remaining layers, then either fine-tune the copied layers or leave them frozen². If the target dataset is small and the number of parameters is large, fine-tuning may overfit, so the features are often left frozen². If the target set is large, or the parameter count is small, the copied features can be fine-tuned².

They measured this on the ImageNet 2012 training set: 1,281,167 labelled images in 1,000 classes². A random split into two groups of 500 classes trained eight-layer convolutional networks². Layers 1 and 2 transferred almost perfectly². First-layer features on natural images resemble Gabor filters and colour blobs². Fine-tuning still helped: the average gain across layers 1 to 7 was 1.6% over the base network, and 2.1% when at least five layers were kept². Controls trained for the same extra time on the same data did not show that gain². A more distant split, man-made classes against natural ones, transferred worse, but those features still beat random weights².

Goodfellow, Bengio and Courville describe the same reuse as ordinary practice: features from a convolutional network trained on ImageNet are commonly used for other computer-vision tasks³.

Example

Yosinski and colleagues copied early layers from a network trained on one random half of ImageNet's 1,000 classes, about 645,000 images, and used them to classify the other half². The first two layers transferred almost perfectly, and fine-tuning the copied layers beat a network trained only on the target half².

Why it matters

Most practical training does not start from random weights. It starts from a model that already saw a large related task, and the open question is whether those borrowed features still help once the new task drifts away from the old one.

Questions people ask

Is transfer learning the same as fine-tuning?

No. Fine-tuning is one choice after the transfer. Yosinski and colleagues copy the first n layers, then either fine-tune those layers or leave them frozen². Pan and Yang's definition does not use the word fine-tuning at all¹.

Can transfer learning make a model worse?

Yes. Pan and Yang call it negative transfer when the source and target are unrelated: a blunt transfer can fail and, in the worst case, hurt learning on the target¹.

Do early layers transfer better than late layers?

On Yosinski and colleagues' random halves of ImageNet, layers 1 and 2 transferred almost perfectly, and layers 4 to 7 dropped more when left frozen². They give two reasons: higher layers specialise, and splitting co-adapted neurons makes optimisation harder².

Is transfer learning the same as training several tasks at once?

No. Pan and Yang say multi-task learning trains the tasks together, while transfer learning cares most about the target task, so the two roles are not symmetric¹.

Related terms

Sources

Back to the glossary