Backgroud

bias-variance trade-off 在 Machine learning中蠻常被提及的一個名詞. 但研究所的時候老是搞不懂這兩個的差別. (可能是我沒好好的在聽老師上課). 直到最近有需要再重新自己好好研究一下. 並做個筆記, 記錄一下.

In this post, you will discover the Bias-Variance Trade-Off and how to use it to better understand machine learning algorithms and get better performance on your data.

The prediction error for any machine learning algorithm can be broken down into three parts:

  • Bias Error
  • Variance Error
  • Irreducible Error

Bias Error

Bias are the simplifying assumptions made by a model to make the target function easier to learn.

如果你要得到Low Bias Error, 你只要把模型搞得很複雜, 把參數搞得多多的. 這很好理解. 但通常這會被隨 High Variance Error 因為training data不是每一個都是適合這總複雜的模型.

Variance Error

Variance is the amount that the estimate of the target function will change if different training data was used.

如果你要得到Low Variance Error那基本上就把模型弄的很簡單, 所以不管對怎樣的training data效果都會差不多. 但是可能都會很爛就是了.

Bias-Variance Trade-Off

Bias反映的是模型在樣本上的輸出與真實值之間的誤差,即模型本身的精準度,Variance反映的是模型每一次輸出結果與模型輸出期望之間的誤差,即模型的穩定性. 那在一個實際系統中,Bias與Variance往往是不能兼得的.

理想的supervised machine learning algorithm通常都希望得到 low bias and low variance, 但通常 bias 和 variance 的選擇是一個 tradeoff, 像下面這張圖一樣.

Example

下面列出幾個例子:

  • The k-nearest neighbors algorithm has low bias and high variance, but the trade-off can be changed by increasing the value of k which increases the number of neighbors that contribute t the prediction and in turn increases the bias of the model.

  • The support vector machine algorithm has low bias and high variance, but the trade-off can be changed by increasing the C parameter that influences the number of violations of the margin allowed in the training data which increases the bias but decreases the variance.

那你可以透過上面的例子得到一些想法:

  • Parametric or linear machine learning algorithms often have a high bias but a low variance.

  • Non-parametric or non-linear machine learning algorithms often have a low bias but a high variance.

Reference