Machine learning can be summarized as learning a function (f) that maps input variables (X) to output variables (Y), such as Y = f(x).
An algorithm learns this target mapping function from training data. The form of the function is unknown, so our job as machine learning practitioners is to evaluate different machine learning algorithms and see which is better at approximating the underlying function. Different algorithms make different assumptions or biases about the form of the function and how it can be learned. Assumptions can greatly simplify the learning process, but can also limit what can be learned. A model parameter is a configuration variable that is internal to the model and whose value can be estimated from the given data.
Algorithms that simplify the function to a known form are called parametric machine learning algorithms. A parametric machine learning algorithm generates a fixed number of model parameters.
Some more examples of parametric machine learning algorithms include:
- Logistic Regression
- Linear Discriminant Analysis
- Perceptron (MLP)
- Naive Bayes
- Simple Neural Networks (ANN)
Benefits of Parametric Machine Learning Algorithms:
- Simpler: These methods are easier to understand and interpret results.
- Speed: Parametric models are very fast to learn from data.
- Less Data: They do not require as much training data and can work well even if the fit to the data is not perfect.
Limitations of Parametric Machine Learning Algorithms:
- Constrained: By choosing a functional form these methods are highly constrained to the specified form.
- Limited Complexity: The methods are more suited to simpler problems.
- Poor Fit: In practice the methods are unlikely to match the underlying mapping function.
Algorithms that do not make strong assumptions about the form of the mapping function are called non-parametric machine learning algorithms. By not making assumptions, they are free to learn any functional form from the training data.
Some examples of popular non-parametric machine learning algorithms are:
- k-Nearest Neighbors (KNN)
- Decision Trees like CART and C4.5
- Support Vector Machines
Benefits of Non-parametric Machine Learning Algorithms:
- Flexibility: Capable of fitting a large number of functional forms.
- Power: No assumptions (or weak assumptions) about the underlying function.
- Performance: Can result in higher performance models for prediction.
Limitations of Non-parametric Machine Learning Algorithms:
- More data: Require a lot more training data to estimate the mapping function.
- Slower: A lot slower to train as they often have far more parameters to train.
- Overfitting: More of a risk to overfit the training data and it is harder to explain why specific predictions are made.