A convex function is a function which features a single global minimum, whereas a non-convex functions presents many local minima.
A good analysis on the mathematical aspects and mathematical definition of convex and non-convex functions can be found at: https://rumn.medium.com/convex-vs-non-convex-functions-why-it-matters-in-optimization-for-machine-learning-39cd9427dfcc.
Some examples of ML algorithms which have restrictions related to convex functions are the following:
- Gradient descent only works with convex functions (global minimum).
- The MSE method works in linear regression but it does not work with logistic regression, because logistic regression creates a non-convex function.