Introduction to Machine Learning Prof V B More

slide1. Introduction to Machine Learning Prof V B More
MET’s IOE BKC Nashik<br>
slide2. Unit 1: Introduction to Machine learning Classic and adaptive machines, Machine learning matters, Beyond machine learning-deep learning and bio inspired adaptive systems, Machine learning and Big data. Important Elements of Machine Learning- Data formats, Learnability, Statistical learning approaches, Elements of information theory. 2 Prof V B More, MET BKC IOE Nashik<br>
slide3. Important Elements of Machine Learning It is important to understand the mathematical foundation of data formats and prediction functions.
In most algorithms, these concepts are treated in different ways, but the goal is always the same.
More recent techniques, such as deep learning, extensively use energy/loss functions 3 Prof V B More, MET BKC IOE Nashik<br>
slide4. Important Elements of Machine Learning Data formats
In a supervised learning problem, there will always be a dataset, defined as a finite set of Real valued vectors with m features each: 4 Prof V B More, MET BKC IOE Nashik<br>
slide5. Important Elements of Machine Learning Data formats
Consider each X as drawn from a statistical multivariate distribution D. All samples are Independent and Identically Distributed (i.i.d).
Multivariate distributions show comparisons between two or more measurements and the relationships among them.

A multivariate normal distribution is a vector in multiple normally distributed variables, such that any linear combination of the variables is also normally distributed. 5 Prof V B More, MET BKC IOE Nashik<br>
slide6. Important Elements of Machine Learning Data formats

What is the difference between univariate and multivariate distributions?
Any linear combination of the variables has a univariate normal distribution. Any conditional distribution for a subset of the variables conditional on known values for another subset of variables is a multivariate distribution. 6 Prof V B More, MET BKC IOE Nashik<br>
slide7. Important Elements of Machine Learning Data formats
This means all variables belong to the same distribution D, and considering an arbitrary subset of m values, 7 Prof V B More, MET BKC IOE Nashik<br>
slide8. Important Elements of Machine Learning Data formats
The corresponding output values can be both numerical-continuous (regression) or categorical (classification) 8 Prof V B More, MET BKC IOE Nashik<br>
slide9. Important Elements of Machine Learning Data formats
Examples of numerical outputs are: 9 Prof V B More, MET BKC IOE Nashik Examples of categorical outputs are:<br>
slide10. Important Elements of Machine Learning We define generic regressor, a vector-valued function which associates an input value to a continuous output and

generic classifier, a vector-values function whose predicted output
is categorical (discrete). 10 Prof V B More, MET BKC IOE Nashik<br>
slide11. Important Elements of Machine Learning If they also depend on an internal parameter, the approach is called parametric learning: 11 Prof V B More, MET BKC IOE Nashik Regressor

Classifier<br>
slide12. Important Elements of Machine Learning On the other hand, non-parametric learning doesn't make initial assumptions about the family of predictors. A very common non-parametric family is called instance-based learning and makes real-time predictions based on hypothesis determined only by the training samples. 12 Prof V B More, MET BKC IOE Nashik<br>
slide13. Important Elements of Machine Learning A generic parametric training process must find the best parameter vector which minimizes the regression/classification error given a specific training dataset and it should also generate a predictor that can correctly generalize when unknown samples are provided. 13 Prof V B More, MET BKC IOE Nashik<br>
slide14. Important Elements of Machine Learning Another interpretation: additive noise: 14 Prof V B More, MET BKC IOE Nashik m – population mean,
s2 – variance, n – additive noise Expect value<br>
slide15. Important Elements of Machine Learning We can expect zero-mean low-variance Gaussian noise added to a perfect prediction.

A training task must increase the signal-noise ratio by optimizing the parameters.
High noise variance means that X is dirty and its measures are not reliable. 15 Prof V B More, MET BKC IOE Nashik<br>
slide16. Important Elements of Machine Learning In unsupervised learning, we have an input set X with m-length vectors, and
we define clustering function (with n target clusters) with the following expression: 16 Prof V B More, MET BKC IOE Nashik<br>
slide17. Important Elements of Machine Learning In most scikit-learn models, there is an instance variable coef_ which contains all trained parameters. For example, in a single parameter linear regression, the output will be:
>>> model = LinearRegression()
>>> model.fit(X, Y)
>>> model.coef_
array([ 9.10210898]) 17 Prof V B More, MET BKC IOE Nashik<br>
slide18. Important Elements of Machine Learning Multiclass strategies
When the number of output classes is greater than one, there are two main possibilities to manage a classification problem:
One-vs-all
One-vs-one 18 Prof V B More, MET BKC IOE Nashik<br>
slide19. Important Elements of Machine Learning Multiclass strategies
One-vs-all
This is the most common strategy and is widely adopted by scikit-learn for most of its algorithms. If there are n output classes, n classifiers will be trained in parallel 19 Prof V B More, MET BKC IOE Nashik<br>
slide20. Important Elements of Machine Learning Multiclass strategies
One-vs-all
This approach is relatively lightweight (at most, n-1 checks are needed to find the right class, so it has an O(n) complexity) and, for this reason, it's normally the default choice and there's no need for further actions. 20 Prof V B More, MET BKC IOE Nashik<br>
slide21. Important Elements of Machine Learning Multiclass strategies
One-vs-one
Training a model for each pair of classes. The complexity is O(n2) and the right class is determined by a majority vote.
In general, this choice is more expensive and should be adopted only when a full dataset comparison is not required. 21 Prof V B More, MET BKC IOE Nashik<br>
slide22. Important Elements of Machine Learning Multiclass strategies
Classification task performed with more than two classes.
Each sample can only be labelled as one class. 22 Prof V B More, MET BKC IOE Nashik<br>
slide23. Important Elements of Machine Learning Multiclass strategies
For ex., classification using features extracted from a set of images of fruit, where each image is either orange, apple, or a pear. Each image is one sample and is labelled as one of the 3 possible classes. 23 Prof V B More, MET BKC IOE Nashik<br>
slide24. Important Elements of Machine Learning Multiclass strategies
Multiclass classification makes the assumption that each sample is assigned to one and only one label - one sample cannot be both a pear and an apple. 24 Prof V B More, MET BKC IOE Nashik<br>
slide25. Important Elements of Machine Learning Multiclass strategies
Multiclass classifiers:
Inherent Multiclass Classifiers:
sklearn.naive_bayes.BernoulliNB
sklearn.tree.DecisionTreeClassifier
sklearn.tree.ExtraTreeClassifier
sklearn.ensemble.ExtraTreesClassifier
sklearn.naive_bayes.GaussianNB
sklearn.neighbors.KNeighborsClassifier
sklearn.semi_supervised.LabelPropagation
sklearn.semi_supervised.LabelSpreading
sklearn.discriminant_analysis.LinearDiscriminantAnalysis
sklearn.svm.LinearSVC 25 Prof V B More, MET BKC IOE Nashik<br>
slide26. Important Elements of Machine Learning Multiclass strategies
Multiclass classifiers:
Inherent Multiclass Classifiers:
sklearn.linear_model.LogisticRegression
sklearn.linear_model.LogisticRegressionCV
sklearn.neural_network.MLPClassifier
sklearn.neighbors.NearestCentroid
sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis
sklearn.neighbors.RadiusNeighborsClassifier
sklearn.ensemble.RandomForestClassifier
sklearn.linear_model.RidgeClassifier
sklearn.linear_model.RidgeClassifierCV 26 Prof V B More, MET BKC IOE Nashik<br>
slide27. Important Elements of Machine Learning Multiclass strategies
Multiclass classifiers:
Multiclass as One-Vs-One:
sklearn.svm.NuSVC
sklearn.svm.SVC.
sklearn.gaussian_process.GaussianProcessClassifier

Multiclass as One-Vs-The-Rest:
sklearn.ensemble.GradientBoostingClassifier
sklearn.gaussian_process.GaussianProcessClassifier
sklearn.svm.LinearSVC
sklearn.linear_model.LogisticRegression
sklearn.linear_model.LogisticRegressionCV
sklearn.linear_model.SGDClassifier
sklearn.linear_model.Perceptron
sklearn.linear_model.PassiveAggressiveClassifier 27 Prof V B More, MET BKC IOE Nashik<br>
slide28. Important Elements of Machine Learning Multiclass strategies
Support multilabel:
sklearn.tree.DecisionTreeClassifier
sklearn.tree.ExtraTreeClassifier
sklearn.ensemble.ExtraTreesClassifier
sklearn.neighbors.KNeighborsClassifier
sklearn.neural_network.MLPClassifier
sklearn.neighbors.RadiusNeighborsClassifier
sklearn.ensemble.RandomForestClassifier
sklearn.linear_model.RidgeClassifierCV

Support multiclass-multioutput:
sklearn.tree.DecisionTreeClassifier
sklearn.tree.ExtraTreeClassifier
sklearn.ensemble.ExtraTreesClassifier
sklearn.neighbors.KNeighborsClassifier
sklearn.neighbors.RadiusNeighborsClassifier
sklearn.ensemble.RandomForestClassifier 28 Prof V B More, MET BKC IOE Nashik<br>
slide29. Important Elements of Machine Learning Multiclass strategies
Multiclass strategies implemented by
scikit-learn,
Visit following link for more info.
http://scikit-learn.org/stable/modules/multiclass.html 29 Prof V B More, MET BKC IOE Nashik<br>
slide30. Important Elements of Machine Learning Learnability
A parametric model can be one of two types: static or dynamic.
Static is determined by choice of a specific algorithm and is generally un-changeable.
Dynamic is based on learning hypothesis and can operate on dynamic set of parameters. 30 Prof V B More, MET BKC IOE Nashik<br>
slide31. Important Elements of Machine Learning Learnability
The goal of a parametric learning process is to find the best hypothesis
Having less prediction error and the avoid overfitting in generalization. 31 Prof V B More, MET BKC IOE Nashik<br>
slide32. Important Elements of Machine Learning Learnability 32 Prof V B More, MET BKC IOE Nashik In example dataset,
points must be classified as red (Class A) or blue (Class B).<br>
slide33. Important Elements of Machine Learning Learnability

Three hypotheses are shown:
the first one (the middle line starting from left) misclassifies one/two samples,
while the lower and upper ones misclassify 13 and 23 samples respectively. 33 Prof V B More, MET BKC IOE Nashik<br>
slide34. Important Elements of Machine Learning Learnability

The first hypothesis is optimal and should be selected;
The dataset X is linearly separable if there exists a hyperplane which divides the sample space into two subspaces containing only elements belonging to the same class. 34 Prof V B More, MET BKC IOE Nashik<br>
slide35. Important Elements of Machine Learning Learnability

Overfitting must also be taken into consideration while separating the classes.
Parametric model adopts only a family of non-periodic and approximate functions whose ability to oscillate and fit the dataset is determined by the number of parameters. 35 Prof V B More, MET BKC IOE Nashik<br>
slide36. Important Elements of Machine Learning Learnability 36 Prof V B More, MET BKC IOE Nashik In example dataset,
the blue classifier is linear while the red one is cubic<br>
slide37. Important Elements of Machine Learning Learnability

For generalization in classification / categorization there should be a function that separate the sample data into respective categories. 37 Prof V B More, MET BKC IOE Nashik<br>
slide38. Important Elements of Machine Learning Learnability

If we use linear function some of the sample points misclassified because of oscillation in data points. Classification function must also consider future trend. 38 Prof V B More, MET BKC IOE Nashik<br>
slide39. Important Elements of Machine Learning Learnability

If we apply cubic approach, it can fit this data almost perfectly but, at the same time, loses its ability to keep a global linear trend. 39 Prof V B More, MET BKC IOE Nashik<br>
slide40. Important Elements of Machine Learning Learnability
Therefore, there are two possibilities:
If we expect future data to be exactly distributed as training samples, a more complex model can be a good choice.
In this case, a linear model will lead to underfitting, because it won't be able to capture all samples for correct classification. 40 Prof V B More, MET BKC IOE Nashik<br>
slide41. Important Elements of Machine Learning Learnability
Therefore, there are two possibilities:
If we think that future data can be locally distributed differently but keeps a global trend, it's preferable to have a higher remaining misclassification error as well as a more precise generalization ability.
If we focus only on training data, it can lead to overfitting. 41 Prof V B More, MET BKC IOE Nashik<br>
slide42. Important Elements of Machine Learning Learnability
Underfitting and overfitting
The purpose of a machine learning model is to approximate an unknown function that associates input elements to the best possible output. Whereas, a training set is normally a representation of a global distribution, but it cannot contain all possible elements; otherwise the problem could be solved with a one-to-one association. 42 Prof V B More, MET BKC IOE Nashik<br>
slide43. Important Elements of Machine Learning Learnability
Underfitting and overfitting
If we don't know the future trend while training, it is necessary to think about fitting the model but keeping it free to generalize when an unknown input is presented.

Unfortunately, this ideal condition is not always easy to find. 43 Prof V B More, MET BKC IOE Nashik<br>
slide44. Important Elements of Machine Learning Learnability
Two different dangers to consider:

Underfitting: It means that the model isn't able to capture the dynamics show by the same training set (probably because its capacity is too limited). 44 Prof V B More, MET BKC IOE Nashik<br>
slide45. Important Elements of Machine Learning Learnability
Two different dangers to consider:
Overfitting: the model has an excessive capacity and it is not more able to generalize with respect to the original dynamics provided by the training set. It can associate almost all the known samples to the corresponding output values, but when an unknown input is presented, the prediction error will be very high. 45 Prof V B More, MET BKC IOE Nashik<br>
slide46. Important Elements of Machine Learning Learnability 46 Prof V B More, MET BKC IOE Nashik<br>
slide47. Important Elements of Machine Learning Learnability 47 Prof V B More, MET BKC IOE Nashik<br>
slide48. Important Elements of Machine Learning Learnability
It's very important to avoid both underfitting and overfitting.
Underfitting is easier to detect by observing prediction error,
while Overfitting may prove to be more difficult to discover as it could be initially considered the result of a perfect fitting. 48 Prof V B More, MET BKC IOE Nashik<br>
slide49. Important Elements of Machine Learning Learnability
Error measures
When working with a supervised learning, we define a non-negative error measure em which takes two arguments (expected and predicted output) for computing total error: 49 Prof V B More, MET BKC IOE Nashik<br>
slide50. Important Elements of Machine Learning Learnability
Error measures
This value is implicitly dependent on the specific hypothesis H through the parameter set, therefore optimizing the error implies finding an optimal hypothesis. In many cases, it's useful to consider the mean square error (MSE): 50 Prof V B More, MET BKC IOE Nashik<br>
slide51. Important Elements of Machine Learning Learnability

Error measures
This measure is also called loss function because its value must be minimized through an optimization problem. 51 Prof V B More, MET BKC IOE Nashik<br>
slide52. Important Elements of Machine Learning Learnability
Error measures
MSE is the average of the squared error that is used as the loss function : It is the sum, over all the data points, of the square of the difference between the predicted and actual target variables, divided by the number of data points. 52 Prof V B More, MET BKC IOE Nashik<br>
slide53. Important Elements of Machine Learning Learnability
Error measures
Another useful loss function is called zero-one-loss and it's particularly efficient for binary classifications (also for one-vs-rest multiclass strategy): 53 Prof V B More, MET BKC IOE Nashik<br>
slide54. Important Elements of Machine Learning Learnability
Error measures
A helpful interpretation of a generic (and continuous) loss function can be expressed in terms of potential energy: 54 Prof V B More, MET BKC IOE Nashik<br>
slide55. Important Elements of Machine Learning Learnability
Error measures
The predictor is like a ball upon a rough surface: starting from a random point where energy (=error) is usually rather high, it must move until it reaches a stable equilibrium point where its energy is null (relative to the global minimum). 55 Prof V B More, MET BKC IOE Nashik<br>
slide56. Important Elements of Machine Learning Error measures 56 Prof V B More, MET BKC IOE Nashik<br>
slide57. Important Elements of Machine Learning Learnability
Error measures
Just like in the physical situation, the starting point is stable without any external disturbance, so to start the process, it's needed to provide initial kinetic energy. However, if such an energy is strong enough, then after descending over the slope the ball cannot stop in the global minimum. The residual kinetic energy can be enough to overcome the ridge and reach the right valley. 57 Prof V B More, MET BKC IOE Nashik<br>
slide58. Important Elements of Machine Learning Learnability
Error measures
If there are not other energy sources, the ball gets trapped in the plain valley and cannot move anymore. There are many techniques that have been engineered to solve this problem and avoid local minima. 58 Prof V B More, MET BKC IOE Nashik<br>
slide59. Important Elements of Machine Learning Learnability: PAC learning
In many cases machine learning works very much better, but is there any method to learn the concept?

In 1984, the computer scientist Valiant proposed a mathematical approach to determine whether a problem is learnable by a computer. The name of this technique is PAC, or Probably Approximately Correct. 59 Prof V B More, MET BKC IOE Nashik<br>
slide60. Important Elements of Machine Learning Learnability: PAC learning
It can be imagined as a classification problem where an algorithm A has to learn a set of concepts. A concept is a subset of input patterns X which determine the same output element. Therefore, learning a concept means minimizing the corresponding loss function, while learning all possible concepts, means finding the minimum of a global loss function. 60 Prof V B More, MET BKC IOE Nashik<br>
slide61. Important Elements of Machine Learning Learnability: PAC learning
It can be imagined as a classification problem where an algorithm A has to learn a set of concepts. A concept is a subset of input patterns X which determine the same output element. Therefore, learning a concept means minimizing the loss function, while learning all possible concepts, means finding the minimum of a global loss function. 61 Prof V B More, MET BKC IOE Nashik<br>
slide62. Important Elements of Machine Learning Learnability: PAC learning
We have many possible hypotheses to consider for study. Probabilistic approach is often necessary for refinement of study. For this reason, we accept good approximations with high probability based on a limited number of input elements and produced in polynomial time. 62 Prof V B More, MET BKC IOE Nashik<br>
slide63. Important Elements of Machine Learning Learnability: PAC learning

Therefore, an algorithm A can learn the class C of all concepts if it is able to find a hypothesis H with a procedure O(nk) so that A, with a probability p, can classify all patterns correctly with a maximum allowed error me. 63 Prof V B More, MET BKC IOE Nashik<br>
slide64. Important Elements of Machine Learning Learnability: PAC learning

This must be valid for all statistical distributions on X which must be greater than or equal to a minimum value depending only on p and me.

Polynomial-time algorithms is can determine the success or the failure of a machine learning problem. 64 Prof V B More, MET BKC IOE Nashik<br>