Date Updated: Feb 25, 2020
Welcome to the Binary Classification Tutorial (CLF101) - Level Beginner. This tutorial assumes that you are new to PyCaret and looking to get started with Binary Classification using the
In this tutorial we will learn:
Read Time : Approx. 30 Minutes
The first step to get started with PyCaret is to install pycaret. Installation is easy and will only take a few minutes. Follow the instructions below:
pip install pycaret
!pip install pycaret
If you are running this notebook on Google colab, run the following code at top of your notebook to display interactive visuals.
from pycaret.utils import enable_colab
Binary classification is a supervised machine learning technique where the goal is to predict categorical class labels which are discrete and unoredered such as Pass/Fail, Positive/Negative, Default/Not-Default etc. A few real world use cases for classification are listed below:
PyCaret's classification module (
pycaret.classification) is a supervised machine learning module which is used for classifying the elements into a binary group based on various techniques and algorithms. Some common use cases of classification problems include predicting customer default (yes or no), customer churn (customer will leave or stay), disease found (positive or negative).
The PyCaret classification module can be used for Binary or Multi-class classification problems. It has over 18 algorithms and 14 plots to analyze the performance of models. Be it hyper-parameter tuning, ensembling or advanced techniques like stacking, PyCaret's classification module has it all.
For this tutorial we will use a dataset from UCI called Default of Credit Card Clients Dataset. This dataset contains information on default payments, demographic factors, credit data, payment history, and billing statements of credit card clients in Taiwan from April 2005 to September 2005. There are 24,000 samples and 25 features. Short descriptions of each column are as follows:
Lichman, M. (2013). UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science.
The original dataset and data dictionary can be found here.
from pycaret.datasets import get_data dataset = get_data('credit')
5 rows × 24 columns
#check the shape of data dataset.shape
In order to demonstrate the
predict_model() function on unseen data, a sample of 1200 records has been withheld from the original dataset to be used for predictions. This should not be confused with a train/test split as this particular split is performed to simulate a real life scenario. Another way to think about this is that these 1200 records are not available at the time when the machine learning experiment was performed.
data = dataset.sample(frac=0.95, random_state=786).reset_index(drop=True) data_unseen = dataset.drop(data.index).reset_index(drop=True) print('Data for Modeling: ' + str(data.shape)) print('Unseen Data For Predictions: ' + str(data_unseen.shape))
Data for Modeling: (22800, 24) Unseen Data For Predictions: (1200, 24)
setup() function initializes the environment in pycaret and creates the transformation pipeline to prepare the data for modeling and deployment.
setup() must be called before executing any other function in pycaret. It takes two mandatory parameters: a pandas dataframe and the name of the target column. All other parameters are optional and are used to customize the pre-processing pipeline (we will see them in later tutorials).
setup() is executed, PyCaret's inference algorithm will automatically infer the data types for all features based on certain properties. The data type should be inferred correctly but this is not always the case. To account for this, PyCaret displays a table containing the features and their inferred data types after
setup() is executed. If all of the data types are correctly identified
enter can be pressed to continue or
quit can be typed to end the expriment. Ensuring that the data types are correct is of fundamental importance in PyCaret as it automatically performs a few pre-processing tasks which are imperative to any machine learning experiment. These tasks are performed differently for each data type which means it is very important for them to be correctly configured.
In later tutorials we will learn how to overwrite PyCaret's infered data type using the
categorical_features parameters in
from pycaret.classification import *
exp_clf101 = setup(data = data, target = 'default', session_id=123)
Setup Succesfully Completed!
|3||Original Data||(22800, 24)|
|8||High Cardinality Features||False|
|9||High Cardinality Method||None|
|10||Sampled Data||(22800, 24)|
|11||Transformed Train Set||(15959, 90)|
|12||Transformed Test Set||(6841, 90)|
|22||Ignore Low Variance||False|
|23||Combine Rare Levels||False|
|24||Rare Level Threshold||None|
|38||Features Selection Threshold||None|
Once the setup has been succesfully executed it prints the information grid which contains several important pieces of information. Most of the information is related to the pre-processing pipeline which is constructed when
setup() is executed. The majority of these features are out of scope for the purposes of this tutorial however a few important things to note at this stage include:
session_idis passed, a random number is automatically generated that is distributed to all functions. In this experiment, the
session_idis set as
123for later reproducibility.
train_sizeparameter in setup.
Notice how a few tasks that are imperative to perform modeling are automatically handled such as missing value imputation (in this case there are no missing values in the training data, but we still need imputers for unseen data), categorical encoding etc. Most of the parameters in
setup() are optional and used for customizing the pre-processing pipeline. These parameters are out of scope for this tutorial but as you progress to the intermediate and expert levels, we will cover them in much greater detail.
Comparing all models to evaluate performance is the recommended starting point for modeling once the setup is completed (unless you exactly know what kind of model you need, which is often not the case). This function trains all models in the model library and scores them using stratified cross validation for metric evaluation. The output prints a score grid that shows average Accuracy, AUC, Recall, Precision, F1 and Kappa accross the folds (10 by default) of all the available models in the model library.
|1||Linear Discriminant Analysis||0.8236||0.7703||0.3813||0.6818||0.4888||0.3923|
|2||Gradient Boosting Classifier||0.8225||0.7887||0.3649||0.687||0.4763||0.3813|
|4||Extreme Gradient Boosting||0.8218||0.7894||0.3595||0.6862||0.4715||0.3767|
|5||Light Gradient Boosting Machine||0.8214||0.7859||0.3878||0.6663||0.49||0.3908|
|6||Ada Boost Classifier||0.8185||0.7783||0.3507||0.6729||0.4607||0.3644|
|7||Extra Trees Classifier||0.8093||0.7533||0.3839||0.61||0.4711||0.362|
|8||Random Forest Classifier||0.8084||0.738||0.3337||0.6254||0.4349||0.3323|
|9||Quadratic Discriminant Analysis||0.7893||0.7392||0.1734||0.6276||0.2378||0.1698|
|11||K Neighbors Classifier||0.7505||0.6099||0.1802||0.3693||0.2421||0.1134|
|12||Decision Tree Classifier||0.7294||0.6197||0.4221||0.3953||0.4081||0.233|
|13||SVM - Linear Kernel||0.653||0||0.2547||0.1056||0.112||0.0121|
Two simple words of code (not even a line) have created over 15 models using 10 fold stratified cross validation and evaluated the 6 most commonly used classification metrics (Accuracy, AUC, Recall, Precision, F1, Kappa). The score grid printed above highlights the highest performing metric for comparison purposes only. The grid by default is sorted using 'Accuracy' (highest to lowest) which can be changed by passing the
sort parameter. For example
compare_models(sort = 'Recall') will sort the grid by Recall instead of Accuracy. If you want to change the fold parameter from the default value of
10 to a different value then you can use the
fold parameter. For example
compare_models(fold = 5) will compare all models on 5 fold cross validation. Reducing the number of folds will improve the training time.
compare_models() is a powerful function and often a starting point in any experiment, it does not return any trained models. PyCaret's recommended experiment workflow is to use
compare_models() right after setup to evaluate top performing models and finalize a few candidates for continued experimentation. As such, the function that actually allows to you create a model is unimaginatively called
create_model(). This function creates a model and scores it using stratified cross validation. Similar to
compare_models(), the output prints a score grid that shows Accuracy, AUC, Recall, Precision, F1 and Kappa by fold.
For the remaining part of this tutorial, we will work with the below models as our candidate models. The selections are for illustration purposes only and do not necessarily mean they are the top performing or ideal for this type of data.
There are 18 classifiers available in the model library of PyCaret. Please view the
create_model() docstring for the list of all available models.
dt = create_model('dt')
#trained model object is stored in the variable 'dt'. print(dt)
DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini', max_depth=None, max_features=None, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0, presort='deprecated', random_state=123, splitter='best')
knn = create_model('knn')
rf = create_model('rf')
Notice that the mean score of all models matches with the score printed in
compare_models(). This is because the metrics printed in the
compare_models() score grid are the average scores across all CV folds. Similar to
compare_models(), if you want to change the fold parameter from the default value of 10 to a different value then you can use the
fold parameter. For Example:
create_model('dt', fold = 5) will create a Decision Tree Classifier using 5 fold stratified CV.
When a model is created using the
create_model() function it uses the default hyperparameters. In order to tune hyperparameters, the
tune_model() function is used. This function automatically tunes the hyperparameters of a model on a pre-defined search space and scores it using stratified cross validation. The output prints a score grid that shows Accuracy, AUC, Recall, Precision, F1 and Kappa by fold.
tune_model() does not take a trained model object as an input. It instead requires a model name to be passed as an abbreviated string similar to how it is passed in
create_model(). All other functions in
pycaret.classification require a trained model object as an argument.
tuned_dt = tune_model('dt')
#tuned model object is stored in the variable 'tuned_dt'. print(tuned_dt)
DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini', max_depth=3, max_features=78, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=3, min_samples_split=2, min_weight_fraction_leaf=0.0, presort='deprecated', random_state=123, splitter='best')
tuned_knn = tune_model('knn')
tuned_rf = tune_model('rf')
tune_model() function is a random grid search of hyperparameters over a pre-defined search space. By default, it is set to optimize
Accuracy but this can be changed using
optimize parameter. For example:
tune_model('dt', optimize = 'AUC') will search for the hyperparameters of a Decision Tree Classifier that results in highest
AUC. For the purposes of this example, we have used the default metric
Accuracy for the sake of simplicity only. Generally, when the dataset is imbalanced (such as the credit dataset we are working with)
Accuracy is not a good metric for consideration. The methodology behind selecting the right metric to evaluate a classifier is beyond the scope of this tutorial but if you would like to learn more about it, you can click here to read an article on how to choose the right evaluation metric.
Notice how the results after tuning have been improved:
Metrics alone are not the only criteria you should consider when finalizing the best model for production. Other factors to consider include training time, standard deviation of kfolds etc. As you progress through the tutorial series we will discuss those factors in detail at the intermediate and expert levels. For now, let's move forward considering the Tuned Random Forest Classifier as our best model for the remainder of this tutorial.
Before model finalization, the
plot_model() function can be used to analyze the performance across different aspects such as AUC, confusion_matrix, decision boundary etc. This function takes a trained model object and returns a plot based on the test / hold-out set.
There are 15 different plots available, please see the
plot_model() docstring for the list of available plots.
plot_model(tuned_rf, plot = 'auc')
plot_model(tuned_rf, plot = 'pr')
plot_model(tuned_rf, plot = 'confusion_matrix')
Another way to analyze the performance of models is to use the
evaluate_model() function which displays a user interface for all of the available plots for a given model. It internally uses the
Before finalizing the model, it is advisable to perform one final check by predicting the test/hold-out set and reviewing the evaluation metrics. If you look at the information grid in Section 6 above, you will see that 30% (6,841 samples) of the data has been separated out as test/hold-out sample. All of the evaluation metrics we have seen above are cross validated results based on the training set (70%) only. Now, using our final trained model stored in the
tuned_rf variable we will predict against the hold-out sample and evaluate the metrics to see if they are materially different than the CV results.
|0||Random Forest Classifier||0.8126||0.7538||0.3212||0.6559||0.4312||0.3345|
The accuracy on test/hold-out set is
0.8126 compared to
0.8229 achieved on the
tuned_rf CV results (in section 9.3 above). This is not a significant difference. If there is a large variation between the test/hold-out and CV results, then this would normally indicate over-fitting but could also be due to several other factors and would require further investigation. In this case, we will move forward with finalizing the model and predicting on unseen data (the 5% that we had separated in the beginning and never exposed to PyCaret).
(TIP : It's always good to look at the standard deviation of CV results when using
Model finalization is the last step in the experiment. A normal machine learning workflow in PyCaret starts with
setup(), followed by comparing all models using
compare_models() and shortlisting a few candidate models (based on the metric of interest) to perform several modeling techniques such as hyperparameter tuning, ensembling, stacking etc. This workflow will eventually lead you to the best model for use in making predictions on new and unseen data. The
finalize_model() function fits the model onto the complete dataset including the test/hold-out sample (30% in this case). The purpose of this function is to train the model on the complete dataset before it is deployed in production.
final_rf = finalize_model(tuned_rf)
#Final Random Forest model parameters for deployment print(final_rf)
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None, criterion='gini', max_depth=10, max_features='auto', max_leaf_nodes=None, max_samples=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=2, min_samples_split=10, min_weight_fraction_leaf=0.0, n_estimators=70, n_jobs=None, oob_score=False, random_state=123, verbose=0, warm_start=False)
Caution: One final word of caution. Once the model is finalized using
finalize_model(), the entire dataset including the test/hold-out set is used for training. As such, if the model is used for predictions on the hold-out set after
finalize_model() is used, the information grid printed will be misleading as you are trying to predict on the same data that was used for modeling. In order to demonstrate this point only, we will use
predict_model() to compare the information grid with the one above in section 11.
|0||Random Forest Classifier||0.8361||0.8189||0.3681||0.7715||0.4984||0.4148|
Notice how the AUC in
final_rf has increased to
0.7538, even though the model is the same. This is because the
final_rf variable has been trained on the complete dataset including the test/hold-out set.
predict_model() function is also used to predict on the unseen dataset. The only difference from section 11 above is that this time we will pass the
data_unseen is the variable created at the beginning of the tutorial and contains 5% (1200 samples) of the original dataset which was never exposed to PyCaret. (see section 5 for explanation)
unseen_predictions = predict_model(final_rf, data=data_unseen) unseen_predictions.head()
5 rows × 26 columns
Score columns are added onto the
data_unseen set. Label is the prediction and score is the probability of the prediction. Notice that predicted results are concatenated to the original dataset while all the transformations are automatically performed in the background.
We have now finished the experiment by finalizing the
tuned_rf model which is now stored in
final_rf variable. We have also used the model stored in
final_rf to predict
data_unseen. This brings us to the end of our experiment, but one question is still to be asked: What happens when you have more new data to predict? Do you have to go through the entire experiment again? The answer is no, PyCaret's inbuilt function
save_model() allows you to save the model along with entire transformation pipeline for later use.
save_model(final_rf,'Final RF Model 08Feb2020')
Transformation Pipeline and Model Succesfully Saved
(TIP : It's always good to use date in the filename when saving models, it's good for version control.)
To load a saved model at a future date in the same or an alternative environment, we would use PyCaret's
load_model() function and then easily apply the saved model on new unseen data for prediction.
saved_final_rf = load_model('Final RF Model 08Feb2020')
Transformation Pipeline and Model Sucessfully Loaded
Once the model is loaded in the environment, you can simply use it to predict on any new data using the same
predict_model() function. Below we have applied the loaded model to predict the same
data_unseen that we used in section 13 above.
new_prediction = predict_model(saved_final_rf, data=data_unseen)
5 rows × 26 columns
Notice that the results of
new_prediction are identical.
This tutorial has covered the entire machine learning pipeline from data ingestion, pre-processing, training the model, hyperparameter tuning, prediction and saving the model for later use. We have completed all of these steps in less than 10 commands which are naturally constructed and very intuitive to remember such as
compare_models(). Re-creating the entire experiment without PyCaret would have taken well over 100 lines of code in most libraries.
We have only covered the basics of
pycaret.classification. In following tutorials we will go deeper into advanced pre-processing, ensembling, generalized stacking and other techniques that allow you to fully customize your machine learning pipeline and are must know for any data scientist.
See you at the next tutorial. Follow the link to Binary Classification Tutorial (CLF102) - Intermediate Level