Data is the core basis of any computer environment. For machine learning systems, data is required to be fetched and first converted into a usable form. To process the raw input data and convert it into a usable form for machine learning is called as Data Preprocessing. Data preprocessing is performed by various techniques. The selection of techniques for data preprocessing depends upon the context of the data. I will discuss the most common practices for Data Preprocessing in this post. I work on Python for machine learning. So the below techniques are relevant to Python. Similar techniques are used in different languages like R.

**Import the Libraries**

The first step is to import the libraries. In python, we require libraries like numpy, matplotlib, pandas, etc. The library – pandas, is commonly used to import datasets and manage dataset. So the first step is to import the libraries and then to import the dataset. To execute the code I am writing, make sure to place this python file inside the same folder as your input dataset file.

#Import the libraries import numpy import pandas as pd import matplotlib |

You can download the complete code as well as the dataset file here.

**Import the Dataset**

#Import the dataset dataset = pd.read_csv('Data.csv') |

Once we have fetched the complete dataset into our python variable, we need to filter the independent and dependent variables. This can be accomplished using the iloc function. If you are using Spyder(Anaconda), you can hover your cursor on iloc and press Ctrl+I to see what parameters this function will require. Variable X will hold the independent variable matrix called observations or features and Y will hold the dependent variable vector.

#Fetch independent variable(observation) matrix X = dataset.iloc[:,:-1].values #iloc[index for rows, index for columns] #Fetch dependent variable vector Y = dataset.iloc[:,3].values |

**The case of Missing Data**

Now let’s assume, the dataset we obtained had some missing values. This can interfere with the accuracy of the predictions that our machine learning program will generate. However, handling of missing data might not be required in all cases, but it is still an important concept to understand. So in case of missing data, the most common technique is to replace that missing value with the mean value of that complete column. To do this mathematical mean calculation we need another python library – scikit-learn.

#Missing Values from sklearn.preprocessing import Imputer imputer= Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0) |

Here we have created an imputer object which will replace any missing value in the data by ‘NaN’. The strategy used to replace the missing value is Mean. Axis = 0 means the mean value has to be calculated along the values available across that column. Axis = 1 would mean calculation of mean value across the rows.

Once the imputer object is created for the missing values, we need to fit this imputer and change the values in the dataset. Since mean can only be performed on the numeric columns, we will assign this imputer to the numeric columns only. The complete code to replace missing values with the mean of the column is below.

#Missing Values from sklearn.preprocessing import Imputer imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0) X[:,1:] = imputer.fit_transform(X[:,1:]) |

**Categorical Variables**

As we just observed above, only the numeric data could be taken into use to perform any mathematical calculation and string data is left abandoned. However, if we need to process the string data, it needs to be first categorized and converted into mathematical values for calculations. For example – France will be represented by 0, Germany by 1 and Spain by 2. To directly convert this, we again use the scikit-learn library’s encoding functions.

#Categorizing data from sklearn.preprocessing import LabelEncoder labelencoder_X = LabelEncoder() # Encoding the Independent Variable X[:,0] = labelencoder_X.fit_transform(X[:,0]) # Encoding the dependent Variable Y[:,] = labelencoder_X.fit_transform(Y[:,]) |

After this step, the first column in our data that was having the country names as Spain, France, etc has been converted into their mathematical representation of 0,1 etc. If we look deeply into this, earlier there was no relation between Spain and France, however, after converting it into mathematical representation they have developed a relationship as 1>0 which might cause an issue to the calculations. To overcome this issue, we use the approach of dummy variables. We create dummy variables equal to the number of categories created in the above step.

Values obtained in Dummy Variables :

**Category **France Spain Germany

France 1 0 0

Spain 0 1 0

France 1 0 0

Germany 1 0 1

To achieve this we use OneHotEncoder function from the LabelEncoder library.

#Create Dummy Variables from sklearn.preprocessing import OneHotEncoder onehotencoder = OneHotEncoder(categorical_features = [0]) #Here 0 is the index of the column for which dummy variables have to be created. X = onehotencoder.fit_transform(X).toarray() |

**Test And Train Sets**

This is again an important part of data preprocessing. Out of the total data, we filter out some data as test data and the remaining data as the training observation data(or features). The train data will be used by the program to observe the behavior and predict the answer while the test data will be used to cross verify the answers. For the generation of test and train sets, scikit-learn library’s cross_validation method is used.

#Test and Train Data from sklearn.cross_validation import train_test_split X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state = 0) |

Test_size refers to the percentage of data to be moved to test set from the parent dataset.

**Feature Scaling**

Scaling is required for mathematical calculations in machine learning to bring all data to a common scale. For example, age = 40 and salary = 90000. Here salary is way bigger than age which will result in the huge gap while performing mathematical calculations on the two variables. To bring them to smaller and easily comparable values, we scale them using feature scaling. Feature scaling might not be required every time, but it might be required in some cases. To perform feature scaling, we import StandardScaler from the scikit-learn python library.

# Feature Scaling from sklearn.preprocessing import StandardScaler scale_X = StandardScaler() X_train = scale_X.fit_transform(X_train) X_test = scale_X.fit_transform(X_test) scale_Y = StandardScaler() Y_train = scale_Y.fit_transform(Y_train) Y_test = scale_Y.fit_transform(Y_test) |

With this our data preprocessing concept is complete and we can have a look at the complete data preprocessing template. The template is very useful and saves a lot of time. In most of the cases, this template can be used directly by just changing few values as per requirements like – dataset name, dataset column index values, and test size value.

**Data Preprocessing Template**

#Import the libraries import numpy import pandas as pd import matplotlib #Import the dataset dataset = pd.read_csv('Data.csv') #Fetch independent variable(observation) matrix X = dataset.iloc[:,:-1].values Y = dataset.iloc[:,2:3].values """ -- This block is optional to use -- #Missing Values from sklearn.preprocessing import Imputer imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0) X[:,1:] = imputer.fit_transform(X[:,1:]) """ """ -- This block is optional to use -- #Categorizing data from sklearn.preprocessing import LabelEncoder labelencoder_X = LabelEncoder() # Encoding the Independent Variable X[:,0] = labelencoder_X.fit_transform(X[:,0]) # Encoding the dependent Variable Y[:,] = labelencoder_X.fit_transform(Y[:,]) #Create Dummy Variables from sklearn.preprocessing import OneHotEncoder onehotencoder = OneHotEncoder(categorical_features = [0]) X = onehotencoder.fit_transform(X).toarray() """ #Test and Train Data from sklearn.cross_validation import train_test_split X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state = 0) """ -- This block is optional to use -- # Feature Scaling from sklearn.preprocessing import StandardScaler scale_X = StandardScaler() X_train = scale_X.fit_transform(X_train) X_test = scale_X.fit_transform(X_test) scale_Y = StandardScaler() Y_train = scale_Y.fit_transform(Y_train) Y_test = scale_Y.fit_transform(Y_test) """ |