Introduction to One-Hot Encoding

What is one-hot encoding?

In digital circuits and machine learning, a one-hot is a group of bits among which the legal combinations of values are only those with a single high (1) bit and all the others low (0).

In machine learning, features are not always continuous and sometimes are categories. For example, the feature gender can be categorized as 'male' or 'female'. In model training, we have to convert these features to numerical values. Consider the following example with 3 features:

• gender: ['female', 'male']
• region: ['Africa', 'Asia', 'Europe', 'US']
• class: ['A', 'B', 'C'']

For a specific sample, like ['female', 'Asia', 'C'], to convert the features to numerical codes, the easier way is to serialize them to [0, 1, 2]. However, these serialized features cannot be used directly in machine learning model training.

In statistics, we usually use dummy variables to represent these categorical variables. Similarly, we can use a representation of categorical variables as binary vectors to extract features and convert them to numbers. In our example, 'female' corresponds to [1, 0], 'Asia' corresponds to [0, 1, 0, 0] and 'C' corresponds to [0, 0, 1], so the overall numerical representation of this sample is [1, 0, 0, 1, 0, 0, 0, 0, 1]. This technique of conversion is called one-hot encoding.

What are the advantages and disadvantages?

Advantage

• It could be better to deal with categorical features;
• It could be better to deal with discontinuous numerical features;
• Possibly increase the number of features. For instance, sex (male/female) is one feature, but after converting into one-hot encoding, it becomes two features: male | female.

Disadvantages

• In NLP, it disregards the sequences of words;
• It assumes features are independent. However, in a lot of cases, features are correlated with each other;
• The data can be extremely sparse after the conversion.

How to convert to one-hot encoding?

As an instance, consider we have 4 samples of 3 features:

In the table above, features are already converted into numerical codes. There are 2 possible choices for feature_1: perhaps male/female, and it's represented as 0-male / 1-female. But how can we convert these numbers into one-hot encoding?

Let one digit in the binary code of a feature represents a choice of the feature. Then, the digit representing the choice of the sample is set to 1 and all the remaining digits are set to 0, as below:

In this way, the original table is converted to:

The feature vectors for the four samples above are now:

This is the one-hot representation of the four samples, in which all features of the samples are converted into one-hot encoding.

One-hot encoding using Python

Using Pandas

pandas.get_dummies is used to convert categorical variables into one-hot encoding.

pandas.get_dummies(data, prefix=None, prefix_sep='_', dummy_na=False, columns=None, sparse=False, drop_first=False, dtype=None)

See the following usage example:

import pandas as pd

df = pd.DataFrame([
            ['male', 'Africa', 'C'],
            ['female', 'Asia', 'B'],
            ['male', 'Europe', 'B'],
            ['female', 'US', 'A']
])
df.columns = ['gender', 'region', 'class']
pd.get_dummies(df)

Before conversion, the data frame is:

After conversion, the new data frame is:

Using sklearn

sklearn.preprocessing.OneHotEncoder is used to encode categorical features as a one-hot numeric array.

class sklearn.preprocessing.OneHotEncoder(*, categories='auto', drop=None, sparse=True, dtype=<class 'numpy.float64'>, handle_unknown='error')

See the following usage example:

from sklearn import preprocessing

enc = preprocessing.OneHotEncoder()
enc.fit([[0, 3, 2], [1, 2, 1], [0, 1, 1], [1, 0, 0]])

array = enc.transform([[0, 3, 2]]).toarray()

print(array)

The result of the code above is:

[[1. 0. 0. 0. 0. 1. 0. 0. 1.]]

Why do we use one-hot encoding in machine learning?

In order to perform machine learning algorithms, like classification, regression, and clustering, we have to compute the 'distances' or 'similarities' between features, and most 'distances' and 'similarities' are defined to be computed in Euclidean space.

To compute the distances in Euclidean space, for discrete or categorical data, it is more rational in the binary representation.

For example, consider we have three feature vectors:
$x_1 = 0, x_2 = 1, x_3 = 2$ .

The distances between these vectors are:
$d(x_1, x_2) = 1 - 0 = 1, d(x_1, x_3) = 2 - 0 = 2$ .

This result indicates that the distance between $x_1$ and $x_2$ are smaller than the distance between $x_1$ and $x_3$ . However, there is no reason to say two features are more separated than two other features in a category.

However, if we convert the feature vectors into one-hot encodings, then $x_1 = (1, 0, 0)$ , $x_2 = (0, 1, 0)$ , and $x_3 = (0, 0, 1)$ . In this way, $d(x_1, x_2) = \sqrt{1^2+1^2} = \sqrt{2}$ , $d(x_1, x_3) = \sqrt{1^2+1^2} = \sqrt{2}$ , and $d(x_2, x_3) = \sqrt{1^2+1^2} = \sqrt{2}$ as well, so the three feature vectors are equally distant from each other in the Euclidean space, which corresponds to the characteristics of non-correlated categorical variables.

Therefore, in the representation of one-hot encoding, categorical features are binarized, and after binarizing these features, we can regard them as vectors embedded in the Euclidean space to calculate the distance for machine learning models.