How to reduce the memory size of Pandas Data frame

After importing with pandas read_csv(), dataframes tend to occupy more memory than needed. This is a default behavior in Pandas, in order to ensure all data is read properly. It’s possible to optimize that, because, lighter the dataframe, faster will be the operations you do on them later on.

So, let’s first check how much memory (RAM) this dataframe occupies in MB.

Check memory usage of pandas dataframe in Mb

# size occupied by dataframe in mb.
df.memory_usage(deep=True).sum() / 1024**2

2.63516902923584

That’s about 2.6MB. That’s not a lot for modern computers, and by reducing the size of this data, you will not see a noticeable difference in processing times.

But, when the dataset size is large, reducing the dataframe size without affecting the content can matter.

So how do we go about reducing the size of the dataframe?

How to reduce the size of the dataframe without affecting the content?

The idea is, for a variable like ‘Age’, it will probably have values less than 100 ever. So, it is sufficient to use ‘int8’ datatype for this column instead of ‘int64’.

Because, int8 can hold values between -128 to 127. Whereas, int64 can hold much larger numbers, thereby requires more memory when stored.

This idea applies for other variables like ‘NumofProducts’, ‘CreditScore’ etc as well.

Certain features occupy more memory than what is needed to store them. Reducing memory usage by changing data type will speed up the computations. So, whereever possible, it’s better to avoid using the largest possible datatype.

How to know how much bytes a given datatype require?

Let’s create a function for that:

int8 / uint8 : consumes 1 byte of memory, range between -128/127 or 0/255. The ‘u’ in uint stands for unsigned.
bool : consumes 1 byte, true or false
float16 / int16 / uint16: consumes 2 bytes of memory, range between -32768 and 32767 or 0/65535
float32 / int32 / uint32 : consumes 4 bytes of memory, range between -2147483648 and 2147483647
float64 / int64 / uint64: consumes 8 bytes of memory

print('int64 min: ', np.iinfo(np.int64).min)
print('int64 max: ', np.iinfo(np.int64).max)
print('int8 min: ', np.iinfo(np.int8).min)
print('int8 max: ', np.iinfo(np.int8).max)

int64 min:  -9223372036854775808
int64 max:  9223372036854775807
int8 min:  -128
int8 max:  127

Function to optimize memory usage

The idea is quite straight: For each column, check the max and min value for each variable and decide what datatype will be suitable. Then change the datatype.

Apply this logic for all columns in the dataset. Doing this often saves the size the dataframe significantly.

# Reduce memory usage
def reduce_mem_usage(df, verbose=True):
    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
    start_mem = df.memory_usage(deep=True).sum() / 1024**2    
    for col in df.columns:
        col_type = df[col].dtypes
        if col_type in numerics:
            c_min = df[col].min()
            c_max = df[col].max()
            if str(col_type)[:3] == 'int':  # for integers
                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                    df[col] = df[col].astype(np.int8)
                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                    df[col] = df[col].astype(np.int16)
                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                    df[col] = df[col].astype(np.int32)
                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                    df[col] = df[col].astype(np.int64)  
            else:  # for floats.
                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                    df[col] = df[col].astype(np.float16)
                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                    df[col] = df[col].astype(np.float32)
                else:
                    df[col] = df[col].astype(np.float64)    
    end_mem = df.memory_usage(deep=True).sum() / 1024**2
    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))
    return df

Apply

# Reduce the memory size of the dataframe
df_o   = reduce_mem_usage(df)

Mem. usage decreased to  2.02 Mb (23.5% reduction)

Now’ lets check the memory usage.

df_o.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 14 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   RowNumber        10000 non-null  int16  
 1   CustomerId       10000 non-null  int32  
 2   Surname          10000 non-null  object 
 3   CreditScore      9999 non-null   float16
 4   Geography        10000 non-null  object 
 5   Gender           9986 non-null   object 
 6   Age              9960 non-null   float16
 7   Tenure           10000 non-null  int8   
 8   Balance          9963 non-null   float32
 9   NumOfProducts    10000 non-null  int8   
 10  HasCrCard        10000 non-null  int8   
 11  IsActiveMember   10000 non-null  int8   
 12  EstimatedSalary  9999 non-null   float32
 13  Exited           10000 non-null  int8   
dtypes: float16(2), float32(2), int16(1), int32(1), int8(5), object(3)
memory usage: 459.1+ KB

Check memory usage again

df.memory_usage(deep=True).sum() / 1024**2

2.0152807235717773

Nice!.

This effect is more pronounced and impactful as for larger datasets.

[Next] Lesson 5: Exploratory Data Analysis (EDA)

Data Manipulation