Remove infinite values from Pandas DataFrame

Pandas

In this post, we are going to learn to Remove infinite values from Pandas DataFrame by using the built-in method replace(),dropna(),option_context(),set_option(),isin().The infinite values are longest positive or negative values.The Numpy library attribute np.inf, -np.inf is used for represent infinite values in dataframe.To remove or drop infinite values convert them into Nan

Remove infinite values from Pandas DataFrame

To remove/drop infinite positive or negative infinite value from pandas dataframe. First, we have to convert them into nan values by using the Pandas dataframe replace() method. After replacing, the Pandas dataframe dropna() method is used to drop all infinite values from the given dataframe. The inplace=True is used to drop from the original dataframe without creating a new copy.

import pandas as pd
import numpy as np
    
data = {
    'Name': ['Jack', 'Rack', 'Max', 'David'],
    'Marks':[np.inf,98, -np.inf,100],
    'Subject': ['Math', 'Math', 'Math', np.inf]
}
 
dfobj = pd.DataFrame(data)
 
dfobj .replace([np.inf, -np.inf], np.nan, inplace=True)

dfobj.dropna(inplace=True)
 
print('dataframe after Drop infinite :\n',dfobj )

Output

dataframe after Drop infinite :
    Name  Marks Subject
1  Rack   98.0    Math

2. option_context() to Remove infinite values from Pandas DataFrame

The pd.option_context() context manager temporary set option in the with statement block.In the python program Firstly, we modify all inf values with nan, and dropna() method is used to drop the infinite values from the whole dataframe.

import pandas as pd
import numpy as np
    
data = {
    'Name': ['Jack', 'Rack', 'Max', 'David'],
    'Marks':[np.inf,98, -np.inf,100],
    'Subject': ['Math', 'Math', 'Math', np.inf]
}
 
dfobj = pd.DataFrame(data)
 
with pd.option_context('mode.use_inf_as_na', True):
  dfobj.dropna(inplace=True)
 
print('dataframe after Drop infinite :\n',dfobj )

Output

dataframe after Drop infinite :
    Name  Marks Subject
1  Rack   98.0    Math

2.1 Remove infinite from Pandas DataFrame

In the above example, we have used pd.option_context() context manager to temporarily set option the with statement block. The pd.option_context() can be used in the whole code that is used set all the infinite values to nan and pd.dropna() to remove infinite values from the existing dataframe without creating a new copy.

pd.set_option('mode.use_inf_as_na', True)
dfobj.dropna(inplace=True)
print(dfobj)

Output

   Name  Marks Subject
1  Rack   98.0    Math

3. Remove infinite from Pandas DataFrame using replace()

In this python program example first, we are replacing all positive and negative infinite values with nan using pd.replace() method, and after replacing the values.Next used pd.dropna() method to drop values along with given axis=0 .

 
dfobj = pd.DataFrame(data)

dfobj = dfobj.replace([np.inf, -np.inf], np.nan).dropna(axis=0)
print(dfobj)

Output

   Name  Marks Subject
1  Rack   98.0    Math

4. Remove infinite from Pandas DataFrame using not-null()

In this python program example we have used pd.replace() to replace negative or positive infinite values with nan and selected non-null rows and all columns. Let us understand with the below example.

import pandas as pd
import numpy as np
    
data = {
    'Name': ['Jack', 'Rack', 'Max', 'David'],
    'Marks':[np.inf,98, -np.inf,100],
    'Subject': ['Math', 'Math', 'Math', np.inf]
}
 
dfobj = pd.DataFrame(data)

dfobj = dfobj[dfobj.replace([np.inf, -np.inf], np.nan).notnull().all(axis=1)] 
print(dfobj)

Output

   Name  Marks Subject
1  Rack   98.0    Math

5. Remove infinite from Pandas DataFrame

In this python program example, First, we have to find all positive and negative infinite values using df.isin() method that returns a dataframe with boolean values, where True for infinite positive or negative values and False for other values. After filtering we have dropped nan values from the dataframe using pd.dropna() method.

import pandas as pd
import numpy as np
    
data = {
    'Name': ['Jack', 'Rack', 'Max', 'David'],
    'Marks':[np.inf,98, -np.inf,100],
    'Subject': ['Math', 'Math', 'Math', np.inf]
}
 
dfobj = pd.DataFrame(data)



filter_inf = dfobj.isin([np.nan, np.inf, -np.inf])
print('Fillter boolean values :\n',filter_inf)


dfobj = dfobj[~filter_inf]
dfobj.dropna(inplace=True)
print()
print(dfobj)

Output

Fillter boolean values :
     Name  Marks  Subject
0  False   True    False
1  False  False    False
2  False   True    False
3  False  False     True

   Name  Marks Subject
1  Rack   98.0    Math

Summary

In this post we have learned how to Remove infinite values from Pandas DataFrame with python program examples by using replace() and dropna(),option_context(),set_option(),isin() method

Related Posts