In this post, We will explore different 6 Methods to Remove duplicates from Python list with code examples. We are going to use Set, NumPy unique() function, List comprehension, for loop, and other ways in python programming.
1.Set() to Remove duplicates from Python list
A Python set is a unique collection that does not contain duplicate values. It is an unordered collection and does not remember the insertion order. By converting a list to a set we can remove duplicates items from the python list and can return a unique list.
These are steps to remove duplicates using the set()
- Convert list(numlist) to set using set().
- It will give all the unique list items.
- Finally, covert set to list to maintain the insertion order.
#Program python remove duplicates from the list
numlist= [1,2,'C#',2,3, 'C','Go', 'Go','C#','Go','C',3,'C']
print('original list :\n',numlist)
uniquelist = (list(set(numlist)))
print ("\n list after removed duplicate:\n",uniquelist)
Output
original list :
[1, 2, 'C#', 2, 3, 'C', 'Go', 'Go', 'C#', 'Go', 'C', 3, 'C']
list after removed duplicate :
[1, 2, 3, 'C', 'C#', 'Go']
2. dict.fromkeys() to Remove duplicate from list
Using dict.fromkeys() will create dictionary keys from the list elements, the keys in a dictionary are unique. So this way, we can remove duplicates items from the list. Finally, convert it to a list to preserve the insertion order. This example works in Python 3+.
#Program python remove duplicates from the list
numlist= [1,2,'C#',2,3, 'C','Go', 'Go','C#','Go','C',3,'C']
print('original list :',numlist)
uniquelist = list(dict.fromkeys(numlist))
print (" \nlist after removed duplicate :\n",uniquelist)
Output
original list : [1, 2, 'C#', 2, 3, 'C', 'Go', 'Go', 'C#', 'Go', 'C', 3, 'C']
list after removed duplicate :
[1, 2, 'C#', 3, 'C', 'Go']
3. Remove duplicate from list using NumPy
We can remove duplicate items from the list using the NumPy unique() method. First, we have to import the NumPy module.
These are steps to install and import and use NumPy.
- Install NumPy using pip install Numpy
- import numpy as np
- Next use the Numpy np. unique() method to remove duplicates from the python list.
#Program python remove duplicates from the list
import numpy as np
numlist = [1,2,'C#',2,3, 'C','Go', 'Go','C#','Go','C',3,'C']
print('original list :',numlist)
uniquelist = np.unique(numlist).tolist()
print ("\n list after removed duplicate:\n",uniquelist)
Output
original list : [1, 2, 'C#', 2, 3, 'C', 'Go', 'Go', 'C#', 'Go', 'C', 3, 'C']
list after removed duplicate:
['1', '2', '3', 'C', 'C#', 'Go']
4. For loop with Temporary List
- We will loop over the original list (orig_list), check for an item that exists in the temporary list.
- if the element does not exist in the temporary list (temp_list), then we will append the item in the Temporary list (temp_list) else we will not append it.
- After appending all unique items to the Temporary list.
- Finally, we assigned the Temporary list (temp_list) to the original list (orig_list) to preserve the insertion order.
Let us understand with code example.
#Program python remove duplicates from the list
orig_list = [1,2,'C#',2,3, 'C','Go', 'Go','C#','Go','C',3,'C']
print('original list :',orig_list)
temp_list = []
for item in orig_list:
#checking item exist in temp_list
if item not in temp_list:
temp_list.append(item)
orig_list = temp_list
print ("\n list after removed duplicate:\n",orig_list )
Output
original list : [1, 2, 'C#', 2, 3, 'C', 'Go', 'Go', 'C#', 'Go', 'C', 3, 'C']
list after removed duplicate:
[1, 2, 'C#', 3, 'C', 'Go']
5.List comprehension to remove duplicates from Python list
It is almost the same as removing elements using for loop with a temporary list, list comprehension. But by using list comprehension we achieve the same with a shortcode.
First, we are iterating over the list, checking if an element exists in the temporary list [new_list] if exists then we are not appending else we are appending using the list built-in append() method.
#Program python remove duplicates from the list
numlist = [1,2,'C#',2,3, 'C','Go', 'Go','C#','Go','C',3,'C']
print('original list :',numlist)
new_list = []
[new_list.append(item) for item in numlist if item not in new_list ]
print('list after removed duplicate :',new_list )
Output
original list : [1, 2, 'C#', 2, 3, 'C', 'Go', 'Go', 'C#', 'Go', 'C', 3, 'C']
list after removed duplicate : [1, 2, 'C#', 3, 'C', 'Go']
6. Using List Comprehension with enumerate()
We can use the enumerate() function with list Comprehension to remove duplicates from the list in python. The enumerate() adds a counter to each element of the list.
#Program python remove duplicates from the list
numlist = [1,2,'C#',2,3, 'C','Go', 'Go','C#','Go','C',3,'C']
print('original list :',numlist)
new_list = [x for y, x in enumerate(numlist) if x not in numlist[:y]]
print('list after removed duplicate =',new_list)
Output
original list : [1, 2, 'C#', 2, 3, 'C', 'Go', 'Go', 'C#', 'Go', 'C', 3, 'C']
list after removed duplicate = [1, 2, 'C#', 3, 'C', 'Go']
Conclusion
In this post, We have explored different 6 Methods to Remove duplicates from Python list with code examples. We can use any of them to remove duplicates from the list in Python.