7 ways to find duplicates in Python List

Python List

In this post, we are going to understand different 7 ways to find duplicates in Python List with code examples. We will return a new list of duplicates items from the Python list. These ways can be used both string and int types of list items

1. Using Set with list comprehension


In this example, we are using list comprehension to check the elements that exist more than once in the original list, later converting it to a SET to get the unique count of elements and printing the newly created list.

# Program to find duplicates in the list and return list of duplicate items in list

original_list = ['C++',13,15,13,15,17,'C#','Go','C#','C#','C#']



duplicate_item_list = set([item for item in original_list if original_list.count(item) > 1])

   



print('duplicate items in list :',duplicate_item_list)

Output

duplicate items in list : {'C#', 13, 15}

2. Using collections module


In this example, we are using the collection module, checking which item represents more than once and creating a new list with duplicate items in the list, and printing the list.

# Program to find duplicates in the list and return list of duplicate items in list

import collections

original_list = ['C++',13,15,13,15,17,'C#','Go','C#','C#','C#']

duplicate_list_item =  [item for item, count in collections.Counter(original_list).items() if count > 1]

print('new list of duplicate items   :',duplicate_list_item)

Output

new list of duplicate items : [13, 15, 'C#']

3. Using for loop and dictionary


In this example, we are using the dictionary for loop to check if the item is not in the dictionary. We are then adding it into the dictionary else incrementing 1 each time item is present more than once and happening to Create a new list with duplicate items in the list, and printing the list.

# Program to find duplicates in the list and return list of duplicate items in Python


original_list = ['C++',13,15,13,15,17,'C#','Go','C#','C#','C#']

dict = {}
duplicate_item_list = []

for item in original_list:
    
    if item not in dict:
        dict[item] = 1
    else:
        if dict[item] == 1:
            duplicate_item_list.append(item)
        dict[item] += 1


print('duplicate items in list :',duplicate_item_list)

Output

duplicate items in list :[13, 15, 'C#']

4. Using iteration_utilities to find duplicate in list


In this example we are using the iteration_utilities modules methods to find duplicates in the list and creating a new list.

# Program to find duplicates in the list and return list of duplicate items in Python


from iteration_utilities import duplicates,unique_everseenfrom 

original_list = ['C++',13,15,13,15,17,'C#','Go','C#','C#','C#']

dupes = list (unique_everseen(duplicates(original_list)))

print(dupes)

Output

duplicate items in list : [13, 15, 'C#']

5. Using Pandas


Using the Pandas module we can easily find the duplicate in the list and create a new list.

# Program to find duplicates in the list and return list of duplicate items in Python


import pandas as pd


original_list = ['C++',13,15,13,15,17,'C#','Go','C#','C#','C#']

pd_vec = pd.Series(original_list).value_counts()

list_dupes = pd_vec[pd_vec > 1].index.tolist()

print('duplicate items in list  :',list_dupes)

Output

duplicate items in list : [13, 15, 'C#']

6. Counter to find duplicate in Python list


In this example, we are using the counter module to find the duplicate and create a new list of duplicate items.

# Program to find duplicates in the list and return list of duplicate items in Python

from collections import Counter

original_list = ['C++',13,15,13,15,17,'C#','Go','C#','C#','C#']



dupvalue = Counter(original_list) - Counter(set(original_list))

duplicate_list_items =  list(dups.keys())

print('duplicate items in list:',duplicate_list_items)

Output

duplicate items in list : ['C#', 13, 15]

7. Using group by to find duplicate in Python list


In this example, we are using the itertools module groupby() method to find duplicates in the list and creating a new list of duplicate items in the list.

# Program to find duplicates in the list and return list of duplicate items in Python


from itertools import groupby

original_list = [13,15,13,15,17,17]
duplicate_item_list = []

for a, b in groupby(sorted(original_list)):
 if len(list(b)) > 1:
     duplicate_item_list.append(a)
print('duplicate item in a list :',duplicate_item_list)

     

Output

duplicate item in a list : [13, 15, 17]

Conclusion

In this post, we have explored 7 different ways to find duplicates and return a list of duplicate items in Python with code examples. You can use any of them as per your requirement. We hope that you find this post useful.