5 ways to Remove duplicates list of Python dictionaries

Python Dictionary

In this post, we will understand the different 5 ways to Remove duplicates list of Python dictionaries with examples. Here we are using simple for loop, list comprehension,frozenset(), and many more techniques.

1. For loop to Remove duplicates list of Python dictionaries


The simple and easy method comes to our mind to loop over lists of dictionaries and compares the value for a duplicate to remove the duplicate value.

In this code example, we are looping over the lists of dictionaries and removing the duplicate dictionary, and appending it in the new dictionary.

Let us understand with code examples and build our understanding from it.

Program Example

#Program to remove duplicates from a list of dictionaries
Langlist = [{"C#" : 1}, {"Rust" : 2}, {"C++" : 3}, {"Rust" : 2}, {"python" : 4},{"python" :4}]

result_dict=[]
#removing the duplicate entry
for i in range(len(Langlist)): 
    if Langlist[i] not in Langlist[i + 1:]: 
        result_dict.append(Langlist[i]) 

print('after Removing duplicate from list of dictionary =\n',result_dict)

Output

after Removing duplicate from list of dictionary =
 [{'C#': 1}, {'C++': 3}, {'Rust': 2}, {'python': 4}]

2. List Comprehension to Remove duplicates list of Python dictionaries


In this code example, we are using list comprehension with enumeration to remove the duplicate from lists of dictionaries. List comprehension helps us use a shortcode and do the job in a single line of code.

Let us understand with code example.

Program Example

#Program to remove duplicates from a list of dictionaries
Langlist = [{"C#" : 1}, {"Rust" : 2}, {"C++" : 3}, {"Rust" : 2}, {"python" : 4},{"python" :4}]

#removing the duplicate entry
unique_dict = [k for j, k in enumerate(Langlist) if k not in Langlist[j + 1:]]

print('after Removing duplicate from list of dictionary =\n',unique_dict)

Output

after Removing duplicate from list of dictionary =
 [{'C#': 1}, {'C++': 3}, {'Rust': 2}, {'python': 4}]

3. Remove duplicate lists of dictionary by convert list to tuple


In this code example, we are converting lists of dictionaries into a list of tuples, and further using the set Comprehension to remove the duplicate from the lists of dictionaries.

Let us understand with code examples and build our understanding.

Program Example

#Program to remove duplicates from a list of dictionaries
Langlist = [{"C#" : 1}, {"Rust" : 2}, {"C++" : 3}, {"Rust" : 2}, {"python" : 4},{"python" :4}]

#removing the duplicate entry
unique_dict = [dict(tuple) for tuple in {tuple(sorted(dict.items())) for dict in Langlist}]

print('after Removing duplicate from list of dictionary =\n',unique_dict)

Output

after Removing duplicate from list f dictionary =
 [{'C#': 1}, {'Rust': 2}, {'python': 4}, {'C++': 3}]

4. Using package iteration_utilities.unique_everseen


In this code example, we are using the third-party package to remove duplicates from lists of dictionaries. We are importing the package iteration_utilities import unique_everseen and using its unique_everseen() function to remove duplicates.

Let us understand with code example below.

Program Example

#Program to remove duplicates from a list of dictionaries
from iteration_utilities import unique_everseen
Langlist = [{"C#" : 1}, {"Rust" : 2}, {"C++" : 3}, {"Rust" : 2}, {"python" : 4},{"python" :4}]

#removing the duplicate entry
unique_dict=list(unique_everseen(Langlist))

print('after Removing duplicate from list of dictionary =\n',unique_dict)

Output

after Removing duplicate from list of dictionary =
 [{'C#': 1}, {'Rust': 2}, {'C++': 3}, {'python': 4}]

5. frozenset() to Remove duplicates list of dictionaries


frozenset() is an inbuilt function of python it takes iterable as an argument and makes it immutable(unchangeable) or it freezes the iterable object and makes it unchangeable. It returns an immutable frozenset() object initialize with the elements from the iterable.

In this code snippet, we are using the frozenset() to remove duplicates from lists of dictionaries.

Let us understand with the code example below.

Program Example

#Program to remove duplicates from a list of dictionaries
Langlist = [{"C#" : 1}, {"Rust" : 2}, {"C++" : 3}, {"Rust" : 2}, {"python" : 4},{"python" :4}]

#removing the duplicate entry
unique_dict = {frozenset(item.items()) : item for item in Langlist}.values()
print("after Removing duplicate from list of dictionary ="+str(unique_dict))

Output

after Removing duplicate from list of dictionary =
 dict_values([{'C#': 1}, {'Rust': 2}, {'C++': 3}, {'python': 4}])

Conclusion

In this post, we understood the different ways to remove duplicates from the list of dictionaries. We hope you will find these ways useful and these will help you to solve your problem.

These techniques will be really helpful when you are working on the Data analysis work and they help you clean your data set samples.