How to Remove Duplicates From a Python List ?

You can remove duplicates from a Python list using various methods. Below are some common approaches:

1. Using set()

A set in Python automatically removes duplicates because it only allows unique values.

my_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = list(set(my_list))
print(unique_list)  # Output: [1, 2, 3, 4, 5]

Note: This method does not preserve the original order of the list.

2. Using a Loop to Preserve Order

To remove duplicates while preserving the order, you can loop through the list and add items to a new list only if they are not already in it.

my_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = []
for item in my_list:
    if item not in unique_list:
        unique_list.append(item)

print(unique_list)  # Output: [1, 2, 3, 4, 5]

3. Using dict.fromkeys() (Python 3.7+)

Since Python 3.7, dictionaries maintain insertion order. You can use dict.fromkeys() to remove duplicates while preserving order.

my_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = list(dict.fromkeys(my_list))
print(unique_list)  # Output: [1, 2, 3, 4, 5]

4. Using List Comprehension and set() to Preserve Order

You can also use a set to track seen elements while using list comprehension.

my_list = [1, 2, 2, 3, 4, 4, 5]
seen = set()
unique_list = [x for x in my_list if not (x in seen or seen.add(x))]
print(unique_list)  # Output: [1, 2, 3, 4, 5]

5. Using pandas (if available)

If you’re working with a large dataset, you can use the pandas library to remove duplicates.

import pandas as pd

my_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = pd.unique(my_list).tolist()
print(unique_list)  # Output: [1, 2, 3, 4, 5]

Summary:

• Use set() for quick removal but lose the order.
• Use loops or dict.fromkeys() to preserve the order.
• Use list comprehension with set() for a more Pythonic solution that preserves order.