How to Remove Duplicates from an Array in PowerShell?

Do you need to remove duplicates from a PowerShell array? In this PowerShell tutorial, I will explain how to remove duplicates from an array in PowerShell using different methods and examples.

To remove duplicates from an array in PowerShell, use the Select-Object cmdlet with the -Unique switch. For example, $uniqueArray = $array | Select-Object -Unique will filter $array to only include unique elements. This approach is efficient and concise for obtaining an array with distinct values only

Table of Contents

Remove Duplicates from an Array in PowerShell

If you are new to the PowerShell array, check out an article on the PowerShell array.

Here’s an example of a simple array in PowerShell:

$array = "apple", "banana", "apple", "cherry", "banana"

In this array, we have some duplicate entries for “apple” and “banana.” Now, I will show you different methods to remove these duplicates from this array in PowerShell.

Method 1: Using `Select-Object` with `-Unique`

One of the simplest ways to remove duplicates from an array in PowerShell is by using the Select-Object cmdlet with the -Unique parameter.

Here’s how you can use it:

$array = "apple", "banana", "apple", "cherry", "banana"
$uniqueArray = $array | Select-Object -Unique
$uniqueArray

This command pipes the $array into Select-Object, which selects each item. The -Unique parameter filters out the duplicates, leaving us with an array of unique values.

Check out the screenshot below after I executed the PowerShell script using VS code:

Remove Duplicates from an Array in PowerShell

Method 2: Using `Sort-Object` and `Get-Unique`

Another method involves sorting the array using Sort-Object followed by Get-Unique. This is particularly useful if you want the resulting array to be sorted as well as unique.

Here’s an example:

$sortedUniqueArray = $array | Sort-Object -Unique

Alternatively, you can use Get-Unique:

$array = "apple", "banana", "apple", "cherry", "banana"
$sortedArray = $array | Sort-Object
$uniqueSortedArray = $sortedArray | Get-Unique
$uniqueSortedArray

First, we sort the array, and then we pipe the sorted array into Get-Unique. The Get-Unique cmdlet compares each item in a sorted list to the next item, eliminates duplicates, and returns only one instance of each item.

You can see the output in the screenshot below after I executed the script using VS code.

Method 3: Using a Hashtable

For larger arrays or when performance is a concern, using a hashtable can efficiently remove duplicates in PowerShell. A hashtable is a data structure that stores key-value pairs and does not allow for duplicate keys.

Here’s how you can use a hashtable to remove duplicates:

$hashtable = @{}
foreach ($item in $array) {
    $hashtable[$item] = $true
}
$uniqueArray = $hashtable.Keys

In this script, we create an empty hashtable in PowerShell. We then loop through each item in the original array and add it to the hashtable, where each item becomes a key with a value of $true. Since hashtables cannot have duplicate keys, this effectively removes any duplicates. Finally, we extract the keys from the hashtable, the unique items from the original array.

Method 4: Using LINQ

For those familiar with .NET, PowerShell can leverage the Language-Integrated Query (LINQ) capabilities to remove duplicates. This method is more advanced and suitable for those needing more control over how the data is processed.

Here’s an example using LINQ:

$uniqueArray = [System.Linq.Enumerable]::Distinct([System.Collections.Generic.List[string]]$array)

In this line of code, we use the Distinct method from the System.Linq.Enumerable class to retrieve distinct elements from our array, which is cast to a generic list of type string.

Remove duplicates from a multidimensional array in PowerShell

Now, let us see how to remove duplicates from a multidimensional array in PowerShell.

Removing duplicates from a multidimensional array in PowerShell can be more complex because Select-Object -Unique does not work directly on arrays of arrays. Instead, you can use a combination of custom comparison logic with the Group-Object cmdlet. Here’s an example that removes duplicates based on a specific property of a multidimensional array:

# Define a multidimensional array with duplicate sub-arrays based on the first element
$multiArray = @(
    @(1, 'apple'),
    @(2, 'banana'),
    @(1, 'apple'), # duplicate
    @(3, 'cherry')
)

# Remove duplicates by grouping on the first element and then selecting the first group occurrence
$uniqueMultiArray = $multiArray | Group-Object { $_[0] } | ForEach-Object { $_.Group[0] }

# Display the unique multidimensional array
$uniqueMultiArray

In this example, $multiArray is a multidimensional PowerShell array with some duplicate entries. We use Group-Object to group the sub-arrays by the first element, and then for each group, we select only the first sub-array using ForEach-Object { $_.Group[0] }. The $uniqueMultiArray will contain the unique sub-arrays based on the first element of each sub-array.

This method assumes that the “duplicate” definition is based on the first element of each sub-array. If your definition of duplicates is different, you will need to adjust the script accordingly by changing the script block { $_[0] } in the Group-Object cmdlet to match your criteria for uniqueness.

Here, you can see the output in the screenshot below after I executed the PowerShell script using Visual Studio code.

Remove duplicates from multidimensional array in PowerShell

Conclusion

Removing duplicates from an array in PowerShell can be achieved through various methods. One of the easiest methods is to use the Select-Object -Unique for simplicity, while more advanced users may opt for hashtables or LINQ for performance and control.

I hope now you have an idea of how to remove duplicates from an array in PowerShell using various methods like: