How to Get Unique Lines from a File Using PowerShell?

When working with text files in PowerShell, I got a requirement to extract unique lines from a file. I tried different methods and thought to share them with you. In this tutorial, we’ll check several methods to get unique lines from a file using PowerShell.

Get Unique Lines from a File Using PowerShell

PowerShell is the best option for this kind of file operation. Let’s check each method with a complete example.

Method 1: Using Get-Unique Cmdlet

The PowerShell Get-Unique cmdlet is designed to compare each item in a sorted list to the next item, eliminate duplicates, and return only one instance of each item. This method requires the input to be sorted first.

Here is how it will work:

1. Read the file and sort the content: $lines = Get-Content “C:\MyFolder\MyExample.txt” | Sort-Object
2. Get unique lines: $uniqueLines = $lines | Get-Unique
3. Output the unique lines to a new file: $uniqueLines | Set-Content “C:\MyFolder\MyExample_unique.txt”

This method ensures you get unique lines, below is the complete script.

$lines = Get-Content "C:\MyFolder\MyExample.txt" | Sort-Object

$uniqueLines = $lines | Get-Unique

$uniqueLines | Set-Content "C:\MyFolder\MyExample_unique.txt"

$uniqueLines

I executed the above script, and it showed me the output in the console and written in the text file. Check out the screenshot below.

Method 2: Using Select-Object with -Unique Parameter

Let me show you another relatively simple method. In PowerShell, you can use the Select-Object with the -Unique parameter. When we use the -Unique parameter, we do not need to sort the input.

This is how it will work:

1. Read the file and get unique lines: $uniqueLines = Get-Content “C:\MyFolder\MyExample.txt” | Select-Object -Unique
2. Output the unique lines to a new file: $uniqueLines | Set-Content “C:\MyFolder\MyExample_unique.txt”

Here is the complete PowerShell script:

$uniqueLines = Get-Content "C:\MyFolder\MyExample.txt" | Select-Object -Unique
$uniqueLines | Set-Content "C:\MyFolder\MyExample_unique.txt"
$uniqueLines

Check out the screenshot below, it is giving the output like the above after I executed the PowerShell script using VS code.

Method 3: Using HashSet

If you are working with a large text file, you can use the HashSet to get unique lines in PowerShell.

Here is how it works.

1. Initialize the HashSet and read the file: $hashSet = [System.Collections.Generic.HashSet[string]]::new() $lines = Get-Content "C:\MyFolder\MyExample.txt"
2. Add each line to the HashSet: foreach ($line in $lines) { $hashSet.Add($line) | Out-Null }
3. Output the unique lines to a new file: $hashSet | Set-Content "C:\MyFolder\MyExample_unique.txt"

Here is the complete script.

# Step 1: Initialize the HashSet and read the file
$hashSet = [System.Collections.Generic.HashSet[string]]::new()
$lines = Get-Content "C:\MyFolder\MyExample.txt"

# Step 2: Add each line to the HashSet
foreach ($line in $lines) {
    $hashSet.Add($line) | Out-Null
}

# Step 3: Output the unique lines to a new file
$hashSet | Set-Content "C:\MyFolder\MyExample_unique.txt"

Method 4: Using Group-Object Cmdlet

Let me show you the final method.

You can also use the Group-Object cmdlet in PowerShell to group and then select unique lines.

Here is how it works:

1. Read the file and group by line content: $groupedLines = Get-Content "C:\MyFolder\MyExample.txt" | Group-Object
2. Select unique lines: $uniqueLines = $groupedLines | ForEach-Object { $_.Name }
3. Output the unique lines to a new file: $uniqueLines | Set-Content "C:\MyFolder\MyExample_unique.txt"

Below is the complete script.

# Step 1: Read the file and group by line content
$groupedLines = Get-Content "C:\MyFolder\MyExample.txt" | Group-Object

# Step 2: Select unique lines
$uniqueLines = $groupedLines | ForEach-Object { $_.Name }

# Step 3: Output the unique lines to a new file
$uniqueLines | Set-Content "C:\MyFolder\MyExample_unique.txt"

$uniqueLines

After I executed the above script, you can see the output in the screenshot below.

Conclusion

I hope now you have a complete idea of how to get unique lines from a file in PowerShell using various methods, such as the Get-Unique Cmdlet, the Select-Object with -Unique Parameter, and the Group-Object Cmdlet.

You may also like the following tutorials:

• How to Count Words in a File Using PowerShell?
• How to Count Files in a Folder Using PowerShell?
• Count Lines in a File in PowerShell
• Get Unique Values from CSV Using PowerShell

Bijay Kumar

Bijay Kumar is an esteemed author and the mind behind PowerShellFAQs.com, where he shares his extensive knowledge and expertise in PowerShell, with a particular focus on SharePoint projects. Recognized for his contributions to the tech community, Bijay has been honored with the prestigious Microsoft MVP award. With over 15 years of experience in the software industry, he has a rich professional background, having worked with industry giants such as HP and TCS. His insights and guidance have made him a respected figure in the world of software development and administration. Read more.