How to Sort and Remove Duplicates in Linux Files with sort and uniq

If you spend much time working with files or data in Linux, you’ll eventually run into the need to tidy up a list, organize information, or weed out duplicates. Maybe it’s a long list of email addresses, some messy CSV exports, or a log file that’s gotten out of control. While graphical tools can help, nothing beats the speed and power of Linux’s built-in command line tools: sort and uniq.

Whether you’re a beginner trying to clean up a simple file or a power user scripting data pipelines, mastering these two commands will save you time and keep your files organized. This guide will show you, in a friendly and practical way, how to use sort and uniq to transform messy text files into clean, sorted, and duplicate-free lists.

Table of Contents

1. Why Sorting and Removing Duplicates Matters
2. What are sort and uniq?
3. Sorting Files with sort
   1. Basic Use
   2. Sorting by Field or Column
   3. Sorting Numerically or by Date
   4. Case Insensitive and Reverse Sorting
4. Removing Duplicates with uniq
   1. Basic Use
   2. Showing Only Duplicates or Uniques
   3. Counting Repeated Lines
5. Combining sort and uniq for Best Results
6. Practical Examples for Real-World Tasks
7. Tips, Tricks, and Advanced Usage
8. Troubleshooting and Common Mistakes
9. Frequently Asked Questions (FAQ)
10. Conclusion

Why Sorting and Removing Duplicates Matters

Messy data is hard to work with. Repeated lines can cause confusion, repeated emails in a marketing list may annoy recipients, and unsorted logs are tough to read. Cleaning up data with sort and uniq makes it easier to read, process, and share.

For example, if you’re a system admin, you might need to find which users logged in most recently—or you might want to create a clean list of hosts that have connected to a service. Developers often process lists as part of a build or deployment script. No matter your job, these tools will help.

What are sort and uniq?

sort is a command that arranges lines in files or streams, alphabetically or numerically, in ascending or descending order.

uniq is a command that removes repeated lines—showing only unique lines, or just the duplicates, depending on the option.

Both are installed by default on nearly every Linux (and UNIX-like) system.

Sorting Files with sort

Basic Use

To arrange the lines of a file alphabetically:

sort mylist.txt

This prints the sorted lines to your terminal, leaving the original file unchanged.

To save the sorted output to a new file:

sort mylist.txt > sorted.txt

Sorting by Field or Column

If your file has columns (like a space- or comma-separated file), you can sort by a specific column.

Sort by the second column (assuming space/tab separated):

sort -k2 mylist.txt

For a comma-separated file (CSV), sort by the second column:

sort -t, -k2 mydata.csv

-t, sets the delimiter as a comma.

-k2 sorts on column 2.

Sorting Numerically or by Date

By default, sort works alphabetically. To sort numbers:

sort -n numbers.txt

For dates or times, you might use:

sort -M months.txt

(-M sorts by month name.)

Case Insensitive and Reverse Sorting

To sort regardless of capitalization:

sort -f mylist.txt

To reverse the order:

sort -r mylist.txt

Combine options as you like:

sort -f -r mylist.txt

Removing Duplicates with uniq

Basic Use

uniq mylist.txt

This filters out repeated lines, but only if they’re adjacent. That means you should always use sort first if you want to find all duplicates in a file.

Showing Only Duplicates

To see only lines that are repeated:

uniq -d mylist.txt

Showing Only Unique Lines

To see only lines that appear once:

uniq -u mylist.txt

Counting Repeated Lines

To count how many times each line appears:

uniq -c mylist.txt

You might see:

   3 apple
   1 banana
   2 cherry

Combining sort and uniq for Best Results

The classic combo to get a sorted list with all duplicates removed:

sort mylist.txt | uniq

Or, to save to a new file:

sort mylist.txt | uniq > cleaned.txt

If you want to see which lines are duplicates and how often they occur:

sort mylist.txt | uniq -c | sort -nr

This counts each unique line, then sorts so the most common lines are at the top.

Practical Examples for Real-World Tasks

Example 1: Clean a List of Email Addresses

sort emails.txt | uniq > unique_emails.txt

Now you have a clean list with one email address per line.

Example 2: Find the Most Frequent Error

sort errors.log | uniq -c | sort -nr > error_counts.txt

You’ll get a list of each error, with the most frequent at the top.

Example 3: Remove Duplicate Rows in a CSV by a Specific Column

sort -t, -k1,1 emails.csv | uniq > deduped_emails.csv

Example 4: Remove Case-Insensitive Duplicates

sort -f names.txt | uniq -i > unique_names.txt

Tips, Tricks, and Advanced Usage

• Remove blanks before sorting: grep -v '^$' mylist.txt | sort | uniq
• Sort in place: sort mylist.txt -o mylist.txt
• Get only the first 10 unique lines: sort mylist.txt | uniq | head -n 10
• Find and count duplicate lines only: sort mylist.txt | uniq -cd
• Sort by multiple columns: sort -k2,2 -k1,1 mydata.txt

Troubleshooting and Common Mistakes

• uniq doesn’t remove all duplicates? Always sort first, since uniq only removes adjacent duplicate lines.
• Sorting numbers as text? Use the -n flag for proper numerical sorting.
• Mixed tabs and spaces? Specify the delimiter with -t for consistent results.
• File is huge? sort and uniq are efficient, but for massive files, ensure enough temp space or split files if needed.

Frequently Asked Questions (FAQ)

Q: Does sort or uniq change my original file?

A: No, unless you redirect output back into the original file.

Q: Can I use sort and uniq on binary files?

A: They’re designed for text (line-based) files.

Q: How do I sort in descending order?

A: Add -r to sort.

Q: Can I remove duplicates without sorting?

A: uniq only removes adjacent duplicates, so sort first for best results.

Q: Can I keep the original order but remove duplicates?

A: For that, try: awk '!seen[$0]++' mylist.txt

Conclusion

Sorting and removing duplicates in Linux isn’t just for programmers or sysadmins—it’s for anyone who wants to keep data clean and organized. With sort and uniq, you have simple but powerful tools at your fingertips. Whether you’re cleaning contact lists, analyzing logs, or scripting automation, these commands can save you hours of manual work.

Don’t be afraid to experiment: try options, chain commands together, and see what works best for your needs. With a little practice, you’ll find organizing and deduplicating files becomes second nature.