uniq command tutorial in linux/unix with examples and use cases

linux uniq command – report or omit repeated lines.

Linux uniq command, which usually requires sorted input, compares and ranks only on adjacent lines.

syntax

uniq [OPTION]... [INPUT [OUTPUT]]

options

• -c, –count
  prefix lines by the number of occurrences
• -d, –repeated
  only print duplicate lines, one for each group
• -D
  print all duplicate lines
• -f, –skip-fields=N
  avoid comparing the first N fields
• –group[=METHOD]
  show all items, separating groups with an empty line;
• -i, –ignore-case
  ignore differences in case when comparing
• -s, –skip-chars=N
  avoid comparing the first N characters
• -u, –unique
  only print unique lines
• -z, –zero-terminated
  line delimiter is NUL, not newline
• -w, –check-chars=N
  compare no more than N characters in lines

Examples

Remove adjacent duplicates using uniq command

➜ uniq test.log

Remove duplicates using sort and uniq commands

➜ sort test.log | uniq

Remove duplicates and count duplicates

➜ sort test.log | uniq -c
      3 hello 99
      5 test 73
      2 ylspirit 78

Show only duplicate rows

➜  uniq sort test.log | uniq -d