In Linux file processing, in addition to statistics, filtering, and sorting of file contents, we often perform multi-file merge, sort and merge, and find non-repeating content.
This Linux tutorial mainly solves the above problems.
Here you will use linux commands in combination: cat command, tail command, sort command, uniq command, pipeline and redirection, etc.
So let’s look at some usage examples.
How to merge multiple files into one file in Linux
Use tail command : Display multiple file contents and corresponding file names.
➜ tail -n +1 file-*
➜ tail -n +1 file-1.log file-2.log file-3.log
- -n, –lines=[+]NUM
output the last NUM lines, instead of the last 10; or use -n +NUM to output starting with line NUM
Use cat command and redirection symbol “>” : Merge multiple files into one file
➜ cat file-1.log file-2.log file-3.log > file.log
➜ cat file-* > file.log
#Redirection symbol ">": Overwrite
#Redirection symbol ">>": Append write
How to sort and deduplicate multiple files and import one file in linux
➜ cat file-* | sort | uniq > file.log
How to sort multiple files and export duplicate content to one file in linux
➜ cat file-* | sort | uniq -d > file.log