pipeline in linux/unix with examples
Pipeline technology is an inter-process communication technology in Linux and other Unix-like operating systems, similar to redirection, which transmits standard output to other target locations.
Linux pipe are used for communication between applications, between Linux commands, and between applications and commands.
Linux pipe symbol is the vertical bar symbol “|”, pipe communication format between commands:
command1 | command 2 | command3 | ... | commandNCommand1 to commandN represent N commands of linux.
These N commands use pipes to communicate. After the execution of command1 is completed, the execution result of command1 is used as the input parameter of command2 through the pipeline, and so on.
Pipeline usage examples
In the following examples, we will use multiple commands to communicate through the pipeline to achieve our ultimate goal.
tail, pipeline and grep command example
When we are dealing with problems, we often need to monitor the content of log files with specific identifiers in real time.
In the following example, we will need to use the tail -f command to monitor real-time log updates, and pass the updated log content as input parameters to the grep command through the pipeline, and search the pipeline input content through the grep command.
➜ ~ tail -f pipeFile.log | grep "test"
awk, pipeline and sort command example
In the following example, we want to count the value of the first occurrence of the string in the first column of the test file divided by “#”.
We first need to use awk to cut and use the awk execution result as the input parameter of the sort command through the channel. After the sorting is completed, the uniq command is executed through the channel.
➜ ~ awk -F'#' '{print $1}' pipeFile.log | sort | uniq -c | sort -r
ls, pipe and grep command example
In the following example, we will use the ls command, pipeline, and grep command to query files or folders with the keyword “test” in the current directory.
➜ ~ ls | grep "test"