How to split large files by line or by file size in Linux

In the daily development process, we occasionally encounter situations in which large files are processed.

At this time, in order to improve the processing efficiency, we often divide large files into several small files for processing according to certain rules.

In this linux command tutorial, I mainly introduce how to use the linux split command to cut large files.

linux split command — split a file into pieces

Syntax

split [-b byte_count[k|m]] [-l line_count] [file [name]]

Options

• -b byte_count[k|m]
  Create smaller files byte_count bytes in length.
  If “k” is appended to the number, the file is split into byte_count kilobyte pieces.
  If “m” is appended to the number, the file is split into byte_count megabyte pieces.
• -l line_count
  Create smaller files n lines in length.

split example(mac)

linux split by line

➜ split -a 2 -l 100000 test.log t1-

linux split by size

➜ split -a 2 -b 1m test.log t2-

Of course, when you split the file, you will also need to use the split of the string.

The linux awk command will be used here.

linux split by delimiter

➜ echo "adsf:ffff:2343:4444:444sfsafadsff" | awk -F":" '{print $5}'

linux split by line number

Export all odd line contents, use awk NR(ordinal number of the current record)

➜ awk '{if(NR % 2) {print $0}}' t2-ag 

Best linux commands articles:

Linux common commands tutorial and use examples
Sed command tutorials in linux/unix with examples and use cases
Awk command tutorials in linux/unix with examples and use cases