How to Read Large Files on Linux With Ease

Want to sift through a massive log file, text file, or dataset? You’re not alone. Reading large files on Linux can be challenging when you want to view and manage content without overwhelming your system resources. However, there are several approaches you can use to read large files on Linux, such as using the less command, Vim, or splitting the document into smaller chunks and others.

In this article, we’ll learn how to read large files or find specific information on Linux using various methods.

Content

Using Less Command

Looking for a lightweight Linux tool to view the contents of large text files or perform quick searches? Look no further than the less command.

I love this utility because, unlike regular text editors, it lets you view files one page at a time without loading the entire file into memory. This makes it faster, especially for large files, and helps you easily review long documents or logs.

To use it, just type less followed by the filename:

less filename.txt

This will open the less interface, where you can scroll through the file line by line using the arrow keys or search for specific terms by pressing / followed by your query.

You can also use less to read the other commands’ outputs using a pipeline operator. For example, to read and display the ls command output, use:

ls -l | less

Furthermore, the less command has several options to adjust its behavior. You can combine these options to fine-tune how less works.

For instance, you can use the -p option with less to search for a specific pattern:

ls -l | less -p "sample"

This command opens the output and jumps to the first occurrence of the word sample.

You can also display line numbers alongside the file content using the -N option:

ls -l | less -N

Split File Using Split Command

Sometimes, breaking a large file into smaller chunks is the best approach – especially when you want to process or read the file in manageable sections. For example, I split the file into chunks whenever its size exceeds 1 GB or contains more than 100 million lines.

You can split files by size or by line count. If you’re working with text, it’s best to split by lines to avoid cutting lines or words in half.

For example, to split a file based on a specified number of lines, run this command:

split -l 10000 samplefile.txt part_

In this example, the file “samplefile.txt” is split into multiple files, each containing 10,000 lines. The resulting files will be named “part_aa”, “part_ab”, and so on. Now, you can open and read smaller chunks of the file without worrying about memory usage or system slowdowns.

If you want to split large files based on size, such as 100 MB, you can run this command:

split -b 100M samplefile.txt part_

Using Midnight Commander

Midnight Commander (MC) is a dual-panel text-based file manager that provides an intuitive visual interface for navigating directories and files.

MC lets you view files directly in the interface, allowing you to quickly scroll through large logs or data without loading the entire file into memory. What I like about MC is its smooth and efficient way of scrolling through large files.

To install MC, simply run:

sudo apt install mc

You can launch it by running mc in the terminal. Once inside, you can navigate to the large file you want to read and view its content.

Using Klogg

Klogg is a fast, open-source GUI-based log viewer that can effortlessly handle large files. Unlike other editors, which load the entire file into memory, it only reads portions of the file as needed, reducing memory usage.

Klogg also offers real-time filtering and searching capabilities, making it easy to find what you’re looking for without excessive scrolling.

Before installing Klogg, you first need to create the “/etc/apt/keyrings” directory and add the GPG key to it. The GPG key is necessary to verify the Klogg package repository.

To create the directory, run:

sudo mkdir -p /etc/apt/keyrings

Now, add the GPG key:

curl -sS https://klogg.filimonov.dev/klogg.gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/klogg.gpg

Next, you need to add the Klogg repository to your system using the curl command:

curl -sS https://klogg.filimonov.dev/deb/klogg.jammy.list | sudo tee /etc/apt/sources.list.d/klogg.list

Update the package list:

sudo apt update

Finally, you can install Klogg with this:

sudo apt install klogg

Once installed, simply open the file through the interface and use its built-in features to search and navigate through the content.

Read Large Files Using Text Editor

Most text editors struggle with large files, but editors like Vim or Emacs handle large files better than standard editors like Nano or Gedit.

For instance, Vim has features that allow you to quickly jump around the file and search for specific terms without loading everything into memory at once. However, searches are limited to the portions already loaded.

To open a file in Vim, run:

vim samplefile.txt

Note: While Vim can manage large files more effectively, it’s still not the fastest tool. For quick peeks, it is generally more efficient to use the less command or the previously mentioned specialized tools.

Search Through File With Grep Command

If your goal is to find specific information within a large file, try using the grep command. It enables you to search through files and output only the lines that match your search query.

When you pipe command output into less, it’s temporary – once you close less, the output is gone. If you want to keep the output for later, redirect it to a file and open it with command-line tool like less.

For example, to print every line containing the word “ERROR” from a large file, run:

grep "Error" samplefile.txt

You can refine your search with additional options, like ignoring case sensitivity (grep -i) or searching for whole words only (grep -w).

Redirect output to a file

If you want to save specific information for later, you can redirect the output of your commands to a new file.

For example, you can search for lines using the grep command and save them to a new file:

grep "Error" samplefile.txt > errors.log

The > operator creates a new file or overwrites it each time. However, if you want to append data to an already present file, use >> instead of > operator.

Using Head and Tail Command

Sometimes, when working with large files on Linux, you may only need to view the beginning or the end of the file. To do this, you can use the head or tail commands.

You can display the first 20 lines of a file with this:

head -n 20 samplefile.txt

Similarly, to view the last 20 lines, run:

tail -n 20 samplefile.txt

In both cases, -n 20 specifies that you only want to see the first or last 20 lines. You can adjust this number to display more or fewer lines as needed. By default, both commands show 10 lines.

You can also combine tail and head to navigate through a particular file section. For example, to see lines 10-14 of a 100-line file, calculate the starting line from the end by subtracting the starting line minus one from the total number of lines (100 – 9 = 91). Then, use this command:

tail -n 91 samplefile.txt | head -n 5

This will display lines 10-14. You can confirm the output by comparing it to what you’d see using less.

If you’re monitoring a file that is constantly updating, like a log file, use tail -f to follow the changes in real time.

Wrapping up

Whether you’re sifting through logs, working with datasets, or viewing a large text file, these methods make the process simple and efficient. You can also learn how to find large files on Linux and how to transfer them to another location in the terminal.

Image credit: Unsplash. All alterations and screenshots by Haroon Javed.

Haroon Javed –
Contributor

Haroon is a lifelong tech enthusiast with over five years of experience writing thousands of articles about Linux, programming languages, and more. He loves exploring new technologies and experimenting with them to find innovative ways to use them. Haroon’s work has been featured on various online platforms, including HTG, Baeldung, and LinuxHint.

Subscribe to our newsletter!

Our latest tutorials delivered straight to your inbox

Sign up for all newsletters.
By signing up, you agree to our Privacy Policy and European users agree to the data transfer policy. We will not share your data and you can unsubscribe at any time. Subscribe