Linux Text Processing: Search, Transform, and Analyze Text
Learn powerful text processing with grep, sed, awk, and essential Linux text tools. Search log files, transform data, and build complex processing pipelines.
Why Text Processing Skills Matter
Linux is built around text. Configuration files are text. Log files are text. Command output is text. Script input and output is text. Knowing how to search, filter, and transform text is fundamental to working with Linux.
When you need to find errors in gigabytes of log files, grep finds them instantly. When you need to modify configuration across hundreds of files, sed does it in seconds. When you need to analyze structured data, awk processes millions of rows effortlessly.
These tools turn complex manual tasks into simple one-line commands. Instead of opening files in editors and searching manually, you process entire directories with a single pipeline. Instead of writing custom scripts, you combine existing tools.
This guide covers six essential text processing techniques. Learn these and you'll handle log analysis, data transformation, and text manipulation with confidence and speed.
Common Text Processing Mistakes
Using uniq Without Sorting First
The Problem: uniq only removes adjacent duplicate lines. If duplicates aren't next to each other, they won't be removed. This leads to confusion when uniq doesn't seem to work.
The Solution: Always use sort before uniq: sort file.txt | uniq. Or use sort -u to sort and remove duplicates in one command. The only time to skip sort is when processing already-sorted data or when you specifically want adjacent duplicates only.
Forgetting to Quote Regex Patterns
The Problem: Special characters in patterns can be interpreted by the shell instead of grep/sed. This causes unexpected behavior or errors.
The Solution: Always quote your patterns: grep 'pattern' not grep pattern. Use single quotes for literal strings, double quotes if you need variable expansion. Escape special characters: \. \* \[ \] \$ when needed.
Using sed -i Without Testing First
The Problem: sed -i modifies files in-place. If your pattern is wrong, you can corrupt files with no easy undo. Many users have accidentally destroyed important files this way.
The Solution: Test sed commands without -i first to see the output. Once verified, use sed -i.bak to create automatic backups. Only use sed -i alone when you're absolutely certain and have backups.
Confusing awk Field Numbers
The Problem: awk uses $1 for first field, $2 for second, etc. But $0 is the entire line. Also, awk splits on whitespace by default, which can surprise users with fixed-width data.
The Solution: Remember: $0 = whole line, $1 = first field, $NF = last field. Use -F to specify delimiter if not whitespace. Test with small samples and print NF to see how many fields awk sees.
6 Essential Text Processing Techniques
These techniques cover the core text processing tools every Linux user needs for searching, transforming, and analyzing text.
Searching Text with grep
grep searches for patterns in files using regular expressions. It's one of the most-used Linux commands, essential for finding specific text in logs, code, and configuration files.
# Basic search
$ grep "error" /var/log/syslog
# Case-insensitive search
$ grep -i "warning" logfile.txt
# Show line numbers
$ grep -n "TODO" app.py
# Recursive search in directory
$ grep -r "function" /path/to/code/
# Invert match (exclude pattern)
$ grep -v "^#" config.conf # Skip comment lines
# Count matches
$ grep -c "ERROR" app.log
# Show only filenames
$ grep -l "import" *.py
# Multiple patterns (OR)
$ grep -E "error|warning|critical" system.log
# Match whole word only
$ grep -w "test" file.txt # Matches "test" but not "testing"
# Show context (lines before/after)
$ grep -A 3 "ERROR" log.txt # 3 lines after
$ grep -B 2 "ERROR" log.txt # 2 lines before
$ grep -C 5 "ERROR" log.txt # 5 lines before and after
# Regex examples
$ grep "^Error" file.txt # Lines starting with "Error"
$ grep "failed$" file.txt # Lines ending with "failed"
$ grep "^$" file.txt # Empty lines
$ grep "[0-9]" file.txt # Lines containing digitsPro Tips: Use grep -E for extended regex (easier syntax with +, ?, |). Combine with other commands via pipes for powerful filtering. Quote your patterns to avoid shell interpretation. Use -r for searching entire directories, but be aware it can be slow on large directory trees.
Stream Editing with sed
sed edits text streams, performing find-and-replace, deletion, insertion, and text transformation. Perfect for automated text processing and bulk file modifications.
# Basic find and replace
$ sed 's/old/new/' file.txt
# Only replaces first occurrence per line
# Replace all occurrences (global)
$ sed 's/old/new/g' file.txt
# Edit file in-place (careful!)
$ sed -i 's/foo/bar/g' file.txt
# Make backup before in-place edit
$ sed -i.bak 's/old/new/g' file.txt
# Case-insensitive replace
$ sed 's/error/ERROR/gi' file.txt
# Delete lines matching pattern
$ sed '/^#/d' config.conf # Remove comment lines
$ sed '/^$/d' file.txt # Remove empty lines
# Delete specific line numbers
$ sed '5d' file.txt # Delete line 5
$ sed '1,3d' file.txt # Delete lines 1-3
# Print only matching lines
$ sed -n '/error/p' log.txt
# Multiple commands
$ sed -e 's/foo/bar/' -e 's/old/new/' file.txt
# Replace only on specific lines
$ sed '10,20s/old/new/g' file.txt # Lines 10-20
# Add text before line
$ sed '1i\Header Line' file.txt # Insert at line 1
# Replace with regex backreferences
$ sed 's/\([0-9]*\)/Number: \1/' file.txtPro Tips: Always test sed commands without -i first to preview changes. Use -i.bak to create backups automatically. sed processes line by line, making it memory efficient for large files. Escape special regex characters: . * [ ] ^ $ \ /
Text Processing with awk
awk is a powerful programming language for text processing. It excels at processing column-based data, performing calculations, and generating formatted reports.
# Print specific columns
$ awk '{print $1, $3}' file.txt # Print 1st and 3rd columns
# Print with custom separator
$ awk -F: '{print $1}' /etc/passwd # Use : as separator
# Add line numbers
$ awk '{print NR, $0}' file.txt
# Print lines matching condition
$ awk '$3 > 100' data.txt # Where 3rd column > 100
# Sum values in column
$ awk '{sum += $1} END {print sum}' numbers.txt
# Average calculation
$ awk '{sum += $1; count++} END {print sum/count}' numbers.txt
# Format output
$ awk '{printf "Name: %-10s Age: %d\n", $1, $2}' people.txt
# Multiple conditions
$ awk '$1 == "ERROR" && $3 > 5' log.txt
# Use built-in variables
$ awk '{print "Line", NR, "has", NF, "fields"}' file.txt
# Pattern matching
$ awk '/error/ {print $0}' log.txt # Like grep
# BEGIN and END blocks
$ awk 'BEGIN {print "Report"} {print $1} END {print "Done"}' data.txt
# Count occurrences
$ awk '{count[$1]++} END {for (word in count) print word, count[word]}' file.txt
# Process CSV
$ awk -F, '{print $1, $3}' data.csvPro Tips: awk is overkill for simple column extraction (use cut instead), but powerful for complex processing. Default field separator is whitespace; use -F to specify others. awk can replace grep, sed, cut, and wc in many cases, but specialized tools are often clearer.
Extracting Columns with cut
cut extracts specific columns or character positions from text. It's simpler and faster than awk for basic column extraction.
# Extract specific fields (tab-delimited)
$ cut -f1,3 data.txt
# Use custom delimiter
$ cut -d: -f1,6 /etc/passwd # Username and home directory
# Extract range of fields
$ cut -d, -f2-5 data.csv # Fields 2 through 5
# Extract from field to end
$ cut -d: -f3- /etc/passwd # Field 3 onwards
# Extract specific characters
$ cut -c1-10 file.txt # First 10 characters
# Extract multiple character ranges
$ cut -c1-5,10-15 file.txt
# Practical examples
# Extract usernames
$ cut -d: -f1 /etc/passwd
# Extract IP addresses from logs
$ cut -d' ' -f1 access.log
# Get file extensions
$ ls | cut -d. -f2-
# Process ps output (fixed-width)
$ ps aux | tail -n +2 | cut -c1-8 # First 8 characters
# Combine with other commands
$ cat data.csv | cut -d, -f2 | sort | uniq
# Output delimiter different from input
$ cut -d: -f1,6 /etc/passwd --output-delimiter=$'\t'Pro Tips: Use cut for simple column extraction, awk for complex processing. cut is faster for basic tasks. Remember: cut uses 1-based indexing (first column is 1, not 0). For space-delimited files with variable spacing, awk handles better than cut.
Sorting and Deduplicating with sort and uniq
sort arranges lines in order; uniq removes duplicate adjacent lines. Together they're perfect for analyzing data, finding unique values, and counting occurrences.
# Sort lines alphabetically
$ sort file.txt
# Sort numerically
$ sort -n numbers.txt
# Sort in reverse
$ sort -r file.txt
# Sort by specific column
$ sort -k2 data.txt # Sort by 2nd column
# Sort by column, numeric
$ sort -k3 -n data.txt
# Sort with custom delimiter
$ sort -t: -k3 -n /etc/passwd # Sort by UID
# Sort human-readable numbers
$ du -h * | sort -h
# Remove duplicates (must sort first!)
$ sort file.txt | uniq
# Sort and remove duplicates in one command
$ sort -u file.txt
# Count occurrences
$ sort file.txt | uniq -c
3 apple
1 banana
2 orange
# Sort by count (most common first)
$ sort file.txt | uniq -c | sort -rn
# Show only duplicates
$ sort file.txt | uniq -d
# Show only unique (non-duplicate) lines
$ sort file.txt | uniq -u
# Top 10 most common items
$ sort access.log | uniq -c | sort -rn | head -10
# Find unique IP addresses
$ awk '{print $1}' access.log | sort -u
# Count unique users
$ cut -d: -f1 /etc/passwd | wc -lPro Tips: Always pipe sort output to uniq, never the other way around. Use sort -u instead of sort | uniq when you only want unique lines. For large files, sort can be slow - consider using LC_ALL=C sort for speed (basic ASCII sorting).
Combining Tools with Pipes
The real power of Linux text processing comes from chaining commands together with pipes. Build complex data processing pipelines by combining grep, sed, awk, cut, sort, and uniq.
# Find most common error messages
$ grep ERROR app.log | awk '{print $5}' | sort | uniq -c | sort -rn | head -5
# Count unique IPs accessing a website
$ awk '{print $1}' access.log | sort -u | wc -l
# Top 10 largest files
$ du -ah /var/log | sort -rh | head -10
# Find most active users
$ last | awk '{print $1}' | sort | uniq -c | sort -rn | head -5
# Extract and count status codes from logs
$ awk '{print $9}' access.log | sort | uniq -c
423 200
89 404
12 500
# Process CSV: extract column, remove duplicates, sort
$ cut -d, -f3 data.csv | tail -n +2 | sort -u
# Complex log analysis
$ cat app.log \
| grep "2024-01" \
| awk '/ERROR/ {print $4}' \
| sort \
| uniq -c \
| sort -rn
# Find files modified today, count by extension
$ find . -mtime 0 -type f \
| sed 's/.*\.//' \
| sort \
| uniq -c
# Process system logs: filter, clean, deduplicate
$ journalctl -u nginx \
| grep ERROR \
| sed 's/^.*ERROR: //' \
| sort -u
# Word frequency count
$ cat book.txt \
| tr -cs '[:alpha:]' '\n' \
| tr '[:upper:]' '[:lower:]' \
| sort \
| uniq -c \
| sort -rn \
| head -20Pro Tips: Build pipelines incrementally - add one command at a time and verify output. Use tee to save intermediate results while continuing the pipeline. For long pipelines, add comments and consider converting to a script. Remember: pipes are processed left to right.
Your First Steps with Text Processing
Build text processing skills through hands-on practice with real files and data.
Practice grep Searches
Start with simple searches: grep 'word' file.txt. Add options one at a time: -i for case-insensitive, -n for line numbers, -r for directories. Search your own log files or code. Within a week, grep will be automatic for finding text.
Learn Basic sed Replacements
Practice find-and-replace: sed 's/old/new/' file.txt. Test without -i to see results safely. Try global replacement (/g), delete lines (/d), and multiple commands (-e). Start with simple patterns before attempting complex regex.
Build Simple Pipelines
Combine commands with pipes: grep pattern file | sort | uniq -c. Start with two commands, then add more. Test each step before adding the next. This incremental approach builds understanding and prevents errors in complex pipelines.
Go Deeper with Text Processing
This guide covers essential text processing. The Practical Linux Handbook includes advanced regex, complex awk programs, text analysis workflows, and complete scripting examples.
What You'll Learn:
- Complete text processing (grep, sed, awk, regex)
- Advanced pattern matching and regex
- Data analysis and reporting with awk
- Log file analysis and monitoring
- Text transformation pipelines
Beyond Text Processing:
- 70+ Linux commands with detailed examples
- File operations and permissions
- Process management and monitoring
- Network administration and troubleshooting
- 3 bonus editor cheat sheets (Vim, Nano, Emacs)
Frequently Asked Questions
When should I use grep vs awk for searching?
Use grep for simple pattern matching - it's faster and simpler. Use awk when you need to process specific columns, perform calculations, or combine searching with data manipulation. grep is specialized for searching; awk is a full programming language.
What's the difference between sed and awk?
sed is for stream editing (find/replace, delete, insert). awk is for processing structured data (columns, calculations, reports). Use sed for simple text transformations, awk for column-based data processing. Both can be used together in pipelines.
How do I replace text in multiple files?
Use sed with find or a for loop: find . -name '*.txt' -exec sed -i.bak 's/old/new/g' {} \; or for file in *.txt; do sed -i.bak 's/old/new/g' "$file"; done. Always create backups (.bak) when modifying multiple files.
Why doesn't grep -r search my text files?
grep -r skips binary files by default. If files are detected as binary (perhaps due to encoding), use grep -a to force text mode, or use grep -rI to actually skip binary files and show warnings. Check file encoding with file command.
How can I count lines/words/characters in a file?
Use wc (word count): wc -l counts lines, wc -w counts words, wc -c counts characters. Example: wc -l file.txt shows number of lines. Combine with other commands: grep ERROR log.txt | wc -l counts error lines.
What's the best way to learn regex for grep and sed?
Start with basic patterns: . (any char), * (zero or more), ^ (start), $ (end). Practice with simple examples. Use regex101.com to test patterns. Learn incrementally - master basic patterns before extended regex. grep -E makes regex easier with +, ?, |.
Continue Your Linux Journey
Ready to Master Text Processing?
Join thousands who've learned to efficiently search, transform, and analyze text with The Practical Linux Handbook.