| layout | post |
|---|---|
| title | GNU Coreutils |
| subtitle | The Standard Command Line Toolbox |
| author | Peter Hill |
| classoption | aspectratio=169 |
| theme | York169dark |
| filter | columnfilter.py |
| tags | terminal shell basics |
If you do anything at all on the command line in either Linux or OS X, at some point or another you've probably had to do something tricky and thought, "I wish there was some tool that just did this", and then discovered to your delight there was a tool that did exactly what you needed! That tool was probably part of coreutils, a set of standard tools available on all UNIX-like systems -- Linux, OS X, BSD, Solaris, TruOS, even Android! This week I read through the entirety of the coreutils manual, and picked out the most useful programs and options and summarised them for your pleasure.
You can find the original slides for this talk here and a plainer "cheatsheet" style pdf here.
- "Standard" set of (mostly) POSIX-compatible tools
- Portable Operating System Interface
- Family of standards for compatibility between UNIX-like OSes (Linux, OS X, BSD)
- Combination of tools for operating on/with files, shells and text
- Has extensions to the POSIX-standard
- 103 separate programs
- Many are used all the time, some have fallen by the wayside
Peter H. Salus in A Quarter-Century of Unix (1994):
- Write programs that do one thing and do it well.
- Write programs to work together.
- Write programs to handle text streams, because that is a universal interface.
Use pipes to connect the output from one program directly into the input of the next, and so build up complex pipelines out of the simple building blocks.
Summarise your top ten most used functions:
sed 's/|/\n/g' ~/.bash_history | awk '{CMD[$1]++;count++;} END \
{ for (a in CMD) print CMD[a] " " CMD[a]/count*100 "% " a;}' \
| grep -v "./" | column -c3 -s " " -t | sort -nr | nl | head -n10info coreutils: coreutils manual?: Key bindings
info <command>: manual for<command>man <command>: manual for<command>- The
infopages tend to be a better for learning about a command, whereas themanpages tend to be better for refreshing your memory
- The
--help: Print a usage message- Even terser than the manpage!
--version: Print the version number- Useful for verifying you have the correct thing installed in scripts
--verbose: Print exactly what operation is being performed (less common)--: Delimit the option list- Useful if you need to operate on files beginning with a dash
-: Read from standard in/write to standard out
- Majority of tools use an exit status of 0 for "success" and non-zero for failure
- Concatenate files together and print them to standard output
- Also useful to just dump contents of a file to screen
- Concatenate files together, reversing them separately, and print them to standard output
- Print the first
n(default: 10) lines of each FILE --lines K: Print first K lines--lines -K: Print all but the last K lines
- Print the last
n(default: 10) lines of each FILE --lines K: Print last K lines--lines +K: Print all but the first K lines--follow: Keep trying to read characters from files- Useful for watching log file(s)
- Count the number of bytes, characters, words and lines in each FILE
--lines: Print only the number of lines
- Computes the MD5 digest of each FILE
- Useful for verifying downloads haven't been corrupted
- Sorts files alphabetically
--numeric-sort: Sort numerically--human-numeric-sort: Sort numerically, but understand SI suffixes (M, G, etc.)--key=POS1[,POS2]: Sort between fields POS1 and POS2 (inclusive) on each line- A field is some text surrounded by whitespace
--debug: Annotate the part of the line used to sort--reverse: Reverse the ordering
- Discard all but the first of adjacent repeated lines
--count: Print the number of times each line occurred along with the line
- Coreutils has the following, but see
awkfor a more powerful tool
- Write to stdout selected parts of each line of each FILE
- Writes all the first lines of each FILE, followed by all the second lines, etc.
- Joins each pair of lines from FILE1 and FILE2 that have identical join fields
- Translate SET1 into SET2
- Input from stdin, output to stdout (i.e. you probably want this in a pipe)
- e.g. Convert output of
some-commandfrom lowercase to uppercase:some-command | tr a-z A-Z
--delete: delete SET1 from input- Useful to remove extra tabs, etc.
- List directory contents
--almost-all: Show files beginning with., except.and..--human-readable: Print human readable sizes (in powers of 1024)--format=long: Print as list, showing file information--sort=size: Sort by file size--sort=time: Sort by modification time--reverse: Reverse sort--color: Use colours to distinguish file types--classify: Append character to indicate file types
alias ls='ls -hF --color' # add colors for filetype recognition
alias la='ls -Alh' # show hidden files
alias lk='ls -lSrh' # sort by size, biggest last
alias lt='ls -ltrh' # sort by date, most recent last
alias lr='ls -lhR' # recursive ls- Output a sequence of shell commands to set up the terminal for
colour output from
ls - Run like:
eval "$(dircolors)"
- Copy, move/rename, delete files
cp/mv [OPTION]... SOURCE... DEST: DEST may be a directoryrm [OPTION]... [FILE]...- Share most of the same options:
--force: Don't prompt before overwriting/deleting existing files--interactive: Prompt before overwriting existing files--no-clobber: Don't overwrite existing files (notrm)--recursive: Copy/delete directories recursively (notmv)
- For complicated copies/backups, look at
rsync - Shell
{}syntax is useful here:cp file.txt{,.bak}: Same ascp file.txt file.txt.bak
- Make/remove directories
--parents: Make/remove parent directoriesmkdir -p a/b/cwill make./a,./a/band./a/b/cif they don't already existrmdir -p a/b/cwill remove./a,./a/band./a/b/cif they are all empty
- Make a link from TARGET to LINKNAME
--force: Replace existing destination files without asking--symbolic: Make symbolic links -- generally the kind you want- Hard links (the default) are another name for the same physical file on disk. They have to be on the same filesystem. A file is only deleted once all hard links are deleted
- Symbolic links are just links to a filename. If you delete the original, the symbolic link is "dangling", not pointing to anything. Symbolic links can be across filesystems
- Print the value of the given symbolic links
- Change the access permissions of the named files
--recursive: Recursively change permissions of directories and their contents- Permissions on Linux:
- Read/Write/eXecute for
- User who owns the file/Group who owns the file/Other users
- MODE should look like USERS OPERATION PERMISSIONS:
chmod u+x ./my_script: Give User eXecute permissionchmod go-w important_file: Remove write permission for everyone but userchmod go= secret_file: Remove all permissions for everyone else
- Change the access/modification times of FILEs
- Default is change the times to "now"
--date=TIME: Use TIME instead, e.g.:--date="2017-01-02 13:00"--date="yesterday"--date="2004-02-27 14:19:13.489392193 +0530"
- Date formats are complicated because human times are way, way, way more complicated than you think
- Report the amount of disk space used and available on file systems
--human-readable: Print with sensible suffixes (powers of 1024)--portability: Among other things, print each file system on a single line--print-type: Print each file system's type- Useful to see network disks, tmpfs, etc.
- Report the amount of disk space used by the specified files and for each subdirectory
--total: Print a grand total--max-depth=DEPTH: Show the total for each directory that is at most DEPTH levels down from the root of the hierarchy. The root is at level 0, sodu --max-depth=0is equivalent todu -s--human-readable: Print with sensible suffixes (powers of 1024)--summarise: Only display the total
- Write each STRING to stdout
- Useful for basic debugging of scripts
- Print formatted text
- FORMAT is mandatory, and is mostly the same as the C
printf()function - Useful for more complicated printing
false: Do nothing, unsuccessfullytrue: Do nothing, successfully- Useful as placeholders in scripts
- Return a status of 0 (true) or 1 (false) depending on EXPRESSION
- Lots of different tests:
- File type (is it a normal file, a directory, a symlink?)
- Access permission (can I read/write/execute this file?)
- File characteristics (does it exist? Is it newer than another file?)
- String tests (is the length of this string zero? Is it the same as another string?)
- Numeric tests (is this number larger/smaller/equal to another?)
- Copy stdin to stdout and also FILEs
--append: Append to the files rather than overwriting them- Very useful for creating logs for a command whilst still seeing the
results on screen
some_command | tee command.log
basename: Remove all the leading directory componentsdirname: Print all the leading directory components- Useful for cleaning up paths got from other programs
$ dirname /usr/bin/sort
/usr/bin
$ basename /usr/bin/sort
sort- Expand all symbolic links and resolves references to "/./", "/../" and extra "/" characters
- Safely create a temporary file or directory based on TEMPLATE, and print its name. TEMPLATE must include at least three consecutive "X"s in the last component (default "tmp.XXXXXXXXXX")
--directory: Create a directory rather than a file--tmpdir[=DIR]: Treat TEMPLATE relative to the directory DIR (defaults to "/tmp")
- Print the current (working) directory
- Print information about USER (default: you)
- Useful to find out what groups you belong to, and the numeric user/group IDs
- Print the names of the groups USER is in
- Print information about currently logged in users
- Useful to see if a system is busy
- Get the current time and date in FORMAT
- Useful in scripts to make uniquely named files
--date=DATESTR: Get the time and date from DATESTR- Can be a string like "1 month ago" or "+1 year"
- Time specifiers: %[HIklMNpPrRsSTXzZ]
- Date specifiers: %[aAbBcCdDeFgGhjmuUVwWxyY]
- Can be used to convert between different periods of time
- Can be used to convert to "seconds since UNIX epoch", 1970-01-01 00:00:00 UTC, useful for sorting dated data
- Print information about the machine and OS
--all: Print all the available information- Very useful when you diagnosing computer problems
- Gives info on: hardware platform, machine name, processor type, OS, kernel
- When you need the exact name/version of the OS, a portable method
that works almost everywhere is
cat /etc/*release
- Print the current time, the system's uptime, the number of logged-in users and the current load average
- Useful to quickly gauge how busy a machine is
- Run COMMAND with modified niceness
- The niceness affects how favourably the command is scheduled in the system
- Values range from -20 (highest priority) to 19 (lowest priority)
- With no options, runs COMMAND with nice of 10
- You generally need sudo to set a negative nice value
- Run given COMMAND with hangup signals ignored, so that the command can continue running in the background after you log out
- You still need to background the process by ending the line with "&"
nohup long_running_command > file.log &
- Pause for an amount of time
[smhd]: seconds, minutes, hours, days
- Print the numbers from FIRST to LAST by INCREMENT
--separator=STRING: Set the separator (default: newline)--equal-width: Print all numbers with same width by padding with leading zeros- Useful for generating lots of sequential names!
- Search the directory tree from STARTING-POINT for EXPRESSION
- Around 100 expressions... Most useful is the basic:
find . -type f -name "*.[ch]xx"- "Starting here, find normal files that end in either 'cxx' or 'hxx'"
- Multiple expressions can be chained
- Expressions are evaluated left-to-right with implied "and"
- The expressions can be tests or actions
-name,-ctime,-ls,-delete,-exec
- Build and execute command lines from stdin
- Useful in conjunction with
find - Reads from stdin and passes as arguments to COMMAND
find . -type f -name "*.[ch]xx" | xargs grep "variable"
- Find files by name
- Reads a local database of file names
- Very handy for tracking down libraries or headers
--basename: Match the pattern only against the last component of the filename
- I relied heavily on the GNU Coreutils documentation to write this
- Copyright (C) 1994-2014 Free Software Foundation, Inc.