Useful Linux Terminal Commands

One of the things that I really like about Linux is how much you can do from the terminal, which is much more powerful and flexible than the DOS/Windows command line. I keep notes on commands that I run frequently so that I don't have to recreate them from scratch.

These all work in Ubuntu Linux 8.10, 9.04, 9.10, 10.04, 10.10, 11.04, 11.10; Linux Mint Julia/Katya/Rebecca/Rafaela/Rosa; Fedora 14; and/or CrunchBang Waldorf. In some cases, the packages are not provided by default with the distribution, so sudo apt-get install <command name> (Ubuntu/Mint/CrunchBang) or su -c 'yum install <command name>' (Fedora) may be necessary.

Bash Usage

Template for building "for" commands

for file in "ls -1"; do $(echo "$file"); done

(as-is, this does exactly the same thing as ls -1 — note that the -1 is a dash-one, not dash-L, and serves to list exactly one file per output line — but can be used to build more complex commands.)

To execute a command 5 seconds from now

sleep 5; echo "hello"

To search bash history for a particular string

history | grep exiv

To grep through the contents of ZIP-compressed files, and output the same of the archive in which the match occurs

find . -name \*.ods -print | while read f; do unzip -p "$f" | fgrep -q something && echo $f; done

Source

to create aliases for commands using common options

alias rm="rm -i"

(Note that this can be made permanent by putting it in ~/.bashrc)

To run the same command a specified number of times

for i in $(seq 1 20); do pcopy -u2 /dev/zero /dev/sdd; done

(the above command runs pcopy -u2 /dev/zero /dev/sdd twenty times. I'm zeroing out an old hard drive repeatedly before throwing it away.)

To execute a command at a certain time

echo "ls -l" | at midnight

To get the return (error) code from the last command

echo $?

Source

To put a process in the background (from a terminal), but keep it running

Ctrl-Z, then use the

bg

command.

Source

To find symbolic links in & below the current directory and replace them with the targets of those links

find . -type l -exec bash -c "echo 'Replacing {} ...';  cp -LR '{}' '{}'.dereferenced;  rm '{}';  mv '{}'.dereferenced '{}'" \;

Source

System Management

to show all processes

ps ax

To find out which package installed a file

dlocate /usr/bin/eps2eps

or

dpkg -S usr/bin/eps2eps

(note that the first is much faster, but the dpkg program is not installed by default under Ubuntu 8.10/9.04.)

To find the UUID (Universally Unique Identifier) for a volume

vol_id --uuid /dev/sdd1

(useful when editing the /etc/fstab file, and much more convenient than using udevinfo.) (Using gparted is almost as easy, though not as convenient from the command line)

Alternately

blkid -o value -s UUID /dev/sda1

To find a bunch of information about a file, including its specific last access time

stat /path/to/file

To install the packages that I like to have installed from the beginning, including making multimedia work well (on Ubuntu/Mint)

sudo apt-get -y install alsa-oss faac faad adobe-flashplugin gstreamer0.10-ffmpeg gstreamer0.10-plugins-bad gstreamer0.10-plugins-ugly gstreamer0.10-pitfdll libmp3lame0 non-free-codecs amarok tilda unrar gnome-mplayer audacity amarok zsnes gfceu flobopuyo libreoffice firefox thunderbird mate-desktop-environment mate-core wesnoth-all vlc audacity kid3 ffmpeg mencoder mplayer devede dia scribus xsane alien atomicparsley epiphany-browser exiftran gftp-gtk gparted ghostscript ghostscript-doc hfsplus hfsprogs hfsutils htop joystick lame lastfm dlocate macutils make mp3gain mp3splt mp3wrap mp3check build-essential joystick lame lastfm dlocate macutils make mp3gain mp3splt mp3wrap mp3check vorbisgain vorbis-tools w32codecs wine xsane-common tilda libgimp2.0-dev gimp=2.6.12-1ubuntu1 bluefish skype hugin qtpfsgui pcmanfm openbox obmenu obconf openbox-themes tidy autopano-sift tidy-doc transmission parcellite exiv2 && sudo apt-get update && sudo apt-get -y upgrade && sudo apt-get -y autoremove && sudo apt-get -y clean

Similar command under Crunchbang:

sudo apt-get install iceweasel konsole tilda libreoffice bluefish zsnes epiphany default-jdk openjdk-7-jdk tidy weblint icoutils vlc gcj-jre-headless hspell pulseaudio-esound-compat oss-compat myspell-en-us myspell-de-de libreoffice-help-en-us cups-bsd mythes-en-us libreoffice-lightproof-en libreoffice-gnome libreoffice-kde openclipart-libreoffice pstoedit imagemagick gstreamer0.10-plugins-ugly libreoffice-officebean libreoffice-style-hicontrast libmysql-java libreoffice-style-tango libreoffice-style-crystal libreoffice-style-oxygen icedtea-plugin phonon-backend-gstreamer openjdk-7-jre faad lame sox kid3 mencoder genisoimage cdrdao dlocate exiftran ffmpeg pdftk alsa-oss alsa-oss gstreamer0.10-ffmpeg gstreamer0.10-plugins-bad gstreamer0.10-plugins-ugly libmp3lame0 amarok unrar gnome-mplayer audacity amarok flobopuyo kid3 mplayer devede dia scribus xsane alien atomicparsley epiphany-browser exiftran gftp-gtk gparted ghostscript ghostscript-doc hfsplus hfsprogs hfsutils htop joystick lastfm macutils make mp3gain mp3splt mp3wrap mp3check build-essential lame dlocate macutils make vorbisgain vorbis-tools wine xsane-common libgimp2.0-dev gimp hugin qtpfsgui pcmanfm openbox obmenu obconf openbox-themes  tidy-doc transmission parcellite exiv2 amarok-doc moodbar cups  wodim cdrkit-doc ghostscript-cups ghostscript-x hpijs gimp-help-en gimp-data-extras imagemagick-doc autotrace html2ps finger oxygencursors tango-icon-theme libwine-sane libwine-print media-player-info libsox-fmt-all mplayer-doc mpg321 sun-java6-fonts icedtea-7-plugin phonon-backend-vlc scribus ufraw gocr; sudo apt-get -y autoremove; sudo apt-get -y clean; sudo apt-get update; sudo apt-get -y upgrade

Assumes already running on a root terminal, i.e. sudo -i has been executed. Also assumes that the MATE repository has already been made available. This can be done with

sudo add-apt-repository "deb http://packages.mate-desktop.org/repo/ubuntu precise main"

(Obviously, this is based on my personal preferences, but I think it makes for a highly usable system.)

To install the packages that I like to have installed from the beginning, including making multimedia work well (on Ubuntu/Mint)

sudo apt-get -y install alsa-oss faac faad adobe-flashplugin gstreamer0.10-ffmpeg gstreamer0.10-plugins-bad gstreamer0.10-plugins-ugly gstreamer0.10-pitfdll libmp3lame0 non-free-codecs amarok tilda unrar gnome-mplayer audacity amarok zsnes gfceu flobopuyo libreoffice firefox thunderbird mate-desktop-environment mate-core wesnoth-all vlc audacity kid3 ffmpeg mencoder mplayer devede dia scribus xsane alien atomicparsley epiphany-browser exiftran gftp-gtk gparted ghostscript ghostscript-doc hfsplus hfsprogs hfsutils htop joystick lame lastfm dlocate macutils make mp3gain mp3splt mp3wrap mp3check build-essential joystick lame lastfm dlocate macutils make mp3gain mp3splt mp3wrap mp3check vorbisgain vorbis-tools w32codecs wine xsane-common tilda libgimp2.0-dev gimp=2.6.12-1ubuntu1 bluefish skype hugin qtpfsgui pcmanfm openbox obmenu obconf openbox-themes tidy autopano-sift tidy-doc transmission parcellite exiv2 && sudo apt-get update && sudo apt-get -y upgrade && sudo apt-get -y autoremove && sudo apt-get -y clean

Similar command under Crunchbang

sudo apt-get install iceweasel konsole tilda libreoffice bluefish zsnes epiphany default-jdk openjdk-7-jdk tidy weblint icoutils vlc gcj-jre-headless hspell pulseaudio-esound-compat oss-compat myspell-en-us myspell-de-de libreoffice-help-en-us cups-bsd mythes-en-us libreoffice-lightproof-en libreoffice-gnome libreoffice-kde openclipart-libreoffice pstoedit imagemagick gstreamer0.10-plugins-ugly libreoffice-officebean libreoffice-style-hicontrast libmysql-java libreoffice-style-tango libreoffice-style-crystal libreoffice-style-oxygen icedtea-plugin phonon-backend-gstreamer openjdk-7-jre faad lame sox kid3 mencoder genisoimage cdrdao dlocate exiftran ffmpeg pdftk alsa-oss alsa-oss gstreamer0.10-ffmpeg gstreamer0.10-plugins-bad gstreamer0.10-plugins-ugly libmp3lame0 amarok unrar gnome-mplayer audacity amarok flobopuyo kid3 mplayer devede dia scribus xsane alien atomicparsley epiphany-browser exiftran gftp-gtk gparted ghostscript ghostscript-doc hfsplus hfsprogs hfsutils htop joystick lastfm macutils make mp3gain mp3splt mp3wrap mp3check build-essential lame dlocate macutils make vorbisgain vorbis-tools wine xsane-common libgimp2.0-dev gimp hugin qtpfsgui pcmanfm openbox obmenu obconf openbox-themes  tidy-doc transmission parcellite exiv2 amarok-doc moodbar cups  wodim cdrkit-doc ghostscript-cups ghostscript-x hpijs gimp-help-en gimp-data-extras imagemagick-doc autotrace html2ps finger oxygencursors tango-icon-theme libwine-sane libwine-print media-player-info libsox-fmt-all mplayer-doc mpg321 sun-java6-fonts icedtea-7-plugin phonon-backend-vlc scribus ufraw gocr; sudo apt-get -y autoremove; sudo apt-get -y clean; sudo apt-get update; sudo apt-get -y upgrade

Assumes already running on a root terminal, i.e. "sudo -i" has been executed. Also assumes that the MATE repository has already been made available. This can be done with

sudo add-apt-repository "deb http://packages.mate-desktop.org/repo/ubuntu precise main"

(Obviously, this is based on my personal preferences, but I think it makes for a highly usable system.)

To determine CPU info

cat /proc/cpuinfo

To recover data from a deleted file

grep -a -B 25 -A 100 'some string in the file' /dev/sda1 > results.txt

Source

To delete all "Picasa.ini" files in current directory and its subdirectories

find -name "Picasa.ini" -type f -print0 | xargs -0 /bin/rm -f

To run a non-exhaustive disk check on an ext2 or ext3 volume

e2fsck -f -p -v -D /dev/sdc1

To prevent packages from being upgraded

sudo aptitude hold gqview amarok

To force enabling of Xinerama

sudo nvidia-xconfig --xinerama --separate-x-screens --no-twinview

Source

Filesystem info/searching commands

To find out how much space the home directory occupies

du -hs ~

To find what files were changed today in the current directory

ls -lrt | grep 'May 22'

Same, but just display file name and modification date

ls -lrt | awk '{print $6" "$7" "$9 }' | grep 'May 22'

Or all files from a past year

ls -lR | grep 2006

To find what changed in the current directory between three and six minutes ago

find . -mmin +3 -mmin -6

To find the top five largest files in the named directory or subdirectory

du -a directoryname | sort -nr | head -n 5

To find the five biggest files on the system, and sort them from largest to smallest

find / -type f -printf '%s %p\n' |sort -nr| head -5

Same, but excluding the /proc filesystem

find / -wholename '/proc' -prune -o -type f -printf '%s %p\n' |sort -nr| head -5

Same, but limited to the current filesystem

find / -xdev -wholename '/proc' -prune -o -type f -printf '%s %p\n' |sort -nr| head -5

Source for these commands

To restore an installation from a compressed backup disk image

gunzip -c installed-mint.img.gz | sudo dd of=/dev/sda5

IMPORTANT NOTE: Doing this thoughtlessly is incredibly stupid and may completely bork an installed system or otherwise destroy data. Make absolutely sure you know what you're doing; read on for more information. It is OF COURSE necessary to be absolutely sure that you're writing to the correct partition, which can be easily checked by running gparted, parted, fdisk, or, if the file system is currently mounted, mount (though the system should be unmounted before you proceed).

RATIONALE AND BACKGROUND: After freshly installing an OS onto my computer, I create a (compressed) image of that installation; rewriting this compressed image onto the device is then faster and easier than reinstalling the OS from scratch, should this ever need to be done.

The compressed image can be created with something like

sudo dd if=/dev/sda5 | gzip - > ~/Desktop/installed-mint.img.gz

I then burn the file onto a data DVD or keep it on an external drive.

Before creating the image, it's useful to zero unused space on the drive; this vastly improves the compression of the disk. (Otherwise, the compression program winds up trying to compress whatever random junk is on the drive, and random junk doesn't compress well. This wastes a lot of space.) The empty space can be zeroed with

dd if=/dev/zero of=/mnt/mint/000.000

Let this run until the disk is full, when dd will quit and complain that there is no more space. Then

rm /mnt/mint/000.000

This ensures that all empty space on the device is overwritten with zeroes, a long string of which compresses down to take up almost no space.

Note that it is incredibly important to check that the device name is correct; this can be done with parted, gparted, fdisk, or perhaps mount (if the drive is currently mounted, though the filesystem being imaged should be unmounted before it is imaged). Note that the sudo before the dd is necessary to get full access to the device in /dev; it's possible to just do sudo -i, then exit, to ensure that the dd command doesn't ask for elevated privileges while running.

To list all filename extensions in a directory and its subdirectories, with the count of files for each extension, sorted most-to-least

find /some/path -type f -printf '%f\n' | grep -o '\..\+$' | sort | uniq -c | sort -rn

Source

To run an OSS-dependent program, redirecting sound to a PulseAudio server

padsp BasiliskII

Source

To launch a process with modified priority ("nice value")

nice -n 10 {some command}

Source

To find all children of a parent process

ps -o user,pid,ppid,command ax | grep {process id of parent process}

Source

To restore the screen to its default resolution after some Windows app running under WINE has crashed at a lower resolution

sudo xrandr --output LVDS1 --auto --rotate normal --pos 0x0

To add my personal public key to root's keyring

gpg --export 505AB18E --output > "/home/patrick/Desktop/keys.gpg"

then,

gksudo seahorse

... and import the key.

Networking

To find the IP address of my router relative to the local network

route -n

(It's the last line under "gateway").

To find the IP address of my computer relative to the local network

ifconfig

(it's under "inet addr")

To restart the nfs server

sudo /etc/init.d/nfs-kernel-server restart

To determine what shares are available on an nfs server

showmount -e {server name/server IP address}

To update Twitter status

 curl -u patrick_mooney -d status="♫♬♪ And some folks loves ham hocks/ And some folks loves po'k chops/ And some folks loves vegetable soup ... ♪♬♫" -d source="cURL" http://twitter.com/statuses/update.xml

— not anymore: Twitter now requires that clients use OAuth.

To restart the Samba server

sudo restart smbd
sudo restart nmbd

Source

To back up my Tumblr site

wget -mpNHk -D .media.tumblr.com,patrickbrianmooney.tumblr.com -R "*avatar*","*\?*","*_[0-9][0-9][0-9].*" http://patrickbrianmooney.tumblr.com -o wgetLog.txt

Source

To sync the files on my remote web server with files in the local directory

rsync -avv --progress --log-file-format="" --log-file="/home/patrick/Desktop/nfs.log" --exclude "*jpg" --exclude "*png" --exclude "*pdf" --exclude "*odt" --exclude "*doc" --exclude "*docx" --exclude "*ppt" --exclude "*pptx" /~patrick/* patrickbrianmooney_patrickbrianmooney@ssh.phx.nearlyfreespeech.net:/home/public/~patrick/

To look up the IP address for a domain

host patrickbrianmooney.nfshost.com

Source

To download an entire website

wget --wait=20 --limit-rate=20K -m -p -U Mozilla http://www.stupidsite.com/restricedplace.html

-m turns on "mirroring" options, including recursion and infinite recursion depth. If wget should convert links to other documents that were downloaded to links to the local files, add -k.

Source

Audio Processing

To transcode .flac to .mp3

for file in *.flac; do $(flac -cd "$file" | lame --replaygain-accurate -t --id3v2-only -V 2 -h - "${file%.flac}.mp3"); done

Source

To transcode .mp4 to .mp3

for file in *.m4a; do $(faad -o - "$file" | lame --tt "" --ta "David Bowie" --tl "ChangesBowie" --tg "Pop" --ty 1990 --pad-id3v2 --replaygain-accurate --id3v2-only -V 2 -h - "${file%.m4a}.mp3"); done

Source

To transcode Monkey's Audio (ugh) to FLAC (both lossless)

for i in *.ape; do mac "$i" - -d | flac -o "${i%*.ape}.flac" -; done

Source

(a Monkey's Audio .deb package suitable for Ubuntu can be found here)

To transcode .wma to .mp3

for i in *.wma ; do mplayer -vo null -vc dummy -af resample=44100 -ao pcm -waveheader "$i" && lame -m j -h --vbr-new -b 160 audiodump.wav -o "`basename "$i" .wma`.mp3"; done; rm -f audiodump.wav

Source — this seems kind of an ugly hack and could almost certainly be done better with a pipe.)

To encode .wavs to .mp3

for file in *.wav; do $(lame -h -V 2 "$file"); done

To transcode .wavs to .m4b audiobooks

for file in *.wav; do $(faac -q 80 --artist "Pimsleur" --title "French Short Course" --genre "Audiobook" --disc 1 -w "$file" "${file%.wav}.m4b"); done

to transcode (wrapped) .mp3s to iPod audiobook format

sox -V0 Heidegger_MP3WRAP.mp3 -t wav -c 1 -r 16000 - | faac -w -q 60 --artist "Hubert Dreyfus" --writer "Hubert Dreyfus" --title "Philosophy 185: Heidegger's Being and Time" --genre "College Course" --album "Lectures, Fall 2000" --compilation --track 1 --year 2000 --disc 1 --cover-art "dreyfus.jpg" -o Heidegger.m4b -

To descend down the directory tree from the current point, converting any .wav files to .mp3

find . -type f \! -iname "*mp3" -exec /usr/bin/lame -m j --replaygain-accurate -V 2 --add-id3v2 --id3v2-only '{}' \;

Rationale for this madness

A CD ripper I used to use (RipperX) used to fail to encode to .mp3 on files with really long paths (I presume that long means over 255 characters, but haven't tested this). Rather than worrying about it, I just leave this running when I go to bed after a day when I've been ripping CDs. Of course, the better option is to use a non-idiotic CD ripper.

It searches the directory tree from the current point for anything that's a file (not a directory or symlink) with a name that doesn't end in .mp3 (note the !, which is POSIX not, preceeded by a backslash to protect it from being interpreted by the shell) and runs lame over it, converting the .wav file to .mp3 (when RipperX fails on a file, it's almost certain that the file doesn't end in .wav, so searching for .wav produces few useful results). The first -exec ... argument is the command to be run, where {} is replaced with the name of the (non-mp3) file. It's quoted to protect the bash shell from interpreting it. The semicolon (backslashed to — again — protect it from expansion by the bash shell) ends the command.

To extract audio from a chapter of a DVD

transcode -i /dev/dvd -x dvd -T A,B,1 -a 0 -y raw -m track-a.mp3

where A is the number of the title to rip from, and B is the chapter number.

to find the title/chapter info

lsdvd

(no arguments!)

To dump audio from a video file created by my digicam

ffmpeg -i 100_4001.MOV -f wav -vn -acodec copy output.wav

(note that with my little Kodak, this creates m-law files, which are hard to postprocess. VLC can play them, though.)

or, somewhat more simply

ffmpeg -i 100_4001.MOV output.flac

To dump a segment of mp3 audio from a video file containing mp3 audio to a file

ffmpeg -i Dummy.2003.avi -acodec copy -vn -ss 71 -t 0:04:09 ~/Desktop/MikeRueckberg-Years.mp3

(This dumps the mp3 audio from 1:11 to 4:09 in the movie.)

To downsample mp3 files

for file in *.mp3; do $(lame --pad-id3v2 --id3v2-only -V 2 "$file" "${file%.mp3}_processed.mp3"; id3cp -1 "$file" "${file%.mp3}_processed.mp3"); done

(Note that this copies only the id3 version 1 tags ... id3cp seems to have a problem with version 2 tags in which the copied text shows up with an incorrect character encoding much of the time. I do realize that lossy recompression occurs under these circumstances, but am willing to put up with this in certain circumstances, such as when some idiot distributes mp3s created with --preset insane. [At this point, just use FLAC. Honestly])

To record audio from the microphone jack (e.g., being played over Internet on another computer and pulled in via out-to-mic-jack cable)

rec -V -S -b 16 -r 48k -c 2 /media/Patrick\!/Mahler3.flac

To transcode (HQ) sound files from my RCA voice recorder to .mp3

 sox "Lecture 05 - The Day the World Went Away (23 January 2013).alaw.wav" -b 16 -t wavpcm - | lame --replaygain-accurate -q 0 -V 8 -b 8 -B 40 --tt "The Day the World Went Away" --ta "Patrick Mooney, M.A." --tl "English 165EW, Winter 2013" --ty 2013 --tn "05/18" --id3v2-only --tg "College Lecture" - "as .mp3/Lecture 05 - The Day the World Went Away (23 January 2013).mp3"

Graphic Processing

To replace JPEG comments

wrjpgcom -replace -c "3 June 2004 - View of people from entrance to Trinity College campanile - Dublin, Ireland" 46.12\ -\ People\ walking.jpg > 46.12\ -\ People\ walking.jpg

(notice the use of output redirection to specify output filename)

To copy JPG comments from all jpeg files in a directory into a single text file

for i in *jpg; do echo $i >> all-comments.txt; rdjpgcom "$i" >> all-comments.txt; echo "--------" >> all-comments.txt; echo >> all-comments.txt; done

To create the above-style all-comments.txt files in each subdirectory below the current working directory

for j in ??\ -\ Roll*; do cd "$j"; for i in *jpg; do echo $i >> all-comments.txt; rdjpgcom "$i" >> all-comments.txt; echo "--------" >> all-comments.txt; echo >> all-comments.txt; done; cd ..; done

(note that the pattern in the first for statement is specifically chosen to match the directory structure in which I last used this command)

The EXIFtool documentation recommends this to dump all tags in a photo, which may be a useful alternate method of accomplishing the same task

exiftool -r -w .txt -common pictures

To convert a video to an animated GIF

mplayer video.nut -vo gif89a:fps=15:output=test.gif

(order of suboptions is, apparently, important)

To construct an animated GIF from a series of photos

First, convert the photos to the appropriate size, e.g. 450 pixels wide

for file in *JPG; do $(convert "$file" -resize 450 "${file%JPG}png"); done

Source

Then, assemble the photos into an animated GIF

convert -delay 20 -loop 0 *JPG result.gif

Source

To dump frames from a video

ffmpeg -i out.avi -r 25 -f image2 image%4d.png

(-r is frame rate; set it correctly.) Source

To crop all image files in a directory

mogrify -crop 640x520+4+21 *png

Source

To copy EXIF tags from one JPEG file to another

exiftool -tagsfromfile 100_0702.JPG Circles\ of\ Stones.jpg

(I do this to embed an EXIF tag from one of the original images that gets stitched into a panorama.)

To OCR all graphic files in a directory

for file in *png; do $(ocroscript rec-tess "$file" > "${file%png}html"); done

To convert all Canon RAW images in a folder to 16-bit TIFF

for file in *CR2; do $(dcraw -w -6 -T "$file"; exiftool -tagsfromfile "${file%CR2}JPG" "${file%CR2}tiff"); done

To rename all photos in a folder according to the timestamp in their EXIF data

exiv2 mv -r "%Y-%m-%d_%H:%M:%S" *

Source

To subtract two hours from a photo's EXIF date

exiftool "-alldates-=2:00:00" "-FileModifyDate-=2:00:00" -overwrite_original *

Source

To downsize and convert all JPG files in a directory to .tiff

for file in *JPG; do $(convert "$file" -resize 700x1000 "${file%JPG}.png"); done

Dealing with ISO Images

Creating from a DVD

dd if=/dev/dvd of=dvd.iso

Creating from a (standard) CD-ROM

dd if=/dev/cdrom of=cd.iso

Creating from a PlayStation (1) CD

cdrdao read-cd --read-raw --datafile ~/ClockTower.bin --device /dev/cdrom --driver generic-mmc-raw ~/ClockTower.toc
    ... but note that this is really best done through PCSX's GUI

It seems not to be possible to create images of audio CDs, probably because the audio CD format is so different from data-based optical discs. (Please correct me if I'm wrong.) — Well, it's not possible to create an .iso. But there are other possibilities, like BIN/CUE, or .wav/.flac+CUE, etc.

Creating from a directory

mkisofs -o /tmp/cd.iso /tmp/directory/

Creating from a directory, and the CD should be readable & as useful as possible in a variety of systems

genisoimage -hide-joliet-trans-tbl -hide-rr-moved -iso-level 3 -J -joliet-long -nobak -r -T -hfs -probe -map /usr/bin/genisoimage-default-mappings.hfs -hfs-creator GKON -hfs-type "????" -o /mnt/Externa/EuroPhotos.iso /mnt/Externa/Photos/pictures from Europe/

Creating from an entire hard disk

dd if=/dev/sde of=~/macHD.dd

(note that this provides an image of the entire disk, not of a particular partition ... which is desirable, for instance, when recovering data from old Mac hard drives, because HFS+ volumes actually include a dummy HFS partition.)

To create a blank disk image for an emulator

dd if=/dev/zero of=~/applications.hfv count=4194304

(count is in blocks of 512 bytes. The above command creates a 2GB blank (zeroed) disk image. It can then be formatted by the emulator.)

Video Processing

To rotate a video clockwise

mencoder 100_1969.mov -o 100_1969-rotated.avi -vf rotate=1 -oac mp3lame -ovc lavc

NOTE: This recompresses video and audio, with all that implies; mencoder doesn't really write Quicktime video ...

Alternately, commandlinefu.com recommends

    mencoder -vf rotate=1 -ovc lavc -oac copy "$1" -o "$1"-rot.avi

To trim a video so that only seconds 56 through 114 are copied, using motion JPEG compression

mencoder 100_1999.mov -o 100_1999-trimmed.avi -ss 61 -endpos 114 -oac mp3lame -ovc lavc -lavcopts vcodec=mjpeg:mbd=1

NOTE: mjpeg compression really isn't that efficient ...

To rotate a video and mute the volume completely

mencoder 100_0778.MOV -o 100_0778-rotated.avi -vf rotate=1 -oac mp3lame -ovc lavc -af resample=8000:0:0,channels=1,volume=-200

(decreases the volume to the minimum possible volume and downsamples it to the lowest allowable samplerate & number of channels.)

Alternately, to remove sound from a video completely, commandlinefu.com recommends

mencoder -ovc copy -nosound ./movie.mov -o ./movie_mute.mov

Resizing a video

mencoder 100_2168.MOV -o 100_2168-shrunk.avi -oac mp3lame -ovc lavc -vf scale=320:240

To concatenate video files

This is a bad way to do it

First, do a simple concatenation

cat VTS_01_1.VOB VTS_01_2.VOB VTS_01_3.VOB VTS_01_4.VOB VTS_01_5.VOB > FORBIDDEN_ZONE.VOB

Then, re-sync audio/video with mencoder

mencoder -forceidx -oac copy -ovc copy FORBIDDEN_ZONE.VOB -o FORBIDDEN_ZONE-final.VOB

Better

As a one-step, just use multiple input files as arguments to mencoder

mencoder -forceidx -oac copy -ovc copy -o "Wristcutters: A Love Story.avi" Wristcutters-\ A\ Love\ Story\ CD1.avi Wristcutters-\ A\ Love\ Story\ CD2.avi

Alternately, commandlinefu.com recommends this for .avi files

avimerge -o output.avi -i file1.avi file2.avi file3.avi

To dump title 1 from a DVD to a file onto a hard disk

from a physical DVD

mplayer dvd://1 -dumpstream -dumpfile /media/Externa/unloading-dock/DVD-rips/ForbiddenZone.vob

(useful if ripping software fails, but the disc is playable. It can then be postprocessed ...)

from an ISO file (if it can be copied to hard drive, but shrinking software fails)

(working on this ... Suggestions?)

the man page for mplayer has this option, which may be helpful:

-dvd-device <path to device> (DVD only)

Specify the DVD device or .iso filename (default: /dev/dvd). You can also specify a directory that contains files previously copied di‐ rectly from a DVD (with e.g. vobcopy).

Also, in this regard, see this example

Here's a sample that does this:

mencoder -forceidx -dvd-device "/home/patrick/Desktop/Adaptation.iso" dvd://1 -alang "en" -ovc lavc -oac mp3lame -sid 8190 -ss 1:06:25 -endpos 0:05:20 -forceidx -o Adaptation.avi

(which means: extract only the English track, only 5:20 of it, re-encoding, from an .ISO, and avoid hard-coding subtitles by specifying non-existent subtitle track ID# 8190)

To convert a file from .3g2 to .avi

ffmpeg -threads 2 -i filefromyourphone.3gp -f avi -r 29.97 -vcodec libxvid -vtag XVID -s 1024x768 -aspect 4:3 -maxrate 1800kb -b 1500kb -qmin 3 -qmax 5 -bufsize 4096 -mbd 2 -bf 2 -flags +4mv -trellis -aic -cmp 2 -subcmp 2 -g 300 -acodec libmp3lame -ar 48000 -ab 128kb -ac 2 fileonpc.avi

Source

Converting a Matroska video into a DVD-ready MPEG file

mencoder -oac lavc -aid 0 -ovc lavc -of mpeg -mpegopts format=dvd:tsaf:telecine -ofps 24000/1001 -vf harddup -lavcopts threads=2:vcodec=mpeg2video:trell:mbd=2:sc_threshold=1000000000:cgop:vstrict=0:vrc_maxrate=7002:vrc_buf_size=1835:vbitrate=1200:aspect=80/33 -srate 48000 -o "Poltergeist.mpeg" "Poltergeist 1982 BRrip 720p x264 [Herakler].mkv"

(based on an invocation of mencoder invoked automatically by DeVeDe. DeVeDe seems to fail on my system when provided with a Matroska file as input, so I manually convert the file to a DVD-ready MPEG and tell DeVeDe to use it as-is.)

To extract subtitles from a Matroska video file

First, find the subtitle stream number

mkvmerge -i Shutter.Island.2010.720p.BluRay.x264.DTS-WiKi.mkv

Then, extract that stream to a subtitle file

mkvextract tracks Shutter.Island.2010.720p.BluRay.x264.DTS-WiKi.mkv 3:Shutter.Island.2010.720p.BluRay.x264.DTS-WiKi.srt

(in this example, the subtitles were in track 3.)

To remux a Matroska file, removing subtitles and attachments

    mkvmerge -S -M -o Crash.mkv Crash\ \(remove\ subtitles\).mkv

To convert a video file to a DVD image, which can then be burned with image burning tools

1. First, convert the video files into an appropriate DVD-ready format:

   export VIDEO_FORMAT=NTSC
   ffmpeg -i inputfile.mkv -target ntsc-dvd {"-aspect 16:9", if widescreen} DVD-output-filename.mpg

   Note: using dumped VOBs? It may help to start off by modifying the ffmpeg command more or less as follows

   ffmpeg -i "StepBrothers (DVD dump).vob" -target ntsc-dvd -map 0:0 -map 0:1 -acodec copy -vcodec copy "Stepbrothers (ready).mpeg

   Note the use of mappings here: in this case, the first and second streams (one audio, one video, because stream numbers are zero-based). Dumped VOBs often contain multiple audio streams, and ffmpeg doesn't always make good choices about which ones to include in the final product. Stream information can be found simply by executing ffmpeg -i [filename] and/or with VLC media player. Note that ffmpeg requires that if any streams are mapped, all streams to be included in the output file must be mapped explicitly.

   Using -acodec copy -vcodec copy prevents ffmpeg from recompressing audio and video. This is not only much faster, but prevents loss of quality.

2. Then, build the DVD image:

   mkdir Stepbrothers
   dvdauthor -o Stepbrothers/ -t "Stepbrothers (ready).mpeg"
   dvdauthor -o Stepbrothers/ -T # can't be combined with the first invocation of dvdauthor
   genisoimage -dvd-video -V STEPBROTHERS -o "Stepbrothers (ready).iso" Stepbrothers/

3. Then, burn to disc with:

   growisofs -dvd-compat -Z /dev/sr0=Stepbrothers\ \(ready\).iso

   Based on this article

Notes toward constructing an alternate method

The first-pass invocation of mencoder that DeVeDe runs is

mencoder -srate 48000 -af lavcresample=48000 -noautosub -oac lavc -aid 1 -ovc lavc -of mpeg -mpegopts format=dvd:tsaf -ofps 30000/1001 -vf scale=720:480,harddup -lavcopts vcodec=mpeg2video:trell:mbd=2:sc_threshold=1000000000:cgop:vstrict=0:vrc_maxrate=5050:vrc_buf_size=1835:vbitrate=2525:keyint=12:acodec=ac3:abitrate=224:aspect=4/3:vpass=1 -passlogfile /media/Backup/movies 2/Sybil (ready)/Sybil (ready).log -o /media/Backup/movies 2/Sybil (ready)/Sybil (ready)_01_01.mpg /media/Externa/movies/Sybil.divx

Second pass

mencoder -srate 48000 -af lavcresample=48000 -noautosub -oac lavc -aid 1 -ovc lavc -of mpeg -mpegopts format=dvd:tsaf -ofps 30000/1001 -vf scale=720:480,harddup -lavcopts vcodec=mpeg2video:trell:mbd=2:sc_threshold=1000000000:cgop:vstrict=0:vrc_maxrate=5050:vrc_buf_size=1835:vbitrate=2525:keyint=12:acodec=ac3:abitrate=224:aspect=4/3:vpass=2 -passlogfile /media/Backup/movies 2/Sybil (ready)/Sybil (ready).log -o /media/Backup/movies 2/Sybil (ready)/Sybil (ready)_01_01.mpg /media/Externa/movies/Sybil.divx

To mux audio and video streams from a screencast for a YouTube video

Ideally, the screencast video is .mp4. The original screencast might plausibly be created with something like

ffmpeg -f x11grab -r 12 -s 1280x800 -i :0.0 -qscale 0.1 ~/Desktop/web.mp4

or

avconv -f x11grab -r 15 -s 1366x768 -i :0.0 ~/Desktop/out.mp4

First, trim the audio and/or video so they're synchronized. Then, mux them with mencoder

mencoder -forceidx "Machines Read, Too (just not well).mp4" -forceidx -audiofile "machines.flac" -oac mp3lame -ovc copy -forceidx -o "Machines Read, Too (just not well) (with audio).mp4"

Then, trim any front matter from the video and put a "fast start" MOOV atom at the beginning of the video for YouTube

ffmpeg -ss 74 -i "Machines Read, Too (just not well) (with audio).mp4" -acodec copy -vcodec copy -movflags +faststart "Machines Read, Too (just not well) (final).mp4"

(note that doing this requires a version of ffmpeg more recent than that in the Crunchbang/Debian repos; this can be found in the repo here.)

To pre-process all video files in a folder so that they are ready for concatenation

for file in *; do $(mencoder -srate 22050 -af lavcresample=22050 -oac mp3lame -ovc lavc -ofps 30000/1001 -vf scale -zoom -xy 720 -lavcopts vcodec=mpeg4:trell:mbd=2:keyint=75:vbitrate=1001:autoaspect -lameopts vbr=4:q=2:mode=1 -o "${file%flv}-processed.avi" "$file"); done

(Of course, parameters can be adjusted as needed.)

To perform a screen capture using ffmpeg

ffmpeg -f x11grab -r 12 -s 1280x800 -i :0.0 ~/Desktop/out.avi

Even better (Source)

ffmpeg -f x11grab -r 25 -s 1280x800 -i :0.0 -qscale 0.1 ~/Desktop/out.mp4

Using avconv

avconv -f x11grab -r 15 -s 1366x768 -i :0.0 -qscale 0.1 ~/Desktop/out.mp4

Capture audio, too

avconv -f alsa -i pulse -f x11grab -r 15 -s 1366x768 -i :0.0 -vcodec libx264 -acodec ac3 -threads 2 ~/Desktop/test.mp4

Document Processing

To tidy and validate an HTML document (making declarations explicit, outputting as XHTML, removing stupid Microsoft coding, pretty-printing, without auto-wrap) ...

for a file for my website (XHTML)

tidy -m -i -w 0 -asxhtml -utf8 --add-xml-decl true --bare true --clean true --doctype strict --drop-empty-paras true --drop-proprietary-attributes true --enclose-block-text true --logical-emphasis true --quote-marks true --replace-color true --word-2000 true  --preserve-entities yes --tab-size 3 test.html

for a file on my website (HTML5)

tidy -m -i -w 0 -utf8 --bare true --clean true --drop-empty-paras true --drop-proprietary-attributes true --enclose-block-text true --logical-emphasis true --quote-marks true --replace-color true --word-2000 true  --preserve-entities yes --tab-size 3 test.html

to clean up Thunderbird's HTML

tidy -m -i -w 0 -utf8 --doctype omit --drop-empty-paras true --enclose-block-text true --replace-color true --word-2000 true  --preserve-entities yes --tab-size 3 temp.html

To rotate all pages in a PDF file 90 degrees clockwise

pdftk CriticalLegalHistories-1.pdf cat 1-endE output CriticalLegalHistories-rotated.pdf

(to rotate counterclockwise instead, change cat 1-endE to cat 1-endW; to rotate 180 degrees, it should be cat 1-endS.)

To concatenate PDF files

pdftk 01.pdf 02.pdf 03.pdf 04.pdf 05.pdf 06.pdf 07.pdf 08.pdf 09.pdf 10.pdf cat output Richard.pdf

To excerpt pages from a PDF file

pdftk "Selections From Early American Writers.pdf" cat 5-7 182-193 output wigglesworth.pdf

To reverse the order of pages in a PDF file

pdftk cummings\ backwards.pdf cat end-1 output "cummings.pdf"

To create a version 1.3 PDF (Acrobat 4.x) from a later version

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.3 -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Source

OR

pdf2ps file.pdf - | ps2pdf13 - file2.pdf

To convert .pdf documents to really janky HTML

pdftohtml -s -noframes -enc UTF-8 -hidden -nomerge "Student Name.pdf" "student-name.html"

To extract attachments from all Thunderbird .eml files in a directory

munpack *eml

To convert an ebook to a Kindle-compatible format

ebook-convert

To spell-check a document for OCR-specific problems

gutcheck

(program must be downloaded from SourceForge)

To dump a man page as pdf

man -t awk | ps2pdf - awk.pdf

Source

Text Processing

To open all html files that comprise my personal website in Bluefish

find /~patrick -iname "*html" -type f -print0 | xargs -0 /usr/bin/bluefish

To find all files in a directory that contain a specific string and open them in Bluefish

grep -ilR --null "<link" "/home/patrick/Documents/current websites/NearlyFreeSpeech.net" | xargs -0 $(which bluefish)

To find all files in a directory and its subdirectories that do NOT contain a particular string, and then open this in Bluefish

find . -iname "*html" -print | xargs grep -iL favicon.ico | xargs $(which bluefish)

Source for this last

To find all files that contain "profile/hcard" but not "vcard" and open them in Bluefish

grep -R -Z -l "profile/hcard" . | xargs -0 grep -Z -L vcard | xargs -0 $(which bluefish) &

Source

To search within files in a specified directory for specified text

grep -R -B2 -A2 "hcite" /~patrickmooney/

(use the -l switch to just print a list of the matching files without spitting out matching lines within the file.) (use the -i switch for a case-insensitive search.)

Source

To extract the first letter of every word in a text file

cat "djibriel's walkthrough.txt" | sed 's/\B\w*//g;s/\s//g' > "1st letters.txt"

To eliminate duplicate lines from a text file without sorting

awk '!x[$0]++' 150.txt > 150-unique.txt

Source

To strip leading and trailing whitespace

echo " test test test " | sed -e 's/^ *//g' -e 's/ *$//g'

Source

To sort lines in a text file by length

cat titles.txt | awk '{print length,$0}' | sort -n | awk ' {$1="";print $0}' | cut -f2- -d' ' > sorted\ titles.txt

Source

To parse through text file 1, outputting all lines that are not also in text file 2, to a new text file

grep -v file2.txt file1.txt > file3.txt

Source