Archive: ‘Command Line (CLI)’

Favorite CLI Linux Apps: Guake

02.19.10

Posted by adamlinuxhelp  |  No Comments »

Guake and Yakuake are drop-down terminal shells

Guake, or (if it’s a K Desktop Environment) Yakuake are “drop-down terminal shells” that appear when you press a button (usually F12).

If you usually open a terminal with every log in to your Linux desktop, there’s a more convenient (not to mention laaazy) way.  Simply tell the OS to run Guake after your user’s successful login. In Openbox, you’d add it to your autostart.sh file

The Guake terminal will notify you that it has started, and then auto-hides itself until you “un-hide it” by pressing the F12 key.  When you’re done with your current command-line stuff, simply press F12 again and Guake gets out of your way.

Favorite CLI Linux Apps: Lynx

02.19.10

Posted by adamlinuxhelp  |  1 Comment »

Lynx is a text-only web browser that runs in the shell.

Lynx is useful tool for those times when you want to extract only the web links from a web page.  Install lynx using the Synaptic (or other) package manager.

To view the hyperlinks of a given web page (google.com in this example), issue the command

lynx -dump http://www.google.com

It can also behave in a similar way to wget when you want to view the HTML source code of a web page.  The command to view the HTML source code is

lynx -source http://www.example.com

Click the following link to view a post where we collected links to mp3 files to build an unattended download list for the wget command.  Another feature of Lynx is that it allows you to view your pages as a web crawler/robot such as googlebot might see them.

The linux find command

01.20.10

Posted by adamlinuxhelp  |  1 Comment »

Using the find command to find and move files

Today I’d like to discuss one of my favorite shell commands: find.
find is awesome.  It locates files on your unix/linux system by using options and criteria.

Many of us are familiar with using the Microsoft Windows “Search Tool” in XP to Find Files or Folders, files by date, etc.  Finding a file this way involves populating text fields, drop-down menus, and (possibly) date-range selectors to find the files you’re looking for.

In unix and linux, the find command is used like this:

find [directory to search] [options] [actions]

Here’s a find command example.  Let’s say we want to find .mp3 files within your “Music” folder (/home/yourName/Music).  For this example, let’s pretend that you have the following files: song1.mp3, song1.wav, song2.mp3, song2.wav, song3.mp3, and song3.wav.

To find files in the Music directory, the command is: [don’t type the dollar sign- it’s the prompt, start typing at the word “find”.]

$ find Music/ -type f [press ENTER]
Music/song1.mp3
Music/song1.wav
Music/song2.mp3
Music/song2.wav
Music/song3.mp3
Music/song3.wav

We used the “-type f” option to find a regular file and not a directory.  The output (above) shows that we have media files (mp3 and wav) in the Music folder.  Remember we’re not interested in listing files so much as performing a ‘find’ on them.

On a Linux system you can use a GUI program to find files, but you can find them faster using the find command.  With its many options, find’s power is in its flexibility.  Flexibility to do what??? Finding files is finding files, right? Yes, I suppose so.

But what if you wanted to perform an action on files that you find? In Windows you’d have to first find them, navigate to them, and then (possibly) perform actions on them one file at a time.  While the steps are few, they can be manual and tedious, thus begging for a shortcut.

Let’s take our example a step further.  Let’s filter the command to only find ‘mp3’ files.  We’ll add the “-name” option to search wildcard patterns for files ending in mp3.

$ find Music/ -type f -name "*mp3"
Music/song1.mp3
Music/song2.mp3
Music/song3.mp3

If you wanted to find only the .wav files, then you’d change *mp3 to *wav.  Now let’s use find to move files to another folder.  For the sake of neatness, let’s create 2 sub-folders (also known as sub-directories) to hold mp3 files and a separate sub-folder for wav files.

To create a new directory use mkdir [folderPath].  If you do not specify a folder path, the shell creates the directory within the current working folder.  More on “current working folder” later.  For now, let’s assume that you’re in the “home/yourName” folder.

Creating folders “Music/mp3” and “Music/wav” requires 2 identical commands.  However, you can create 2 sub-directories in the “Music” folder with the command below.

mkdir Music/{mp3,wav}

Brackets tell the shell to expect a sequence of characters.  The comma separates each new directory.  This is the same as issuing mkdir Music/mp3 and mkdir Music/wav.

With the sub-folders created, we have a neater storage arrangement for our music files.  But we’re not done yet.  Remember that our “Music” folder currently stores both mp3 and wav files.  Let’s move each type into their respective sub-directory.  To make this happen we add an action onto our command known as “-exec” (followed by the “command terminator” \;) and the mv command to move the files as seen in the example below:

mv fileOldLocation fileNewLocation

Note: mv also renames files and can overwrite them—so use caution because mv actions cannot be undone.  For safety, use mv -i which is interactive in the event of a possible naming conflict or unintentional overwrite—it may save you anguish.

We then add brackets {}, BUT brackets behave differently in this context.  When brackets are part of -exec they represent each found file.  Also, when moving a file to another directory, put the trailing slash / on the target directory name.

To move the mp3 files from directory level “Music” to “Music/mp3” issue this command:

$ find Music/ -type f -name "*mp3" -exec mv -i {} Music/mp3/ \;

Since there weren’t any files in Music/mp3 we didn’t get any warnings from mv -i
Result: mp3 files moved from /home/yourName/Music to /home/yourName/Music/mp3/

To check it, you can use the ls command in the shell to “list” files in a given directory

$ ls Music/mp3/
song1.mp3  song2.mp3  song3.mp3

To move the wav files from directory level “Music” to “Music/wav” issue this command:

$ find Music/ -type f -name "*wav" -exec mv -i {} Music/wav/ \;
$ ls Music/wav/
 song1.wav  song2.wav  song3.wav

Summary

  • find (and its options) allow you to find files and perform actions on the found items
  • mkdir allows you to create new directories
  • ls lists files (and directories, subdirectories) within folder(s)
  • mv moves or renames files

Download several files: part 2

01.14.10

Posted by adamlinuxhelp  |  2 Comments »

In an earlier post we used wget to download a single image file, and then used it to get all of the ‘gif’ and ‘jpg’ files from a single command.  Multi-download commands of this type are helpful when you know the URL and exact directory where the image files exist.  Let’s now take it a step further, and get lazy too.  Lazy?? Yes, lazy.  Since we’re looking to use Linux for time-saving shortcuts, the less work we have to do to get to accomplish our task, the better.

As previously mentioned, I like podcasts.   Podcasts are (usually) available in an RSS feed in the form of a web URL.  Programs such as ITunes, Amarok, Rhythmbox (or other) use feed URLs to get info about the available audio files and you can manually download them or set up preferences that do this for you.

We’re going to look at this from a “get me all the files—now” approach using the Linux command line.

To perform a multi-file “unattended” download…

  1. Make sure that “lynx” (a terminal-based web browser) is installed.  To check if lynx is installed, type which lynx at the prompt.  If the shell responds with nothing but the next prompt, then it’s not installed.  To install lynx and you’re on a debian-based OS such as Ubuntu (or similar) type “sudo aptitude install lynx” at the prompt.  If you’re using a redhat-based system type “yum install lynx” to accomplish the same.   When lynx is installed, the shell will return the executable path of lynx (it might appear as /usr/local/bin/lynx) when you type “which lynx” at your prompt.
  2. Make sure you have wget installed.  In the terminal, type “which wget” and see what the shell returns.  If it’s not on your system then install it.  Items one and two only have to be done once, if at all.  I think wget will be there, but  lynx is probably not included out of the box at install time.
  3. A URL (or RSS feed URL) where the desired files exist.

Here’s our practical example.  Let’s download all the mp3 files at Steven J. Cohen’s “Doctor Who” RSS Feed. You should view this link in a web browser to make sure that the page/feed is still there.

Time for a “trial-run” (this next command will not download, just list the mp3 files at the Feed URL).

lynx -dump http://www.stevenjaycohen.com/audio/drwho/feed | egrep -o "http:.*mp3"

lynx -dump [URL] returns a numbered list of web links from a given web page (for the complete HTML source, use lynx -source [URL]). Since we only want the links, (and not the numbering) we need to filter this list using the UNIX pipe character “|” and the search tool egrep -o [pattern].  We put in “http:.*mp3” as our pattern which will capture any link that starts with http and ends with mp3 (note the .* is a wildcard meaning `any character`). A word of caution. It’s ALWAYS a good idea to do a trial run so that you have an idea of what you will request for download and if your command will succeed in building the list properly.  This is a very important preliminary step.

Now, let’s do this for real.  The following command downloads files into the current directory of the shell.  So if you execute the command from “/home/myUserName/music” then the files get saved into “music”.

lynx -dump http://www.stevenjaycohen.com/audio/drwho/feed | egrep -o "http:.*mp3" | xargs -n1 wget

And that’s it.  The shell shows progress of each file as it downloads.  When it’s done with the first file, it downloads the next one, and so on.  It runs unattended, allowing you to do other things with your time.

To perform the “unattended” download of all the files specified in the list, we needed another pipe, and another command structure known as “xargs”.  Why xargs?  Sometimes the shell runs into a problem of having “too many arguments” in its list to act on.  xargs is your friend should this happen.

xargs [options] [command].  The option and the command work together as follows.  Option “-n1” directs the command “wget” to work one time per each url from the list resulting from the “lynx -dump” part of the command.  Like many shell commands, there’s usually more than one way to do it.

Download several files: part 1

01.14.10

Posted by adamlinuxhelp  |  1 Comment »

How to use wget; download many files with one command.

A typical way to download a file is to “right-click” on it and “save as” to a folder on your computer.  Downloading a few files this way is not tedious.  But if an audio book has 25 to 30 files you can bet I don’t want to do those manual moves over and over again.

Using a terminal, there’s a faster way to download files.  I’ll introduce now one of my often-used commands: wget.  This command has many useful options.  For example, you can download files, set up custom directory structures for your download(s), or see if a file exists without actually downloading it.

Using the command (in simple terms): Open a terminal and type wget [options] [urls] at the prompt (usually a dollar sign).  You can use one or several URLs.  Options are (well…) optional.

Here’s a practical example where you can download a gif image from the O’Reilly site linked below.  When you open a Linux terminal, you are usually in your user’s “home” directory.  This is fine for the purpose of this example.  Issue the command

wget http://oreilly.com/catalog/covers/0596009305_bkt.gif

Here’s what will happen: the file 0596009305_bkt.gif gets downloaded and saved to your home folder.  Cool right? But it was a bit of work (typing) just to download one file.  How does this save me time?

Yes, the above example is overly-simplified.  You can, if you wish, download any “.gif” or “.jpg” files from a given web address in the example below.  It’s a time-saving single command, borrowed from the commandlinefu website mentioned in the “cool and advanced uses of wget” link below.

wget -r -l1 --no-parent -nH -nd -A".gif,.jpg" http://example.com/images

*Change the “example.com/images” to a valid web address.  The options above (explained) are:

  • -r for “recursive”
  • -l1 only get files in the “images” directory (don’t dive into subdirectories)
  • –no-parent and -nH and -nd : ignore directory structure (no directories—just get the files)
  • “-A” is the “accept list” for files of type [.gif and .jpg].  It’s case-sensitive, so it would not download files ending in “.JPG”, so if you needed those too, specify with -A”.gif,.jpg,.JPG”

You can find more wget info and options here.  For really cool and advanced uses of wget, see this page.

I’ll post another awesome usage of wget in another post.  Thanks for reading.

Useful Commands: introduction

01.12.10

Posted by adamlinuxhelp  |  No Comments »

Learning useful shell commands help save time & effort.

Do you have to use the command line?  No, you don’t have to use it.  I didn’t use it much when I started using Linux.  But, in my opinion, learning useful shell commands helps you get the most out of Linux.

What are some benefits of using the Command Line?

  • When you need help from the Linux community, many helpful solutions are expressed as commands to be run in the terminal.  It’s done this way for simplicity, accuracy, and consistency.  Many graphical-based (GUI) programs are “front-ends” where a user triggers events (via menu choices & button clicks) that execute terminal-based commands in the background.
  • Many jobs in the IT and web development field require candidates to be comfortable on the command line.  This implies the ability to issue shell commands and use console-based text editors such as vi and emacs.  Some jobs will also require you to understand (and perhaps troubleshoot) pre-written shell scripts in many languages.
  • For repetitive tasks, using the command line is just plain faster.  Why wait for a GUI program to open, click on things, or browse for file(s) to manipulate one file at a time?  You could simply type one or more commands [and options] into a prompt to accomplish the same.  When you find yourself issuing the same commands a few times, it becomes apparent to save these command calls in a text-based file (a shell script) to make the process even faster.  More on this to come.

Save time on tasks

01.08.10

Posted by adamlinuxhelp  |  No Comments »

Learning basic Linux commands helps save you time when working on repetitive tasks.

Imagine you had a folder of images in *.bmp format.  File sizes of bmps are larger than jpg files because they contain more information.  To convert these bmp files to jpgs you could open a .bmp file in a graphics program (such as photoshop), select processes (such as ‘save for web’), set the format to convert to, and finally, save the file “as” filename.jpg.  Converting each image takes about five to six “manual” steps, depending on how you opened the source files and where you save the destination files.  If you had a folder with 100 images, it becomes tedious repetition.  There are better ways to do this.

You could create a batch process or “macro” in your graphics app that lets you record the individual steps performed on a single image, and then point the macro at a folder of source images and a folder to save the converted images.  Photoshop has “batch processing” that handles this fine.  Other programs (such as THE GIMP) are “scriptable” but some knowledge of Python is probably a must.

So we now have a way to convert images en masse with a GUI app, but is it the most efficient way? Is it reusable? It is reusable but you’ll probably have to reset the macro/batch settings if your source and destination folders change the next time you had to convert a lot of images.

Is there a quicker way to accomplish the same thing more or less?  There is, in my opinion, a better way perform mass image conversion in a predictable, reliable way.  It requires the command line terminal and an application known as “ImageMagick“.

Using Linux (the bash shell and ImageMagick) here are the 2 steps:

1. Use the “cd” command to get to the directory that has the bmp images
2. Issue this command:  mogrify -format jpg *.bmp
The above command was found at http://www.ofzenandcomputing.com/zanswers/1016 and I thank them for posting it.  It accomplished in one line the same work as a 50-line shell script.
Result: All of your bmp files have been converted and saved-as jpgs in the same directory
Warning: the “mogrify” command usually replaces the source, but when changing formats, Imagemagick is intelligent enough not to destroy the original bmp file.

2a. To store the new files in a folder other than the source (e.g. “bmpFiles/jpgs”), add the -path option in front of -format.  The command becomes:  mogrify -path jpgs/ -format jpg *bmp
The folder specified by -path must first exist, or the command will fail.