{"id":319,"date":"2010-01-14T21:08:51","date_gmt":"2010-01-15T02:08:51","guid":{"rendered":"http:\/\/nylinuxhelp.com\/blogs\/?p=319"},"modified":"2010-01-17T08:34:13","modified_gmt":"2010-01-17T13:34:13","slug":"download-several-files-part-2","status":"publish","type":"post","link":"https:\/\/nylinuxhelp.com\/blogs\/command-line\/download-several-files-part-2","title":{"rendered":"Download several files: part 2"},"content":{"rendered":"

In an earlier post we used wget<\/strong> to download a single image file, and then used it to get all of the ‘gif’ and ‘jpg’ files from a single command.\u00a0 Multi-download commands of this type are helpful when you know the URL and exact directory where the image files exist.\u00a0 Let’s now take it a step further, and get lazy too.\u00a0 Lazy?? Yes, lazy.\u00a0 Since we’re looking to use Linux for time-saving shortcuts, the less work we have to do to get to accomplish our task, the better.<\/p>\n

As previously mentioned, I like podcasts. \u00a0 Podcasts are (usually) available in an RSS feed in the form of a web URL.\u00a0 Programs such as ITunes, Amarok, Rhythmbox (or other) use feed URLs to get info about the available audio files and you can manually download them or set up preferences that do this for you.<\/p>\n

We’re going to look at this from a “get me all the files\u2014now” approach using the Linux command line.<\/p>\n

To perform a multi-file “unattended” download…<\/h2>\n
    \n
  1. Make sure that “lynx” (a terminal-based web browser) is installed.\u00a0 To check if lynx is installed, type which lynx<\/strong> at the prompt.\u00a0 If the shell responds with nothing but the next prompt, then it’s not installed.\u00a0 To install lynx and you’re on a debian-based OS such as Ubuntu (or similar) type “sudo aptitude install lynx<\/strong>” at the prompt.\u00a0 If you’re using a redhat-based system type “yum install lynx<\/strong>” to accomplish the same.\u00a0\u00a0 When lynx is installed, the shell will return the executable path of lynx (it might appear as \/usr\/local\/bin\/lynx) when you type “which lynx” at your prompt.<\/li>\n
  2. Make sure you have wget installed.\u00a0 In the terminal, type “which wget” and see what the shell returns.\u00a0 If it’s not on your system then install it.\u00a0 Items one and two only have to be done once, if at all.\u00a0 I think wget will be there, but \u00a0lynx is probably not included out of the box at install time.<\/li>\n
  3. A URL (or RSS feed URL) where the desired files exist.<\/li>\n<\/ol>\n

    Here’s our practical example.\u00a0 Let’s download all the mp3 files at\u00a0Steven J. Cohen’s “Doctor Who” RSS Feed<\/a>. You should view this link in a web browser to make sure that the page\/feed is still there.<\/p>\n

    Time for a “trial-run” (this next command will not download, just list the mp3 files<\/strong> at the Feed URL).<\/p>\n

    lynx -dump http:\/\/www.stevenjaycohen.com\/audio\/drwho\/feed | egrep -o \"http:.*mp3\"<\/pre>\n
    lynx -dump [URL] <\/strong>returns a numbered list of web links from a given web page\u00a0(for the complete HTML source, use lynx -source [URL]). Since we only want the links, (and not the numbering) we need to filter this list using the UNIX pipe character “|” and the search tool egrep -o [pattern]<\/strong>.\u00a0 We put in “http:.*mp3” as our pattern which will capture any link that starts with http<\/em> and ends with mp3<\/em> (note the .* is a wildcard meaning `any character`). A word of caution.<\/strong><\/em> It’s ALWAYS a good idea to do a trial run so that you have an idea of what you will request for download and if your command will succeed in building the list properly.\u00a0 This is a very important preliminary step.<\/p>\n
    Now, let’s do this for real.\u00a0 The following command downloads files into the current directory of the shell.\u00a0 So if you execute the command from “\/home\/myUserName\/music” then the files get saved into “music”.<\/p>\n
    lynx -dump http:\/\/www.stevenjaycohen.com\/audio\/drwho\/feed | egrep -o \"http:.*mp3\" | xargs -n1 wget<\/pre>\n
    And that’s it.\u00a0 The shell shows progress of each file as it downloads.\u00a0 When it’s done with the first file, it downloads the next one, and so on.\u00a0 It runs unattended, allowing you to do other things with your time.<\/p>\n
    To perform the “unattended” download of all the files specified in the list, we needed another pipe, and another command structure known as “xargs”.\u00a0 Why xargs?\u00a0 Sometimes the shell runs into a problem of having “too many arguments” in its list to act on.\u00a0 xargs is your friend should this happen.<\/p>\n
    xargs <\/strong>[options] [command].\u00a0 The option and the command work together as follows.\u00a0 Option “-n1” directs the command “wget” to work one time per each url from the list resulting from the “lynx -dump<\/strong>” part of the command.\u00a0 Like many shell commands, there’s usually more than one way to do it.<\/p>\n","protected":false},"excerpt":{"rendered":"
    In an earlier post we used wget to download a single image file, and then used it to get all of the ‘gif’ and ‘jpg’ files from a single command.\u00a0 Multi-download commands of this type are helpful when you know the URL and exact directory where the image files exist.\u00a0 Let’s now take it a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14,7,23],"tags":[26,34,31,33],"class_list":["post-319","post","type-post","status-publish","format-standard","hentry","category-linux-apps","category-command-line","category-use-linux","tag-commands","tag-pipe","tag-wget","tag-xargs"],"_links":{"self":[{"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/posts\/319","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/comments?post=319"}],"version-history":[{"count":31,"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/posts\/319\/revisions"}],"predecessor-version":[{"id":348,"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/posts\/319\/revisions\/348"}],"wp:attachment":[{"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/media?parent=319"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/categories?post=319"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nylinuxhelp.com\/blogs\/wp-json\/wp\/v2\/tags?post=319"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}