Using find and HandBrakeCLI to convert AVI for iMovie

I love using iPhoto for keeping track of my pictures and on occasion I use iMovie to manipulate one of the movies from iPhoto.  I like the convenience of being able to see all my movies that are in iPhoto.

However some do not show up.  Some AVI videos I have work just fine in iPhoto but cannot be viewed in iMovie.   They need to be converted to mp4 to work.

I am going to use the linux find command to find all the .avi files and use HandBrakeCLI to convert them.

I warn you upfront this is a long article if you want the simple answer just scroll to the bottom (after you install HandBrakeCLI that is).    But you may miss some interesting linux one liners I had to figure out along the way.

Installing HandBrakeCLI (OSx)

I am actually going to be using an Ubuntu Box to run the conversions.  My wife and kids would get a little upset if I took their macbook away from them to do conversions for the next week.

But if you do not have that option here is how to install HandBrakeCLI on OSX.

Go to http://handbrake.fr/downloads2.php [1]

Download the program.

Open up a terminal window and run the following commands

> cd /usr/bin > open .

This will open the /usr/bin folder  (a good location to put the HandBrakeCLI program.

Drag and drop the HandBrakeCLI program to the bin directory.

Now you should have access to it via the command line.

Installing HandBrakeCLI (Ubuntu)

HandbrakeCLI is not directly available via apt-get.

To install it run the following commands. (I installed it on 12.04 and 10.04)

Ubuntu 12.04 is missing apt-add-repository to add it run the following command first

> sudo apt-get install python-software-properties

> sudo apt-add-repository  ppa:stebbins/handbrake-releases > sudo apt-get update > sudo apt-get install handbrake-cli

Then as a quick test run

> HandBrakeCLI --help

The Find command

I copied my entire iPhoto library over to my Ubuntu machine.  I ran this command from the Pictures/iPhoto Library folder

> find $PWD -iname "*.avi"

This returned a long list of AVI files

To get a count I ran the following command

> find $PWD -iname "*.avi" | wc -l

This gave me back 303

So I have 303 files to convert.

It would be nice to know how much space all these files take up.

Running

> find "$PWD" -iname "*.avi" | xargs -I{} du -h {}

Works up to a point then gave me an unmatched single quote error.

Looking around I found that some of my folders have ' single quotes in them

Without them I could run the following command

> find "$PWD" -iname "*.avi" | xargs -I{} du -hs {}

And get an honest total as it stands it tells me its 109MiB, but I think it stops counting after it finds the first single quote '

I found this site http://stackoverflow.com/questions/5036735/xargs-unterminated-quote [2] where Ian posts about using -d to termintate the line

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} du -chs "{}"

This works to list all the files as it uses the return '\n' as a key to indicate a new command being passed in.

But, it really separates them counting each one and not giving a running total.

… I looked around and tried a few different things, but I failed to find a simple command line that would work

I even tried sed, which did replace the quote but xargs handled them all as individual lines

> find "$PWD" -iname "*.avi" | sed "s/'/\\\'/g" | xargs -I{} du -chs {}

… Oh well, if anyone has a way to do it please tell me and I will update this manual.

Getting details about the avi files

Before I start converting files I want to dig into them to see what they are made of.

exiftools seems to do the trick. To install it do the following

> sudo apt-get install libimage-exiftool-perl

Now to test it.

> exiftool MVI_4054.AVI

Which returns

Looks like the file is encoded in Microsoft PCM

HandBrakeCLI command

HandBrakeCLI has lots of really cool features, such as resizing and getting very specific with codec features.  But all I want is to convert an AVI to an MP4 format with h.264

Here is my first attempt at encoding

> HandBrakeCLI -i MVI_4054.AVI -o ~/MVI_4054_2.mp4

You can watch the progress of the tool

At the bottom right it will display the ETA.

When it is done converting Look at the file with exiftools results in the following

At this point I got a little lost on what HandBrake was exactly doing…

I used ffmpeg tool to get more detail on the file output.

> ffmpeg -i MVI_4054.mp4

Here you can see the total bitrate is 1085 kb/s and the width/height is 640x480.

Looking around I found this site http://www.scribd.com/doc/87043367/HandBrake-CLI-Guide [3]

This site explains the default behavior is to the mpeg-4 for the codec and to encode at 1,000 kb for the video and 160 for the audio.

Looking at some video from my camera recorded at 1280x720 I get a bit rate of 23,000 kb/s using h264.   The is the quality I would like to approach.

For a video that is 640x480 has 3 times fewer pixels that 1280 x 720 so it would require only 1/3 the bit rate or 7666 Kb/s. 

I ran this quick test to see how many videos have a width of 640

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} exiftool "{}" | grep Width

Almost all of mine are 640 a few are 1024.  So I think I will stick with a bit rate of 8000Kb/s for now and an audio bit rate of 192 kb/s using h264.

To accomplish this I need to run a command like this.

> HandBrakeCLI -i MVI_4054.AVI -o ~/MVI_4054.mp4 -e x264 -b 8000 -B 192

Looking at this newly converted file with the following command

> ffmpeg -i MVI_4054.mp4

Now I am getting the bit rate I want and the h264 encoding.

I tried to run the following command

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} HandBrakeCLI -i "{}" -o ~/videos/{}.mp4 -e x264 -b 8000 -B 192

But it failed

That was when I realized the output file is using the PWD version of the file name (the entire location as a name)  and its failing because of this.

If I set the output name to a static name like this.

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} HandBrakeCLI -i "{}" -o ~/videos/test.mp4 -e x264 -b 8000 -B 192

It works….

But every single output file has the same name so they keep overwriting each other.

The best command I came up with is this one.

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} HandBrakeCLI -i "{}" -o {}.mp4 -e x264 -b 8000 -B 192

This create the file I want but it creates it in the same directory as the .avi file and it ends in .AVI.mp4 .  I would rather the file be in another folder and have the .AVI removed….

I tried a lot of tricks but never got it to work

I ran this one liner to see how many .avi files share the same name.

> find "$PWD" -iname "*.avi"  | sed 's/^.*\///' | sort | uniq | wc

This finds all the .avi file, strip of the list of directories so you only have the file name.   Sots the list of file names,  then uses uniq to remove any duplicates (that are next to each other in the list) and finally does a word count.

The results of running this for me were 203.  100 files of mine have duplicate names.

So converting them all, and keeping the same names with .mp4 added, and placing them in the same directory won't work.   You could add a timestamp to the name I suppose and that would work.

I also ran this command to see how much space the current .avi files occupy.

> find "$PWD" -iname "*.avi"  | du -hs

Its 255 GiB,  and I do not have enough extra space on my machine to convert them all in one go….

So I will stick with running the following command in subfolders

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} HandBrakeCLI -i "{}" -o {}.mp4 -e x264 -b 8000 -B 192

But I need to modify it a little to add a timestamp to the converted file, something like this.

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} HandBrakeCLI -i "{}" -o {}.`date +%H:%M:%S`.mp4 -e x264 -b 8000 -B 192

But that is not quite right.  The date command is executed at the time the command is run and does not change.  As a result all date stamps are the same.

I found a site that discusses this issue at http://unix.stackexchange.com/questions/37418/random-is-not-random-within-finds-exec [5]

So how do I fix this???

Fixing the date stamp

To show the problem run this problem using $RANDOM

> find "$PWD" -iname "*.avi"  | xargs -d'\n' -I{} echo {}.`echo $RANDOM`.mp4

$RANDOM is only executed once and the result used each time the xargs is run.

The solution,  force xargs to use the bash shell by adding "bash -c"

After a lot of trial and error I got this command to work to echo out a random number each time.

> find "$PWD" -iname "*.avi"  | xargs -d'\n' -I{} bash -c 'echo "$0".$RANDOM.mp4' {}

Here is the final command I came up with to convert the file that adds a random number.   But there is the off chance you could get the same random number on a file that has the exact same name… but its pretty low.

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} bash -c 'HandBrakeCLI -i "{}" -o "$0".$RANDOM.mp4 -e x264 -b 8000 -B 192' {}

This works with a timestamp that goes down to the nanosecond!  This has no fear of creating a duplicate file.

> find "$PWD" -iname "*.avi" | xargs -d'\n' -I{} bash -c 'HandBrakeCLI -i "{}" -o "$0".`date +%H:%M:%S%N`.mp4 -e x264 -b 8000 -B 192' {}

This makes uniquely named files that are created in the same location as the original .avi file.

This command did not work on my macbook, it did not like the -d'\n' part.  So I removed it and it seemed to work.

> find "$PWD" -iname "*.avi" | xargs -I{} bash -c 'HandBrakeCLI -i "{}" -o "$0".`date +%H:%M:%S%N`.mp4 -e x264 -b 8000 -B 192' {}

Moving the files

To move them all at once I ran the following command

> find "$PWD" -iname "*.avi.*.mp4" | xargs -I{} mv {} ~/videos

Touch command

I also want to use touch on these files to set their date to match the original file.  I found this useful site http://www.thegeekstuff.com/2012/11/linux-touch-command/ [4]

You can update the timestamp of a file to match another file using -r

For example.

> touch MVI_3662.AVI.mp4 -r /backup-drive/Pictures/iPhoto\ Library/Originals/1903/Dec\ 31\,\ 1903/MVI_3662.AVI

But If I run

> ls -alh MVI_3662.AVI.mp4; ls -alh /backup-drive/Pictures/iPhoto\ Library/Originals/1903/Dec\ 31\,\ 1903/MVI_3662.AVI

You can see they now share the same timestamp

I want to do this, because when I import them back into iPhoto, iPhoto uses their timestamp to date them.

Unfortunately the timestamp I have on the copied files on my Ubuntu box are not the original dates, but the date I backed everything up.   I need to wait until I transfer these files back to my Macbook to run the touch command against the original files.

I could not get the -r working on OSX.  I was forced to use the -t and just write out a timestamp

So I had to use a command like this

> touch -t 201212251003 MVI_3662.AVI

This sets the date to 12/25/2012 10:03 AM

Last little Tweak

As a last little tweak I found this command, "basename"

Which when given a full path will return only the file name

Here is the command I came up with that will convert the .avi videos and place them in the directory of my choosing.

> find "$PWD" -iname "*.avi" | xargs -I{} bash -c 'HandBrakeCLI -i "{}" -o ~/videos/`basename "$0"`.`date +%H:%M:%S%N`.mp4 -e x264 -b 8000 -B 192' {}

Or a better one for an OSX environment might be (to put it on the Desktop)

> find "$PWD" -iname "*.avi" | xargs -I{} bash -c 'HandBrakeCLI -i "{}" -o ~/Desktop/`basename "$0"`.`date +%H:%M:%S%N`.mp4 -e x264 -b 8000 -B 192' {}

… But I had one last issue…

I had folders with single quotes in them I added a sed command to fix this issue.

> find "$PWD" -iname "*.avi" | sed -e "s/['\'']/\\\'/g" | xargs -I{} bash -c 'HandBrakeCLI -i "{}" -o ~/Desktop/`basename "$0"`.`date +%H:%M:%S%N`.mp4 -e x264 -b 8000 -B 192' {}

But on OSX I found a problem.  BSD unix does not understand the %N command and some of the smaller videos convert in less than a second.   So I changed the command just to put a simple timestamp (seconds since epoch) and to pause 1 second between conversions.

> find "$PWD" -iname "*.avi" | sed -e "s/['\'']/\\\'/g" | xargs -I{} bash -c 'HandBrakeCLI -i "{}" -o ~/Desktop/`basename "$0"`.`date +%s`.mp4 -e x264 -b 8000 -B 192; sleep 1' {}

Final Command set up

I changed my mind at the last minute and decided to run this command from my OSX machine rather than Ubuntu (Although the command should work fine in Ubuntu)

I created a folder on my desktop called videos to place all the converted videos.

This is the command!

> find "$PWD" -iname "*.avi" | sed -e "s/['\'']/\\\'/g" | xargs -I{} bash -c 'HandBrakeCLI -i "{}" -o ~/Desktop/videos/`basename "$0"`.`date +%s`.mp4 -e x264 -b 8000 -B 192; sleep 1' {}

This will convert all movies it can find that end in .avi 

Here is what the command will do

·         It will leave the original untouched

·         It will make a new .mp4 file at ~/Desktop/videos

·         The new file will be encoded in h.264 at a 8,000 kb rate

·         The videos will have a timestamp in their name, to assure they are all uniquely named (ex. FILENAME.AVI.timestamp.mp4)

That is it!

But after I created all these files I had to reset their creation time to match the timestamp of the original files.  This I did by hand using the touch command.

> touch -t 201210121245 MOV00639.AVI.1379782621.mp4

This sets the date to 10/12/2012 12:45 AM

I only have 300 files to change a date on so I am OK doing it …manually….

References

[1]        HandBrakeCLI download page

http://handbrake.fr/downloads2.php

            Accessed 09/2013

[2]        xargs: unterminated quote

http://stackoverflow.com/questions/5036735/xargs-unterminated-quote

            Accessed 09/2013

[3]        Using HandBrakeCLI

http://www.scribd.com/doc/87043367/HandBrake-CLI-Guide         

Accessed 09/2013

[4]        5 Linux Touch Command Examples (How to Change File Timestamp)

http://www.thegeekstuff.com/2012/11/linux-touch-command/         

Accessed 09/2013

[5]        `$RANDOM` is not random within `find`'s `-exec`

http://unix.stackexchange.com/questions/37418/random-is-not-random-within-finds-exec    

Accessed 09/2013