Search and replace in multiple files with Sed

Back on the 30th of May, I put up a blog post on how to do a search and replace in multiple files with Vim.
I think this is a great way of doing substitutions in multiple files because you get to see a preview of what it looks like before you actually writing it to disk.

This is great for a small number of files, but to do a large directory of files, you don't really gain any advantages as you're probably not going to open every file in a vim buffer and review each one separately.

This is where sed and find come in handy!

This command will replace every occurrence of the word "foo" with the word "bar" in all html files recursively from the current directory:

find . -name "*.html" -exec sed -i "s/foo/bar/g" '{}' \;

The explanation:

This might look a little confusing, but it essentially 2 parts:

The first part will list all the .html files recursively and for each one, execute the command specified by the -exec option:

find . -name "*.html" -exec ...

The second part is the command to execute for each found file, which in this case is sed:

sed -i "s/foo/bar/g" <TheFile>

The '{}' represents the current file and the \; represents the end of the command to execute as part of the -exec option.


Doing more:

As you can see, the -name option to the find command takes a glob pattern.
If you wanted to be more specific about which files should have the substituion, you could use a regular expression instead, like this:

find . -regex ".*[.]html\|.*[.]js" -exec sed -i "s/foo/bar/g" '{}' \;

This regex will match all files that end with ".html" or ".js".

If you wanted to preview your changes before they are actually written to disk, there are a few ways of accomplishing this, but the simpliest is probably to modify the command as follows:

find . -name "*.html" -exec sed "s/foo/bar/g" '{}' \; | less

Removing the "-i" will mean that instead of writing the changes "in place", the changed output will be written to STDOUT instead of the files.
Then to capture this (potentially large) output, we pipe it to less.
From within less you will be able to do searches (with the "/" key) and scroll through the output to do a quick sanity check.
Once your happy, you can remove the pipe and add the "-i" option again to make it permanent!

ssh-copy-id with an alternative port

If you use ssh on a regular basis, you probably already know about setting up ssk-keys.
If you dont know about ssh-keys, you should! - its worth reading up on.

ssh-copy-id is a helper script that will copy your public key across to the remote machine and append it to the authorized_keys file.

This is all well and good if the server is on port 22, but for non-standard ports, this script does not support a -p option.

Instead of copying across the key manually, I googled for workarounds and came across this post where the author explains that the arguments to ssh-copy-id are simply passed on to the ssh command.

This means by using quotes, everything inside of them will be passed straight through and that allows a command like this to achieve the desired result:

ssh-copy-id -i ~/.ssh/id_rsa.pub ‘-p xxxx username@host’

Kudos to Mike Gerwitz for the tip.

Pencil for UI design

I'm designing software!

I'm currently in the early stages of design and have developed some basic requirements, a feature list, some rough UI sketches and a very rough roadmap.

I'm putting a fair bit of thought into the user-friendliness of the app and I have gone through quite a few different idea's for the UI.
During this sort-of brainstorming activity, I got really sick of trying to change my sketches on paper and asked my self "why aren't I doing this electronically!?" - so I started looking into freeware mockup software.

My searches lead me to this page:
http://stackoverflow.com/questions/54606/what-free-software-can-i-use-to-create-ui-mockups

Which lead me to Pencil:
http://pencil.evolus.vn/en-US/Home.aspx

Pencil is an XUL based application for making UI mockups - and its free!
For those of you that have used professional UI mocking tools, I'm not sure you'll find this to suit your needs, but as I have never used any software like this, this was basic enough for me (and faster than using The GIMP).

Pencil is advertised as a firefox plugin, but the .xpi is configured for FF3.5 (which is a little old!) so instead of hacking at this, the downloads page offers a "stand-alone" option that you can run with xulrunner.

Please note at the time of this blog, this app requires xulrunner 1.9.
In Ubuntu, xulrunner can be easily installed with this command:

sudo apt-get install xulrunner-1.9.2

Once you've downloaded and extracted Pencil, and you have xulrunner 1.9 installed, you can run Pencil by giving xulrunner the application.ini file, for example:

xulrunner /opt/Pencil-1.2-0-linux-gtk/application.ini

The UI for Pencil is pretty intuitive and judging by the menus, there isn't a whole bunch of advanced features (or at least from what I can tell), but I do like the simplicity and I was able to sketch up my UI design quite quickly.

By deafult (on Ubuntu) it comes with a number of "Collections" of widgets to start playing with.
I particularly like the "Sketchy GUI" collection for mockups, so the end result does not allow the viewer to let any preconceived opinions about the look and feel affect their judgement (some people dont like the look of the Windows XP widgets and may get turned off straight away).

This also means if you change the implementation (from desktop app to web app for example) the mockups may possibly stay the same without too much confusion.

Screenshot of the "Sketchy GUI" collection from the Pencil Website

If you are after more complete mockup software, that StackOverflow article has some good references, and I have heard Balsamiq is quite good, but Pencil suited my needs just fine.

Python-fu for The Gimp

I've been meaning to learn how to script The Gimp for a while now, but have never bothered to put the time aside to do it. Well, I finally did it, and this is why and how :)

The story:
 
When I had my 21st birthday, my brother made the party invites for me. He added a whole bunch of photos and I've still got a copy of that with all my photos as a keepsake.
Now his birthday is coming up and I thought I would repay the favour.

The Idea I had was to scan a bunch of old photos (like he did), then use The Gimp to make them look like printed photos (white border, drop shadow, kind of like polaroids) and scatter them around a "Happy Birthday" title.

I spent quite a while scanning the photos, adding them all as layers to a single canvas, resizing them to suit and then individually adding white padding, a black stroke around the outside, dropping a shadow, rotating and moving each individual layer to lay it out nicely, I was finally happy with the end result and saved a .xcf and then exported a .jpg.
At this stage I had all the scans, a font, a background image, the .xcf and the .jpg all saved in a folder on my desktop.
I quit The Gimp, got a flash drive and copied the .jpg over so I got get it printed, shutdown the pc and went for dinner.

When I got back quite some time later, my pc hadn't quite shutdown yet, it just had a black screen.. So I left it a little longer and it still hadn't shutdown, so I held down the power button to force it.
The next day, I got asked to make a small adjustment to the invite, and to my horror, I could not find the .xcf anywhere! I had the exported .jpg from the flash drive, but that wasn't suitable to make the adjustment... something had happened and to this day I'm still not sure what, but I lost all my work... When I realised I had to redo all that work, that's when I put time aside to learn how to script The Gimp!

Scripting The Gimp:

I took a look at some code examples for The Gimp's script-fu functionality and having never learned Lisp, it looked pretty foreign to me! That's when I noticed there was a Python-fu equivalent (luckily).

Using the Python Console from within Gimp is really easy!
It has a "Browse" button which brings up the "Python Procedure Browser" - which is a list of functions that interact with The Gimp.

Using the filterbox at the top, you can easily find the python function that matches the action you would normally execute using The Gimp's menus (such as a layer resize, or filter).
When you select the function, it will show you a basic man page on what it does and the parameters it takes, and applying this function will insert the code into the console with place holders for the parameters, which you can then simply modify and execute.

After getting the basics from Googling around, I made a basic script that I just copied and pasted into the console when I wanted to execute the repetitive actions.
I then optimised the script by wrapping it in a function, doing all my actions in a separate context (as not to disturb the users context for things like selected brush, etc.) and also group all my actions in a single undo (to simplify the history and make it easy to undo what my script does).

Work-arounds:

There are a couple of hacks in the script that I couldn't work out the nice way around in a short period of time (I'm still a n00b).
Rather than working out how to fix it, I thought I would just describe them here.

One of those being that to add a white border around a layer, I need to resize the layer with an offset, and this new extra space would not always be white (regardless of my selected background color). My work around was to add a white layer behind it and merge it down.

The other hack was with stoking. To add a 1px black stroke around a layer, I normally just specify 1px black from the stroke dialogue (as opposed to using the selected brush). I could not find an equivalent to this in the Python API - it always used the selected brush.
As the default brushes include a 1px brush (and a 3px brush, but no 2px brush) I thought I'd use that, but to do a stroke around the outside of a layer, I needed to select-all, then stoke, but with a 1px brush, I could not specify I wanted that 1px to be on the inside of the selection. So stroking with this method put the line on the top and left sides, not on the bottom and right.
The hack I used here is to rotate 180 degrees, stroke and repeat.

Not the most elegant of scripts, but this was my first attempt and got the job done.

Registering the script with The Gimp:

So now that the script was working (albeit with work arounds) and I added the polish (function definition, separate context, grouped undo, etc.) I was ready to add it to the menus just like any other filter.
I used an existing script (located in ~/.gimp-2.6/plug-ins/) as a baseline and adjusted it for my new script (that I called "photofy").
This part of the code allows you to specify the name of the script, where in the menu it should be registered and if there are any user-specified inputs to the script you can even specify a simple GUI for a dialogue box to get them from the user - which I did to collect the desired size of the white margin.
You can see this part of the code at the bottom of the python script.

Demo:

So now the fun part, this is how you use it :)
Here is a simple image with 2 layers: a white background layer and a smaller subject layer (the gradient).

With the subject layer selected, navigate to the menu in which the "Photofy" script is registered.

The dialogue box that I specified in the script-fu code segment at the bottom of the script will appear and allow the user to enter the size of the white margin.

When you press "OK", the script will run through its actions and the user's environment will remain the same, and the there will be a single new undo step in the history.
This is the final product:

The code:

If you're interested in seeing what the code looks like, or want to use the script yourself (either as is, or as a baseline to write you own), it is available here:
https://github.com/stevenocchipinti/Gimp-Python-Fu/blob/master/photofy.py