underdevelopment

I recently had a project with xs2theworld to help create the mobile websites for intel asia. Because this project was quite important and I wanted to step up my game I created a proper deployment strategy. No more sweaty palms while running custom scripts, pressing svn up or switching symlinks. I wanted a fully automated deployment. A deployment I could test, run and always get the same result.

Because I’ve been hearing about phing and dbdeploy from dragonbe and harrieverveer I looked into them. They ended up being excellent tools to reach my goal.

Phing is a deployment tool in which you can create a deployment “script” made up out of a ant like syntax using xml.

<copy todir="${buildDir}" >
  <fileset dir="${projRoot}">
    <include name="**" />
  </fileset>
</copy>

DBdeploy is a tool that compares your patches to your database and creates a forward SQL patch and a backwards SQL patch, aggregating your SQL patches in the forward file and the undo statements in the backwards file.

In this blog post I will highlight some of the things I did.

I created an actual ‘build’ stage, where all the website elements were processed and copied into a separate build directory. The reason for this was two-fold. Firstly I wanted to be able to check the result of a ‘build’ without it being deployed, especially the SQL patches that dbdeploy generated. Secondly, I wanted to only copy those files that were needed for the site to run. So no .git directory, no sql patches directory, etc..
This has really been a great choice, because of the separate build stage i’ve at least had two instances in which i caught a problem before deployment. Saving me from the embarrassment of having to make quick fixes while the site was in offline mode.

I created separate phing property files for different environments. (production, staging, development) this combined with a simple wrapper script that called phing resulted in a very pleasant way of deploying by just issuing a command like “./deploy build development” or “./deploy rollout production” and the inevitable “./deploy rollback production”. Much better then “phing -Denvironment=staging build”.  Property files are basically ini files that contain key/value pairs that can be referenced from within your phing build file.
Then in phing you can say “<property file=”deploy/${environment}.properties” />” and it will read the property file. Please note that “${environment}” refers to a variable. which in my case was set when calling phing. (the -Denvironment=)

Dbdeploy is a piece of software that can read your SQL patch files, compare them to the database and create a single SQL file you can run to update your database. Unfortunately dbdeploy is very fussy about the separator you use between your patch and your undo patch. Yes, undo patch. At some point you want to rollback a deployment and at that time you really do not want to find out that you can’t because the new table structure breaks the old code.
It only takes very little time to create undo statements and you will avoid excruciating minutes of frantically applying changes manually when things break.
Also when creating undo statements be sure to set them in the reverse sequence of your normal sql patch statements.
e.g.

ALTER TABLE `myrecords`  ADD `rank` int NOT NULL;
RENAME TABLE `myrecords`  TO `myrecord`;
-- //@UNDO
RENAME TABLE `myrecord` TO `myrecords`;
ALTER TABLE `myrecords` DROP `rank`;

Also, that is how you should write the undo separator. Exactly like that. If you don’t dbdeploy will simply add the undo section to your deployment SQL file as well. Which is very much unwanted.
Also be sure to always, ALWAYS, ALWAYS! run both your forward SQL patch as well as your backwards SQL patch to make sure it works. Preferably not on production.

That’s about it I guess. There are many wonderful guides that will explain how to use phing and dbdeploy in detail, which is the reason I didn’t. I just wanted to pass along some things I used and thought worked nicely.
I would like to point people who want to read more about phing to the following blog posts that helped me heaps:
- Diving into Phing I http://groups.drupal.org/node/4363
- Diving into Phing II http://groups.drupal.org/node/5400
- Phing Build File http://sean.gravener.net/blog/web-design/phing-build-file/134/
- How To: Simple database migrations with Phing and DbDeploy (*this was a bit outdated*) http://www.davedevelopment.co.uk/2008/04/14/how-to-simple-database-migrations-with-phing-and-dbdeploy/

most important though.
http://phing.info
http://dbdeploy.com/

admin PHP, software development best practice, dbdeployment, deployment, phing

So a little while back ibuildings had this fun contest to build a path finding program that would solve a traveling salesman like problem. The constraints where pretty simple, you got a CSV file with latitude/longitude locations. You started at a certain location and you should end at a certain location. The application should then find the shortest route that touched all locations.

Now I’ll be the first to say that PHP is really not the language for that. A few years back I wrote a pathfinding tool for use with a game called EVE online in which I calculated certain trade routes based on data you could export from the game. After seeing PHP’s performance I switched to Python and more or less sliced the processing time in half if not more. Mainly because Python has specific array like types and PHP just has generic array’s, which with large data sets matters a lot. Perhaps also because my pathfinding-foo was still rather weak

However, back to now and the ibuildings challenge. The challenge would be rated on 3 criteria. Speed, lines of code and code complexity. Personally I could care less about the latter two and focused purely on speed. In the weeks that followed I had a great time comparing execution times with Remi and Andries. I think this was also key to keep diving into it and tweaking it until it was as fast as I could get it.

Sadly, my submission actually had a off-by-one bug in it which more or less instantly disqualified my entry. Yes, bit of a bummer, but such is life.

Now I had already decided to publish my code after the contest, however sadly I never really found the time to type this up. So a bit late but here is the code for my solution for the ibuildings elephpant challenge.

Below is the submitted code, and a link to download the code and original test data file so you can try it out for yourself.

download the code – Just unpack and run via php contest.php elephpant_landmarks.csv

-edit-

Ok scratch the code, I’m having some trouble with getting it to play nice. Just download the tar.gz and view the code in your favorite editor.

admin PHP, software development code, performance, spielerij, techniques, traveling salesman

Sometimes in applications you will have certain user settings that you want to apply, even when the user is not logged in. Take for instance these examples:

• “welcome back <name>” msg on return.
• You have a portal type page where the user can control what content is shown where
• You want to track where the user was when he last visited the site, perhaps to offer him the option to return to there.

I recently needed some functionality like that. So I’ve created a object that can help me with that.

I thought about it for a moment and created a singleton settings object for me to call upon to set and retrieve certain settings. Now I have to warn you that there is a small problem with singletons, if you use unit testing it can be difficult to control the behaviour of singletons over multiple tests. So be wary of this when you are running unit tests.

I also wrap all data in a separate array. This isn’t really necessary, but it makes handling the data a lot easier. If you wanted you could also add some sort of encryption to the cookie data so that users couldn’t easily tamper with it.

admin PHP, software development design patterns, object oriented programming, PHP, singleton, techniques

Well I figured a provocative title like that would catch your attention, so please read on to find out I am not as stupid as you might think.

Unit tests are great, they reduce unknown bugs and arguably lead to better object design. The reasons for this are simple; Well thought out unit tests will force you to think about the input and output of any method your objects may have, forcing you to correctly decomposition the needed functionality within your object, because methods that have too much responsibility are a pain in the ass to write unit tests for. Simple one task methods however are easy to write unit tests for. Because your writing unit tests you are (hopefully) actively thinking about what can go wrong so that methods can actually fail.

So taking for granted that the above is true, why does test driven development suck?
Well the answer to that is a market one and really only applies to professional business. When a client comes to your business and asks for a offer for his project, you will quote him X, where X is roughly the amount of hours it takes to create the unit tests and then fill in the code + other stuff.

A development house that doesn’t use test driven development will quote that same client only for the time it takes to write the code + other stuff. We all know that after that he will spend a few weeks/months bugfixing, but that’s not what he quotes or tells the client.

Naturally the client will more often then not choose your competitor, unless you can positively convince him that the extra hours you spend will actually save him money in the long run. Which from the clients standpoint is a bit of an unproven claim if he is not technically inclined. So from a business standpoint, test driven development sucks as a selling tool.

Secondly, while this heavily depends on the morality of your business, those bugfix hours can be profitable. Depending on how smooth your talk is, you could very well be billing the client for those hours.

Thirdly, test driven development often suffers when deadlines need to be met. We all know that estimated hours is an imperfect science and often goes wrong, be it overconfident developers or overselling sales people, deadlines tend to be missed in our industry. So when deadlines are creeping, unit tests are often one of the first things to be dropped off schedule. Yes, we all know that shouldn’t happen, but it does. Once this has happened and no extra time is allocated to bring the tests up-to-date again (good luck getting that approved), the test driven development has failed and you’re just plainly developing again.

So for these reasons I think test driven development sucks. I might sometimes use it for private projects or open-source projects (for which I think it is absolutely brilliant), but certainly not for business ones. With the possible exception of in-house developed products, but I haven’t had much experience with that.

For anyone who now still thinks TDD is a sound development methodology, I would be very interested to hear you thoughts.

admin PHP, software development best practice, object oriented programming, PHP, pragmatic programmer, TDD, test driven development, unit test

I’ve recently changed the way I work so I figured I would detail it here.

First off all let me describe the setup. Basically all my code resides in a subversion repository on server X. My development environment is a virtual server run on virtualbox, a default ubuntu server with a LAMP stack. On my client I run the PDT edition of eclipse with subclips installed.

The client side:

• Install PDT eclipse, you can download it from http://www.eclipse.org/pdt/downloads/.
• Go to http://subclipse.tigris.org/ and follow the text to install it into your eclipse installation.
• In your eclipse choose import from the file menu and select “checkout project from SVN”
• You now should have a checkout on disk inside a project to work from in eclipse

Development environment side:

• Create and setup a LAMP based virtual server. (I’m going to assume ubuntu because that’s what I’ve used)
• Create a virtual host for your project.

The actual interesting bit:

Now the above is really standard stuff, so that’s why it’s a bit brief, but now comes the interesting bit.

What we want to do is mount the code checkout from your client machine as the serving directory of your development environment. Because I’m lazy I want to be able to do this with one click of the button, so we are going to break some security things to be able to do this via the browser.

We want the apache user (www-data in my case) to be able to mount via sshfs and unmount. This will enable it to mount the checkout from my client pc to the serving directory of apache on the virtual server.

First we want to add www-data to the fuse usergroup, we do this via the following command.

sudo usermod -G fuse www-data

Now the www-data user will be allowed to use sshfs.

Next we create the file /var/www/index.php and put some code into it. The following code is really simple and could probably be made more elegant but currently I don’t care


<?

if ($_GET['mount']=='myproject' )
  `sshfs user@192.168.1.20:/home/demo/workspace/myproject /var/www/myproject`;

if ($_GET['umount']=='myproject')
  `sudo umount /var/www/myproject/`;

if (is_file('/var/www/myproject/index.php'))
{
?>
myproject is currently mounted -- <a href="?umount=myproject" />umount</a>
<?
}else{
?>
myproject is currently not mounted -- <a href="?mount=myproject" />mount</a>
<?
}


What this allows us to do is to go to the default hosts index.php and remotely tell the server to mount or unmount the environment. There are a few things we need to do to actually make the above code work though.

First the mounting.
sshfs uses ssh, so since we want to use it without user interaction we need to create a private and public key for www-data.


sudo su - www-data
/bin/bash
ssh-keygen


After that you should have a set of keys, then copy the public key to your client and add it in ~/.ssh/authorized_keys of the user you need to log into.

Now the mounting of the repository should work.

However for the unmounting we need to use sudo to get the rights to do that. So for that we need to edit /etc/sudoers on the virtual server and add the following line somewhere.

www-data ALL = NOPASSWD: ALL

Yes, it would be much safer to not allow ALL, but only the umount, but for my virtual development box i’m really not concerned with security, I just want a trouble free and quick fix.

So now I created a simple script that will let me mount and unmount my working copy of the code on the virtual server.

I typed this all up rather quickly so if anyone wants some parts explained in some more detail just leave a comment.

admin apache2, PHP, software development LAMP, PHP, pragmatic programmer, sshfs, subclipse

Now this piece of wisdom is actually not mine, but from a colleague at ibuildings. He basically states the following; “If a client or user ever finds a bug or a security hole, then don’t ever say ‘Oh, i knew about that’.”

Now the observation here is that sometimes while you are reading older code you will see a bug or security leak. Often enough you will think “Oh well, this is old code and has been running for years”. That however is not the right thought to have.

The right thing to do is to estimate how many hours would be needed to fix it and what the impact of the problem is. If you then have enough time, you fix it. If you don’t, you register it and escalate it to the person responsible of the project.

At least then, you don’t have to say “Oh, I knew about that”. This phenomena is closely related to that of the broken window effect*, which teaches us that it is human nature to not fix the window. It is therefore wise to always keep the above in mind, and consciously force yourself to do the right thing.

So next time you see a bug, don’t say to yourself that you will fix it next week, because we both know that will probably never happen.

* This effect is wonderfully described in the book ‘the pragmatic programmer‘, required reading for any programmer if it was up to me.

admin software development best practice, broken window, pragmatic programmer