Suppose you know somewhere in the multitude of stored procedures in the database you’ve inherited that there’s likely a reference to a field or table, and you don’t know where it is. How do you find it without going through each stored procedure and checking it?

Easy. The definitions are stored in the system tables, just like pretty much all structure of a MSSQL database, so we just take a look in those tables, or the INFORMATION_SCHEMA views.

There’s a few different ways of getting at the information, but let’s say you’re looking for references to the field email, and the table Organisation:

SELECT		routine_name,
		created,
		last_altered,
		routine_definition
FROM		information_schema.routines
WHERE		routine_definition LIKE '%email%'
AND		routine_definition LIKE '%Organisation%'

SELECT		NAME,
		create_date,
		modify_date,
		OBJECT_DEFINITION(OBJECT_ID) as Routine_Definition
FROM		sys.procedures
WHERE		OBJECT_DEFINITION(OBJECT_ID) LIKE '%email%'
AND		OBJECT_DEFINITION(OBJECT_ID) LIKE '%Organisation%'

or all references to a different email field that include no reference to the Organisation table? This one’s a bit more likely give false negatives – e.g. the procedure may reference the Organisation table in some way not related to the email field. But you get the idea.

SELECT		routine_name,
		created,
		last_altered,
		routine_definition
FROM		information_schema.routines
WHERE		routine_definition LIKE '%email%'
AND		routine_definition NOT LIKE '%Organisation%'

I’ve been getting more and more depressed with what I suspected was happening in IT education – certainly in the UK, and it appears too here in NZ – I recently started looking into schools here for my 6 year old (some years away) – and was saddened, but not surprised, to find a class masquerading as IT but really sounding more similar to the European Computer Driving License, basically teaching office computing skills.

The driving lesson analogy works quite well here – it’s like claiming you’re teaching mechanic and metal work skills, when what you’re actually doing is teaching someone how to drive a car. Both valuable skills, but I wouldn’t like to live in a country that didn’t have any mechanics.

I’ll admit, teaching kids ECDL type lessons may still be useful – I’m sure that there are still a significant (but I suspect continuously decreasing) number of young people who still are intimidated by IT, but realistically, computers are one hell of a lot less intimidating than say even 10 years ago, and I imagine these classes are frankly a waste of time for 80+% of pupils.

I’m one of the generation that benefited from the BBC – I remember programming on Commodore PETs back in the mid 80′s, and then on the ground breaking BBC Master Series – these machines I’m sure encouraged a fair few of us to get really interested in computers.

It seems strange that when comparing how quickly you can both learn how to, and actually implement, vastly more complex and cool functionality than we could dream of 20-25 years ago, teachers seem to be shying away from programming and real IT, which should surely be far more accessible to young kids?

Bruce Clement has posted similar thoughts too.

Tim Foster also pointed me to a great article in the Guardian – Kids today need a licence to tinker

So anyway, I’ve started showing my daughter scratch – it seems to be the Logo of the 2010′s. Although secretly I suspect she’ll be a musician anyway!

There’s nothing really complex here, but I found it confusing in a number of places, so hopefully this may be of use to others – it will certainly be a memory aid for me in the future.

Background:

We have an old Windows server the free running VMware Server 2 administered via VMware Infrastrutcure Web Access. There are some peculiarities if you want to run a console window in Firefox, but it generally works well, and is free, so I can’t really complain.

Whilst we’re doing this migration, we’re also looking at consolidating software licenses, an obvious candidate is the host machine running VMs – in this case I could either run VMware Server on Windows or Linux, or look at VMware vSphere based on ESXi. I’ve run VMware Server 2 on Ubuntu before – it was a bit of a nightmare and took a couple of days to get working properly, so I was keen to see if ESXi was any better. ESXi also runs bare metal as opposed to on top of another operating system, so there should be some performance benefits too.

So, VMware vSphere ESXi it is then.

All of the VMware software we’re using below is free, but you’ll have to register in order to download it.

Downloading and installing ESXi

Head on down to https://www.vmware.com/tryvmware/index.php, sign your life away with EULAs and T&C’s, and grab a binary – I manually downloaded the ESXi 4.1 Installable Update 1 (CD ISO) image file. Also make a note of the license key.

Once I got the ISO file, I blindly burned a CD, and hit my first obstacle. My HP DL 360 G7 doesn’t have a CD drive. No worries – we’ll create a bootable USB drive.

I already had a bootable Ubuntu thumb drive, so I used 7-zip to extract the contents of the ISO file onto the USB drive. Not too surprisingly, this wasn’t quite sufficient to get a functioning ESXi boot drive working, so instead a quick Google turned up many pages, among them this excellent one by Ivo Beeren detailing how to Install VMware ESXi 4.1 from bootable USB stick, which basically consisted of downloading Syslinux, editing 2 files, copying 2 files, and booting of the USB stick.

After kicking of the installation, a couple of 5 minute pauses and a restart, I was presented with EXSi’s very simple text interface, telling me to administer the server from 192.168.1.10.

Sure enough, on browsing to this address I can see a welcome screen allowing me to access a Web-Based Datastore browser, but also suggesting I download the vSphere Client, which I do.

Administering the host server

Once we’ve downloaded and installed the vSphere Client, we can connect to the server – the credentials set in Ivo’s page above include a root password of VMware01 in the ks.cfg file:

rootpw VMware01

Log into your server, e.g. 192.168.1.10, user: root, password: VMware01.

The first thing that confused me was mention of a 60 day trial – hang on, I’m a cheapskate, and chose ESXi as I thought it was free?!? Don’t worry – it is – just plug in your license key you noted earlier to get rid of the pesky warning.

vSphere -> Home -> Inventory -> Configuration -> Software -> Licensed Features -> ESX Server License Type -> Edit.

Copying pre-existing VMs

OK, so next I’ve got a load of VMs on my old server that I want to get onto the new box. But there’s no network file access, no FTP access, and the Web-Based Datastore browser is read-only. How do we get our VMs onto the new box?

Well, first off, don’t use the approach below – it probably won’t work if your copying old VMware Server 2 VMs. But hidden away in vSphere is a method to upload files to the server:

vSphere -> Home -> Inventory -> Summary -> Resources -> Datastore, right click, Browse Datastore…

Click the Upload files to this datastore toolbar button, and upload either a file or folder as required.

If you upload old VMware Server 2 images using this method, you’ll likely run into the following error when you try to power up your guest machines:

DevicePowerOn power on failed.
Unable to create virtual SCSI device for scsi0:0,
Failed to open disk scsi0:0: Unsupported or invalid disk type 7.
Make sure that the disk has been imported.

So how do we copy VMs and get them to work? Instead of copying the images manually, head back over to VMware and download the vCenter Converter. Install and run this software (again free) and select Convert Machine.

1. For the Source System, select VMWare Workstation or other VMware virtual machine – and select your .vmx file.
2. For the Destination System, select VMware Infrastructure and specify the IP address and credentials to connect to the ESXi box.
3. For the Destination Virtual Machine, specify the name for the machine, likely the same as the original guest, and  review Destination Location, Options and Summary.

Enable networking

vSphere -> Home -> Inventory -> Configuration -> Networking -> Add Networking…

1. Under Connection Types – specify Virtual Machine, and click Next.
2. I’m dealing with internal development servers, so I’m not worrying about security here – you may need to. I simply chose the already configured vSwitch0, then clicked Next, specified a Network Label of VM Network, clicked Next again, and then Finish.
3. Stop your VM if necessary – configure to use the above created Network Connection (VM Network) by going to vSphere -> Home -> Inventory -> <Guest Machine Name> -> Summary  -> Edit Settings, selecting Network Adaptor 1, and then specifying the VM Network network label created above under Network Connection -> Network Label.
4. Click OK and restart your guest VM.

Your guest machine should now start up, and be on the network!

The web interface in Redmine makes it pretty easy to bulk move issues to new projects, as I was doing earlier today. Whilst it manages issues, relationships between issues, and time entries against issues, one minor thing it doesn’t seem to let me do through the web interface is move time entries that are not associated with an issue/task to a new project (for example, I have certain project management and deployment activities that are not related to a particular issue or task in Redmine).

In order to do this, I had to get into mysql. This is all some pretty basic SQL, but I thought I’d jot it down incase it’s of use to anyone.

BACKUP!!!

First, and it goes without saying, but I will anyway – take a backup!

lhunt@vm-srv052:~$ rsync -a /var/www/redmine/files /backup
lhunt@vm-srv052:~$ /usr/bin/mysqldump -u redmine -predmine redmine > /backup/db/redmine.sql

Now, get a list of Projects:

mysql> select id, name, parent_id from projects;
+----+-----------------------+-----------+
| id | name                  | parent_id |
+----+-----------------------+-----------+
|  1 | Project X             |      NULL |
|  2 | Sub project 1         |         1 |
|  3 | Silverlight migration |         1 |
|  4 | Version 2             |         1 |
+----+-----------------------+-----------+
4 rows in set (0.00 sec)

Then review all those entries without the correct sub-project associated to with them, in this case, everything still associated with the original parent project, Project X.

mysql> select id, project_id, spent_on, comments  from time_entries where project_id =1 ;
+-----+------------+------------+-----------------------------------------+
| id  | project_id | spent_on   | comments                                |
+-----+------------+------------+-----------------------------------------+
|  22 |          1 | 2010-10-29 | comments related to ticket...           |
|  27 |          1 | 2010-11-01 | Some more comments related to ticket... |
|  .. |         .. | .......... | ....................................... |
|  .. |         .. | .......... | ....................................... |
+-----+------------+------------+-----------------------------------------+

In my case, I had a minority of entries that should now be associated with the new Silverlight migration sub-project I’m working on….

You could do this by picking individual entries, or by searching the comments…

mysql> update time_entries set project_id = 3 where id in (102, 106, 109, 110);
Query OK, 4 rows affected (0.03 sec)
Rows matched: 4  Changed: 4  Warnings: 0

mysql> update time_entries set project_id = 3 where comments like '%silverlight%';
Query OK, 5 rows affected (0.03 sec)
Rows matched: 9  Changed: 5  Warnings: 0

And the rest should be associated with a new sub-project for a piece of stabilisation work…

mysql> update time_entries set project_id = 2 where project_id = 1;
Query OK, 38 rows affected (0.04 sec)
Rows matched: 38  Changed: 38  Warnings: 0

Done!

Update 24 Dec

Err – I lied – it wasn’t completely done – I had some Documents that were referenced by Title rather than ID, which then failed to link across projects.

Quick fix:

mysql> select id, project_id, title from documents;
+----+------------+----------------+
| id | project_id | title          |
+----+------------+----------------+
|  1 |          1 | V1.3.3.3 ..... |
|  2 |          1 | Training Env.. |
|  3 |          1 | V1.3.3.4a .... |
|  4 |          1 | Flow diagram.. |
|  5 |          1 | 20101203...... |
|  6 |          1 | Training Env.. |
|  7 |          1 | V1 3 3 4b UAT..|
+----+------------+----------------+
7 rows in set (0.00 sec)

All the documents needed to be moved to the Stabilisation project, so again a really simple bit of SQL…

mysql> update documents set project_id = 2;
Query OK, 7 rows affected (0.10 sec)
Rows matched: 7  Changed: 7  Warnings: 0

N.B. At the end of this you should have an Amazon Machine Instance running GeoServer suitable for playing around with. What you will not have is a strongly secured performance optimised GeoServer box – hopefully that will be covered in a following post.

This post also assumes you already have an Amazon Web Services account.

Creating our instance

I’ve tried doing this setup with the default Fedora AMIs from Amazon, but I haven’t used Redhat in a few years, and I’m more familiar with Ubuntu, so the first step is to get a trusted Ubuntu instance to start from.

First step – grab an AMI with Ubuntu 10.4 – Ubuntu’s Public Cloud Deployment documentation lists the Amazon EC2 published AMI’s – I’ll be using the 32bit EBS image on the US West Coast (ami-cb97c68e), as I’m just wanting a small instance (the 64bit are larger, and cost more), I may want to be able to stop the instance without loosing it completely, and as I’m in New Zealand, I suspect the US West Coast will have lower latency than Singapore.

So, let’s create our Instance. Head over to Amazon Web Services EC2 Console Dashboard, and click Launch Instance.

• In the Choose an AMI tab, click Community AMIs, and enter ami-cb97c68e in the text box next to Viewing / All Images.
• Press Enter, and the Ubuntu AMI ami-cb97c68e should be listed. Click Select.
• The Instance Details tab appears, you can leave the defaults selected here. Click Continue.
• Under Advanced Instance Options,  you may want to click Enable CloudWatch Monitoring for this instance, though note that this costs extra – you can leave this unchecked and add it later if you wish. Click Continue.
• In the Create Key Pair tab, either select a preexisting key pair, or create a new pair. I’ll creating a new key pair called geoserver – enter geoserver in the name for keypair text box and click Create and Download your Key Pair. Save the .pem file to somewhere safe.
• In the Configure Firewall tab, as we’ll be running GeoServer on top of Tomcat on port 8080, we’ll select Create a new Security Group. Name the Group, and add the following ports: HTTP (TCP, 80), and SSH (TCP, 22). We’ll also need to add Tomcat (TCP, 8080), but we’ll need to set this up as a custom rule later on.
• Click Continue, and we’ll be at the Review tab. Check everything looks OK, and click Launch. You can now click Close, and click on the x Security Group(s) link under My resources.
• Select the GeoServer security group we created earlier, and add the Tomcat rule (Connection Method: Custom, Protocol: TCP, From Port: 8080, To Port: 8080, Source (IP or Group): 0.0.0.0/0). Click Save.

Within a few minutes, the machine should be up and running on the web. Now we need to connect to it and do some installation.First we need to get the public DNS address of the machine:

• Go back to the EC2 Console Dashboard, and click on the x Running Instance(s) link under My Resources.
• Right-click your new instance, and select Connect.
• In the popup that appears, the public DNS will be listed – something like ec2-184-72-xx-xx.us-west-1.compute.amazonaws.com.

Depending on what platform you are using locally, there are two ways to connect. (Well, actually there are loads, but I’ll focus on Windows and Ubuntu).

Connecting from an Ubuntu Linux box

• Locate your private key file, geoserver.pem
• Use chmod to make sure your key file isn’t publicly viewable, ssh won’t work otherwise:
chmod 400 geoserver.pem
• Connect to your instance using instance’s public DNS.
  ssh -i geoserver.pem ubuntu@ec2-184-72-xx-xx.us-west-1.compute.amazonaws.com
• Answer yes when warned about authenticity of the host and asked if you want to continue connecting.
• You should see something like this:

leigh@deep-thought:~$ ssh -i geoserver.pem ubuntu@ec2-184-72-xx-xx.us-west-1.compute.amazonaws.com
The authenticity of host 'ec2-184-72-xx-xx.us-west-1.compute.amazonaws.com (184.72.xx.xx)' can't be established.
RSA key fingerprint is f5:f5:0c:2e:77:9f:6a:82:3a:33:8c:99:5a:65:e2:09.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-184-72-xx-xx.us-west-1.compute.amazonaws.com,184.72.xx.xx' (RSA) to the list of known hosts.
Linux ip-10-160-43-6 2.6.32-305-ec2 #9-Ubuntu SMP Thu Apr 15 04:14:01 UTC 2010 i686 GNU/Linux
Ubuntu 10.04 LTS

Welcome to Ubuntu!
* Documentation: https://help.ubuntu.com/

System information as of Wed Jun 16 21:59:44 UTC 2010

System load: 0.06 Memory usage: 2% Processes: 54
Usage of /: 4.6% of 14.76GB Swap usage: 0% Users logged in: 0

Graph this data and manage this system at https://landscape.canonical.com/
---------------------------------------------------------------------
At the moment, only the core of the system is installed. To tune the
system to your needs, you can choose to install one or more
predefined collections of software by running the following
command:

sudo tasksel --section server
---------------------------------------------------------------------

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@ip-10-160-43-6:~$

Connecting from a Windows box using PuTTY.

• PuTTY doesn’t use .pem files for authentication – instead we need to create a PuTTY private key file (.ppk) using the PuTTY Key Generator (PuTTYgen). A guide to doing this can be found here, but the steps required are.
  • Download and run puttygen.exe from the PuTTY download page.
  • Click Conversions->Import Key Pair, and select your .pem file. Click Save private key. You can password protect it at this point if you like, however in my humble opinion, you probably don’t need to if you treat this file with the same care as you would the .pem file. We’ll save the file as geoserver.ppk.
• Fire up PuTTY, and enter the public DNS in the (Host Name or IP address) text box.
• In the Category tree, select Connection->SSH->Auth, and Browse for the Private key for authentication, selecting our newly created.ppk file.
• Go back to the Session category and save this session if you like, and then click Open.
• Click Yes when warned about verifying the identity of the server, asked if you want to continue connecting.
• Login as the user ubuntu.
• You should see something like the following:

login as: ubuntu
Authenticating with public key "imported-openssh-key"
Linux ip-10-160-43-6 2.6.32-305-ec2 #9-Ubuntu SMP Thu Apr 15 04:14:01 UTC 2010 i686 GNU/Linux
Ubuntu 10.04 LTS

Welcome to Ubuntu!
* Documentation: https://help.ubuntu.com/

System information as of Wed Jun 16 22:32:12 UTC 2010

System load: 0.99 Memory usage: 2% Processes: 56
Usage of /: 4.6% of 14.76GB Swap usage: 0% Users logged in: 1

Graph this data and manage this system at https://landscape.canonical.com/
---------------------------------------------------------------------
At the moment, only the core of the system is installed. To tune the
system to your needs, you can choose to install one or more
predefined collections of software by running the following
command:

sudo tasksel --section server
---------------------------------------------------------------------

0 packages can be updated.
0 updates are security updates.

Last login: Wed Jun 16 21:59:45 2010 from xx-xx-xx-xx.dsl.sta.inspire.net.nz
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@ip-10-160-43-6:~$

Installing necessary packages

We’ve now got a vanilla Ubuntu instance running, and we’re connected. Now let’s install the packages necessary to get GeoServer up and running.

• From the terminal, execute the following:

• sudo apt-get install unzip lynx tomcat6 tomcat6-admin

• sudo vim /etc/default/tomcat6
  Find, uncomment and modify the following lines:

JAVA_OPTS="-Djava.awt.headless=true -Xmx512m"
TOMCAT6_SECURITY=no

• sudo vim /var/lib/tomcat6/conf/tomcat-users.xml
  Modify to contain the following, substituting your own super strong password. Be sure to remove any comment block surrounding the <tomcat-users> section if one exists. (Thanks to jvangeld)

<tomcat-users>
<role rolename="admin"/>
<role rolename="manager"/>
<role rolename="tomcat"/>
<user username="tomcat6" password="some super strong password" roles="admin,manager,tomcat"/>
</tomcat-users>

• sudo /etc/init.d/tomcat6 restart
• We’re now ready to login to Tomcat’s administration interface and install GeoServer.
• Download the Web Archive from GeoServer’s Stable download page.
• Unzip the downloaded .war.zip file.
• Browse to your new instance’s tomcat administration interface, e.g. http://ec2-184-72-xx-xx.us-west-1.compute.amazonaws.com:8080/manager/html
• Log in using the tomcat6 and some super strong password username and password combination specified earlier.
• Under WAR file to deploy, upload the unzipped geoserver.war file you have just downloaded, and click Deploy.
  • This file is about 40-50MB in size, so this step can take a while depending on your connection speed. You’re waiting to see both the message OK in the top of the window, and /geoserver being listed under the Applications list on the Tomcat Web Application Manager.
• Browse to your GeoServer instance, e.g. http://ec2-184-72-xx-xx.us-west-1.compute.amazonaws.com:8080/geoserver/web/
• To check things are OK, lets do a sanity check by clicking on Layer Preview link in the left hand pane.
• At the bottom of the list of configured layers, there are some entries named tasmania, spearfish and tiger-ny, next to which are some OpenLayers links. Click the spearfish link, and you should get a simple OpenLayers interface showing some test data.
• Click on the tiger-ny OpenLayers link, and you should get a simple map of New York…
  • …except you won’t – you’ll get a bunch of blank image tiles in an empty map. This threw me for a bit, so I had to delve into the GeoServer error log. You can skip to the solution below, but if you want to see where some of the error information is logged, see the steps below.
    • Open up a terminal session to the server if necessary, and type the following:
      

      vim /var/log/tomcat6/catalina.out

      You should see, near the bottom of the log file, some telltale entries like the following:

      16 Jun 23:29:42 ERROR [geoserver.ows] -
      org.vfny.geoserver.wms.WmsException: org.vfny.geoserver.wms.WmsException: Rendering process failed
      at org.vfny.geoserver.wms.responses.GetMapResponse.execute(GetMapResponse.java:447)
      ...
      ...
      Caused by: java.lang.Error: Probable fatal error:No fonts found.
      at sun.font.FontManager.getDefaultPhysicalFont(FontManager.java:1088)
      at sun.font.FontManager.initialiseDeferredFont(FontManager.java:960)

    • Google came to the rescue – this being a headless server, it has no fonts installed.
    • In order to find out what fonts Java is expecting, type the following:
      

      vim /etc/java-6-openjdk/fontconfig.properties

    • This yields the following line amongst others:
      

      serif.plain.latin-1=DejaVu Serif

  • To install the missing DejaVu font:
    

    sudo apt-get install ttf-dejavu
    sudo /etc/init.d/tomcat6 restart

  • Try the New York map again, and you should get a map of Manhattan.

• Note that the maps being displayed are non-tiled, and generated every request. Click the small menu icon in the top left of the map to enable tiled mapping (taking advantage of local caching), and change the map canvas size.

Next steps

I’ve kept this short so that it details the absolute minimum steps required to get GeoServer up and running on an AWS Ubuntu instance.

So the next few bits – I need to get working are:

• Getting some base map data – ideally I’d like to get this from OpenStreetMap‘s OSM export facility, probably using osm2pgsgl. I haven’t got this working yet,  so in the meantime, I’ll probably download some Shapefiles from the great koordinates repository, which has loads and loads of data, especially for New Zealand.
• Running direct from PostGIS is going to be a better solution than from Shapefiles, and more manageable.
• Then, I need to import some TFW raster files, probably using something like GDAL.
• Making use of GeoWebCache to pre-render and cache map tiles on the server.
• Last, but not least, securing and optimising the installation.

Hopefully I will write up these steps in the coming weeks.

Some other useful references:

• GeoServer documentation
• Building a Headless Linux GeoServer Box by Dave Smith