Jeremy Jarrell is Stretching Out for some Breadth. Here here!

Popularity: 16% [?]

K. Scott Allen recently published a five part series on the importance of version control in creating and maintaining the database behind your product. This starts with something pretty important and fundamental, three rules for database work. Rules #2 and #3 are vital. No argument there. Rule #1 I’ll come to later. Jeff Atwood has also written on the subject before, and highlighted his previous post and a few further comments in support of what K. Scott Allen had written in an apparently unscheduled post on his blog. Looking at the trackbacks and comments on the posts, this appears to have generated a lot of interest, and I feel compelled to critique the posts.

Firstly, you must get your database under version control if you ever plan on releasing more than one version. You must have an authoritative source of schema and procedures etc, you must do versioned releases. This is not in contention. But other points in K’s series and Jeff’s unquestioning support for this are.

Secondly, some background, I’m a Development Manager for a Microsoft Technology Stack web based application which is maintained and released as a product. We have a lot of tables, a lot of stored procedures and other database entities. And a lot of developers.

Taking the approaches outlined by K in his posts on Change Scripts and Views, Stored Procedures and the Like on face value, they are hopeless. They’ll get you into trouble. Fast.

K’s process is one script for the database schema and one script per stored procedure in the master definition. He then references Phil Haack’s Bullet Proof Sql Change Scripts post for an idea of how to provide better change programs than his simplistic ones.

Phil’s stuff is good in principle, making sure that your SQL Change Scripts can execute many times, but, they are laborious. We’ve rolled all the checks you could possibly want into a set of functions dbo.ColumnExists(TableName, ColumnName). dbo.IndexExists(IndexName, TableName) etc.

But with a system like ours with thousands of stored procedures (literally) and doing a re-deploy of all sps is painful. So we script only the updates and roll them out. But that will be a lot of updates in a release designed to take any version of our oldest supported version up to our newest major release (iApplication XP Panoramic Web 2.0 Version). Executing each of these individually and tracking the results is a hassle.

The next step is to roll these scripts together into one to make a quick deployment, just get the installer/dba to execute “Update_1.1.0.1.sql” on the target database and there you go. *

Only that’s when it gets really problematic. SQL Query Optimiser pulls the rug out from under your feet. What it will do is change the order of execution to optimise it. It’ll create new tables, add rows and script stored procedures at times to suit it’s own optimisation desires. And then everything will fail. Columns won’t exist, so stored procedures won’t compile. Views will fail. Commands to add indexes will explode. There will be bits of schema all over the place.

So then you have to wrap every alter table command, every DROP usp_ and CREATE usp_ in sp_executeSQL. Which means you have to escape every ‘ character correctly. So then you need to write a tool to generate the change scripts cleanly.

Then, you find that if you are executing hundreds of changes in batch in a single change program, you’ll find that it’s hard to stop a change logging itself when only part has fallen over, so then you have to wrap each item in a check for @@ERROR to see if anything went wrong. And wrap the whole lot in a transaction and roll that back if any errors happened at all.

Right, so now we have something approaching bullet proof. Now we need to make sure our team of developers do it right every time. We write a process document and run a training session. We explain how each development starts with getting the latest SP/Table definition from Source Control making your changes, generating your change program. That your change program must be tested and shown to execute cleanly leaving the full trace etc, and must contain the right assigned version number.

Whoever writes this change script will test it thoroughly and against a variety of test data, then commit the change script into source control. The schema change is officially published. The schema change will start to appear in developer workspaces as they update from source control, and on test machines as new builds are pushed into QA and beyond.

That just doesn’t work. You will despair as even your best guru programmers, the ones who really care about their craft, make mistakes and take short-cuts because the process is onerous and is seen as a tax. So then you need to have a nightly build process that restores a clean database, executes every change against it, checks the results and the database. That runs a parse on the change program beyond the SQL Execution to make sure the rules are adhered to and Change_1_0_1_0_2.sql actually records itself as 1.0.1.0.2 and not 2.0.1.3.4 which is it’s number on a different branch etc etc. K’s text is incredibly hand-wavy.

1. Never use a shared database server for development work.

The convenience of a shared database is tempting. All developers point their workstations to a single database server where they can test and make schema changes. The shared server functions as an authoritative source for the database schema, and schema changes appear immediately to all team members. The shared database also serves as a central repository for test data.

Like many conveniences in software development, a shared database is a tar pit waiting to fossilize a project. Developers overwrite each other’s changes. The changes I make on the server break the code on your development machine. Remote development is slow and difficult.

Avoid using a shared database at all costs, as they ultimately waste time and help produce bugs.

So having a single instance per developer on their local machine is a panacea for all your “shared database” problems. All those bugs created by people over-writing each other’s schema changes etc.

What about when a developer neglects to update their local instance and produces a fix based on an out-dated schema or stored procedure? Same thing. How do you ensure all your developers are keeping their local instance sufficiently up to date? Auto-update their instances with each commit as it’s stabilised? What if that wipes out their changes?

I’m not saying a shared database solves these issues, or that you won’t run into the issues he mentions on your shared database. But it’s only one point to control. We periodically re-stabilise our test and development databases where developers are patching their work for peer testing. We’re working on improving this process all the time. Some development requires an isolated environment to avoid breaking everyone’s ability to work, and they do have isolated local environments, but only for the length of that development.

Database version management is an incredibly hard problem to resolve. And although getting people started with it as K and Jeff have done is good, it’s not enough. You have to go further. And if you’ve gone further than us, please tell me where we go next!

* - (Using isql with the right voodoo to supress all the line numbers and pointless messages just leaving us the completion state, piping the output to a text file which we can parse to ensure that everything completed AOK and the database is not left in an inconsistent state, prior to us then needing to validate that the database IS actually in a decent state and the scripts haven’t falsely reported success…)

Popularity: 22% [?]

I was reading a blog entry about hot technology in Java over at Manageability.org. The second paragraph in the entry slapped me out of my non-blogging frenzy with it’s wrongness.

It’s been suggested that Polyglot programming be in the list. Even though I do subscribe to the notion that learning other languages are beneficial to one’s craft, it simply is not pragmatic advice. It is not practical to recommend that someone study Ruby, Groovy, Scala and who knows what other language is vying for your attention. Stick to a couple of languages and do it well. Some languages are better than others for certain tasks. However the biggest fallacy of all is that, a dynamic language is not considerably better than a static one. It’s no magic bullet.

The main thing to note, is the paragraph is not completely wrong. The author does note that “the notion that learning other languages are beneficial to one’s craft”, but unfortunately caveats that with it’s just not pragmatic or practical and that the reader should stick to a “couple” of languages.

I strongly disagree.

As a professional programmer, in your day job you should code in one technology set. Note I say technology set, not language. For some that may be one language. For others that may be several. For me, the last time I was a hands-on-programmer as my day job, that was Javascript, CSS, XHTML, ASP(VBScript), Visual Basic 6 and T-SQL.

You should strive for a deep and complete depth of understanding of that technology set. This should clearly start with a basic competency of the limited set of those technologies that relate to the product/project you are working on. But you should deepen and broaden this understanding as fast and well as humanly possible.

You should know how to do anything that VB6 can do, not just within the context of your web development. You will need to learn the aspects of VB6 programming that can never be used in a web context, but along the way, you will learn many things you would not have otherwise learnt. These things may be things you can directly use in a web context, or things that just improve your approach to problem solving, design and development issues, a fresh perspective on the language.

The next step from here is to take that solid grounding in your primary weapon and mature and expand it with exposure to other languages and technologies.

The development communities around each language are akin to separate nations. Sometimes diplomatic channels are open and citizens freely move between the nations. Other times there is open hostility. Each nation has it’s own way of life. There is always some common ground between all languages, but, often between them they have vastly differing ways of approaching a common problem. Continual exposure to these different languages opens you up to more ways to solve the problems you are faced with. You will be able to deal with a vastly wider range of problems as a result.

And this is a critical skill to develop.

Do not let yourself become an island nation. Have that deep mastery of a key language/technology set and use it daily, but make sure you are also constantly looking around for other languages and ideas to broaden your understanding of your craft. Travel widely. Use the languages for “real” in anger development to really understand the different pain points. Ruby may solve one pain for you at the cost of other deeper pains.

Without you, and people like you, doing this language tourism, building this breadth, there won’t be a new top 5 interesting technologies in [whatever language] in 2009, because the [whatever language] community will stagnate as it examines it’s own navel.

Popularity: 24% [?]

The Zend Framework provides an interesting set of PHP5 libraries for caching. There’s a nice architecture to it, providing a number of different backends and frontends for caching. However, I recently found it very frustrating to try and figure out a decent caching strategy for the new version of a site I was working on.

And the documentation did not help at all. So, allow me to elaborate for the benefit of the huddled masses.

The introduction in the caching section of the manual gives a decent enough overview of the basics, if I want to cache a page with a nice simple ID, such as “Page1″ with a set lifetime I can do so in a few lines of code.

However, another page goes on to mention how you can also “tag” records with multiple tags. Another page talks about how to clear the cache by a single tag, or combination of tags.

But nothing explains the relationships between tags and ids, and how the clear works with tags or ids.

Now, I’m working on a system which has two views of a music catalogue for a radio station. There is the requests system and there is the discography system. They both present differing views of the same data. The request system filters the list of artists, albums and tracks on the station to those that can be requested and displays a “request optimised” set of screens with some of the data. The discography section shows everything about an artist, all their albums, all tracks, reviews and so forth, without the cruft needed for the request sub-system.

We cache these screens for obvious reasons. What we need is the ability to clear the cache of an artist, album or track possibly within the requests or the discography, or both. So, I figured on a system of unique keys like artist_123 and album_123 etc then to use the tags to “lump” things together. So album_123 would have the artist_123 tag in both discography and request view plus the discography tag in the discography and the requests tag in the requests view. Something like:

$cache->load("album_123", array("album_123", "artist_123", "requests", "en_GB", "album");

I could then simply invalidate the cache of all albums, request pages, artist_123 pages or British English generated pages or any combination of those items

This does not work.

The first important thing is that the ID is the unique key. Not the combination of the ID and the tag(s). So if you save page1 to the cache with the tags tag1 and tag2, then try and load page1 from the cache with tags tag3 and tag4, you’ll get the result of saving with tag1 and tag2!

Insane, but true. Try it. The tag has no effect whatsoever it seems on the load code. If it can find an item by ID, then it loads it, irrespective of tags. I’m not sure if this is behaviour by design, or a bug on my system using the file backend, but it is consistent. I just think it’s mad.

To get what I desire, I’m going to have to cache with:

$cache->load("requests_artist_123_album_123_en_GB", array("requests", "artist_123", "album_123", "en_GB", "albums");

Then I can still do a clear on the appropriate tags, if for example I want to remove all albums from the requests cache:

$cache->clean(Zend_Cache::CLEANING_MODE_MATCHING_TAG, (array("requests","albums"));

It’s a bit of a pain in the rear, but, once you’ve figured out the $tags argument on the load method is pointless, it’s fine.

Of course, figuring all this out was further compicated by the fact that the examples in the manual often use load() with $key and $tags and save() with no arguments. I assumed therefore that the point of the $tags argument on load() was to set the tags that would be used auto-magically on save(). Only, if you don’t pass $tags to save() it saves with no tags. Which is also silly, since it does respect the $key used in the load() method.

Popularity: 40% [?]

I’m often faced with the need to do a one off crunch of data to provide answers to questions management ask about raw data. Not the kind of thing they’ll be asking for on a regular basis. Just a need to scratch an individual itch. One off reports on specific aspects of code metrics. Calculate some predictions of data growth in the application across different aspects of it’s user base.

On these occasions, I either turn to Query Analyser to mine our SQL Server databases directly, or use the data import tool in Excel and try and crunch the data swiftly in that. Sometimes, those needs become a long running need to manage some data, where Excel is often the preferred format, because I can do some initial crunching and manage the data in there and the rest of management can then take a copy and further manipulate it and play with it to get additional information as and when it occurs to them they need it.

The problem I face is that Excel is designed for accountants and management types with no programming knowledge to manage spreadsheets of data they understand. It’s too damn user friendly. I find it very hard sometimes to find a good way to manage my data in Excel. I often throw my hands up in dispaire and lash up a software tool specifically to manage the data. I’m talking about a full on database driven web application in most cases. It’s so much faster for me to work with the data that way, and I can then use the Data Import tool in Excel to shove the data in raw forms into spreadsheets if the people asking for the data want to take it away and play with it.

There has to be a better way. There has to be a more productive way for me to do this. A more developer focussed tool for doing this, that allows you to achieve with scripting/programming what you would achieve in Excel by mucking around with excessively user friendly wizards and obscure dialogue screens.

John Udell thinks the answer might look something like Resolver, which is a new spreadsheet application written in Python that allows you to use Python directly in cells and to have full access to .NET and IronPython through the whole application.

This just seems to be the wrong solution to me. With Excel we have a spreadsheet product that is so good it’s destroyed all competition that non-programmers use and love. It can be extended by programmers with add-ins and macros. You can write .NET code or VBA code (easy for non-programmers to learn) in the Macros etc. However, the formulae are restricted to the old style “icky” functions. Stick=if(condition,forumlae,formulae) in, which just makes programmers recoil from the keyboard in horror.

The right answer to the question is to have a simple option to enable direct access to the .NET runtime in cells. Then people can code formulae in any .NET enabled language they choose, including IronPython.

Do not throw the baby out with the bathwater.

Popularity: 31% [?]

Having said only the other week that there is no correct answer to the question of what should go in your résumé, Steve Yegge has chimed in with his article ten tips for a (slightly )less awful resume.

Steve starts out, much like I did by pointing out that what he says only relates to recruiting people for the kinds of places he works, and may not apply outside of the recruitment of technical people for companies that produce their own software, and even within that niche, may not apply to everyone:

I’m just talking about software engineer resumes today, and specifically just the subset intended for applying to companies that build their own software. I have no idea how much (if at all) this stuff applies to resumes for other kinds of positions, or companies. Maybe not much. Sorry!

It seems the companies he’s worked for take a different approach to reviewing CVs to any company I’ve ever worked for. It seems they have a large CV screening panel, phone screens (possibly multiples) and multiple panel interview sessions. We do it differently.

Steve’s first point is right, in bullet form, no-one cares about you yet, but he also suggest stripping out anything with personality in it. Which I disagree with strongly. I do want to know what you’re like other than as a professional software engineer. I recently came across a candidate who was a Reiki healer and psychic reader. Who offered his services for £25 an hour over MSN or email. This information was gleaned from “personal interests” on his CV and listing his personal webpage address. Valuable information saving me from interviewing a total fruit-cake.

Steve’s point 2 asks for unformatted ASCII CVs. This is because where he works they pump them through automated manglers. That is not true anywhere I’ve applied for a job or worked in my life. It’s true that Pimps screw them up, something I was planning on writing about soon, but, in general I’d prefer to see a nicely formatted word doc, that has had some care and attention lavished on it than a nasty plain .txt file.

Points 3, 4 and 5 have merit. Don’t set off my bullshit detector with too much weasel and wank talk. Make sure it’s spelling and grammar checked, if it’s a word doc, and a UK install f Word has put a red squiggle under anything other than an obscure techie term, you’re a moron. But, don’t rely on the spell checked, I’ve seen far too many CVs with spell-checking mistakes.

However, my biggest contention is point 6, certified looser. If you are a contractor, get certifications, do them in your own time, it shows your skill, gives me more confidence you have at least a basic level of understanding of the subject and technology. A safety net. it’s also quite common for UK companies to put people through certification courses to ensure their staff have that rounded skillset, that they know the right way of doing things, and to give the employee something back to move their career on. Certification is great.

But, just a few things there, the certifications, the format of the cv, the lack of personal information. Submit a “perfect for Steve” CV to me and I’ll reject you. Submit a perfect for CV to Steve and he’ll reject you. It’s impossible to get this right. The single best tip in Steve’s article?

You can apply for 18 jobs, but you should send 18 different resumes, each targeted at that job, and you shouldn’t send them all at once.

Tailor it for the job. It’s web development? Emphasis your experience in that area, and trim down the references to the Linux Device Driver work you’ve done. And try and get a feel for the company you’re applying for. If it’s a Big One, like IBM, Microsoft, Amazon, Google etc, there is plenty you can find out about what the recruiters are looking for. Talk to the Pimp, ask them what kind of CV the recruiting managers like to see.

Do your research first, it really helps.

Popularity: 24% [?]

A very popular question people ask is “what should I put in my C.V?”, the problem is, there is no correct answer to that question. There are a few consistent formats I see to the documents I see, but no set pattern I can figure out behind what has caused people to use that pattern.

Two of the guys I’ve worked with in the past and have equally successful careers, following similar tracks, who are of similar age and position in their career, have totally different approaches. Clive goes for the “everything in a good level of detail” approach. Pete’s CV is two sides, and he thinks it needs trimming down.

I think Pete’s CV comes from the approach that it seems a number of managers take, they want to get the details quickly and don’t want to wade their way through a load of self-promoting rubbish to get to them. Clive’s approach comes from the desire to give all the information so that every base is covered and people won’t miss any of your brilliant skills.

Personally, as a recruiting manager, I like to see a good level of detail on a CV. I want enough to really know whether it’s worth seeing the candidate or no. I want to get a feel for the candidate in advance. I want a chance for them to make mistakes.

But not all managers want that. They follow the Pimp Path, they want to find a few keywords, such as 10+ Years C++ experience on Windows Drivers. Or whatever. If they can’t get this information, they’ll reject the C.V.

The big problem is you don’t know what kind of manager your C.V. is going to.

I’m starting to come down in favour of a hybrid approach. The first page of the C.V. should be a totally lightweight highlights and basic facts page. With detail to follow.

Something that is also important to understand is what information you do need on the C.V., what order and level of detail to present it, and unfortunately that changes based on where you are in your career. For a graduate, clearly education history should come first, it’s the most relevant information. For an experienced engineer, the education history gradually becomes less and less important. But I still feel it’s necessary to include that information.

I think a front page should probably detail personal details, in a small contained area, current role and a brief statement of what you’re looking to achieve in your career. If there is room, a brief summary of key technical skills. Brief please, and the key ones. Not an exhaustive list of everything you’ve ever worked with.

Then, follow this with a career history, education history and some personal stuff.

With the later part, people are rather conflicted on too. Some people think a bit of “personal interests” adds something to a CV, others think it’s a no-no. Which seems odd, because technical skills are not everything, you also need to know if that person is going to fit in with your team and company. Are they going to have the right attitude to work? All information that can be indicated very roughly from the information given there.

So, summary, and then a set of detail. Attempting to keep as many people happy as possible. Really, an example would help. But I don’t have one to hand. And this is the worst article I’ve written ever. I’ll try and come back to it when I’m more alert.

Popularity: 29% [?]

I don’t know what it’s like round the rest of the world, but, in the UK most recruitment is done by asking recruitment companies to find candidates. The recruitment companies place the adverts, find CVs that are relevant and submit them to the hiring company.

This makes them salesmen. A typical recruitment company will charge anything from 15% to 40% of the candidates starting salary as a finding fee. That can be a lot of money. The role tends to attract younger people. It’s very high pressure as well as high margin. Very competitive. This all means that your recruitment consultant is worse than a double-glazing salesman.

They are much more like Pimps.

So, that is how they are referred to in my world. I deal with a lot of different agents. They all give you the same spiel, they’re different from the other agencies. They work this way and that way. Not like the others. Which is always a lie. They are all the same at the base of it, they are salespeople. They all work in much the same way. Their ethical boundary is slightly off from most peoples.

That is not to say they are all the same. Some agencies are better than others. Some agents are better than others. The larger agencies tend to work on volume. They want me to give them half a dozen keywords, some as “essentials” some as “Nice to haves”. They plug this into a propriatory keyword search engine and blast you everything they find without speaking with the candidate. Other’s actually take time to understand what you are looking for and try harder to find them.

But, once you’ve shown interest in a candidate, nearly all of them will do anything to close the sale. They’ll miss-represent me to you and you to me. They can see the sale just out of reach and go for it.

Take a little while to read up on hard-sales techniques. Then see how many you spot after an interview where the company was interested, but you weren’t. It’s shocking really.

You must remember, this is your career, the wrong job for you could send you down a dead-end, or a path you don’t want to take. Of course, you might not be too bothered about switching technologies or career path. But if you were then you wouldn’t be reading this.

Popularity: 14% [?]

The single most important document you will ever write is your Résumé or Curriculum Vitae (C.V.) . That bears saying twice. The single most important document you will ever write.

Ok, maybe your Will seems more important to you as a family person. But I’m not convinced. Your CV gets you the next job. The next step up. It enables you to be in a position to write a better will.

There will always be people who never ever need to write a CV. That just network from role to role. But that’s an edge case.

I’m talking about applying for software engineering jobs. So hopefully you reading this are a software engineer. I’m sure you therefore understand the importance of accurate, complete and up-to-date documentation for any software you are working on.

And if you don’t, go away and find out and don’t look for a job until you do.

Your CV is the same, only the accuracy and attention to detail of your CV is far more important to you. Forget to update the entry for your previous role so the “to date” says “present” and things like “My current job is” in a job 4 jobs back, then you look like a Muppet and won’t get an interview for any role that needs attention to detail on documentation or otherwise. i.e. a job as a software engineer.

Get a collegue, ex-colleague or friend who is a software engineer to read it through and make sure you’ve not messed anything up. I’ve seen CVs where people are proud of working on three tire systems (that would be tier then), sequel server and various other blatantly stupid things.

Your CV is the only tool you have to get you through the door. Sometimes there is a covering letter, an email to send it in, an online application form to attach it to. Make sure that what you write there has the same level of attention as the main document. Don’t let your CV down with a poor submission of the CV.

Pay attention to detail.

Pay attention to detail.

Popularity: 14% [?]

I have been recruiting for my company for two years. We had 6 software engineers in 2004 in my department. We now have 45. During the last two years, 2 of the original 6 have left and I have had 3 new hires quit. One of the 45 wasn’t recruited by me. So doing the maths, that’s 43 people that I have hired. Add to that the 10 contractors. 53.

But to hire 53 people, I have interviewed over 100 people. To interview over 100 people, I have seen over 500 candidate submissions. I’m dealing with at least 10 different recruitment agents. Candidates referred by existing staff. Candidates applying via our web site.

And on top of that, I’m managing the development of a major software product. On the support rota. One of the key firefighters on issues that arise. Sorting out office logistics. Buying development servers and software. Recruitment is not my full time role.

I got involved because they needed a technical person involved in the recruitment and ended up running the show. I have it down to a fine art. I can tell with a great degree of accuracy whether or not it’s worth interviewing someone from their initial application. We follow up with a technical test to be sure, then an in-person interview with me and someone else. But I would say at this point, I’d be 75% certain that I could say yes or no from the application and skip the rest.

So what I’m going to set out over the next few weeks is basically going to be how to get me to hire you.

The fundamental problem with all this advice is I can only be sure it applies to me. Every company has it’s own methods of recruitment. My advice might also work with other small to medium software engineering companies that rely on the technical managers to do the recruitment for the technical teams. But it certainly will not map to applying for a major company such as a massive multi-national bank.

So, before I start on anything else, perhaps we ought to look around and see what else is there to help you outside of my field. Because if you aren’t looking for a job in a company like mine, no advice I can give you is going to help. Because it’ll be too specific to working in a company like mine. Hopefully it will be in the right direction, and will help, but there will be some other different stuff.

So where else to go? There are numerous websites around with advice on your application and so on. Google them. There is a lot of information around about how the Google Recruitment process works. A lot of information about the Microsoft process. But, that’s all written from the point of view of giving the applicant generic advice. My advice is to look at it from the other point of view. Find articles about “How to recruit effectively”, written by people who do the recruitment for other recruiters. Find the inside track on how they’re being advised to find people.

When I started recruiting I looked for this kind of advice first and foremost. I started with Joel’s Guerilla Guide to Interviewing. I found this to be essential reading for me. I respect Joel as an expert on software. He’s an engineer leading some great development, and he needs to find more great engineers to provide that development. I’ve taken on some of his core points. I’m looking for Smart People who Get Things Done. I’m ensuring there is at least two people in every interview. If anyone says no, it’s a no.

But I’m not taking it all on board. Some of it doesn’t, and will not, work for my situation. His article on Phone Screening also has good information in it. I don’t phone screen however.

He also has a number of articles on your side of the fence too. I’d start with Getting Your Résumé Read.

Anyway, the point is, figure out what we’re trying to figure out. How do we ensure we find and recruit good engineers. Know your enemy. That’s a starting point.

Popularity: 12% [?]