I recently spoke at Code Slinger on the subject of scaling engineering teams through periods of rapid growth.
Hiring quality developers is a hot topic, you can’t move for guides on how to get the very best to work for you. In fact it is such a hot topic that many writers on the subject forget why recruitment is on the table in the first place.
In order to be successful, a business needs to be able to build highly productive, durable teams that can adapt to changes to the business of over time. To do this, good recruitment is essential, but is only a small part of the whole.
Start ups pride themselves on disrupting the status quo, reimagining products and processes to gain an edge over incumbents. This has led to rapid innovation in engineering, product and marketing but HR, and relatedly talent management, has for the most part remained static.
What would happen if talent management was re-imagined from the ground up where the only metric to optimise for is retaining quality teams? How would the roles of recruiters, HR and functional managers evolve in response?
I’ll go through four suggestions of what a retention optimised system would look like. The conclusions are counter-intuitive and in some cases directly conflict with current recruitment good practice. Optimising for retention means that a company will undoubtedly lose out on quality staff in the short term.
I should say that every suggestion is something that I have implemented myself in multiple teams. I have run each over a sufficiently long period to believe that it supports retention in the long term. For arguments sake I will consider ‘long term retention’ to mean a company tenure of greater than five years.
Since I’m writing from my own experience, there is a bias towards software engineering organisations in a startup environment. Though I think many of the lessons can be applied more generally.
Before I get there though, let’s just recap on why retention is important.
Why is retention important?
- Long term Investment in staff possible
- Get you through the hard times
- Much cheaper…
Retention is critical for building a strong culture. If you want a culture to persist you need continuity, especially from your most influential staff. I’ll talk about culture in detail later.
Secondly, an assumption of long term retention makes it easier to invest in people. This means that they can improve at their job, also improving the team and subsequently improving retention… It also means that every part of the business is likely to have good industry domain knowledge. So not just an expert on Software Development, but also knowledgeable about the fashion industry or the Telecoms industry (say). This domain knowledge allows the team to make better decisions on a day to day basis.
When everything is going well in start ups, the team can forgive most things, but every company goes through rough patches. Perhaps there is concern over a new direction or growth is slowing. It is at these times when you really need a team to pull together to get the company through.
Also, it’s cheaper. All manner of management consultancies have tried to put a cost on replacing a member of staff, they seem to be landing at between one and two year’s salary – I’ll not argue.
In short, companies that can develop and retain their best staff will have a clear advantage over those that do not.
Hiring for retention
With that in mind, there are four areas that I’d like to discuss further.
- Embedding recruitment into the culture
- Sell the company, not the role
- Hiring exclusively for fit
- Don’t mess about with money
Embedding recruitment into the culture
Every recruiter I’ve ever talked to says that they really benefit when the hiring manager is engaged in the process. How do you actually go about making this happen? There is only so much a recruiter can do.
Recruitment has to be everyone’s business, and management should make sure recruitment activity is a first class citizen.
This means two things:-
- Make working in recruitment aspirational, reward recruitment skills like any other skills. If one of the team is spending 20% of their time on recruitment and their peer is spending 0% then comparing them solely for their engineering output sends a very strong message that recruitment doesn’t matter.
- Recruitment is hard, so you need to find ways to help people improve. This could be as simple as adding a high level of transparency to the process and give constant feedback. Whatever you do, people must feel that they are learning a skill that is valued highly in the company.
To be honest I think that embedding recruitment into the culture is something that is always a good idea, regardless of how you are recruiting. So not especially controversial, it just requires sufficient on the part of management. Onwards.
Sell the company not the role
The role will undoubtedly change, the company will not. If the goal is to focus on retention rather than simply getting the best candidates through the door, it is most important that the candidate is sold on the company.
This can create a conflict since a great way to close a candidate is to help them imagine what their new job will be like. This means going into detail on the specifics of the role, the immediate project etc.
If instead the bulk of the sell is around the company, it is entirely likely that you will lose out on quality staff, though those that you do hire will be that much more durable through the inevitable twists and changes that will occur in coming years.
It is worth noting that this idea is well established. It is relatively easy to do in a startup context, since a company’s mission is likely to be extremely important to the candidate, but it is also possible at much larger organisations. Facebook, for instance, wait until after their 6 week bootcamp before pairing staff up with their eventual teams.
Selling the company not the role will help with long term retention but it will definitely mean losing out on capable staff in the short term. If a recruiter is assessed solely on the number of new hires they can get through the door, it is unreasonable to expect them to do this.
Hiring exclusively for fit
In 1974 following Mick Taylor’s departure, the Rolling stones were in need of a new guitarist. The Rolling Stones were hot, they could have talked to anyone they liked:- Clapton, Beck … but they went for Ron Wood of the Small Faces. Perfectly competent but the not the best guitarist available. The reason they went for him was that being guitarist in the Rolling Stones was much more than simply playing guitar, just being in the band was a pretty tough gig. Ron got the job because he could handle all the other baggage that came with being in the Stones. A pretty good hire in the end and it is fair to say that the Stones know a thing or two about retention.
So here I want to talk about fit and culture. These are terms that abused quite a lot and different people mean different things by them. So here’s what I mean when I say culture. It is simply what is rewarded, what is tolerated and what is punished and it is the company’s leadership, formal leaders as well as informal leaders that set it and maintain it.
Strong culture is essential for retention because it is culture that bind people together. Life is easy when all the graphs go up and to the right, but it is during the periods of slow growth that you really need the team to pull together and it is culture that will get you through.
The most important thing before starting to hire for fit is know very clearly what culture it is you want to propagate. You don’t get to choose of course you just need to take a look around and look at the traits most likely to be supported by leadership (again I stress both formal and informal).
It is really important to write down what is meant by fit and then live it. If you don’t then fit just becomes ‘people like me, or people I like’ – this has implications for diversity which will likely ruin your business. If you can focus on just a few traits then it should actually help build a diverse team since all that matters is satisfying these specific traits, everything else is open.
So this means that fit needs to be a first class citizen to any other area you are selecting for. As important as coding, as important as architecture.
Through experience, I’ve found was that until you pass on a really good candidate purely on grounds of fit, your team won’t really believe that you are serious. If fit is the reason for rejection, then call it out loud and proud.
Once the team really understand how important a factor fit is, then you need to help them get better at assessing fit through interview. This goes back to training and feedback.
If you hire for fit, and reject those who do not, then you will lose good people. I’m not talking about passing on just really obnoxious, toxic people (though you should). I mean people who are perfectly capable, but won’t help support your culture in the long term.
Don’t Mess About with Money
What this means is that you need to put the needs of your current staff (who, you know, you’re trying to retain), ahead of someone who doesn’t work for you yet.
It means that if you are bringing in someone to work as a peer with existing staff, their salary must be in line. And sometimes this means you will lose out on good people, either because you assess them at a lower level than they assess themselves, or because you are simply not paying enough. In the latter case, since you care about retention you need to have a serious think about giving everyone a boost. Expensive stuff.
But it is worse than that. You’ve just spent weeks, perhaps months unearthing a unicorn, you’ve sold them on the mission, they’re a great fit, you offered them a package that is realistic and in line with the rest of the team and, quite reasonably, they want to negotiate.
For the sake of a few thousand dollars you could lose this candidate. But if you do so you are rewarding people joining the company not for their skills to do the job but their skills as a negotiator. It is an understatement to say that your best engineers will not be your best negotiators.
Some people say, ‘oh it is fine’, negotiate and then balance things out at the next review. But this means penalising someone who has already worked for you for perhaps a year. Salary doesn’t really work in absolutes, it is much more about relative change.
This is really tricky stuff to get right, and you will certainly lose good people to money if you peg to your current team and stand firm on negotiation.
Companies that can hire for retention have a clear advantage, over those that do not. But optimising for highly productive durable teams, means losing out on good candidates capable of doing the job.
Traditional approaches to recruitment typically do not consider the long term goal of building highly productive durable teams. In fact optimising for this goal would lead to recruiters appearing less effective since they would take longer to hire.
The question at the heart of the article is this. If I’m right about retention being key, and I’m right about existing structures actively encouraging short term gains at the expense of long term retention, then how should organisations evolve to support this need. How should recruiters, HR and management adapt to a world where retention is considered key?
Over the past month a few people have asked me for some advice on where to advertise for software developers. In truth, advertising is a tiny component of a successful recruitment function, but if you are at the point where you want to spend some money on adverts, you might find the following useful.
Many of the sources are internationally relevant, but all I’m doing here is reflecting my own experiences which are exclusively drawn from hiring for start-ups in London.
It is amazing how little companies consider the costs involved in recruitment. For some reason they happily sign off on a recruitment agent with a fee of 18-25% of first year’s salary then balk at the idea of a few hundred pounds on a job board. This is especially important when considering the costs of your own time. Even in small nimble companies it is entirely possible to spend 4 figure sums in staff time on each successful candidate hired.
Speaking of agencies
Recruitment agencies get a bad press, really it comes down to the incentives in the industry. It is possible to get results, especially for more junior staff who lack a pre-existing network but I generally think there are better ways to spend the time and money. YMMV.
Most people reading this will already know Hacker News. It is a community curated content aggregator backed by start-up accelerator Y-Combinator. It focuses mainly on start-ups and technology but has a bit of everything. Every month it runs a ‘Who is hiring?’ thread where companies can advertise their roles.
Be warned, it is not the most user friendly experience and you’ll need to wait for 12pm EST on the first of the month to post. The signal to noise is very good meaning that you can invest more time on each candidate. Given the nature of the board it is common to receive very strong applications from overseas candidates wishing to move to London, so it is worth figuring out your position on relocation/visas ahead of time.
I’ve hired most of my current team through HN and recommend it very highly – though be mindful that it’s very start-up focussed, if that’s not you you’ll need to work that much harder to be attractive.
Also worth a mention is the complementary thread ‘Who wants to be hired?’ where potential candidates place ads about themselves. This thread sees much less traffic but it still an interesting path and I would recommend it as a supplementary source of candidates.
Again, anyone in software will know Stack Overflow, through their Q&A service they have built a formidable community of smart and helpful developers. As a company you can pay to post ads for a few hundred pounds a month which appear along side the Q&As. There are various options to suit a range of budgets including bulk buy and featured posts. SO also offer analytics and applicant tracking tools – though you are free to use your own email to contact candidates.
You’ll come up against some fierce competition from other companies who do a great job of selling themselves. You’ll need to have a hard think about how to position yourself, to this end SO have produced a range of docs to get you going. It is also worth investing the time in creating a company page and using that to further promote your culture. I can speak from personal experience that you will see a big uptick in traffic if you take the time to tweak your content to work well in SO’s framework.
My results were mixed in that SO introduced me to many interesting candidates (and very few inappropriate ones), however to date I’ve not hired through this channel. When I was using it most heavily I was looking explicitly for people with a demonstrable entrepreneurial background, I suspect that with more generalist requirements I would have had more success.
It is also worth noting that you can also pay to contact SO users directly through their Candidate Search product, this means access to the SO database and then reaching out to people with good standing in the skills that you are looking for. I’ve not tried this myself though it sounds like an interesting approach.
London has a handful of startup centric job boards very much worth taking a look at. I’ve used Hacker Jobs and Unicorn Hunt (it was previously 3-Beards) previously, but they all serve a similar purpose. Since they are somewhat under the radar they get a better signal to noise ratio than larger generalist boards like Indeed/Dice/Monster etc.
- Work In Startups – Free with premium options.
- Hacker Jobs – Free with £150 premium option.
- UK Startup Jobs – £5 for 90 days. £30 to post and feature the ad.
- London Startup Jobs – Free with premium options
- Unicorn hunt £200 – choose your own discount
I previously worked at an American company with a bulk deal on ads. They used it to good effect to hire technical staff in Seattle, but I’ve never found LinkedIn to be a useful way to advertise for developers in London.
I’ve not done this myself, simply because I’ve never clicked on an ad from either. That said they both allow for targeting specific groups and I’ve heard anecdotally that people have had success through this route.
Honourable mention – Silicon Milk Round
Silicon Milk Round is a recruitment fair run out of the Old Truman Brewery on Brick Lane aimed specifically at start-ups. It is a phenomenal event and the standard of attendees is very high. With so many other interesting companies present you will need to work hard to differentiate yourself. I’ve found it to be an excellent place to either hire directly, or simply to start a relationship that may lead to a hire further down the line.
A final word
Good recruitment is all about building relationships over time, this means working through multiple channels, consistently and in a way that ultimately attracts the right people and and gently (or otherwise) dissuades the everyone else. Placing adverts alone will not get you very far but can form part of a broader strategy.
A final final word
I didn’t discuss ad content but for the avoidance of doubt. No ninjas, no pirates, no rock stars, no unicorns.
So, Mr Johnson, what would you say is your greatest weakness?
Worst. Interview. Question. Ever.
In isolation, it’s an interesting question to ask yourself, it requires a great deal of honesty and openness on the part of the individual and it requires trust that there will not be repercussions for the answer.
Sadly, an interview is a place for none of those things. Candidates need, and are expected to, go out of their way to impress their interviewers and actively admitting weakness is a high risk strategy.
And yet it’s popularity remains, company’s still ask it and candidates still find a thousand and one tortured methods to say “I’m a workaholic”. In a novel attempt to get around this, one company used to ask for your three greatest weaknesses. I’d love to know what they gained from this.
So how should this question be answered? From personal experience, responding ‘My left elbow’ sadly does not work.
In comes down to framing, strengths and weaknesses are pretty much alternate views on the same characteristic. Above a certain level of competence it is rare to find people truely good at certain pairings.
For instance, think those people who you admire for their vision. These people are inspiring, charismatic they lift you up and show you a world that you dimly knew existed but couldn’t quite realise. Wonderful people, but tell me, how are they on detail? How good are they are dotting the Is and crossing the Ts and meticulously going through the minutiae of a project, ensuring that everything is as it should be. Not so much huh?
Clearly there will exceptions but Richard Branson, just isn’t a details kind of guy and nor should he be. Conversely I want an Accountant to be exactly the opposite, they need to get down to the details.
Since the question is now framed in terms of strengths it is much easier to answer. Just figure out what strength of yours you wish to highlight, determine how that makes you weak and then go for it.
Not convinced? Here is an example.
Imagine for instance the Great Remallo, Lion Tamer extraordinaire, sitting for an interview. His CV/Resume contains plenty of past experience dealing with many different types of lion and as the inventor of the lion proof cape he clearly knows his stuff. As the final question of the interview the Circus owner, Mr Top, fires out the big kahuna:-
Mr Top: Mr Remallo I have one final question for you. Tell me, what is your great weakness?
The Great Remallo: Well, Mr Top, if I had one weakness it would be that I seem to have trouble accurately assessing the level of danger I might be in at any given time, even as a small child when I used to play in traffic. Actually it’s something that really helps me as a lion tamer, I should think that if I ever truley understod the peril I put myself in everyday I would be petrified and never even enter the ring.
Still not convinced? Let’s try the same question for a Software Developer.
My biggest weakness is that I can’t stand not knowing how things work, it used to drive my family crazy as took everything I owned (and somethings I didn’t) apart in order to see what was going on under the hood. To this day I really struggle to take things at face value if I can’t look and see what is going on. I suppose it’s one of the reasons I became an engineer in the first place, this drive to understand how things works stands me in good stead when wrestling with a gnarly code base or diving deep into library code, bug hunting.
I’m sure you can come up with something better yourself. Just remember if you reframe the question as an opportunity to talk about strengths, it’s then just a question of determining the flip side to that strength and using it as an in.
Anyone else got a good answer to this question?
Phone screens are a great way to address this problem, they are typically shorter and often run solo. They allow a company to take more risks and consider candidates from further afield.
My company is still pretty new to phone screening, we’ve been trialling it out in cases where it is difficult for the candidate to attend in person – perhaps they are based overseas. As a result I’ve been doing a lot of reading on how best to construct a decent phone screen. By far the best writing I’ve found is Steve Yegge’s take. I’m not sure how practical it is to fit everything Yegge mentions into a 45 minute call, but I consider it an excellent resource.
A common fear I have seen in other discussions seems to be that candidates will use google to somehow game the system. If this is a genuine concern then one of two things has gone wrong. Either:-
- The questions are purely fact based and will tell the interviewer nothing about how the candidate thinks.
- Or, the questions are fine but the interviewer is focusing on the wrong part of the answer.
A question like ‘In Java what is the difference between finally, final and finalize’ will tell you very little about the candidate. Plenty of terrible programmers could answer that without problem and what’s worse, a talented but inexperienced developer might stumble. In short these type of quick fire questions add little value to the overall process.
Something like ‘How does a Hash Map work? How would you write a naive implementation?’ is more interesting, it’s open ended but forces the candidate to talk about a specific area of knowledge – even if they don’t know, you’ll learn how good they are at thinking things through from first principles. The only way that it can be gamed through googling is if the interviewer simply waiting to hear specific terms and is not asking free form follow ups.
I’ve just googled Hash Maps on wikipedia and could probably quickly extract ‘Associative array’, ‘key-value pair’, ‘Collision’ but really if that’s all the interviewer wants to hear then the question is of limited value.
So what I’m saying is that if you’re concerned about googling, then it’s probably the questions or desired answers that are the problem. Furthermore if one in a hundred people do manage to game the system you’ll pick them up in the face to face in an instant.
Recruitment is all about relationships and trust, whichever way you look at it common recruitment practices support neither. While there are countless articles focusing on how hard it is to hire good developers little is said about how to find good companies. Trust work both ways and in order to ‘fix’ recruitment both sides of trust equation must be balanced. Examining each in turn:-
Employer-> Candidate trust
Employers have low trust in external recruiters, low trust in CVs, and low trust that candidates can complete a Fizz Buzz question. This means that it’s not possible to invest sufficient time in individual applications, which in turn makes it less likely they’ll ever attract really good people.
Services like LinkedIn and Stack Overflow have made some gains in solving the employer-> candidate trust problem. In LinkedIn’s case they have scaled the ability to ‘ask around’ for recommendations and Stack Overflow provides a feel for someone’s knowledge. Neither is perfect and in truth the best they can do is give me confidence that the candidate is not a total waste of time.
Candidate -> Employer trust
The Candidate -> Employer problem is more interesting not least because it’s generally ignored. Unless you happen to be Google or Facebook candidate->employer trust is a major stumbling block. How can a candidate be sure that they are dealing with a good company? They can’t trust their agent to have a clue (or care) and they themselves will not be aware of a host of interesting companies. As such applications tend towards the bland and generic since candidates cannot afford to spend days tailoring individual introductions, this in turn fuels the employer perception that passionate interested candidates do not exist.
As an example, I work for a small B2B Telecoms company, our work is on the public eye, but our brand is not. Most developers will not be aware of us. Once hired, developers tend to want to stay with us, with working environment and the freedom to ‘get things done’ playing a big part in that. However as a company I have no easy way to express this. It’s not even a case of saying ‘Isn’t my company great!’ it’s much more about about describing the trade offs. Not everyone will appreciate the chaos, pace and variety of working at a small company, some will prefer the promise of a well defined career path, security and greater opportunity to specialise typicallly afforded by a larger organisation. It’s down to personal opinion.
Individual companies can solve this problem by publishing an engineering blog, sponsoring community events, getting people speaking at conferences and generally exposing their culture and values. It could be argued that companies willing to go to these lengths clearly value recruitment more highly than others and deserve the rewards. However, if there was someway that candidates could pull that information rather than have it pushed then that would be hugely valuable.
The closest example I can see is the Joel Test. To me the Joel Test is starting to show it’s age and could benefit from an update, the best it can say is ‘this company is less likely to be a horrific place to work’. Glass Door also addresses this in part, though practically speaking companies must be a of certain size before it becomes useful.
I’m not sure what the solution might be. Perhaps a curated job board/job fair is the way to go, the curator finds a way to characterise companies and makes sure it only backs good companies. This builds trust with candidates, and should mean that it attracts the top people, especially those for whom money is not the top driver. Companies are happy to pay decent rates because they know how good the candidate pool is, further more there is prestige in being associated with the agency.
So, world, here is my challenge to you. How can I, as a company, express my culture and values in a meaningful and standard way so that candidates can approach me with confidence.
Ask anyone about hiring developers and the advice is always the same ‘only hire the best’. The principle reasons being that
- Developer productivity can vary by an order of magnitude between apprarently similarly qualified candidates.
- The best people only want to work with the best people. A graders hire A graders, B graders hire C graders.
On the face the face of it this seems like great advice, who wouldn’t want to hire the best? It turns out pretty much everybody.
For instance, how long are you willing to wait to fill the position? What if you are really really stretched? What if you’re so stretched that you worry for existing staff? What if hiring a specific individual will mean huge disparities in pay between equally productive staff? What if not making the hire is difference between keeping a key client or losing them? At some point every company has to draw a line and elect to hire ‘the best we’ve seen so far’.
The difference between the great companies and the rest is how to deal with this problem. Great organisations place recruitment at the centre of what they do. If hiring is genuinely everyone’s number one priority then hiring the best becomes more achievable. For starters you might even have half a chance of getting ‘the best’ into your interview room in the first place.
So you decided that you need to hire a new developer , advertised in the right way , vetted the CVs and even thought of some questions . Unless you’re in the 0.1% of people that think that a 90 minute chat is the best way to assess the skills of someone you may end up working with for the next 5 years you’re going to need to think really hard about how to give the candidate the best chance of showcasing their skills in a face to face interview.
This post won’t help with what questions to ask, but it will help with structure and maximising the chance of getting a good performance from a candidate.
Calm them down
First off let’s start right at the beginning, you’ve just met and are about to commence on some small talk, perhaps give an overview of the company – to be honest it really doesn’t matter what you say at this point, the candidate simply isn’t listening. Years of evolution are telling them to fight or flight and it’s your job to calm them down sufficiently that you can move the interview forward.
Your end goal is to get to the point where you can ask them some really hard questions with a fighting chance of doing themselves justice, but first off there’s ground work to do.
Get them talking
Having helped them settle into their surroundings, it’s time to turn things up a notch and actually get them talking. A pretty good way is to go through their CV, their past work experience, university projects etc. This is stuff that they definitely know more about than you do, and should help them ease into the interview. The key point to remember here is that by asking very open questions and listening to what they consider to be most important or interesting, you can learn a lot about them. Good candidates will start explaining cool and interesting tech that they have used, while others will start talking about co-ordinating this and managaing that. Give people a chance and press them on the tech, but if they’re not confident or worse disinterested, then this is your first warning sign.
During this period you should be thinking about what sort of person you have in front of you, people handle stressful situations in different ways and if you don’t pick up on this you’re likely to miss some good candidates. e.g. if they are shy they’ll need some encouragement or if they are brash and perhaps little arrogant you need to be challenging them and forcing them to justify their assertions.
A final point, and one that I am most guilty of committing, is to make sure that you don’t go too far and overly relax the candidate. Doing so will make it harder for them focus once you start firing technical questions at them.
Before we get into asking any questions, I’d advise the following approach. Present the problem in a form that will probably be too difficult to answer, expect the candidate to need some help and judge their performance on quite how much help they need. This means that you’ll get an idea for how they think, smart candidates will have stretched themselves to solve the problem and poor candidate can at least solve the problem eventually, meaning that they won’t be so traumatised that they can’t concentrate on future questions.
As an initial technical question I’d recommend something pretty straight forward. Something like Fizz Buzz, I find candidates answer these sorts of questions in three different ways.
- They write out the answer without even having to pause to think and you move on quickly.
- They struggle a little bit, think things through and get the answer out. This isn’t necessarily a bad thing and might be down to inexperience, but it does mean that they are unlikely to hit the ground running.
- They really can’t get the answer together and need a lot of leading. Alarm bells are ringing.
Next up I’d raise the difficulty, I’d suggest something wordy and open ended, perhaps some sort of systems level discussion. This means that the candidate can still push the discussion in a direction that they are comfortable with, but make sure that have at least of few things in mind that you will definitely discuss. Again judge them on the amount of leading they need before completing the question.
Now you’re ready to unleash the killer coding challenge, in many respects the whole interview has been leading up to this point, you’ve learnt a lot about them including how to get the best from them, but the next question will really set the best candidates apart. At this point in the interview you also need to consider whether it’s time to bail, if the candidate struggled with the toy programming question and couldn’t impress with your follow up question, then you really have to ask yourself whether it’s worth skipping the final challenge.
I would chose something that is initially tightly bounded but has the potential for significant extension should the candidate be up to it. Exactly what you go for is up to you, but if you don’t stretch the best candidate they won’t want to work for you.
Lastly it’s time to ask the candidate if they have questions, this in itself can be illuminating, though to be fair at this stage, the best they can do is reinforce a good performance. Questions about working attire and hours are generally a bad sign especially if not a final round, questions about version control, project management practices or something that came up as part of the interview are generally good things and provide an opportunity to sell the role.
After the interview
Once the interview is over, it’s time to come up with an decision. How you do is will be entirely down to your specific circumstances. For first rounds I tend to allocate a point for their programming, their systems and general technical common sense and finally a point for how well I think they’ll fit in. Anything over two and half, i.e. a strong performance in two out of the three and a good performance in the third is a thumbs up. For final rounds you can’t afford to let them leave the room if you still have any concerns unanswered. This is someone who you could end up working with for years so err on the side of caution.
Ultimately though, the only way to arrive at a good system is to have a lot of practice. Make sure technical staff are involved at the earliest possible stage and if you make a bad hire, don’t blame them, blame your process and learn from you mistakes.