Hiring for Retention – Or how to Suck at Recruitment

Hiring for retention
Unicorns!, 10x Engineers!, We only hire the best!, the War for Talent!

Hiring quality developers is a hot topic, you can’t move for guides on how to get the very best to work for you. In fact it is such a hot topic that many writers on the subject forget why recruitment is on the table in the first place.

In order to be successful, a business needs to be able to build highly productive, durable teams that can adapt to changes to the business of over time. To do this, good recruitment is essential, but is only a small part of the whole.

Start ups pride themselves on disrupting the status quo, reimagining products and processes to gain an edge over incumbents. This has led to rapid innovation in engineering, product and marketing but HR, and relatedly talent management, has for the most part remained static.

What would happen if talent management was re-imagined from the ground up where the only metric to optimise for is retaining quality teams? How would the roles of recruiters, HR and functional managers evolve in response?

I’ll go through four suggestions of what a retention optimised system would look like. The conclusions are counter-intuitive and in some cases directly conflict with current recruitment good practice. Optimising for retention means that a company will undoubtedly lose out on quality staff in the short term.

I should say that every suggestion is something that I have implemented myself in multiple teams. I have run each over a sufficiently long period to believe that it supports retention in the long term. For arguments sake I will consider ‘long term retention’ to mean a company tenure of greater than five years.

Since I’m writing from my own experience, there is a bias towards software engineering organisations in a startup environment. Though I think many of the lessons can be applied more generally.

Before I get there though, let’s just recap on why retention is important.

Why is retention important?

  • Culture
  • Long term Investment in staff possible
  • Get you through the hard times
  • Much cheaper…

Retention is critical for building a strong culture. If you want a culture to persist you need continuity, especially from your most influential staff. I’ll talk about culture in detail later.

Secondly, an assumption of long term retention makes it easier to invest in people. This means that they can improve at their job, also improving the team and subsequently improving retention… It also means that every part of the business is likely to have good industry domain knowledge. So not just an expert on Software Development, but also knowledgeable about the fashion industry or the Telecoms industry (say). This domain knowledge allows the team to make better decisions on a day to day basis.

When everything is going well in start ups, the team can forgive most things, but every company goes through rough patches. Perhaps there is concern over a new direction or growth is slowing. It is at these times when you really need a team to pull together to get the company through.

Also, it’s cheaper. All manner of management consultancies have tried to put a cost on replacing a member of staff, they seem to be landing at between one and two year’s salary – I’ll not argue.

In short, companies that can develop and retain their best staff will have a clear advantage over those that do not.

Hiring for retention

With that in mind, there are four areas that I’d like to discuss further.

  • Embedding recruitment into the culture
  • Sell the company, not the role
  • Hiring exclusively for fit
  • Don’t mess about with money

Embedding recruitment into the culture

embed recruitment into the culture
Every recruiter I’ve ever talked to says that they really benefit when the hiring manager is engaged in the process.  How do you actually go about making this happen? There is only so much a recruiter can do.

Recruitment has to be everyone’s business, and management should make sure recruitment activity is a first class citizen.

This means two things:-

  • Make working in recruitment aspirational, reward recruitment skills like any other skills. If one of the team is spending 20% of their time on recruitment and their peer is spending 0% then comparing them solely for their engineering output sends a very strong message that recruitment doesn’t matter.
  • Recruitment is hard, so you need to find ways to help people improve. This could be as simple as adding a high level of transparency to the process and give constant feedback. Whatever you do, people must feel that they are learning a skill that is valued highly in the company.

To be honest I think that embedding recruitment into the culture is something that is always a good idea, regardless of how you are recruiting. So not especially controversial, it just requires sufficient on the part of management. Onwards.

Sell the company not the role

Sell the company not the role

The role will undoubtedly change, the company will not. If the goal is to focus on retention rather than simply getting the best candidates through the door, it is most important that the candidate is sold on the company.

This can create a conflict since a great way to close a candidate is to help them imagine what their new job will be like. This means going into detail on the specifics of the role, the immediate project etc.

If instead the bulk of the sell is around the company, it is entirely likely that you will lose out on quality staff, though those that you do hire will be that much more durable through the inevitable twists and changes that will occur in coming years.

It is worth noting that this idea is well established. It is relatively easy to do in a startup context, since a company’s mission is likely to be extremely important to the candidate, but it is also possible at much larger organisations. Facebook, for instance, wait until after their 6 week bootcamp before pairing staff up with their eventual teams.

Selling the company not the role will help with long term retention but it will definitely mean losing out on capable staff in the short term. If a recruiter is assessed solely on the number of new hires they can get through the door, it is unreasonable to expect them to do this.

Hiring exclusively for fit

Hire for fit
In 1974 following Mick Taylor’s departure, the Rolling stones were in need of a new guitarist. The Rolling Stones were hot, they could have talked to anyone they liked:- Clapton, Beck … but they went for Ron Wood of the Small Faces. Perfectly competent but the not the best guitarist available. The reason they went for him was that being guitarist in the Rolling Stones was much more than simply playing guitar, just being in the band was a pretty tough gig. Ron got the job because he could handle all the other baggage that came with being in the Stones. A pretty good hire in the end and it is fair to say that the Stones know a thing or two about retention.

So here I want to talk about fit and culture. These are terms that abused quite a lot and different people mean different things by them. So here’s what I mean when I say culture. It is simply what is rewarded, what is tolerated and what is punished and it is the company’s leadership, formal leaders as well as informal leaders that set it and maintain it.

Strong culture is essential for retention because it is culture that bind people together. Life is easy when all the graphs go up and to the right, but it is during the periods of slow growth that you really need the team to pull together and it is culture that will get you through.

The most important thing before starting to hire for fit is know very clearly what culture it is you want to propagate. You don’t get to choose of course you just need to take a look around and look at the traits most likely to be supported by leadership (again I stress both formal and informal).

It is really important to write down what is meant by fit and then live it. If you don’t then fit just becomes ‘people like me, or people I like’ – this has implications for diversity which will likely ruin your business. If you can focus on just a few traits then it should actually help build a diverse team since all that matters is satisfying these specific traits, everything else is open.

So this means that fit needs to be a first class citizen to any other area you are selecting for. As important as coding, as important as architecture.

Through experience, I’ve found was that until you pass on a really good candidate purely on grounds of fit, your team won’t really believe that you are serious. If fit is the reason for rejection, then call it out loud and proud.

Once the team really understand how important a factor fit is, then you need to help them get better at assessing fit through interview. This goes back to training and feedback.

If you hire for fit, and reject those who do not, then you will lose good people. I’m not talking about passing on just really obnoxious, toxic people (though you should). I mean people who are perfectly capable, but won’t help support your culture in the long term.

Don’t Mess About with Money

Don't mess about with moneyThere is nothing more poisonous to culture and therefore to retention than a sense that people are being treated unfairly money wise. You simply can’t afford to mess about here.

What this means is that you need to put the needs of your current staff (who, you know, you’re trying to retain), ahead of someone who doesn’t work for you yet.

It means that if you are bringing in someone to work as a peer with existing staff, their salary must be in line. And sometimes this means you will lose out on good people, either because you assess them at a lower level than they assess themselves, or because you are simply not paying enough. In the latter case, since you care about retention you need to have a serious think about giving everyone a boost. Expensive stuff.

But it is worse than that. You’ve just spent weeks, perhaps months unearthing a unicorn, you’ve sold them on the mission, they’re a great fit, you offered them a package that is realistic and in line with the rest of the team and, quite reasonably, they want to negotiate.

For the sake of a few thousand dollars you could lose this candidate. But if you do so you are rewarding people joining the company not for their skills to do the job but their skills as a negotiator. It is an understatement to say that your best engineers will not be your best negotiators.

Some people say, ‘oh it is fine’, negotiate and then balance things out at the next review. But this means penalising someone who has already worked for you for perhaps a year. Salary doesn’t really work in absolutes, it is much more about relative change.

This is really tricky stuff to get right, and you will certainly lose good people to money if you peg to your current team and stand firm on negotiation.

Conclusions

Companies that can hire for retention have a clear advantage, over those that do not. But optimising for highly productive durable teams, means losing out on good candidates capable of doing the job.

Traditional approaches to recruitment typically do not consider the long term goal of building highly productive durable teams. In fact optimising for this goal would lead to recruiters appearing less effective since they would take longer to hire.

The question at the heart of the article is this. If I’m right about retention being key, and I’m right about existing structures actively encouraging short term gains at the expense of long term retention, then how should organisations evolve to support this need. How should recruiters, HR and management adapt to a world where retention is considered key?

Where to advertise for developer jobs in London

Over the past month a few people have asked me for some advice on where to advertise for software developers. In truth, advertising is a tiny component of a successful recruitment function, but if you are at the point where you want to spend some money on adverts, you might find the following useful.

Many of the sources are internationally relevant, but all I’m doing here is reflecting my own experiences which are exclusively drawn from hiring for start-ups in London.

Budget

It is amazing how little companies consider the costs involved in recruitment. For some reason they happily sign off on a recruitment agent with a fee of 18-25% of first year’s salary then balk at the idea of a few hundred pounds on a job board. This is especially important when considering the costs of your own time. Even in small nimble companies it is entirely possible to spend 4 figure sums in staff time on each successful candidate hired.

Speaking of agencies

Recruitment agencies get a bad press, really it comes down to the incentives in the industry. It is possible to get results, especially for more junior staff who lack a pre-existing network but I generally think there are better ways to spend the time and money. YMMV.

Hacker News

Most people reading this will already know Hacker News. It is a community curated content aggregator backed by start-up accelerator Y-Combinator. It focuses mainly on start-ups and technology but has a bit of everything. Every month it runs a ‘Who is hiring?’ thread where companies can advertise their roles.

Be warned, it is not the most user friendly experience and you’ll need to wait for 12pm EST on the first of the month to post. The signal to noise is very good meaning that you can invest more time on each candidate. Given the nature of the board it is common to receive very strong applications from overseas candidates wishing to move to London, so it is worth figuring out your position on relocation/visas ahead of time.

I’ve hired most of my current team through HN and recommend it very highly – though be mindful that it’s very start-up focussed, if that’s not you you’ll need to work that much harder to be attractive.

Also worth a mention is the complementary thread ‘Who wants to be hired?’ where potential candidates place ads about themselves. This thread sees much less traffic but it still an interesting path and I would recommend it as a supplementary source of candidates.

Stack Overflow

Again, anyone in software will know Stack Overflow, through their Q&A service they have built a formidable community of smart and helpful developers. As a company you can pay to post ads for a few hundred pounds a month which appear along side the Q&As. There are various options to suit a range of budgets including bulk buy and featured posts. SO also offer analytics and applicant tracking tools – though you are free to use your own email to contact candidates.

You’ll come up against some fierce competition from other companies who do a great job of selling themselves. You’ll need to have a hard think about how to position yourself, to this end SO have produced a range of docs to get you going. It is also worth investing the time in creating a company page and using that to further promote your culture. I can speak from personal experience that you will see a big uptick in traffic if you take the time to tweak your content to work well in SO’s framework.

My results were mixed in that SO introduced me to many interesting candidates (and very few inappropriate ones), however to date I’ve not hired through this channel. When I was using it most heavily I was looking explicitly for people with a demonstrable entrepreneurial background, I suspect that with more generalist requirements I would have had more success.

It is also worth noting that you can also pay to contact SO users directly through their Candidate Search product, this means access to the SO database and then reaching out to people with good standing in the skills that you are looking for. I’ve not tried this myself though it sounds like an interesting approach.

Job Boards

London has a handful of startup centric job boards very much worth taking a look at. I’ve used Hacker Jobs and Unicorn Hunt (it was previously 3-Beards) previously, but they all serve a similar purpose. Since they are somewhat under the radar they get a better signal to noise ratio than larger generalist boards like Indeed/Dice/Monster etc.

LinkedIn

I previously worked at an American company with a bulk deal on ads. They used it to good effect to hire technical staff in Seattle, but I’ve never found LinkedIn to be a useful way to advertise for developers in London.

Twitter/Facebook Adverts

I’ve not done this myself, simply because I’ve never clicked on an ad from either. That said they both allow for targeting specific groups and I’ve heard anecdotally that people have had success through this route.

Honourable mention – Silicon Milk Round

Silicon Milk Round is a recruitment fair run out of the Old Truman Brewery on Brick Lane aimed specifically at start-ups. It is a phenomenal event and the standard of attendees is very high. With so many other interesting companies present you will need to work hard to differentiate yourself. I’ve found it to be an excellent place to either hire directly, or simply to start a relationship that may lead to a hire further down the line.

A final word

Good recruitment is all about building relationships over time, this means working through multiple channels, consistently and in a way that ultimately attracts the right people and and gently (or otherwise) dissuades the everyone else. Placing adverts alone will not get you very far but can form part of a broader strategy.

A final final word

I didn’t discuss ad content but for the avoidance of doubt. No ninjas, no pirates, no rock stars, no unicorns.

 

 

The Human Side of Agile

If I have a criticism of the agile movement it is that not enough time is given over to the role of management, and furthermore not enough care is taken over the distinction between project and line management.

In a study by VersionOne that looked to asses the reasons for failed agile adoptions, almost all could be traced back to reluctance or failure to engage on the part of management and little wonder when the role of leadership is left so ambiguous.

Gil Broza recently wrote a book entitled the Human Side of Agile that aims to address the human aspects of agile implementations, in particular offering practical guidance to how an agile team leader might incorporate these ideas into their role.

Prior to publication, Gil asked that I contribute an anecdote and I was only too happy to oblige – the chapter was on the subject of making the most of your immediate environment. I drew anecdote from my old office where space was in short supply.

It may not always be possible to create the perfect working environment, however it is important to make the most of what is available. My team were looking to map their work flow using a white board and sticky notes. Unfortunately we were situated in the middle of an open plan office without access to walls, nor did we have the necessary space for a for a free standing white board. In the end we bought a roll of white board sheeting and applied it to a nearby structural pillar. Work items flowed from top to bottom and space was tight, but it served our purpose and is still in use years later.

Elsewhere in the book Gil also references How to do Nothing.

Great stuff Gil.

Compliments from a Sales Guy

Why do sales guys find it so hard to pay an engineer a compliment?

Imagine this exchange

Jeff: Hi John, how was your weekend?

John: Great, I ran the London marathon and finished in 3 hours dead

Jeff: That’s amazing I can’t believe you managed to run all that way!

Jeff is trying to pay a compliment, but because he has such little idea of what John considers to be important he misses the mark and if anything is more likely to have caused offence.

Now take this into a professional environment. Jeff is a superb engineer, he sweats and bleeds over his code, he does this quietly and in the background, he takes pride in his work and just wants to see people benefit from using his systems. No-one outside of the engineering org really notices his impact because, due to him, everything ‘just works’  with almost zero drama.

One day he receives a request from John in sales who needs him to twiddle knob X, it’s a config change and he has it in production later the same afternoon – really nothing special. That said Jeff has really helped John out  and John is very genuinely grateful. He writes him an email thanking him, he even cc’s his boss. “Jeff I really appreciate all the hard work you put into twiddling knob X, it’s guys like you crushing the knob twiddler that make this company awesome’. Now Jeff should pleased, it’s nice of John to take the time, he recognises this, but it’s a terrible compliment since Jeff took no pride in the change and what’s more it’s clear from John’s email that he has no idea what Jeff does all day – if anything the mail is a de-motivator despite John’s best intentions.

So what should John have done, how do you compliment an engineer? The best possible way is to take the time to understand some of the complexities of their work, and thereby uncover some of what drives them and their passion.

This is of course uncomfortable and frustrating, a bit like trying to write with your weaker hand – the benefits are huge though. Let’s forget for a second about expressing thanks being a nice thing to do  – it also a way to influence and build trust across the organisation.

If as an individual, you can build trust and rapport in multiple parts of the org, you much better placed to get things done – especially when you really need a favour to get you out of a jam.

In this example, I’ve used a Sales guys and an Engineer, simply because I see this as the classic case, but it applies equally in reverse. It’s easy (and lazy) for an Engineer to dismiss Sales or Marketing as brainless – when in reality this attitude is simply highlighting ignorance of what it takes to be successful in these fields.

The point of this post? Try and figure out what those folks with the suits and nice hair are actually doing, it will benefit both you and your company. If they are open to it, try and share some of your stuff too.

On the flip side insults work in the same way, you might enjoy Professor Douglas Comer‘s essay  ‘How to Criticise Computer Scientists’

What would happen if you stopped paying people?

So I want to tell you about a chap called Rob Ashton. I don’t know Rob but he appears to be extremely excellent. Having got a bit narked off with his enterprise consultancy gig, Rob decided to toss it all in. Without a firm plan in plan in mind, Rob decided to make an offer to the world – anyone willing to cover his expenses could have Rob come and work for them for free. Initially I think the plan was to stick to Europe but things seem to have got out of hand and Rob got all over the place. You can read about it here.

Rob was very strict on the whole ‘not receiving payment’ thing, noting:-

I was offered pay for a number of roles while I was doing this, and turned it all down because I felt it would sully what I was trying to do. Also – I felt it would muck up the balance where the people I was working for really wanted me to be happy because it was all they were giving me.

And this me thinking, companies want to hire the best people they can. As a means to achieve this some companies come to the startling conclusion that if you want to hire and retain good people you need to be prepared to pay for them.

But this isn’t enough. Staff can only be as effective as the company allows them to be. If the company culture stifles productivity and those same staff, while more productive that others, are still not able to fully deliver.

What’s worse is that the company doesn’t realise this is happening, no company deliberately aims to clamp down on productivity. Things tick on the way they always have done, the star hires perform well relatively speaking, and staff stick around because taking a pay cut is difficult. So the warning signs are less obvious.

An interesting thought experiment would be to ask yourself

What would happen if you staff worked for free, what would you need to change?

Let’s ignore the practical implications of this statement, all I’m saying is, if you take money off the table what would keep your staff wanting to work at your company? This isn’t about extra perks and a ball pool, I’m including all extrinsic motivators. This is about identifying what it is about your organisation that demotivates, and identifying how the organisation could improve intrinsic motivation.

  • How would your approach to flexible working change?
  • How about performance review and professional development?
  • Most importantly, what would be the implications for your org chart?

The answers will highly dependent on context, but if we make the assumption that in many cases the goals of the staff member and company are aligned, then why wouldn’t a company want to act on the conclusions?

I think we are starting to see the results of this already in the form of a shift towards flatter hierarchies, ad-hoc work groups and acknowledgement that people are more important than process. Some companies lead the way such as Valve and Gore but any company could benefit from asking themselves this question.

Annual Performance Review: Emperor Palpatine & Darth Vadar

Darth Who?The value of Annual Reviews is a contentious subject, just ask Microsoft. In my view, so long as they are kept separate from pay reviews and form just a small part of the overall act of providing continuous feedback, I can see some value in them as a means to discuss long term progression.

A few years ago my company re-wrote our review template – I was concerned that my team might not take a great deal of interest in the new form, and heaven forbid, might not even read it before the reviews came about.

I felt that everyone would get more from the process if they had some idea of what to expect and how to use the template, so I filled in the form detailing the annual review that no doubt occurred between Emperor Palpatine and Darth Vadar in the aftermath of the original Stars Wars.

I share it here simply because I wish there was more HR material in the world based on Darth Vadar and Emperor Palpatine.

For those of you who consider the annual review to be an outdated process, bear in mind that this happened a long time ago in a galaxy far, far away.

Part 1: Overview of Role and Recent Projects

Vadar writes:

During the past six months I have focusing on the construction of the Death Star with a view to using it as a tool to dominate the Galaxy. The project is extremely complex and I worked closely with key stake holders to ensure that the design and subsequent deployment match business goals and directives.

Separately I have been working closely with the Galactic Fleet to isolate and crush what remains of the Rebel Alliance. This has involved detaining and interrogating key figures as well as embarking on an extensive search across the out reaches of the Empire.

Palpatine writes

I concur with Vader’s comments. It’s also worth noting that Vader has been actively recruiting and restructuring the Galatic fleet leadership team.

Part 2: Performance Overview

Objective #1 – Deliver a fully functional Death Star

Vadar writes:

How well did I perform against this objective?

I delivered a fully operational Death Star, which passed all user acceptance tests. It is true that the delivery was delayed and I put this down to the project management techniques employed in the early stages. Towards the end of the project I took a much more hands on role implementing a more iterative and incremental approach to delivery. In doing so it was possible to gain rapid feedback and ensure that value to the customer was maximised. Furthermore I made some tough decisions over poor performing members of the management team, introducing them to my ‘claw of death’.

Next Actions

I consider this objective complete, though note that following the destruction of the Death Star, we may need to build a new one.

Palpatine writes:

I agree that a functional Death Star was delivered. I am pleased that I was able to set a vision and direction and that the manifestation of that vision became a reality. I am also very pleased with the aesthetics, it almost looked like a real Moon.

While I was disappointed by the late delivery of the project I am much more concerned how such an obvious vulnerability found its way through the design reviews. I would have expected that this would have been picked up pretty much form the off. I am also deeply concerned by how it became possible for the Rebel Alliance to steal plans for the Death Star such that they were able to exploit the weakness. Furthermore the destruction of the Death Star has damaged our reputation as a Galactic force – how a single manned fighter was able to implode an entire space station is the stuff of science fiction.

Objective #2 – Crush the Rebel Alliance once and for all

Vadar writes

How well did I perform against this objective?

It’s fair to say that the Rebel Alliance cannot be described a ‘crushed’. I am happy with my overall approach, the Empire is very much in control of the Galaxy and the Rebel Alliance have not been able to secure an outpost, meaning that they are forced to move from temporary base to temporary base to evade a the might of the Galactic fleet. I feel strongly that it is only a matter of time before I have them trapped. That said I failed to account for key figures within the alliance, as well over looking a key vulnerability in the Death Star itself.

Next steps

I will embark upon a Galaxy wide search for the relocated rebel base, and continue with the aforementioned crushing

Palpatine writes

I agree that your general approach was thorough and methodical. I would have like to see more outright annihilation of innocent people but overall I am pleased with your execution.

Where I am less pleased is that you failed to adapt your plan to take into account key events, specifically the rebel attack on the Death Star and Luke Sky Walker’s ability to evade attack. It is unclear to me why you were not able to land a direct hit even when pursuing him along a narrow trench and supported by two of the finest pilots in the galactic fleet.

Part 3: Overall Rating

Vadar writes:

3 out of 5 – Effective contributor

Palpatine writes

2 out of 5 – Low Contributor, I feel that your overall performance was not up to the standard that I would expect from a Lord of the Sith.

Part 4: Feedback on Galactic Empire Inc

Vadar writes:

What I enjoy most about working here? I really like the almost unbounded opportunities to inflict misery and despair on pretty much anyone I like.

One thing I would change about Galactic Empires Inc. I think we need to review the weapons issued to our Storm Troopers, they seem faulty and rarely inflict any damage at all.

One thing I would like to see remain at Galactic Empires Inc. I think that our uniforms are pretty much the best in the industry. This is especially true for the Storm Troopers

Part 5: Future Development/Business Goals

Vadar writes

What I do well: Inspire terror in others.

What I want to improve: Despite being a master of the dark side of the force, I am still unable to consistently smite Knights of the Jedi.

How I would like my career to develop over time:Long term I’d like to move into a Supreme Leader of the Universe role

Palpatine responds

I agree with Vader’s comments, in particular I am keen to help him to improve his smiting skills. While I do not see Vader’s wish to move into Supreme Leader of the Universeship as unrealistic, such roles do not come up often.

Part 6 – Objectives

Objective #1: Create a new death star

Next Actions: Review the design docs to remove unlikely failure modes   Time frame for achievement:  This coming October

Objective #2: Crush the rebel alliance

Next Actions:  Discover the location of the rebel base Time frame for achievement: Next April

Palpatine summarises objectives:

Long term, Vader is looking to rule the entire universe, I see building the most devastating weapon ever created and crushing the only viable opposition as key stepping stones towards this goal.

Part 7: Record of discussion arising from the review discussion

Palpatine summarises

During the meeting we discussed my rating of Vader’s performance. While disappointed he understands that the loss of the Death Star played a big part in my overall decision.


Remember kids, appraisals should be:
  • Separate from pay reviews
  • A small part of a wider mechanism to provide staff with continuous
  • A two way street

I thought that 7 Digital had some good things to say on the subject

Twitter Clone Tutorial in Web2py Part 3 – The Wall and Search Pages

Welcome to my getting started with Web2py tutorial.

This tutorial assumes very little on the part of the reader, some knowledge of a programming language will certainly help, but if not don’t worry. I’ll take you from installation through to v1 of your application.

If you don’t want to copy from the tutorial, the full source is hosted on github.

Last time we created the Models as well as the Controller and View for the home page. The next page to look at is the Wall. The Wall is specific to a given user and contains their profile details as well as their weet history.

Wall_reduced

As before we start with in the default.py Controller, add the following method to manage the behaviour for the wall.

# show user's wall, showing profile and posts
def wall():
   #Determine which user's wall is to be displayed
   user = db.auth_user(request.args(0) or me)
   #If user is invalid, return to the home page
   if not user:
       redirect(URL('home'))
   weets = db(db.weets.posted_by==user.id).select(orderby=~db.weets.posted_on,limitby=(0,100))
   return locals()

The first thing to do is determine whose wall it is that we need to display. The line responsible is worth a closer look.

user = db.auth_user(request.args(0) or me)

We have already seen db used as a means to access the database but we have not yet come across ‘request’. Request is an object available to all Controller actions and provide details of the http request that instigated the Controller action being called.

In this case we check the first (technically the zeroth) argument of the request. An argument is best described by example

http://127.0.0.1:8000/witter/default/wall/myargument/myotherargument/

In this specific instance the argument denotes the user id and so the line in question checks if a user id has been specified, if not it assumes that the requester are themselves logged in and trying to access their own wall. If neither case is true then the browser is redirected to the home page.

After that we pull out all the weets for the given user and return local variables, which are then made available to the View.

We will need a new View file called default/wall.html. As before you can create it via the application admin page.

Replace the default code with :-

{{extend 'layout.html'}}
<h2>Profile</h2>
{{=crud.read(db.auth_user,user)}}
<h2>Messages</h2>
{{for weet in weets:}}

{{=name_of(weet.posted_by)}} on {{=weet.posted_on}}:

{{=MARKMIN(weet.body)}}
{{pass}}

We’ve met crud before, it provides a nice high level abstraction to access database information, in this instance we are using it to pull the profile information about the user. The next section loops through all the weets generated in the Controller and presents them in chronological order – this very much the same logic as we saw on the home page. You can a look for yourself at http://127.0.0.1:8000/witter/default/wall or just navigate there from the Witter home page using the navigation bar at the top.

Search

For the grand finale let’s look at search.

The purpose of this page is to allow a user to search for their friends and then follow them. In some senses it is the simplest of the pages, but as we shall see it contains some interesting functionality behind the scenes.

Firstly, let’s take a look at the Controller, add the following to default.py

# a page for searching for other users
@auth.requires_login()
def search():
   form = SQLFORM.factory(Field('name',requires=IS_NOT_EMPTY()))
   if form.accepts(request):
       tokens = form.vars.name.split()
       query = reduce(lambda a,b:a&b,
                      [db.auth_user.first_name.contains(k)|db.auth_user.last_name.contains(k) 
                           for k in tokens])
       people = db(query).select(orderby=db.auth_user.first_name|db.auth_user.last_name,left=db.followers.on(db.followers.followee==db.auth_user.id))
   else:
       people = []
   return locals()

The first thing we do is create the search form, we do this like so

SQLFORM.factory(Field('name',requires=IS_NOT_EMPTY()))

The SQLFORM.factory can be used to generate a from an existing database table, or, as in this case can be used to generate a form from scratch. The form is very simple and takes a single input ‘name’.

The rest of the Controller handles the action once the input has been received  – this is what the follow is checking for:-

if form.accepts(request):

The remaining code searches for the the search terms amongst other users registered in witter and plugs them into ‘people’, it also determines which of those people the user already follows – the View is going to rely on this follower information.

The Search View

So far so good, now let’s take a look at the View, you’ll need to create a file called default/search.html under the Views section. Replace the default code with:-

{{extend 'layout.html'}}
<h2>Search for people to follow</h2>
{{=form}}

{{if people:}}
<h3>Results</h3>

{{for user in people:}}
{{=A(name_of(user.auth_user), _href=URL('wall',args=user.auth_user.id))}}
{{if user.followers.followee:}} Unfollow {{else:}} Follow {{pass}}
</div> {{pass}} {{pass}}

Much of this will now be familiar – we pull in the standard template, then the form and then iterate over the ‘people’ (if any) returned by the Controller. Where it get’s interesting is the button definitions.

Firstly we check if the searched for user is already being followed by the search, this means the button can offer to follow unfollowed witterers and unfollow the followed. So far so good, but what does this do?

<button onclick="ajax('{{=URL('follow',args=('follow',user.auth_user.id))}}',[],null);$(this).parent().html('Followed')">Follow</button>

Before I can answer that I need to explain about AJAX. AJAX stands for Asynchronous JavaScript and XML, but this name is largely misleading since it needn’t be asynchronous and needn’t use XML. In short AJAX is a technique for creating dynamic web pages, it allows pages to be updated without needing to reload the entire page, and it does this by exchanging small amounts of information with the server behind the scenes.

Strictly speaking I could have passed on using AJAX in this tutorial, but it is a really useful technique a very simple to implement.

In layman’s terms the button definition says, when the button is clicked make a poke to another Controller action called ‘follow’ and send the arguments ‘follow’ and the user id. Once this is done replace the button (which read ‘Follow’) to be some text that read ‘Followed’.

A final note about the search View is the use of the css classes ‘row’ and ‘span’. These are provided by Bootstrap which is a a popular front end frame work. Bootstrap is not part of Web2py but Web2py does support it out of the box. In this tutorial I’ve deliberately avoided focusing on the look and feel of the app, but Bootstrap provides all the tools necessary to create an attractive web application.

AJAX Callback

And that’s it for the , however, what about the follow action in the Controller? We’ve not implemented that yet. Here goes, add the following to the default.py Controller:-

# this is the Ajax callback
@auth.requires_login()
def follow():
   if request.env.request_method!='POST': raise HTTP(400)
   if request.args(0) =='follow' and not db.followers(follower=me,followee=request.args(1)):
       # insert a new friendship request
       db.followers.insert(follower=me,followee=request.args(1))
   elif request.args(0)=='unfollow':
       # delete a previous friendship request
       db(db.followers.follower==me)(db.followers.followee==request.args(1)).delete()

The most interesting line is the first one.

if request.env.request_method!='POST': raise HTTP(400)

When a browser makes a request to read a web page it uses an HTTP method called GET, this is exactly what you do everytime you point your browser at Google. There are however other HTTP methods, POST being one of them, POST is used to transmit that is expected to change the state of the application. The AJAX call that we initiate in the search,html View is a POST request.

Since this Controller action is there solely to support AJAX calls from search.html, it does not expect to handle HTTP GET requests and rejects them. You can try this for yourself by trying to View http://127.0.0.1:8000/witter/default/follow which should result in a blank page and a error message complaining of a ‘bad request’.

Search

Right so there we have it – all pages are complete, in order to try out the search and folllow/unfollow functionality you’ll need to register to extra users and have them follow each other. Be sure to notice that it is not necessary to refresh the search page every time to make a follow/unfollow request.

So in conclusion what have we achieved?

We created a new web2py application called Witter, created three pages (and an AJAX method) supporting a home page, a user wall and a user search and all backed by a database that will persist the data over time.

Are we done?

Well perhaps, I certainly am.

This tutorial has taken you from nothing at all to a functioning web2py application, that can be extended as you please.

If you are searching for ideas, you might want to move the search functionality into the home page and extend it so that search covers all weets (much like Twitter does). The UI could certainly do with a spruce up (and there are plenty of CSS tutorials out there) and as a final task you could put the thing live, gain awesome traction (due to your custom tweaks) and take over the social media world.

Feedback on this tutorial much appreciated, I’ll leave you with some useful links.

Twitter Clone Tutorial in Web2py Part 2 – The Home Page

Welcome to my getting started with Web2py tutorial.

This tutorial assumes very little on the part of the reader, some knowledge of a programming language will certainly help, but if not don’t worry. I’ll take you from installation through to v1 of your application.

If you don’t want to copy from the tutorial, the full source is hosted on github.

Last time we set up Web2py and created our application, Witter. This time around we’ll down and dirty with writing our own functionality, but before we do that, let’s register ourselves since it will save us a job later.

Register a new user

From http://127.0.0.1:8000/witter Click ‘Login’ on the top right and then ‘Register’, enter your detail into the form and hit submit. Welcome! You are the first ever user of your application.

While this is extremely cool, it will be even cooler once Witter actually does something, and for that we need to work with the Admin interface.

The Admin Interface

The Admin interface can be found at from http://127.0.0.1:8000/admin/default/design/witter and provides access to all the files you will need to create your application, there is a lot to take in, and for now we are are going to focus on three sections that we will be editing extensively –  The Models, the Views and Controllers.

Admin_reduced

Model-View-Controller MVC

Before we go any further I need to explain some theory. Model-View-Controller or MVC is highly popular web application design pattern. This means that over many years of sweat and toil it is generally considered to the be the best way to lay out your code. It is certainly not the only way, and there are many MVC variations, but Web2py is rather fond of the MVC pattern and this is what we’ll use to create Witter.

So what does working with MVC mean?

We let’s start by defining some terms, I’m going to steal directly from wikipedia

Model–view–controller (MVC) is a software architecture pattern which separates the representation of information from the user’s interaction with it.[1][2] The model consists of application data, business rules, logic, and functions. A view can be any output representation of data, such as a chart or a diagram. Multiple views of the same data are possible, such as a bar chart for management and a tabular view for accountants. The controller mediates input, converting it to commands for the model or view.[3] The central ideas behind MVC are code reusability and separation of concerns.[4]

It goes onto to define the role of each component

In addition to dividing the application into three kinds of components, the MVC design defines the interactions between them.[5]

  • A controller can send commands to its associated view to change the view’s presentation of the model (e.g., by scrolling through a document). It can also send commands to the model to update the model’s state (e.g., editing a document).

  • A model notifies its associated views and controllers when there has been a change in its state. This notification allows the views to produce updated output, and the controllers to change the available set of commands. A passive implementation of MVC omits these notifications, because the application does not require them or the software platform does not support them.[6]

  • A view requests from the model the information that it needs to generate an output representation to the user.”

MVC-Process

In summary, we define how our data will be laid out in the Models, define how it will be presented in the Views and define how the user can interact and modify it in the Controller.

Right now, you might be wondering why this is all so important. For the moment I’m going to ask you to trust me, and work safe in the knowledge doing so will save you time, effort and save the lives of puppies.

Getting going – the Models

Okay, so we have the application up and running, we are now armed with the necessary theory, it’s time to dive in. This is where is really starts to get interesting.

We’ll start with the Models, we can actually reuse the existing ‘db’ Model file, so click edit and we’ll take a look around. You’ll notice that there is a lot of information here, along with housing our data Model the db file also provides some initialisation and sensible defaults for our application.

By all means take a look through but I want to call two lines of special interest:-

db = DAL('sqlite://storage.sqlite',pool_size=1,check_reserved=['all'])

DAL stands for Database Abstraction layer, and provides an API that maps Python objects into database objects. We will make extensive use of the DAL in our application. In this case we are using an SQLite database to back the DAL. Using SQLite will make absolutely no difference to how we write our application since we talk to the DAL not the database. This means that we can swap in and out databases as we please. To start with SQLite is the perfect choice to get going quickly, though we would want something meatier for a production environment.

auth = Auth(db)

Auth is a class that implements, the Web2py role based authorisation API, this will allow us to provide for user registration, login as well as restricting access to certain pages. For instance, once I have registered, it will provide me with a form to let me change my password, and at the same time ensure that I am the only person who can view it.

At the beginning of this tutorial I referenced the Web2py stated goal of Security. Auth and the DAL abstract many of these concerns on behalf of the developer. For instance the DAL makes it impossible to succumb to an SQL injection attack.

In addition to the concept of a user, we will need two additional entities – (t)weets and followers.

Add the following to the bottom of the db.py Model file

# a table to store posted weets
db.define_table('weets',
    Field('body','text',requires=IS_LENGTH(140,1),label="What's going down?"),
    Field('posted_on','datetime',readable=False,writable=False),
    Field('posted_by','reference auth_user',readable=False,writable=False))

#a table to store follower relationships
db.define_table('followers',
   Field('follower','reference auth_user'),
   Field('followee','reference auth_user'))
#Convenience methods to make code more compact
me = auth.user_id

def name_of(user): return '%(first_name)s %(last_name)s' % user

So for a weet we store the body that must not exceed 140 characters, when it was posted and by whom.

Separately, in followers, we store which users follow which other users, each relationship has it’s own entry in the same. ‘auth_user’ is reference to the auth_user table defined by Auth and by explicitly calling out the relationship we ensure that it is not possible to add bogus data into the followers table.

We’ll side step ‘me’ and ‘name_of’ for now and revisit them as they come up later.

Save the default.py and click on ‘edit’ that sits on the top nav bar, this will return you to the Witter admin page from here we need to edit the menu.py Model file. This file configures the look of the menu and also houses meta data about the application.

Remove the existing content and enter the following in menu.py

response.title = "Witter"
response.subtitle = "Super Awesome Twitter Clone in Web2py"
response.menu = [
(T('Home'), False, URL('default','home')),
(T('Wall'), False, URL('default','wall')),
(T('Search'), False, URL('default','search')),
]

This code sets the apps name and sub-title as well as defining the links on the navigation bar.

With the data Model out of the way, it is time to look at Controller file. Once again, save the menu.py and click on ‘edit’ that sits on the top nav bar, under Controllers click edit on ‘default.py’.

default.py also comes with some pre-existing content, you’ll notice a user method – this ties in with the Auth class we came across in the Model exposing functionality to manage user registrations and logins.

So far everything we have seen, has focused on definitions. In the Controllers we will start to use python in anger to determine how our application should behave. I will not attempt to provide an introduction to python in this tutorial but will recommend the Web2py python introduction as well as Zed Shaw’s Learn Python the Hard Way(‘The hard way is easier’).

The Home Page Controller

Our first custom Controller method will be the home page, this is the page from which we can post a weet as well as listing weets from other users that we follow – it also contains our own weets.

Every method in the Controller is exposed by the application, so let’s have a play. Add a new method called ‘home’ like so, adding to the bottom of the file is fine.

def home(): return "Hello from Witter App"

Now point your browser to http://127.0.0.1:8000/witter/default/home

You should see a page empty but for your message. Take a look at the url for a moment – witter is the application, default is the Controller file and home is the specific method. This means that we can write multiple Controller methods and reach them via the same URL scheme changing the final element to match the new method name.

Getting the home method working is good going, but if Witter is going to challenge Twitter on the big stage we are going to need to make some changes.

Replace your previous home method with the following, this is the Controller action for the home page.

@auth.requires_login()
def home():
   '''First point of call, contains all weets and all weets of those you follow'''
   #configure defaults, so they not appear on the form
   db.weets.posted_by.default = me
   db.weets.posted_on.default = request.now
   #create form with which the user can submit weets
   crud.settings.formstyle = 'table2cols'
   form = crud.create(db.weets)
   #determine who the user follows
   my_followees = db(db.followers.follower==me)
   me_and_my_followees = [me]+[row.followee for row in my_followees.select(db.followers.followee)]
   #Pull all weets to be displayed
   weets = db(db.weets.posted_by.belongs(me_and_my_followees)).select(orderby=~db.weets.posted_on,limitby=(0,100))
   return locals()

Woooh! Hold on now, there is a lot going on here. Let’s go through this slowly.

This method uses both ‘db’ and ‘me’ that we defined earlier in the db.py Model. Any variable defined in a Model file is automatically available in all Controllers (and Views). This why we can use these variables without needing to formally import them.

So every time we see ‘db’ we know that some activity against the DAL and therefore the database is going to occur. Similarly every time we see ‘me’ we know it refers to the user who is logged in.

So, back to the home method. The very first line

@auth.requires_login()

Is a method decorator that ensures that this page is only accessible to logged in users. Users not logged in, will be redirected to the login page.

Next we assign some default values to fields in the weet entity before using the crud plugin (also defined in db.py) to auto-generate a form for us. This is the form that we will use to post weets. Crud stands for Create, Read, Update, Delete I don’t want to dwell on it too much for now, other than to say it is a useful tool to create forms and validation logic.

Next we need to pull out the weets that those that we follow have weeted, as well as pulling out our own. The final lines act on the DAL to select out the relevant details to pull out only the weets we are interested in.

Finally we return ‘locals()’ this makes the method’s local variables available to the View, which we will go into in just a second. We have one final task before leaving the Controller and that is to replace the index method with:-

def index():
   if auth.user: redirect(URL('home'))
   return locals()

The index is the default Controller and is where you end up after naviagting to http://127.0.0.1:8000/witter This method simply checks if you are logged in and if so redirects to the ‘home’ page. If not you are presented with the default landing page that you saw when you first started up Web2py.

Now to learn more about Views

The Home Page View

Remember the View is there to present the information stored in the Model, after processing via the Controller. Separately, as we shall see, it also provides the user a means to interact with the application via the Controller.

In the first attempt at the home method ‘’Hello from Witter App” we made use of the default View. The default is useful for checking that everything is working but for a high quality application such as Witter we will need something more sophisticated.

From the application adminstration page scroll down until you see the Views section and create a new file called default/home.html The path of the file denotes that it is the View of the default/home Controller.

Replace the default code with the following:-

{{extend 'layout.html'}}
{{=form}}
$('textarea').css('width','600px').css('height','50px');
{{for weet in weets:}}

{{=name_of(weet.posted_by)}} on {{=weet.posted_on}}:

{{=MARKMIN(weet.body)}}
{{pass}}

I’ll explain what the View is doing in second but for now, just save the file and point your browser at http:127.0.0.1:8000/witter/default/home If you are no longer logged in you will be prompted to provide your username and password.

This is your landing page – what a difference! Along the top left we have the Witter navigation controls, which we specified in the menu.db Model and immediately underneath a text area in which to enter weets.

Home_reduced

Initially, there isn’t very much to see. The next step is to go ahead and make a weet (you can see weets from multiple users in the picture above). Enter some timeless phrase into the text area and hit submit. The page will refresh and low and behold your weet is preserved for eternity.

So how on earth has this happened? Let’s go back to the default/home.html View.

The View is standard html file and as you might expect contains html tags to mark up the content, but it also contains some lines enclosed by ‘{{ }}’. The contents of these lines are Python and allow you to dynamically create the page content.

The first line draws in the layout.html file, this is base template that provides things like a nav bar and a footer and generally makes our page look a bit more like a website. You can take a closer look from the application admin interface under Views.

Next up, we enter the form as created in the ‘home’ Controller, before modifying the size of the text area in the <script> tags.

Here’s where it starts to get interesting. In the Controller we specified a variable called weets that contained details of all the weets to display on the page.

The View loops through all the weets

{{for weet in weets:}}

For each one it displays how weeted, when, and then the body of the weet. The reason this is so interesting is that the content has been created dynamically and is entirely dependent on the weets pulled from the Controller.

So we have out first dynamically generated page on Witter. All that remains is to implement the remaining functionality such as the Wall and the Search page which we’ll pick this up in the next post.