It’s been two months since I wrote Part I on how the technical exercise, if done correctly, is good for all parties involved.  During those two months I’ve had an opportunity to do several more exercises plus a handful of “technical” interviews.  I’ve gotten to see some impressive ones as well as see one where it isn’t set up to be of much value.  My original thoughts were to incorporate these experiences into the original article as an update, but after some thought I think they deserve their own post.  Here is some more of what I learned on how to properly structure or go through this vital process!

Photo by Chris Ried on Unsplash

One really exciting one gave me ideas for when the time comes that I provide input on creating a technical exercise for an SE candidate.  The exercise started off with five simulated pre-sales questions about their technology that an SE might have to answer.  I don’t want to give away what they had, but here are some excerpts:

• I’m new to <firm’s tech space>, and there are a lot of concepts I’m not educated on. To make my onboarding smoother, it’d help if you could provide me with some definitions of the following concepts:  (This was followed by 4 terms.  After that was a single question about a specific radio box in their UI.  If you were able to explain that setting you understood a key point about how their logic worked.)
• I had a look at the article you sent me <URL>After reading it, I’m still not convinced that the approach you have is better than the approach of <open source competitor> that is using <logic>. In addition to this, it also feels as though the <said logic> allow for better ways to <business value>? It may be because of my engineering background, but using <algorithm method> like it’s a lot more precise than your approach. And <tech space> are all about precision.Am I missing something here? Can you convince me of the opposite?
• A detailed question where the potential client wanted to hear more on the security method of one of the features.  They needed more details than what they had read.

There were a few others, but I think this really gives you the feel.  The candidate got to see the types of questions they might get, as well as how good is the documentation and SEO for the firm.  The hiring firm got to see how the candidate likes to communicate, as well as what foundation knowledge they already possess.   All of the sample emails had a valediction with the job title of the person asking.  This allowed the SE to decide at what altitude to answer the question as well as what business value was relevant to mention.

I would have two pieces of feedback for the hiring firm on this one.  My first take after reading the example questions was that I didn’t have enough base skills in this specific industry and had concerns on if it was even worth attempting.  There were sections that I had no knowledge or familiarity with.  I did email the technical contact, admitting I was probably gun-shy because I had spent a lot of time on a prior exercise only to have someone else (who had already been flown out to the East Coast for onsite interviews) get hired prior to me getting a second interview.  (And who can blame them?  A bird in the hand these days – when you find the right person pull the trigger ASAP!)  That person quickly let me know that no one at the company came in super strong in these aspects and all had to learn.  They were more interested in my ability to research and figure things out.  So in this example I’d advise having a sentence or two letting the candidate know that perfection is not expected and to just do their best.

Another area that I might consider is adding a section for the candidate to rate their confidence in each answer in a “out of 10” manner.  This was something we used at one of the companies I worked for – when you made a statement you could follow it with something like “and I’m 3 out of 10 (3/10) on that one” which let others know you weren’t confident in your opinion but it was open for discussion.  There were a couple questions where I felt really good about my answers, and two where I felt like it was a bit of a stretch.  I would have been even more confident in turning in the work being able to position how I felt about it.  And of course if I’d gone the wrong direction on my positioning (confident when I shouldn’t have been) it would have given them even more insight.

Those 5 example emails were followed by a good technical exercise.  There were two data sets to import into their solution, plus several items to try and solve that were true “day in the life” experiences.  It was a great chance to see how solid their UI was, as well as how fast their system is.  I even leveraged support to get a question answered.  Of course I did let them know that I was not a potential customer, since I didn’t want to waste their time.  Response was quick and very helpful.  It also gave me a chance to offer feedback to the company, which was well received.

One spot that I found very funny was that the exercise here was from a github repository.  The candidates were told explicitly not to fork the documents to create their answers in.  It claimed there as a bot watching for this that would delete the fork and also archive your application in their applicant tracking system.  Very entertaining compared to the firm in Part I that wanted you to fork the document for your answers – giving everyone else access to a collection of answers.

This technical exercise and 5 sample email responses were followed by a simulated discovery call.  They sent me two LinkedIn profiles and suggested I review probably a dozen specific long pages in their documentation.  After creating an 11 page document for the first part I almost begged off for the next step.  I’d had enough experiences with opportunities having me put in extensive time and then move on with other candidates farther along or say no after not properly evaluating me.  But this one was a different sort of company and I felt drawn to continue down the path.

I think I’d have some other feedback for them on this process.  Besides updating the instructions around the exercise and the “out of 10” section, I’d suggest the following:

1. If you know you are going to do the discovery call simulation, it is getting pretty heavy as far as the amount of time you are asking the candidate for.  Only ask them to pick 3 of the 5 possible emails to respond to.  Let them know they can do all five – but you are only expecting 3 for this step of the process.
2. Let the candidate know that the simulated discovery call will be laid back and easy going.  Not to spend too much time studying.  You are looking for soft skills at this point, not how well they can cram on your solution in 24 hours.
3. For simulated call or demo simulations, let them know that they can drop out of character to discuss how you think you would handle it – but to specify ahead of time if you are doing this.  I actually did this 2 or 3 times during the exercise.  At one point the person pretending to be with the opportunity mentioned scaling was an concern.  I ended up saying “Hey everyone, just dropping out of character here.  For that one if I worked here I would have memorized the details from your case study for <a specific large firm that I knew> where I could specify how many RPS (requests per second) they were doing as an example of how large we can scale.”  This was well received and made the whole process more exciting and collaborative.
4. And if you are going to also ask for a coding exercise on the whiteboard – specify it can be with pseudo code.  I almost put in several hours brushing up on JavaScript to prepare for this one.  I know how to code in it, and have even taken a college course – but my skills are cloudy because of all of the other languages I have worked in since using it for work.  For this one – the pseudo code exercise was exciting and I think really let their technical people see how I think.  (It also confirmed that I really know how to code as well as think like a developer.)  And any generalist that has worked in a large number of languages should be comfortable taking an impromptu discussion like this.
5. This goes for any opportunity – if you have a hiring process that includes technical exercises and/or demos, limit the number of actual interviews where your employees are grilling the candidate.  After 4 or 5 most candidates have reached their limit and probably have concerns about your ability to attract other employees once they are on board.  Most solid candidates will work hard, but the good ones recognize when they are setup to be burned out.  They’ll move on before that happens, leaving your buried staff even more overwhelmed.

For one technical interview I prepared for, I was told that there were two features I was to study and be prepared for talking about to simulated customers.  I read everything that I could find online, even made around 30 3×5 cards with extra details to memorize.  Instead most of the time was spent with me being asked to explain how I might architect or build these features and to persuade a coworker that it was worth doing.

The entire discussion gave very little insight to the conversation you might have with a potential customer, or the soft skills that are key for an SE to have.  Even so I thought I had done well.  It was surprising to get the call the next day that they were not going to continue the process.  Getting the call sure did beat the form emails that firms like to use these days (or the ghosting that goes on.)  The process did give me insight to a process that was broken.  Their current SEs were heavily overworked, and were going to stay that way for a long time while they employ a broken process to evaluate potential candidates.  Good to not end up on a team headed quickly for burnout.

Speaking of a process that was broken, in Part I I mentioned an APM firm I interviewed for where the process also felt immature.  I was probably being harsh since that interview was 3 and a half years ago, and the processes have evolved a lot.  The really interesting part of that experience was that the VP from that experience had moved onto another firm that I ended up interviewing with.  He remembered me, and also had read my blog post.  (And he was sharp enough to know I was talking about his past firm.)  He brought it up in a good way that really told me this was an organization I’d be happy to work in.  And their current process was the best I’ve had so far when it comes to evaluating a technical resource balanced with not asking too much of their time.  Right now it is a crazy time (I applied for 10 roles last Friday alone – pretty normal every week or two for me right now) and have backed out of opportunities where the ask was too intensive or signaled a broken process.

So the suggestions I’d add after the past two months:

1. Consider sending some sample pre-sales emails that they might have to answer as an SE.  Have a “out of 10” section where they can self-evaluate how they think they did on each one.  Don’t ask too much and have a few soft pitches in there.  Remember you are looking for soft skills here, not how well they can research your site and learn your solution in a time crunch.  You are not their only opportunity and they are balancing their time as best they can.  Limit the work load here.  Perhaps let them pick 3 out of 5 possible emails to respond to.
2. If you have a coding exercise I really feel pseudo code is the way to go.  It puts the candidate at ease and you can see how they think.  A poser really can’t quickly do this with true “hands on” experiences being highlighted.
3. For a simulated discovery call, look for the soft skills and let them have notes.  For this opportunity they did suggest notes which saved me a lot of time and 3×5 cards.

It is an exciting time to be a Sales Engineer or Solutions Consultant right now.  The space is maturing plus more and more firms see the value of this role in their efforts to make sales and gain market share.