Shannon Hogue from Karat and David Moise Discuss: “Better Technical Hiring By Having Technical Interviews

We also talk about equitable hiring, How to do a better tech interview and transparency in the interview process.

— Contents of this Video—

00:00 – Intro

1:08 – Why Qualifying Tech Talent is Tough

1:50 – Better than Technical Trivia and Gotcha Questions

2:50 – Let Candidates Know What to Anticipate

3:45 – Equitable Hiring Practices

4:40 – Why Tech Interviews are Better than Tests

5:35 – A Human Experience

6:55 – How to do a better Technical Interview

7:50 – Google “How to do a Tech Interview..”

10:10 – The Evolution of the Tech Interview

11:00 – Flexibility in Tech Interviews

11:50 – Expand the Applicant Pipeline

12:45 – Transparency in the Interview Process

13:45 – The Technical Interview Struggle

14:15 –

— End of Contents—

Shannon Hogue –

Karat –

David Moise –

Technical Talent Strategies –

Today we are talking with Shannon
Houg, with Karat with a K.
She is the global head of solutions engineering.
There Karat gets into
technical interviewing for companies.
But Shannon, just go ahead and tell us
a little bit more about what you do.
And Karat gives us the quick introduction. Yeah.
So like you said, Karat is
the world leader in technical interviews.
We have created the first technical interviewing cloud
which is an always on, scalable and consistent
human plus tech solution that is used for
conducting engineering interviews on behalf of company 24/7.
We conduct technical interviews for more engineering candidates
than any company in the world in 2022.
And the data from those interviews is helping
us develop more strategic insights that companies can
use to improve their hiring decision.
Okay, great.
Now we’re talking to a lot of different people
about just what’s going on with technical talents getting
more and more hard or harder to hire the
right people to find the right people.
There’s 500,000 open it roles here in the US.
There’s competition for that talent and companies need
to do different things to bring them in.
And there’s a problem with a lot of companies.
How do we qualify who this person is and how
we go through technical interviewing is a tough thing and
it’s a problem for a lot of companies.
You guys have a different answer and this
is something that’s very close to me.
I have done at least 500 technical interviews.
I have a long time, long
career as a software developer.
I was always the person that the recruiters
went to to do the technical interview.
I’ve been doing recruiting for myself
and I’ve continued to do that.
And you guys have done something very different.
I’m just going to start off a thing
I see in a lot of companies.
There are a lot of tests out there that you
can have candidates do and a lot of them wind
up just being technical trivia or got you questions.
You all are different. Proceed.
Tell us about it.
I think the most important thing to keep in mind
right now is that we’re in a candidate first world.
I mean, first I can relate.
I have been writing software for a quarter
of a century and I asked the same
tangle questions for ten years in a row.
Count the number of lower case
as in the null terminated string. Right.
But what was I actually testing for?
I never been trained to think through what
are you looking for in a candidate?
So the first question I would have
is what are you asking a candidate?
Like what are you looking for when you’re
trying to got you with some question that
may or may not have an answer.
Are you trying to get them to do your work for you?
Are you just trying to trip them
up and see how they handle it?
How do you even measure that metric?
In a candidate first world, the majority of folks
are going to be like, that’s not okay with
me because I don’t know what your expectations are.
One of the first things that we tell folks
is it’s really important to let candidates know exactly
what they should anticipate in an interview.
Hi, I’m going to ask you these questions,
and all of us know there are million
ways in which we can answer these questions.
But what we’re looking for is
the optimality of your solution.
Or, hey, we’re looking for you to build
tests inside of your solution, but without letting
the candidate know what they should be working
through and what competencies that you’re looking for.
Whether it’s optimality efficiency, debugging and testing,
it’s pretty impossible for them to succeed.
And candidates need to be set up for success.
You can’t afford to have false negatives today.
And those got you questions 100% do that.
And then, David, we talked about this earlier.
What’s really interesting about the test, we often see
that as a take home, and we see it
as, hey, this may take six to 8 hours.
You’re actually building out a product.
Well, think about equitable hiring practices.
People have lives.
They have multiple jobs.
Sometimes they’re going to school at night.
They have families.
You’re really focusing on a core group of
developers who have that extra time in their
lives where they can spend 6810 hours.
Think of all the folks who can’t.
And so really, in today’s world, candidate first, there’s
not enough talent on the market you cannot afford
to be inequitable and not set expectations.
And think about the candidate experience first.
You’re 100% right on that.
And I think I often say if there’s a candidate, you have
a decent skill set, you can go to LinkedIn, say open for
work, and you’re going to get ten calls this week.
And if three of them are saying, hey, we
want you to take this six hour assessment, those
are three that they’re not going to return.
But along these lines, even just with the six hour
assessment, there are a lot of people out there.
The moment that they hear take home
test assessment, they just shut down.
I’m not going to apply for this
because I’m not going to do that.
And then talk to people and find out about the company.
Then the way that Karen has its
structure, it’s more of a technical interview.
Do you want to speak to that a little bit?
Yeah, 100%.
So what data has told us
we’re a data driven organization.
That candidate always comes first.
And that’s just really important to keep in mind.
And what we learned was through our data and
through our experiences that candidates with a nontraditional background
or minority candidates are more likely to drop out
of the process if there’s a take home test.
And there are a lot of historical reasons
we could talk about the implications of standardized
testing for under represented communities and the like.
But at the end of the day, the fact is that these folks
are going to drop out and you need them to get hired.
So what Care did instead of two things.
One, it needs to be a human experience.
There needs to be somebody there
that you’re working with today.
About over 40% of the candidates that get jobs
out of carrot, especially in senior positions, need some
sort of back and forth, some sort of hand
holding or communication throughout the interviews.
Imagine how many people you would be
missing out on without that human experience.
So what we did is that we went
and we recruited all over the world for
engineers with, on average, seven years experience.
We actually only take about one to 2% of
the folks that want to work with us as
interview engineers because it’s really important for the candidate
to feel comfortable and at ease so that they
can showcase their best selves.
We have, I would say 18 or 19 countries in the
world now where we have human beings, they go in.
They also go through a pretty
strict and rigorous certification process.
They need to have EQ.
They need to make sure that there’s
no bias inject in the process.
We need to ensure that they’re following the rubric.
Fun fact, our interview engineers fill out about
a little over 70 inputs throughout a 1
hour interview process, which is insane, but that’s
how extensive our roof work is.
That’s why we can make data distinct,
data driven decisions around the efficacy of
the content that we’re actually producing from
an interview question point of view.
But our interview engineers, they won’t know
anything about the candidate coming through.
They’ll fill out this extensive rubric.
We have teams that will align how to weigh the
different competencies for what the client is looking for.
We have a second interview engineer to come
through and make sure that the first one
did their job and didn’t inject that bias.
And then we only pass the recommendation that comes
from our engine, not from the interview engineer.
So really doing everything we possibly can to be
completely data driven and not inject any bias in
the process, all but still marrying that with that
human interaction of setting up that five minutes, making
folks feel comfortable, ensuring that they understand what we’re
looking for with the questions that we’re asking and
really just being there if they have to ask
questions or work through a problem together, great.
A lot of companies that are still
doing all their technical interviewing in house.
And I’ve seen this 100 times, hey, development team,
you need to go talk to this person.
And then someone is immediately Googling
questions to ask and.NET interview.
And we can do better.
Everyone can do better.
And tell me a little bit more, because
some companies are not looking for the rock
star, machine learning data scientist person.
Some companies are looking to validate
this is an entry level person. They’re going to be doing
some SSRS programming inside SQL.
You guys can cater a
little bit toward different levels.
Can you speak to that? Yeah.
The secret to that is a few things.
One is that we’ve done well over 100 and
I think 20,000 interviews now across the industry at
different levels, anywhere from University recruiting to L Five,
L six, and hands on director and manager level.
Essentially what we did is we
built a modular interview format.
And so we have different components of an
interview format that we can move and change
out in order to align with two things.
One, what are the competencies
that folks are looking for?
And two, how do they want to weigh
those competencies in order to determine level. Right.
And then we work with them.
We have some more senior content that
we can bring in for senior folks.
We have some more entry level content for
folks that are for interns and University recruiting.
And then we can modularize that and switch and
swap things out to see what works for each
of the companies that we’re working with.
So it is a very white glove.
Make sure that we’re looking for what that specific client
and that specific team would like to focus on.
And we can level based on the
experience that we have in the market.
So it’s really a true partnership between the two.
We’re using our data and our experience to help them
focus on the competencies that are important to them.
And you said it best.
A lot of folks get into this problem where
they’re going and they’re looking online like, oh, my
gosh, I have to get this done.
I don’t know what I’m going to ask. Okay.
I’m just going to look online, pick
three or four questions and go in.
Well, companies, we’re starting to see them work
with us to think about the process because
they’re having trouble, especially in the market today.
And so what we like to do there is just
encourage your team to get to the other, have a
conversation about what competencies you’re actually looking for, and then
come up with a question or two and how you
would level that question and just start with one question,
maybe two per person who’s going to interview.
The minute that you start having even a
checkbox or some sort of defensible position, you
start to think through, oh, okay.
This person came in, they
answered this question this way.
How would I actually level that?
I think I love that a
little bit more junior than senior.
I better add that to my rubric.
And it becomes kind of evolution that can
translate directly not just to your interviews, but
also to performance on site and lovely.
So you’re working with a lot of companies all
over the place with that as a background, what
do you see that companies can do to compete
in the global talent marketplace here?
What can they do more to get
the right people that they want?
Yeah, it’s all going to be about
flexibility in working with a candidate.
So making sure that Carrot is a
24/7 service and this is really appreciated.
In fact, over 60% of the interviews
are done on nights and weekends.
You need to give candidates that
flexibility to interview when they want.
I guess, like I said, more than
60% elect to do nights and weekends.
Speed is another piece that we want to focus on.
Interview at speed of your
candidates is extremely important.
You need to dedicate interviewing time to
create an on demand scheduling for candidates.
Our data shows that a candidate who interviews in
the first 72 hours not only is more likely
to perform in the top tiers, but they also
have better close rates because you have the opportunity
to give them an offer before other companies have.
I think the other pieces to keep
in mind is expanding your applicant pipeline.
You need to diversify your candidate.
We talk about that a little bit, and
you’ll see it interspersed in our conversation.
But essentially, you need to
filter in more direct applicants.
One tactic that we have recommended to clients
is to review the technical interview pass through
rates for each of your recruiting sources.
So if your direct application pool is
close to your proactively recruited candidate pool,
experiment by increasing the number of direct
applicants that you let through. Right.
Like, try loosening your pedigree requirements and
send through 15 or 20% of the
direct applicants to the interview.
There’s a lot more work that needs
to be done on the job description.
Anyways, we talked a little bit
in the beginning about ten X.
When you add ten X to a job, most likely
you’re having people drop out even before you know.
And then the last piece of
something that we touched on. Right.
You need to make your process a lot more transparent.
Once you do start that sourcing and you’re
sourcing more direct applicants, it’s really critical that
they’re being set up for success.
This is why we use human beings.
We don’t send home folks with a test where
they don’t exactly know what the expectations are.
A candidate who has no connection to the
big tech world, they’re not going to have
the same knowledge of the hiring process or
interview questions as a candidate who, say, has
a family member that’s working at your company.
So really just let’s create an equal place for
everybody to get started on the same foot.
Make sure folks know what’s expected so that they’re not
spending time trying to do something in interview that isn’t
going to give you any signal and tell you whether
or not they’d be capable of doing the job.
Great stuff.
So I appreciate it.
I’m a big fan of Carrot.
Technically, interviewing is just a place where a lot
of companies struggle how to get that right.
You have to do it.
You want qualified people, but you don’t want to
limit that candidate pool because they don’t want to
look at this got your technical trivia test what
a lot of them are, and a lot of
good people out there just won’t do it.
But if it’s an interview with a third party, hey, I
will have a conversation with somebody and talk about it.
So I think that what you’re doing is a great thing.
A lot of companies need to be made aware of that,
and hopefully you can spread the word a little bit.
All right.
So who else should reach out to?
Karen, who are you looking for?
Whether companies or people to do interviews?
Who would you like to reach out to or find you? Yeah.
So first, we would love to reach out to candidates.
We’d love to hear about your experience
with Carrot and get your feedback.
I think the second folks that we would
love to reach out to, obviously are going
to be engineers who are interested in learning
and conducting technical interviews, helping folks get opportunities.
We always like to say interviews are
opportunities to change people’s lives, and it’s
a great and rewarding job.
And we have an excellent full of interview
engineers with mentorship opportunities and so on.
And so I always encourage that.
And of course, for companies, anybody who is
looking to hire engineers today, I would say
reach out, talk to us about some of
the trends that we’re seeing in the market.
Talk to us about equitable practices and hiring.
Talk to us about the impact on your business.
If you’re sending out job descriptions that say Ten
X engineer and have very traditionally masculine language, these
are things that we deal with every single day.
And the fact is that engineers a couple of
years ago spent on average, about four years at
their company before they switched jobs, that in the
last couple of years has dropped down to two.
So the engineers that are on
the market, they’re your engineers.
That’s why it’s so important to get this process right and
also to be the culture that you’re trying to build.
That’s an expansion of Carrot.
But let’s say that you hire a
bunch of developers and you have these
equitable practices, and these diverse candidates are
coming through, and they’re getting opportunities to
change their family’s lives with your company.
And they’re really excited.
And then they get inside the job and
they’re like, oh, we don’t have equitable practices.
We don’t see that same rubric
for leveling or growth potential.
So these are really important conversations in
the interview is just the beginning.
So let’s get that conversation
started as soon as possible.
Let’s get you to talk about equitable
hiring practices and learning from all the
things that we’ve seen in the market. Right.
So I pre appreciate your time today.
Thank you very much.
And we’ll be in touch. All right. Thank you. All right.