Insights Podcasts The Testing Show: How to Identify Testing Talent

The Testing Show: How to Identify Testing Talent

January 27, 2020


As it is a new year, we decided it would be interesting to discuss the hiring of testing talent as well as being the testing talent and looking to be hired. What do recruiters actually look for? Do I have the right skill set? What do hiring managers actually look for? How important is a culture fit and why does it matter? Also, what are the areas that we can’t fake on our interviews?

These questions and many more are being asked by Matthew Heusser and Michael Larsen as they talk about the details of interviewing testing talent with Yadira Arevalo and Elle Gee.













Michael Larsen: Hello and welcome to The Testing Show. Happy new year everybody. It is 2020, we are getting started with a brand new year and we decided since it’s a new year, new us, new… whatever we want to do, we thought it would be a good idea to talk about hiring of testers. And to that end we have our regular MC, mr Matt Heusser.

Matthew Heusser: Hey, thanks for listening. Welcome to the show.

Michael Larsen: We’d also like to welcome back a Qualitest regular, Elle Gee.

Elle Gee: Hi, happy 2020

Michael Larsen: I’m Michael Larsen and we’d also like to welcome a new guest, first timer for the podcast. Yadira Arevalo welcome to the show.

Yadira Arevalo: Hi. Happy to be here.

Michael Larsen: and with that Matt, you want to do your thing?

Matthew Heusser: Yeah. So today we’d wanted to talk about how to identify select talent, specifically testing talent and then maybe if we get time, hopefully how, not just talent, but are they a good fit for your organization. We see this as a couple of different ways. How do you tell if their resume is any good? How do you structure the interview processes? Is it one interview is one hour and then you make an offer. Is it multiple hours? How do you do it and what questions do you ask? How can you tell if they’re going to cut the mustard and even how do you know how to customize those questions for your environment? That’s a lot to unpack. Let’s start at the resume and maybe I should flip it just to start it easy. What does a bad resume look like? How do you look at a resume and just go, Oh, they’re not going to be a fit?

Yadira Arevalo: Honestly, a good chunk of time. I don’t focus on looking at resumes. I trust the talent team to go through those and do the first certification of candidates and then I focus on the interview questions themselves, but when I do look at a resume, obviously if it’s 10 pages long, I’m not going to go through reading all of that. A good resume for me has a quick concise summary on their experience at the beginning and a table with specific skillsets that are relevant to the job position.

Matthew Heusser: Maybe we should take this so we haven’t met Yadira yet, you’re the VP of delivery for the LA office for Qualitest, right?

Yadira Arevalo: AVP San Diego office.

Matthew Heusser: San Diego. Okay. And you came up through the ranks at Qualitest, so you had a computer engineering degree. It started out as a tester and you’ve made VP pretty fast. Four or five years, something like that.

Yadira Arevalo: Yeah. Started as a graduate test engineer, moved into a lead position about three, four months in and then into a test manager position and now AVP.

Matthew Heusser: Okay, great. Ellie, do you have any thoughts on resumes? How do you filter out the bad candidates?

Elle Gee: Sure. First I’d speak a little bit about the recruiters and how we use them at Qualitest. We actually have an onboarded recruitment team, so when we identify a job opening, we raise a requisition, we send it through to our recruitment team and they do the first round of identifying candidates on our behalf. Once they have identified a set of candidates that they think meet the criteria that we’ve put out there, we go through a round of interviews within our organization. We try to have at least two people on every interview. For me personally, I believe that that makes for a very well rounded interview. Having said that, that’s where we get to the point where we don’t focus as strongly on the resume. We place our trust in our recruitment team to have used that as a stepping stone to bring the right level of candidates to us so that we can focus on delving into whether or not the skills that we ask for are represented in the candidates presented to us. When I am looking at a resume, the thing that I’m looking for is attention to detail. Testing is all about finding the small mistakes, finding the issues, looking closely, being able to review what you’ve done. So when I come across a resume that’s full of mistakes, that has me wondering about a person’s ability to be detail driven enough to look at what they need to do for a testing position. And the other thing that I focus on when I’m looking at resume is potential. Is there something in there that gives me an indication that this is the right path forward for them and that they might stick because quite often we are looking to people with a computer science background and they generally are looking to move into a development role. I’m looking for anything that could give me an indication that the transition to testing and quality assurance might be something that takes hold and can turn into a career rather than a first job or a stepping stone.

Michael Larsen: So I have a question that I’d like to put out here, and this is something that I have said a number of times. I don’t know if there’s any empirical evidence to back this up or if it’s just a nice opinion that I have because it’s happened to work for me. In the last number of years… I can’t say that anything empirical about this, but we’ve oftentimes said that the resume itself is limited or doesn’t really tell a full story. If somebody is interested in services for a job and they say, “Hey, can you send me your resume?” I actually step back and I say, “you know what? I’ve got a better idea. I have a blog. I’d like you to go and take a look at that because I have a feeling in like the first three or four things you read there, you’re going to know if I’m somebody that is worth talking to or hiring or not.” My question to that is I can say that because I’ve been in this industry for 30 years, but I’ve never really asked a hiring manager or somebody who is actually dealing with hiring how good that advice actually is.

Elle Gee: I’d like to jump in on that one. I can’t say that from our point of view, having a blog per se would make a huge difference to us, partly because on the West coast for our company, we’ve focused really strongly on hiring juniors. We hire individuals straight out of the university. We bring them in, we train them, and we help them to grow in a testing career. However, one of the things that has been common among a number of the juniors or the entry level testers that we’ve hired in the last several years have been those who have created their own webpage, whether that be a webpage for personal use to try building a game application or as some have even built a webpage as, for example, a step for their resume. We’ve definitely gone and looked at that, but I don’t know that I’ve seen many of the candidates that are coming through for us offering any information about a blog, so it hasn’t affected specifically into our hiring to think about the use of blogs and that publication element that comes with that.

Yadira Arevalo: I agree with that. It has not played a factor, at least in my experience thus far, like Elle said, I have seen and taken into consideration where I see the people put in the effort to put a website together or something of that sort. But blogs in particular, haven’t played a role.

Matthew Heusser: So I’m a little bit different. Excelon is pretty boutique when I’ve done recruiting and hiring. It depends on the need and the ask. If we’re looking for someone to come into an established organization that has a way of doing things and they just want someone to follow the process. I don’t know that a blog does much, but if we’re looking to bring in someone more on a consultant basis to help create or standardize or do some training in their spare time, then a blog can be indication that they are thinking and engaged and I’ll go for thinking and engaged every time. And that’ll enable me to have a discussion to see if they’re well-read. And if they’re well read and they’re thinking and they’re engaged, we’re talking about a different kind of critter than someone who it’s a job… and there’s nothing wrong with, it’s a job. There are some organizations where that’s what they want to hire for and that’s fine. The other thing I’d note is that on the resume, a lot of people write, “create and execute test cases and filed bugs”. That doesn’t tell you what they actually did. So I look for a little bit of the kind of bugs that they found and the work that they actually did. But that’s sort of the standard default resume for a tester these days, so I don’t hold it against people too much if they just write, created and executed test cases and filed bugs. Of course you did. That’s the job.

Elle Gee: I really liked hearing what you were saying about what you’re looking for and how the blog could play a role when you’re looking for people, for that role that is more consultancy base. One of the challenges that we often have is because we do hire so many people to entry level and we grow them internally, one of our struggles is finding the right tone in interviews when we are hiring for specific roles that have much stronger requirements are at a predetermined skillset. It was with great interest to me that you spoke about how it could impact your thought process on a higher to see how well read people were or what they were focusing on in the blog realm. It’d be great to sort of dig in a little bit more into how to identify skills in experienced testers because one of the challenges I’ve found in the interview process is, looking at a number of years is not a great thing. You can have been testing for seven years. That doesn’t make you a good tester or a QA analyst or test engineer. Sometimes it can actually make you a bad one. So I wonder, too, if the same sort of thing can be said of being able to write about testing practices. Does that, in your opinion, from your experience, transition into someone who can actually implement those in the workplace and influence the people around them or is it more intellectual and theoretical and not so practical?

Michael Larsen: I think I can give a little bit on this as I’ve written my various blogs, especially on TESTHEAD, which is my personal one, what I oftentimes do is I talk about either my frustrations or the things that I think could be better explained that aren’t, that I actually struggle with, and what I found interesting. When people have read my blog and they’ve commented on it, especially from a hiring perspective, what I had asked them, “Hey, you know, what made you consider me being a good [inaudible] person for this?” is, “well, I read your blog and I read the fact that you were going through and talking about some of the same things that we had dealt with or that we were having frustrations with and the fact that you were experiencing and explaining those frustrations really gave me an indication that you also knew what you were talking about in that regard”. Instead of just saying, “Oh, I do this and I have a perfect view of it and everything’s fine and I can just go in and do your work”, is it a guarantee that I’m going to be perfect for whatever it is they’re doing? No, of course it’s not, but at least it gives them some idea of how I would approach a problem.

Matthew Heusser: One of the things I’ve noticed in interviewing and it’s very subtle and you you often need multiple rounds to pick up on it is when they’re trying to tell you what you want to hear and you typically need to get to a point where you find our contradiction or they use a word and then you ask them to explain it and they can’t. They literally are just repeating something they heard someone in the office say earlier and I think with a blog you can say, “Hey, you took a stand against test cases. Like you said, test cases are dumb. We use test cases here. How are you going to reconcile that?” Anyone that takes a stand is actually in a good shape with me. Hopefully at that point they would say, “well, test cases can mean a lot of things. I don’t know exactly what you mean, so let’s talk about that and see where the middle ground is”. We use these words all the time and assume everyone else knows what we mean by them. Sometimes we don’t. I think that’s when, when culture can come in. Elle touched on the interview process a little bit that you said interviews with two people. How many interviews do you do?

Elle Gee: For most of our hiring in the West coast, we generally have a recruiter level interview and then we’ll have a hiring manager interview. Very infrequently we might introduce an additional layer of interviewing, but two rounds is usually what our candidates will go through.

Matthew Heusser: Yeah, I like that too.

Yadira Arevalo: I just wanted to pedal back. Both Ellie and I were talking about how sometimes we don’t focus too much on the resumes. I think where our focus lies in the process a little bit more, that allows us to trust the talent team to do the first scan of resumes is what we’re asking for when we open the requisition. For example, for junior resources, instead of asking for them to have X number of years of experience or a specific skillset, most of the time what we’ll be asking for is more along the lines of communication, making sure that the recruiters can understand clearly that they can solve just basic logic problems, not necessarily testing skills, more so what we’re looking for in terms of potential. We always open up requisitions with a three must haves and we’ll usually focus on those as the three must haves and then the rest of the requisition might be the skills that most people will have written on there when they open a rec but they’re not the areas that we’re focusing on.

Matthew Heusser: What’s a good basic logic interview question?

Speaker 4: He’re in the San Diego office, we really liked logic puzzles. For example, the marbles within the boxes, we have three boxes with marbles in them and one of the boxes has all black. One of the boxes has all white. One of them has a mixture and all of the boxes are labeled. The one thing you know for a fact is that all of the boxes are labeled incorrectly. The question, can you tell us by pulling one marble out of one box what the correct labels are on all of the boxes? What we want to see is how they work through the problem. Even if they don’t get to the right response. Just seeing how they’re working through the problem, whether or not they can transfer knowledge from one thing that they’ve found to the next, how will they communicate what they’re thinking as they’re working through the problem?

Matthew Heusser: That’s fun. I can see it in my head. One things that has come and gone in testing circles is the use of games. There’s a famous dice game. It’s really about inferring generalized rules from data. And I’m sure Michael has played it a lot. What do you think of things like the dice game? Michael?

Michael Larsen: What I find interesting about these games and what I find compelling to play them in the first place is the conversation that happens while you’re having it. You see their thinking process and you see what type of questions they ask. It’s interesting because sometimes you get people who really get into these games and they get very analytical, sometimes overthinking them or you get some people who approach it and have logical, straightforward manner and they pick up on things fairly quickly. In general, what’s neat is to be able to see how they adapt. If they’re building a model in their head and something that they learn along the way forces them to change how they’re thinking about it. I’ve found that interesting. What you get is you’re watching somebody go through and then they realize that the path they’ve been going down is wrong. Watching them resolve that or try to resolve that, it’s helped me realize what is a pain point for somebody or how do they actually interact with it in a way that helps me understand how they go about their thinking process so that when I work with them or if I get involved with them in a project, I know that going in and I know some of the things that I can either do to help them be at the best of their ability or I know what things are going to be real sticking points for them.

Matthew Heusser: Yeah, you bring up a couple of good points if they are engaged and happy and having fun. The curiosity I think is really important for a tester like, “That’s weird. It’s not on my script but when I do this, when I pull this lever that comes down, that shouldn’t be happening. Hmmmmmmm…” is a thought process that you really want to encourage. I wouldn’t let it be the only thing because some people just aren’t into games and that doesn’t make them bad testers. When I’ve hired for more consulting type roles, you tend to run into the person at events and keep in touch with them over the phone and you sorta throw each other brain teasers and you see how you do and if someone just doesn’t want to engage, that’s not the best sign. Are there any just straightforward questions you would ask?

Elle Gee: I’m terrible. I hate going into an interview is structured questions. That shuts down the interview process before it starts. So for me, I will generally start by asking a candidate why they think they are good for a role. And I will quite often time that. I’ll ask them to give me a response in an amount of time so that I can get an idea if they can apply that on the fly and so on. But I inevitably interview with somebody who has a structured set of questions. So it’s always quite interesting to say the balance of how that goes. Usually what I will do is let them take the lead and I’ll quite often listen because I want to dig into what the candidate says. If somebody says, “Oh, I found hundreds of bugs” and whatever, I want to dig in on that and say, “Hey, so give us an example of a bug” and then I will quite often take that and wander on a bit of a path with them into how they’ve had experiences and what they’ve done in a practical sense to something that they’ve alluded to in a part of the interview.

Yadira Arevalo: I usually don’t take a list of questions but more so a list of topics that I want to hit and if during the interview starts taking a different path, I might not necessarily take it back to make sure I hit all of those points. They might be hitting all the other stuff I’m looking for, but when I am asking questions I tend to go with more scenarios rather than straightforward questions. I might be asking you something like, “we have a long list of requirements. How would you map them to test cases?” Or something of that sort to try to get through the data dump. I have been in situations where after a while, so I have asked for example, whether somebody knows what a requirement traceability matrix is and then dug into ask further like, “can you describe it to me? What would it look like to you?” Et cetera. And they can’t answer. So it depends.

Matthew Heusser: I like to ask, explain something they claim to be good at and that can get that from a resume. You say you do test automation or in conversation, come up with something that they’re interested in and then ask about it. The resume’s usually good because they’ll say TCP/IP and I’ll be like, “really? Tell me about that”. And if they say, “well, I played with it, I’ve experimented with it, I set up my home network”. “Okay, well what is TCP IP?” “Uh, it’s a networking thing.” What does that mean? If you’re going to put it down on your resume as a thing, you know, you should be able to describe it. And then if they can’t, I’ll say of all the things on your resume, pick the thing. You’re the strongest stat. Oh, probably C++. Okay. Then they can give them a problem in C++. That was the single thing on the resume they said they were strongest in and if they really do poorly at that, then we have a conversation. What’s really happening here? I just find the thread to tug on.

Elle Gee: I find that having that thread really makes a big difference, particularly since one of the things we encourage all of our hiring managers to think about, I don’t know about you guys, but when we get particularly more in the experienced tester area, we get a lot of people who come in and they just sound like they’re sprouting Google. I always asked my hiring managers two, look for something that is more than just a data dump. Somebody’s talking at great length. Did they actually at some point in that discourse address concisely the question that you put out there, or did they just spew information at you untilMatth you felt like you had an answer? You need to be able to share information in a concise way that results in an outcome that you can use. So I like to think that there’s some success in the fact that rather than just asking a bunch of questions, we’re trying to dig in and see what the person knows and understands in a way where they can use examples to sell that to us.

Matthew Heusser: Well, that’s real important. Communication and influence skills. If you’ve been around testing long enough, you’ve always seen the technical person who is absolutely correct, but their manner of trying to communicate it actually does a lot more harm than good. How do you figure out, how do you assess someone’s communication skills?

Yadira Arevalo: Yeah, I think it ties back to what Elle was saying about making sure that you’re conscious of whether or not somebody’s actually answering your question. If they can’t really get the response that I’m looking for in terms of not necessarily the content but just the structure, then I would assume that they don’t really know what they’re talking about.

Elle Gee: Some other things we might look for in that same vein is if we have to ask a question multiple times, if we have to redirect multiple times, if the information share can’t be relatively quick and easy in a dynamic working environment, that’s going to be a problem. We’re looking at both the structure of what they’re giving us. We are looking at how long it takes us to pull the answer out of somebody. So for us, we’re always looking for any suggestions on how in a relatively short amount of time and a quick communication with an individual, can we assess that communication factor. It’s so hard to do that in a one hour interview, but it changes the dynamic of the office for however long they’re working with.

Matthew Heusser: Michael, any thoughts before we move on?

Michael Larsen: Very often one of the things that I try to do is I step away and say, “Pretend that I don’t know what’s going on here.” Let’s say that I’m somebody from the organization that you have to explain a problem to and I’m not immediately on your team. I don’t know the ins and outs of your product. I don’t know the details about what you’re looking at. Try to tell me something so that I can quickly assess what’s going on and what we can do to fix it. Advocate for your product for me. It changes the dynamic of the discussion. Now it’s not, “I’m here to look for a gotcha in your tech abilities.” Instead I want to see how do you communicate about something that matters to you.

Matthew Heusser: I’ve done a lot of real simulations. Here’s a piece of software. We want to ship it to production today and I can take on different roles. I can be the vice president of it. I can be the product owner, I can be the other developer. You’re the tester, test it and then after the simulation has ended and they’ve found five bugs, go ahead and write up these bugs as if you would have for the bug tracking system and email them to me at your leisure and then we’re going to see how they actually perform on the job. That brings up, I think the next step, which is how do you customize for job role fit?

Elle Gee: That’s certainly one of the biggest challenges. Once you are looking for those really experienced people, your ETL tester, your strong performance tester, you really have to bring in some more of those structured questions/scenarios, but again with the goal to be able to dig into what they say. That’s where we most often we’ll go down the path that Michael sort of talked to. Give us an example of where it went wrong and how you fixed it might be a pathway we’d go to dig more into the experience level for a tester in a specific skillset.

Matthew Heusser: Yeah. Somebody, I think Joanna Rothman told me, if you asked them how they would handle something, they can make up some awesome answer, but if you ask them to tell you a story about a time or you make them go through a audition, then the real them comes out. If you can’t tell you a story about using a technology they say they have expertise in, maybe they haven’t used it.

Yadira Arevalo: I still feel like communication, critical thinking slash logic or the base for anybody that you hire for any position. So I still am a strong advocate for throwing in there like a logical type of question. Something that might not be testing related. If you can follow the thread of their answer for those types of questions. Most of the time where we’ve been fairly successful using hiring managers that might not actually have the same skill set that we’re hiring for, but they’re still pretty good at digging out the BS essentially. I think it’s still good to come in with more structured questions that have to do with the subject. Obviously you do want to talk to them about that, but I think that, baseline, the questions are pretty much the same.

Elle Gee: One thing we very much want to avoid is thinking that years of experience means that they have the skill set we’re looking for. That’s pretty much our target. Whether or not we’re hiring for a entry level position or a senior is to come out of the interview to be able to have a discussion and say with this person, I think their strengths were here. I think this is how it would fit into our culture and this is where we might need to make compromises. There is no perfect tester because wow, this field is so big. You know, there’s mobile testing, there’s medical device testing, there’s ETL, there’s performance, there’s security. I would love to find the unicorn that had all of those skills, but since they don’t, going into the interview process and then assessing afterwards is really, for me, about identifying where can we compromise, where do we need to really be on top of the skills in order to be able to make that successful? Culture plays a huge role in our hiring. When we are interviewing a candidate, we’re not only thinking about the skillset, will that person be a great fit for the client, but more because we don’t hire just for one client. We don’t hire people to go and work on this account and “Oh no, that account’s gone. They’re gone”. They then need to be able to and back into our pool of employees, transition into another project, So thinking about the cultural, how do they fit in with the particular clients that we’re filling a need for right now? How do they fit in with our office culture? How ready might they be to take on new project challenges? That’s a whole other element that you can always ask a lot of key questions about, but it’s an underlying flow of the whole interview process in every case that we are interviewing.

Matthew Heusser: So I’ve taken up a lot of your time. Really enjoyed it. I think the audience will too. Before we shift out, we do a couple of things in the podcast every time. One of them is final thoughts. Do you have any final thoughts Michael?

Michael Larsen: I think on my end, again it’s just to say that there is no cookie cutter process for this and you’re going to get different people with different takes on it. In my personal view, I think that I’ve had the best success both interviewing and being interviewed where I was able to step away from just the, you know, “how can you fill the need” to “what is it that you bring to the table and can you communicate that well?” If you are able to make a position about something and defend it and also show that what you are working with is reasonable and is actionable and something that other people can either get onboard with or if they can’t, at least you can make an accommodation for it that everybody is if not happy with that at least can work with. I think that tends to go a long way.

Matthew Heusser: Okay. Elle?

Elle Gee: I think the interview process is not about tripping people up. It’s not about right or wrong. I think it’s really important to go into an interview process and create an environment where you can have a conversation where you can talk about the skill set and the questions and the answers and you can take information out of it without putting the person in the position 100% where they feel like, if I don’t get this exactly right, I’ve got no chance because they’ll shut down and then you lose the potential to find out what they can really do. You lose the opportunity to assess their potential. The interview process should be about encouraging a dialogue where we talk through the logic of something and it’s okay to make a mistake. Quite often in a calculation, you can make a mistake at one point that will change the entire outcome. If the logic of what you’re doing is correct. Then you’d be able to go back and find that mistake and fix it. You just have to have the chance to do that and I think that’s helped us to be successful in hiring people who are good at what they do and love what they do rather than people who can just check the boxes.

Speaker 2: Great, thanks. Yadira?

Yadira Arevalo: I think we need to be careful about us hiring managers going into interviews, trying to fill a specific position only we need to be conscious that once they project ramps down, they’re going to be moved to a different position. So we need to think about it from the perspective of what’s the type of team member that we want to gain. Not just filling a position per se, because once they join our team, we have to continuously gauge their potential for other positions as well. It doesn’t end once the person has been hired, we need to be very conscious. I think of those three things, the communication, the critical thinking and overall the cultural fit.

Matthew Heusser: Great. Yeah. I think that sums it up really, really well, but I will say it’s time for where can we find out more about you and what do you have going on? So right now I’m working on my submissions for a couple of conferences. Michael, anything special going on with you?

Michael Larsen: You literally said exactly the same thing. I have in the sense that I’ve got a couple of conference submissions that I am putting together at this point, but otherwise this is quiet time and I am grateful for it, so I’m going to enjoy it for the next couple of months.

Matthew Heusser: Yeah. One thing that I can talk about publicly, but I don’t have out yet is I’m working on a class on how to teach testing using simulations and games. Yadira, what we typically do here is talk about webpages. We’ve got out articles where we’ve got coming out where they can go to learn more about us. In your case, what’s going on in LA, maybe even what you’re hiring for? If you think that hiring is going to last more than the next 20 days or so, so we can get the podcast up.

Yadira Arevalo: Let’s see. Well, right now I’m focusing more so on growing my technical skills so I don’t have anything specific coming out. In terms of hiring in the San Diego area right now. We do have some open positions for juniors. Actually. Hopefully we’ll continue to see the team growing. 2020 had a lot of growth this past year and hopefully that trend continues.

Matthew Heusser: Okay, and Ellie

Elle Gee: Qualitest West coast is always looking for entry level positions. We encourage anyone who’s interested in taking a path into QA and testing to consider coming and working for us. We’ll post a link to the quality test website in the notes for this podcast. For me personally, I can be found on LinkedIn for anyone who’s interested to find out more about Qualitest. Again, we’ll post another link to the quality test website to see what it is we do and how we go about it and thank you.

Matthew Heusser: Thanks for being on the show, Elle.

Elle Gee: It was a pleasure to join you guys again. Thank you very much for having me back.

Yadira Arevalo: It was great meeting you, Yadira. Thank you.

Yadira Arevalo: Thank you for having me.

Michael Larsen: All right, let’s call this a day. Thanks everybody. We will see you all on the testing show next month.