Noah Sussman knows a bit about bringing Dev-Ops to fruition at a variety of organizations. To that end, we asked him to come join us and tell us a bit about what Dev-Ops is, and what it isn’t. We also discuss the ways that testers can develop their skills and become more technical (hint: it’s not as difficult as many may think, but it will require work and interaction with others to really be successful at it).
Also, how does Bing Maps manage to put Melbourne, Australia in the Ocean somewhere near Japan?
Panelists:
References:
- Microsoft accidentally ‘lost’ a city on Bing Maps and its excuse is bizarre
- A/B testing
- statsD
- New Relic
- OSI stack
- Chrome Developer Tools
- Firebug
- Infinite Undo
- How to Teach Yourself to be a Technical Tester: Some Thoughts
- PHP
- Test Master’s Academy
- NYC Tester’s Meetup
- NYC Selenium Meetup
- NYC Continuous Delivery Meetup
Transcript:
MICHAEL LARSEN: Hello and welcome to The Testing Show. Today we are joined by Perze Ababa.
PERZE ABABA: Good morning, everyone!
MICHAEL LARSEN: Justin Rohrman
JUSTIN ROHRMAN: Hey there, good morning.
MICHAEL LARSEN: I’m Michael Larsen, your host and producer, and today we are joined by Noah Sussman. Say “Hi” Noah.
NOAH SUSSMAN: Hi, how’s it going?
MICHAEL LARSEN: It’s great. Thanks for joining us, we’re glad to have you on the show. For those who don’t know you, which I’m guessing with our audience is probably next to nobody that would be listening, can you give our audience a bit of an introduction and tell us a bit about yourself?
NOAH SUSSMAN: Sure, thanks for having me. So I did a lot of the continuous delivery architecture at Etsy, from 2010 to 2012. After that, I consulted for about three years, and now I’m at a company called Tutors Pay Teachers, also doing continuous delivery architecture, and working with some of the same people I worked with at Etsy. So, I’ve spent a lot of time thinking about how to fit testing particularly into continuous delivery and rapid release cycles… and I expect that’s a fair amount of what you want to talk to me about today.
MICHAEL LARSEN: Awesome, and… yes, as a matter of fact that is [laughter] right up the alley of what we want to talk about. Also, as is common with us, we like to take one news story or one item that is currently floating around that we found amusing, and riff on it for a little bit, so our news segment today comes to us courtesy of Microsoft. The recent Bing Maps “confusion”, to put it in a good way, is that Microsoft recently misplaced the Australian city of Melbourne on its Bing Maps, by placing it right in the middle of the Pacific Ocean, right near Japan. Needless to say, something was clearly wrong here. What it turns out had happened was that the glitch was due to a missing negative sign in Wikipedia data… and on that, let’s go!
JUSTIN ROHRMAN: So how are they generating these maps? Are they just siphoning off data from Wikipedia or whatever source and putting it in their map generator?
PERZE ABABA: That’s what I suspect, right? There’s gotta’ be, like, thousands or maybe even millions of data points that they consider, this is most likely a machine learning engine that kind of mis-pointed, and then I’m going to also safely assume that there’s specific weights from these, what we’d probably consider as oracles, when it comes to gathering data. But yes, so I guess we can see now that Wikipedia is very important in the Bing Learning Engine. [laughter]
MICHAEL LARSEN: I mean it sort of stands to reason considering the fact that, on one side, Wikipedia is, of course, an area where there’s a lot of data that has… I mean, if you think about it, over the course of twenty some odd years now, I think. I don’t remember when Wikipedia actually went online, but it’s been quite a while. There’s a good chance for a lot of this stuff to be authoritatively covered, but yet at the same time, it can be edited by anybody. So at any given point in time, somebody can go in and tweak with that data, and it just seems that that being the core place for getting your coordinate data for a city for something like Bing Maps, just seems a little irresponsible to me [laughter]. I don’t know, is it just me?
JUSTIN ROHRMAN: They do have a pretty rigorous fact checking group, like a group of volunteers that do fact checking on every article edit, but there is some temporary amount of time for every change, where bad data could be inserted. Also it seems like there’s an argument that Bing did what it’s supposed to do; it took data and siphoned it in and then just built the map off of it. It’s data source was just bad.
MICHAEL LARSEN: It does say that they don’t rely Wikipedia data much, only for rich description on the Map web site. Yet at the same time, if it’s going to place the city right in the middle of the ocean, it seems that it obviously looks that that data much more closely [laughter] than it says that it will.
JUSTIN ROHRMAN: Yeah, I guess that bug is fixed now, so I can’t go to Bing and see Melbourne near Japan?
MICHAEL LARSEN: Yeah, that’s a shame… if nothing else, it could be very entertaining to say “OK, can you give me driving directions?” and see what it would tell us.
PERZE ABABA: Yeah, that would be great. Remember that time in Google when they would allow you to walk from one continent to the next, and in between bodies of water it actually tells you to swim from point A to Point B.
MICHAEL LARSEN: [laughter]
PERZE ABABA: It would have been great if we would have seen that [laughter]. I think they’ve fixed that now, that was around maybe 2006, 2007.
MICHAEL LARSEN: I haven’t had a chance to see anything that entertaining in a while when it comes to plotting out those courses, but there’s always room for improvement or, if nothing else, another chance for a bug to pop in so that we can be entertained. All right, let’s go on and give Noah a chance so that we can ask him about his area of expertise and the current changing world that we are facing, and that is the evolution and growth of Dev-Ops in organizations. So Noah, you feel in the mood to talk about this with us this morning?
NOAH SUSSMAN: Oh yeah!
MICHAEL LARSEN: Awesome! I guess for many of us, Dev-Ops is something that some organizations are actively doing, or that want to be doing it, but were stumbling on our way to get there, and I think it’s safe to say in my organization we’re somewhere in between those two states. We’re getting to the point to where a lot of what we do is… I don’t want to say “hands off”, but we do push more frequently, we’re able to try out experiments, and see if they work in somewhat real time, and we’re able to learn from it and fix it fairly quickly, but there are other areas that, of course, we’re struggling with. For those who are not necessarily familiar with Dev-Ops, which I assume is a fair chunk of our audience, where do we start with this conversation?
NOAH SUSSMAN: There’s a lot of different definitions of Dev-Ops floating around. At this point, as with any process improvement framework, there’s going to be a lot of different versions. If you just go with the Dev-Ops brand, and Google at this point, it’s become like Agile, to the point where you can find a definition for Dev-Ops that fits whatever you’re doing. It’s been watered down to the point that the term doesn’t mean a whole lot. The goals that the organizations that I have worked with have really been focused on, like you said, the experiment. I think that’s the single biggest value to focus on. I think it’s really easy, in my experience at least, for managers to get lost in some of the more abstract LEAN ideas, like the value chain, and bringing it back to “we’re supposed to be able to learn fast, and that’s the impetus for wanting to move fast”. I found that to be a good sort of focuser.
The company that I work for now, for instance, the focus has all been building on an experiments framework. That informs the rest of the activities we do that are commonly associated with Dev-Ops that are hard to sort of figure out where to start and where to end. A lot of people like to talk about Dev-Ops as breaking down organizational silos so you can go the way we did in the days of Etsy when I was there and go for a really flat organization, and just break down all the silos, but without a focus on “what’s the end goal?”, all we really wound up finding out is some of those silos are valuable. People do have skills that they focus in, and it is a modern society.
People are specialists, just going out and blanket declare “everybody in Ops needs to learn to code” and “everybody who writes code needs to debug production”. I mean, those are great goals, but you can certainly wind up in a situation where you’ve taught your whole Ops team to code and you’re still not deploying rapidly. So I think keeping the focus on being able to experiment, and being able to get the results from those experiments back to the people who are doing the design, who are building the systems… I guess what I’m saying overall is that it’s useful to think about Dev-Ops as shortening feedback loops rather than just breaking down organizational walls or moving faster, because the focus on experiments, the focus on feedback loops, that immediately leads to asking “who’s supposed to be in this loop?”. And that leads to figuring out where maybe there’s too many steps in a process, or too many channels to go through between different groups in an org.
Again, a lot of the work I’ve been doing lately is focused on getting the right information to the right people as fast as possible. A lot of what I’ve been focused on the last couple of years is working with Customer Support to get feedback of our production behavior, funneled back to Dev-Ops, QA, whosever’s most qualified and available at that moment to respond.
The focus on Customer Support, for me, only comes because, for the companies I work with, the impact of changes that we are making are pretty good. If we push out a tweak to a search algorithm or a new check out button, we’re going to have a lot of instrumentation on that. We’re going to be watching as it goes live, those feedback loops are actually pretty well covered, and where we’re finding we still have work is all the rest of the production behavior. So that’s been pretty interesting to sort of feel like in my career I’ve gone through shorting the feedback loop between dev and ops, and like Dev-Ops and test into the loop and that works pretty well, and then this sort of final level of dev, ops, test and support, all working together seamlessly, you know just continuously, you know just continuously improve the web site.
JUSTIN ROHRMAN: So you used the phrase “experiments framework” a little bit earlier. I’ve never heard that before, and I was wondering if you could talk a little bit about that and maybe give a concrete example of what an experiment framework is?
NOAH SUSSMAN: So I think that the experiment framework that everyone is most familiar with right now is Facebook. I would say “the experiment framework used by Facebook” but the reality is Facebook is just a big platform for iterating on Facebook. That’s one of the interesting things about the Facebook model; for all of us who have been building sites for a long time is, they really don’t have to care very much about consistency. The UI is constantly improving and changing out from under you and that’s become part of their contract with their users. That’s part of what you’re getting when you use Facebook is that constant evolution.
All of those changes are tracked. They’re monitored in production. There are dashboards associated with when you click a button, there’s a dashboard somewhere that has an uptick because you clicked the button, and there are product teams making decisions based on that data in real time, and then feeding that back to you, the user, by improving the UI. It’s really just the capability to push out a change, make the feature live and then see the user impact, in some kind of quantifiable way at scale, and then to be able to roll out incremental improvements to that feature, or just turn it off, without impacting the rest of the site.
Another important aspect of experiment frameworks is to do what we call partial rollouts. You know, when you A/B test something, you show a new version of the feature to a small number of users, in production, and then you look at how… usually it’s something that’s easily measurable, like convergence, so you would show the new version of the feature to a small number of people and see if they convert at a higher rate, and if they do, then you might want to show more people that feature and see if the conversion holds steady, and if it does, then you might want to roll that feature out to everybody and have the old feature go away. That’s a really critical part of an experiments framework. The idea of an experiments framework probably comes from that practice that we’ve all done of A/Bing something that just happens to be measurable and important like convergence or like money going into the company’s account, new user signups. The experiments framework is really just n extension of that where you take this modern capability of statsD or New Relic or any of these tools that sprung up in the last five years that let you arbitrarily monitor any part of your application stack, and with those you can now do these A/B partial rollouts on any trivial aspect of the site, and that’s very much in the spirit of continuous delivery; we want to be working with trivial changes. We don’t want to have big scary changes. Lots of little, non-scary changes that are easy to reason about, easy to understand. So that’s what an experiments framework ultimately buys you is every rollout is where you just happen to have a metric going in real time that’s important to the business that you can check. You have the capability now to create that kind of context for just anything you’re working on.
Final optional piece of an experiments framework could be that you allow your users to opt in to really bleeding edge experiments. So Google Labs was a great example of this. Maybe they still have this, I’m not sure, where you could go and just sign up for some experimental features of Google Maps, with their contract of like “yeah, there might be some bugs, but you get this bleeding edge feature”. For companies that are really good at doing this, that can become a really powerful tool to have this pool of committed early adopters who have a tight feedback loop with you. You know you can push out something a little bit experimental, a little bit controversial, and you can find out how users actually interact with it. I think that’s a really important end goal, because there’s so much value in being able to bounce an idea off of a large group of early adopters when you’re a large company. I mean, that’s one of those things that, I think, large companies lose is that early adopter pool. So the end game of an experiments framework can also be that you recover a little bit of that capability that you had when you were a smaller company, and you really had access to those dedicated early adopters and could get their feedback.
PERZE ABABA: From what I’m gathering from these points that you are mentioning, Noah, is that it seems that Dev-Ops is not just something that we do. It’s a culture of some sort that you build within your organization that pretty much answer both the questions “how” and “why”. Most marketing material you see out there just addresses the “how” part, and not necessarily the “why” part, so I think this is a really good insight into how we should actually go about developing our identity as an organization that’s trying to get into Dev-Ops. My question, as a follow-up for that is, as a tester, not many organizations have a training program that can get you to bring your skills up so that you can actually run with the developers and the ops folks. Is there a specific set of skills that you expect testers to have, whether it’s black box testers and transition them into something more, or us to be able to contribute well in a Dev-Ops type culture?
NOAH SUSSMAN: I don’t think there’s a general answer to what skills does a tester need to learn. It’s kind of like “tell me what kind of tester you are, and I’ll tell you what skills I think you need to learn, and what your path to them would be”. One thing I’ve seen a lot is that there’s a lot of good interaction to be had with Operations people because Ops is historically also not a programming role, but at the same time, Ops are very technical. They’re expected to understand networking. They’re expected to deeply understand the OSI stack. They’re tasked generally with understanding the environment in which the application runs. Looking at the environment in which the application runs is a piece of what a lot of testers do. There are a lot of tools to be learned just from learning about systems operations and asking how they would solve a testing problem. An early example of where I started to realize this was debugging CSS dependencies. This is something that now you could do all with Developer Tools or Firebug, but those weren’t around at the time that I’m talking about. This is like the early 2000’s and you’d have these situations where there are style sheets being built on the fly, there are style sheets that use the style sheet include mechanism to include other style sheets, so you have this hierarchy on a large website. The impact of any particular CSS depencency not loading can have this really non-deterministic impact on the layout of the page. It might be anywhere from nothing to a broken column. So that got me thinking “is there a way to see what requests this browser is making?” and on the server there is a way to see which CSS files have been served, why is it do hard for me to look at that?
Just by asking that question, I quickly learned that, sitting just down the row from me are these systems operations people and, lo and behold, they have the capability to see those attributes on servers. So I was able to start to pick up some of those skills. Today a lot of that is built into the browser inspector tools, Firebug and Chrome Inspector, so those also can be areas where people who are doing a lot of browser testing today can build their skills, going through and understanding what all of those tabs can do for you in Chrome would be a real learning project. I don’t pretend to understand what every single tool in there does, and I use it for work quite a bit.
MICHAEL LARSEN: I think it, which is another whole conversation we can have here is the distribution of skills that we ultimately end up developing, especially when we take on additional roles, because when you look at a Dev-Ops environment, of course, the idea isn’t that you’re individual silos and that you do one thing and push it off to someone else to do their thing, etc. I guess the question I would ask here is “how do you encourage, whether you’re testers or you’re expanding into new roles, or you’re getting that opportunity. How do you facilitate those conversations?” How do you encourage people to step out of their comfort zone, and get involved with us, basically? How is somebody who is “well, that’s not really my role or my job!”, but, if you’re going to survive in a Dev-ops environment, you kind of have to make it your job, right?
NOAH SUSSMAN: That kind of interaction that you’re describing, I mean, I think that’s how the transfer of knowledge is supposed to work. There’s no condition where you’re just going to sit the dev and the ops and the testing team down together in a bunch of training sessions and they’re all going to trade off information and then everybody’s going to go back to working without talking to each other. One of the biggest cultural changes in terms of Dev-Ops vs. how we’ve done these things in the past is the use of chat. Chat, I think, is so popular because it facilitates having these quick lateral conversations with people. So how do you facilitate that I think is not really a question that is answerable by any one person in the trenches other than the sort of obvious, canonical inter-personal stuff like “be nice, listen, make small talk”, but in terms of the organization, I think that’s one place I really look to a CTO or a VP of Engineering to help set the tone of “we never should be jumping through hoops to get things done”. Everything that someone feels that they need to do in the course of their work should be something that they can just do without having to check in across a bunch of different silos. For a lot of organizations, that’s a huge shift to placing a lot of implicit trust in individual employees making changes to the website. Even for organizations that are newer, and are more formed around this idea, everybody who has experience is coming in with experience in organizations that were much more conservative about letting people just do stuff. The longer your career is, the more likely you are to have spent a lot of your career in those kind of environments because that’s all there was up until five years ago.
It really is critical to have management that, one of their management values, should be “create this spirit of cooperation” and make sure that that’s valued. And again, I think the most direct and easiest way that they can do that is just to say “if you’re going to do your job, you shouldn’t be waiting for anybody”, and then make that happen. If it is hard to make happen, then tell your manager. It’s really as simple as that, but I think it does have to come from the top.
MICHAEL LARSEN: All right, we’re getting to the point where we need to wind down the podcast, but I did have one question that I wanted to ask Noah since I’ve got him here. For those who are not familiar, Noah also has a blog that he writes called “Infinite Undo”, and the blog post I’m talking about was written a few years ago called “How to Teach Yourself to be a Technical Tester”, and one of the things I really liked is Noah presents it as a table of contents idea with individual suggestions. For those who might say “I’m not a technical tester” or “I’ve not really had a chance to work in this environment” you give a very prescriptive from the start to an end goal that’s pretty comprehensive. Now this was written three years ago. Here it is, it’s 2016. Were you to go back and say “let’s go for a Second Edition”, is there something you would want to put in this 2016 Edition of “How to Teach Yourself to be a Technical Tester”? What could we do to make this even better?
NOAH SUSSMAN: I think I tried to pick tools that weren’t going to date, so I don’t think I’d change any of the tools or the specific recommendations. I think one thing that I definitely learned once I put that up is that it could use another post that’s an introduction and how to use this, because I got a lot of different questions online, and a lot of them were along the lines of “what if I want to change part of this and what would that mean?” that’s something I’d just take as read, I guess, with these kind of articles on the Internet is that, of course, you’re supposed to take what you like and leave the rest, but I could have made that clearer. I think I probably could have also made it clearer that, for the most part that list is not about “learn these specific tools”, it’s a list of tools and practices that you could do if you wanted to do everything for free, and if you didn’t have anyone around you to learn from. But that’s not true for most people, so for most people, it would be more useful as just kind of a rubric that they could share with their friends and say “OK, I’m kind of trying to get here, kind of get this general assortment of skills together, how would you do this?” If I went back and revisited it, that’s a lot of what I would add as just sort of context and scaffolding around it. For instance, I picked PHP as the “hey, go teach yourself” language because PHP is marketable. It’s important. But mostly, PHP is the fastest way to get a web site up and running. It’s very satisfying. You just basically type “Hello World!” and a couple of brackets and now you have a web site. And that hasn’t changed, I mean, that’s what PHP is. Nobody’s out there trying to replace PHP.
MICHAEL LARSEN: As a reminder, Facebook is written in PHP, just for those curious.
NOAH SUSSMAN: Yeah. There are people trying to do better than PHP and improve on it, but really, PHP is a solid choice if you just want to throw a web site together and poke at it. So I mean, again, that’s the kind of stuff I would make a lot clearer. I’m not saying you need to be a PHP developer. I’m not trying to take a side in like a language architecture position and say that PHP is a good language. I’m just saying that, for the purposes of this tutorial you need to have a website up, and PHP is the fastest way to do it. There’s a lot of foot-noting and context I could do there, all the way through. But I think, looking at the piece like two years later, I mean it does what it was intended to do which was to be a really, kind of impossibly long list of things that you could do, and that it’s really intended to inspire the reader to go “OK, well, I’m not going to do all of this, but what could I do?”, and it seems to work well in that respect.
PERZE ABABA: You know, Noah, this actually is a really good, I guess, entry point into coming up with like a forty-five day testing challenge, and just go through each of the steps… I think it’s pretty cool.
NOAH SUSSMAN: Yeah, if you did… Thanks! If you did all of that stuff, you would definitely be in a pretty good place to be hirable because I’ve walked people through basically the same stuff a couple of times at work. We didn’t start from the very beginning, it’s like you sit down with them and say “well, what have you done? OK, well, let’s go here or let’s go…” I think it would be great because a lot of being able to be hired at a more technical level is just being able to get through the interview. I would think anybody who’s listening to this show and, like, interested in this already knows all the stuff about testing and test design to get through the interview. So you really just need to be able to get through the parts where it’s like “OK, well, how would you count how many PHP files are in a directory? How would you tell if this web page isn’t loading? How would you tell if the request is going through?”
When I’ve guided people at work to getting to that level, then all of a sudden they’re like “oh, I’m really hirable now!” I look at it and it’s like “well, you don’t really change much, you learned a couple of new skills around visibility”. The rest of it is sort of the same thing they brought, that they had coming through the door.
PERZE ABABA: Yeah even like the section that you were talking about with regards to headers, like the complicated caching mechanisms that we put up on our sites, I’m still very surprised that, when I interview a lot of candidates who’ve been around for awhile, that still seems to be something that they don’t care about. “Oh, it’s just there, and it’s information as something that I can hardly derive anything from. You’d be surprised when you tell them about “well, take a look at cache hits, take a look at all of these other things, and that can actually tell you why your site is loading a lot slower now than what it was before, and stuff like that. This is actually… maybe I should take that challenge [laughter], and jump into this.
NOAH SUSSMAN: Yeah, I mean I’d say if it makes sense, if you can see a way to write that, you should definitely write it.
MICHAEL LARSEN: Thank you, Noah, that’s awesome! All right so we are at our “Shout Outs and Shameless Plugs” portion. So we want to give everyone a chance to either talk about something that they are doing, where they are going to appear, or something they found pretty awesome or cool. Justin, you got anything you want to share?
JUSTIN ROHRMAN: Yeah, I’ll be at Test Master’s Academy in New York, there’s a peer workshop on Sunday and then a Conference Monday through Wednesday. It looks like those dates are September 25th to 29th. So I’ll be at the peer conference and the conference conference.
MICHAEL LARSEN: Excellent! Perze, how ‘bout yourself?
PERZE ABABA: All right, so piggybacking on that, there is also a corresponding NYC Tester’s event that will be happening hand in hand with Test Master’s Academy. So during the conference, which I believe is on that Wednesday, we will have a dinner, or something that just gives us the ability to say “hi” to everyone that participated in the conference and NYC Testers is helping out for that. That’s pretty much it for me as well.
MICHAEL LARSEN: Noah, how about yourself?
NOAH SUSSMAN: I will also be at Test Master’s Academy, and I’d also like to just plug the couple of testing meetups in New York that people who are interested in these kind of questions should be attending. So there’s the NYC Tester’s meetup that Perze mentioned. There’s also the NYC Selenium meetup, and there’s also the NYC Continuous Delivery Meetup. So I think all of those are great venues for learning more about these kinds of issues, and I am there at each of those from time to time as well.
MICHAEL LARSEN: And for my side, seeing as Noah is our special guest for this, my shout out and my plug as a resource is going to be a recommendation for the article that he wrote about “How to Teach Yourself to be a Technical Tester: Some Thoughts”.
That’s the end of another “The Testing Show”. Thanks for joining us, and we’ll talk to you again in two weeks.
JUSTIN ROHRMAN: See ya’!
NOAH SUSSMAN: Bye!
PERZE ABABA: See you, everyone!
