The Testing Show: Virtual Conference Testing
The title this week may seem odd but it is indeed correct, as Michael Larsen participated in and in effect helped to test out the various challenges and solutions that go with putting on a virtual conference in this current period of time.
In this episode, Michael talks with two long time veterans of the Pacific Northwest Software Quality Conference, Philip Lew (Program Chair) and Bill Opsal (a long time volunteer and troubleshooter) about the ups and downs of putting on a virtual conference, things that went well, challenges faced and some lessons learned that may help anyone with putting on a virtual conference in the future.
- Philip Lew (LinkedIn)
- Bill Opsal (LinkedIn)
- Pacific Northwest Software Quality Conference (PNSQC)
- Get The Balance Right: Acceptance Test Driven Development, GUI Automation, and Exploratory Testing
- Test Guild Conferences
- Slido: Audience Interaction Made Easy
Michael Larsen (00:00):
Hello, everybody and welcome to The Testing Show. It is the middle of October, actually, the day that we are recording this, it is Tuesday, October 13th, 2020. It is 12:09 p.m. on my clock. As I’m looking at this and we are doing a special kind of a show in the sense that we are doing a show live from the show floor… or in this case, what constitutes a show floor. At this moment, the Pacific Northwest Software Quality Conference is running. We are currently in the lunch break for day two. This is a different kind of experience for all of us that have been participating with it, so I thought it might be interesting to get a bird’s eye view of this from two people that are actively involved in helping to put on this conference. One from a coordinator and chair side of things, and also from the logistical side of things. So we’re going to hear from two people today, however, as is often the case with technical details. There’s a challenge at the moment. So we are currently talking to Philip Lew.
Philip Lew (01:05):
Hey there. Hi everyone. Thanks for inviting me on, Michael.
Michael Larsen (01:09):
Thanks for coming. Now, Phillip, I know that you’ve been involved in PNSQC for a number of years. I believe you are the content or the program chair.
Philip Lew (01:17):
Right! Well, my first experience at PNSQC was in 2011 or 2012, I believe. I submitted a paper and I was lucky enough to get accepted and was grilled quite thoroughly by my reviewers to turn out a decent paper and went there and presented. You know, I went up to the volunteer table. I said, “Hey, can I have one of those t-shirts?” And she said, “Well, yeah, if you volunteer.” And so here I am. Now I’m the Program Chair. [laughter] And like you were saying, I handle all the, getting the program together, which includes reading all the papers that come in and so on. So it’s quite a job.
Michael Larsen (01:56):
That’s an interesting place for us to start with because PNSQC is a little different than many of the other conferences, at least from my experience. PNSQC prides itself and has a very specific attribute to its program in that it expects (and up until a number of years ago, I think it required) if you’re going to be a speaker at the event that you had to submit a technical paper. That was part of the process. I attended the conference the first time in 2010. Same deal; I volunteered for it because it was one of the first software testing conferences I’d ever attended. Part of the volunteer was exactly, as you said, it’s like, “We do papers.” And part of that, getting the papers ready for publication is to do paper review. So I’ve been reviewing papers for PNSQC since 2010. It is definitely a different experience. So from the program side, what is the benefit of having a speaker present a technical paper?
Philip Lew (02:58):
Yeah, you know, compared with those other conferences, having a technical paper, cause some speakers are not good public speakers and that’s just the way it is, but that doesn’t mean that they don’t have some great ideas and content. So we wanted to be able to give folks the opportunity to share their ideas, some really great ideas that they can put down on paper and writing, but maybe they’re not such great speakers. And in that case, the attendees are a little bit more forgiving, I think, in that they don’t expect professional speakers, people that are on the speaking circuit, but that we have some great content. If you don’t have such a smooth talker, then you can go to the paper and really get some good stuff out of the papers. And so for that reason, we produced proceedings every year and so on. And so I think that’s really makes our conference special.
Michael Larsen (03:48):
I agree. I’ve actually explained this to people that, this has actually been a springboard for me a couple of times in regards to job opportunities and promotional opportunities. The current company that I’m working for right now, one of the things that when they were asking, “what is your experience with…” They’ll fill in the blank. Instead of my just saying, “Well, you know, I’ve worked with X or I did something here”, I was able to say, “Would you be interested in a paper that I have published through the proceedings for the Pacific Northwest Software Quality conference?” And that particular paper was the one that I did back in 2011 called “Get The Balance Right.” And I said, “Here, if you want, you can go to their website and, listed under Proceedings, here’s the paper listing. And that was really cool, at least from my perspective, because I was able to say, “Hey, look, I have a published paper with this conference body. It’s been peer reviewed. It meets academic standards”. And it helped me get the job. So I’ve always been very proud of that association. Have you had a similar experience with others who’ve said that by writing these papers… it’s definitely, I think, a net benefit for their presentations… but I mean, beyond the conference?
Philip Lew (05:00):
Well, you know, the thing about like you’re saying there was a paper in 2010 and 2012 and think about the proceedings is that they live on forever and they’re actually referenced very heavily by Google. So because we publish these proceedings and the associated keywords with each of these individual papers, your paper is actually referenced by others, which really adds to your professional credibility. And like you said, it stands for something. Other than that, you can just talk a good story that you’ve produced the paper on, especially one of the things about our conferences that focus on hands on assistance for somebody that’s trying to get stuff done, showing your experience, how you coded this, or did this work with acceptance testing or in your case, accessibility work and made a difference. And the results that you’ve got, those are the kinds of things that potential employers, they really get excited when they see that you really did some real work and that people are acknowledging your work and referencing your work.
Michael Larsen (06:00):
I definitely agree on that. And also, thank you for the plug on that, because on day three… though, this is a little different. In the past, usually there is a general program that runs for two days and the general program has four or five, six different tracks running at the same time. There is a whole squad of volunteers that help run those tracks, moderate them, introduce the speakers, make sure that the room is set up correctly. And for those who’ve not attended, for the time that I have been attending the conference, it’s been held at the Portland World Trade Center this year. This conference is entirely virtual that’s because of COVID, of course. As the program chair, how did you have to adapt? What would be the normal approach to the conference to being able to present it as it is being presented now? And we’re talking about it right now, cause it’s running [laughter]. So…
Philip Lew (06:54):
Yeah. Adapt. One of the things that I talked about in January when we kicked off our conference call for papers was that one of the things about being humans is being able to learn very quickly. And so we had to learn very quickly just to adapt. And doing this conference online, that’s for darn sure. I was speaking at another conference and they messed up on some links and somebody called me and said, “Oh, they messed up your link. And geez, I don’t know what’s wrong with these guys”. And I said, “Well, let me tell ya, this is a lot of stuff going on in the background in terms of organizing a virtual conference. Not that it’s any harder, but it’s all new and we’d have to learn how to do it”. You know, if we’re going to do it again, I’m sure that things will go smoother for not only ourselves, but other folks that run conferences, just learning how to do this stuff and organize everything and sending out the lengths and getting the speakers on board and engaged ahead of time, practicing their video and their audio. I mean, it’s just not like, you know, tell them to show up at their room at 1:30 p.m. and make sure that they have their own laptops. You have to think about a lot more detailed oriented, technical stuff.
Michael Larsen (08:04):
Another feeling ,since tomorrow on Wednesday, In fact, I’m one of the closing acts for the conference… I will be presenting a workshop based around Accessibility testing and Accessibility tools. And part of that process was to determine do I want to do a full recording of the talk which I have done, or do I want to be doing it in real time with the people that are there? This forced me to say, “Hmm, what happens if we have the workshop come up? And for some reason, people can’t connect, what do I do?” Well, the positive side is I have a recording. I’ve already walked through all the details that I can present to them and say, “If you go here and click on this, you can at least watch this”. I’m hoping we’ll be able to do this in real time. And so far, it seems that the way that the conference has been going, there haven’t been too many technical difficulties on that end, but we never know. There’s always a chance that something could go sideways and it’s better to be prepared than to be caugh unprepared and say, “Well, sorry, there’s no workshop happening.” That would be bad. [laughter] So in this case, we actually have recordings of all the stuff that I’ve done and I’m still kind of going through and I’m actually looking “Well, you know…”, and that’s also something that I thought was interesting. So as I went through and I looked at my first recording, I had sat down and I was going, “Hey, I know I’m going to use a green screen and I’m going to be able to blank it out. And then I can make something look very nice and clean.” I realized when I was looking at the green screen, the green screen wasn’t quite flat. Like it wasn’t perfectly flat and taut. So when I tried to do an overlay of my background on it, it looked distorted. And I thought, “Ugggghhhh!” So that was one of those times I thought, “Okay, let me redo this”. And so I read it another where I was just saying, “I’ll sit in front of the computer, like it would, and I’ll just do it like a screencast.” So I have that. I’m like, “All right, cool.” And as I was watching what I was saying and listening to it, I thought, “Okay, I can edit out a few things here. Maybe I can cut these into segments so that instead of them having to wait and download a two hour video, instead, each little clip is cut up into 10 minutes walking through”. It was like, “I could do that better. I could do that better. I could do that better.” So at least from my perspective, I feel that this whole situation has forced me to be better. Cause it wasn’t like, “Oh, I’m just going to show up and wing it.” I don’t have that option to show up in a way I need to prepare for this in a way I’ve never really prepared before, which I think is good. So I don’t know if that’s something that anybody else has mentioned.
Philip Lew (10:36):
No, I think that one of the things that we did to avoid possible technical difficulties, as you know, is we did require folks to submit a video for the paper presenters. Some of them didn’t like that, maybe they thought it was unnecessary, but to eliminate the possibility of them saying, “Oops! Microphone doesn’t work!” or whatever we wanted to do that. I think also that in the process of making the video, like you said, it makes you think about things and makes you organize your thoughts carefully. Otherwise you’re going to be spending several days making a half hour video. Right? [laughter] So, and I think people enjoyed it. I think I’ve, you know, you enjoy learning new things and how to use video editing software. And as you said, the whole green screen thing. And to me, it’s fun to learn new things.
Michael Larsen (11:22):
One of the definite differences this go around is that, of course, there are the people that help run the talks and introduce the speakers and all that. And this time, of course, we have a dedicated MC. Joe Colantonio has jumped in. And for those who are familiar with Joe through Test Guild and Automation Guild, this is an environment that he’s been doing for quite a while. So he has experience with doing virtual conferences. Something I thought was interesting, I wanted to comment on it and get your feedback for what you think about it because I thought this was interesting. Instead of just piling through the chat and having a whole bunch of people type questions, what I thought was neat was that they used Slido. Now Slido is an application that you can submit questions to. And also there’s a voting system for it. If you submit a question and there’s a number of questions submitted, other people can go look. And if they think, “Oh, that’s a really good question”, they can click and upvote for it. The questions that get the most upvotes, slide up to the top of the queue and those that don’t get as many upvotes or don’t get any upvotes or at the bottom. I appreciate the fact that Joe has been going through that, but of course that gives a good indication of, “Hey, this is the question that most people want to have asked. Let’s make sure we ask this first.” Was that something that Joe had recommended or was that something that the conference committee researched and decided that that was a neat thing to use?
Philip Lew (12:41):
Yeah. That’s something that he uses in his conferences. And if you notice he has some kind of an integration as well. So he’s not actually typing that question again into his own screen. You notice how the question comes up on the screen? Yeah. There’s some integration between his software and Slido that takes it out of Slido and puts it on the screen. So that’s a pretty nice integration. That was definitely his idea. And I really like it, especially for virtual conference, being able to upvote questions, that even makes it better than a real conference or a large conference. Cause you can’t really do that in a live conference. Whoever jumps up and down the most, I guess, gets their question answered.
Michael Larsen (13:20):
Yeah, that was my take on it too. And that was actually quite pleased. “Oh, that’s neat”. The other benefit too, was that the presenter… Raj is also able to see the questions as they’re being submitted rather than saying, “Okay, we’ll work through each one of these questions”. And Raj was able to look at all of those and say, “Hey, I appreciate this. And it looks like there are a number of other questions that are kind of variations on this theme. Michael submitted one and Jeff submitted one. So I want to kind of group these together” . And I thought that was really good.
Speaker 2 (13:52):
Right. That was good. That he was able to handle it that way.
Michael Larsen (13:57):
If you had to pick any one challenge that you’ve faced that you’ve either learned a great deal from, or that has been a noteworthy headache, what might that be?
Philip Lew (14:06):
I think all of the coordination of the speakers with this online stuff, just a lot of emails and trying to figure stuff out. And that’s probably been the hardest part. Yeah. Just more communication, more emails, right? Cause everything is, you know, electronic. Emails probably multiplied by a factor of two or three [laughter].
Michael Larsen (14:28):
Yeah. I definitely can appreciate that. Again, we are literally in the middle of the conference, we have another activity that is scheduled for right now that we’re going to need to get to. So I want to respect your time. I want to say thank you for joining us today and I wish you the best of luck with the continued rest of the conference.
Speaker 2 (14:47):
Thanks Michael. Enjoy being here. Thanks.
Michael Larsen (14:50):
All right, thank you very much.
Michael Larsen (14:52):
Hey everybody. Thanks very much. We are literally now at the tail end of PNSQC the Pacific Northwest Software Quality Conference. I just finished doing a virtual workshop with 27 participants and some interesting challenges to make that workshop come off. But also I was only able to talk to Phillip Lew who was the program chair. And what we had originally intended with that call was to also have the other side of the equation, which is the “in the trenches volunteers and helpers and people who do a lot of the day to day stuff that makes sure that a conference actually run”. For that end. This is a first timer on the show Phillip’s a first timer, t0o, but I like to introduce a good friend of mine and fellow PNSQC veteran, Mr. Bill Opsal.
Bill Opsal (15:46):
Hi, it’s great to be here.
Michael Larsen (15:49):
So I will mention for anybody listening to the show that Bill and I have a laugh about this… because I met Bill in 2010, the first time that I went to PNSQC. We were both part of the volunteer pool and that’s how we first met. I think we spent the first night before the conference out at one of the beer pubs in Portland with a few other people. We realized that we had a lot in common, well testing-wise and background-wise and the things that held our attention, let’s just believe it that way. The irony of the situation is that, though we both live in the Bay Area over the past 10 years, we’ve only physically seen each other at PNSQC [laughter].
Bill Opsel (16:30):
Yep. That’s right.
New Speaker (16:32):
Which I’ve always found amusing… but again, so that just goes to show the bond that we have for this conference has gone on for a long time and we’ve been actively engaged in it at a number of levels. So I thought it would be interesting to talk about PNSQC from a ground level. It’s safe to say you’ve probably done every volunteer thing that can be done at this conference [laughter]
Bill Opsal (16:54):
Yeah. I’ve been joining this conference since 2005, 2006 timeframe and quickly started going into the volunteer area of it with reviewing papers, helping set up. You know, I’ve done almost every aspect that I believe that I can, it’s been quite eventful.
Michael Larsen (17:12):
So if you had to pick from a volunteer’s perspective, what’s the most challenging thing that you’ve done?
Bill Opsal (17:17):
I think, probably, the most challenging thing was one year…. I don’t remember exactly what year it was…. we were changing some of the processes a little bit around how we were doing the paper reviews and how we were using the Open Conference tool. And I think when we were making some of those changes, that was probably one of the hardest years because we were finding things that, yes, sounded good on paper as a process. But when you got right down into it and you’re reviewing something with someone, email might be the quickest way for one person, texts might be for another person. The open conference tool might be for another way that someone might like to work. So some of those years where we’re adjusting that a little bit, those are some of the challenging things because everybody works a little bit differently and it is a volunteer conference and we set guidelines. But at the same time, some of the communication challenges, you have to learn how to work with all the different types of people.
Michael Larsen (18:10):
So, I mean, you mentioned a few things about that, like open conference and different ways of communicating. What are some of the things that let’s take this year, for example. T his year, without question was a very big change because we couldn’t hold the conference in person. We had to go completely virtual for it being the fact that you’ve done this multiple years. What was the biggest challenge of that, that you found either leading up to, or during?
Bill Opsal (18:36):
I think probably two things. Ensuring that people were still going to be involved given all of the dynamics that are going on right now with COVID-19, how folks are working and they’re working remote or things are just in kind of constant flux. So making sure we were getting the message out was huge, but also, and as we can see today, how happened with the workshop, making sure we had all the Zoom stuff set up correctly because we were relying on that very heavily. There are probably some test cases that we just didn’t think of or some scenarios, but we did the best we could with what we had and the amount of time that we had, I think we did Okay but I think that was probably one of the bigger challenges. And then, of course, today, when we discovered some Zoom limitations that we needed to get figured out, we had to really think quick on our feet. Case with you, you had a Zoom link, we got that out pretty quickly and you had a good turnout on there. So I think that kind of shows the dedication and the ability of this group and the people that helped make this happen to adjust quickly, to make sure we got through what we needed to get through. And then of course the learnings from that, right? But that was probably one of the tougher areas.
Michael Larsen (19:45):
Yeah. I will definitely say to that… I’m not tattling and I’m not trying to call any disparagement. We found that there was a limitation to how many meetings could effectively be running from the same environment. And I thought, “Okay, right when I started, they kicked us out of the meeting”. I’m like, “No, no, what am I going to do?” To be fair, I did anticipate just in case something went completely sideways. So I broadcast to everybody before the meeting started. I said, “Hey, I’ve got a link here. I’ve recorded my workshop”. A lot of the things I wanted to experiment with people and be able to, “Hey, let’s, you know, play around with this”. I want you to play around with this and look at it.
Bill Opsal (20:23):
Michael Larsen (20:24):
Here It is, this is what it looks like. This is how you can do this. Instead of bouncing it around. I said, “Of course I do want to actually do this in person with you”. So that was it. We could have just said, “Well, no, I’m sorry. You’re not going to work. Go ahead and take the link and can call it a day.” But I said, “No, let’s do the best that we can.” I thought, “Well, okay, I guess I need to host this”. And then right in the middle of the meeting, I remembered, “Oh wait, I still got a free account. It’s going to shut off at 40 minutes”.
Bill Opsal (20:49):
Michael Larsen (20:49):
In the middle of the session, I went, “hang on everybody. I’m sorry”. Boom. I jumped over to give myself a premium account just so that it wouldn’t shut off for the time being. I was like, “Hey, I might as well use this”. So, uh, there might be a lot of screencasts in my future.
Bill Opsal (21:05):
Bill Opsal (21:06):
Yeah. I mean, that was one of the things that you had to adapt to that really fast. And I had to lob you a, “Hey, here’s a new link”. And you had to get that out to everybody and they had to jump in. So I mean, other than that, other things of interest, like maybe not just challenges, but what did you think went really well with this conference?
Bill Opsal (21:22):
I do think the kickoff went fine. We had great keynotes again. And of course, great speakers all around. And there was activity with folks joining the calls. There’s Q and A, I think all of that went really well. It allowed people to do what they would probably normally do if we were all in person. So they were able to, you know, you had to get your Q and A into a chat area, but at least they were getting to get that done. One area that I was a little bit worried about, but I think it went okay. Actually it went well, is the workshops are always very interactive and it’s a little bit different this year with all the virtual effort. Everybody did the best they could with what we had at the time. The positive thing that I really love about this is that there was still that feel of, “yes, it is PNSQC, people were involved. People were asking questions.” That feeling, even though we’re not all able to see each other, you could pick that up through the talks. So that was good.
Michael Larsen (22:20):
So I know that if this continues, if our COVID-19 situation continues, we might be looking at this kind of situation again, I certainly hope not. I would like to have an in-person conference again, but just in case this happens again, now that you’ve had a chance to be in the thick of it and here you are, it’s all fresh in your mind [laughter]. What would you maybe say, “Oh, if I have a chance to do something different, I would recommend we do this?”
Bill Opsal (22:50):
There’s a lot of things there, right. Probably spend some more time around the Zoom aspect to it and making sure we’ve really got everything set up the way we really need to do. And Joe did a great job MCing, but I think we would really dig in a little bit more the virtual aspect of it with Zoom and make sure we’ve really got all of our bases covered. We’re pretty good with having backups with things and, you know, a backup plan, like the videos and recordings. But I think I, personally, for this would dive more into that just to really make sure we’ve got it. You know, I had a, an experience here where my laptop wasn’t acting like it should. So I did my part with you on Zoom, on my iPhone. It worked, but it was a little bit more difficult to do than a bigger screen on a laptop. Just having those backup plans in place is something I want to dive into more.
Michael Larsen (23:37):
Yeah. I can definitely appreciate that.
Bill Opsal (23:40):
Michael Larsen (23:41):
Okay. Well honestly, Hey bill, I know that you’ve been active for three days trying to keep sanity here. I just finished my workshop less than half an hour ago. I think both of us probably need some food and some rest. So I want to say thanks for participating. Thanks for being part of the show. Hey, I hope that we’ll get a chance to maybe on a different topic have you come back with us.
Bill Opsal (24:02):
Yeah, I’m looking forward to it. So thanks again.
Michael Larsen (24:04):
All right. Thank you. Bye.