What Could Go Wrong? Lessons From System Outages to Streamlined Operations ft. Rachel Clark

00:00:00:04 - 00:00:19:06
Unknown
I cannot inside. If this was a tool that you put me. Can sit behind my thought provoking presentation to say, oh, 1st May want to be a CIO. Really think about that. Took a lot away from that. Like, I'm definitely going to be following up and talking to you. And as I think about my cousin today and what I lay out for you.

00:00:19:08 - 00:00:59:12
Unknown
It's a lot about transition shoe. I'm at a point in my career where in Tennessee, the election that you sponsor in the lot is changing. An governor for the state of Indiana. And I'm currently the CIO for the BMA, which means just a lot of change. I would say this moment at the state feels a lot like, where everyone is or a sock rack position on the, back position side of a need where you're going through due diligence and there's just this really intense period of someone new coming in and looking at every single thing you do, how you do it, and let the problems are where the faults are.

00:00:59:16 - 00:01:23:14
Unknown
That's really what's happening at the state right now. Times every single agency and party looks about and says, that's so cool, done this many times I've been on that due diligence ride. Many times you have to come up to speed so quickly on a company. Cool. But we don't know. On the other side of the transition, what that's going to mean for anybody in any role, in any agency at the BND or equity issues.

00:01:23:16 - 00:01:43:17
Unknown
So I've been doing a lot of self-reflection. What have I accomplished on the last two years? What I'm proud of, what are they going to find? And they start off from that due diligence deck and looking at the DMV. And when Doug, after you speak, you where my mind was, was, you know, kind of taking this retrospective back on what he accomplished, done at the DMV.

00:01:43:19 - 00:02:00:02
Unknown
So that's really at pair for you today. And it kind of sweats for me. Like, why did I take the job, which is parents perfectly with you, Mike your presentation, we did not plan into this. But in the interview process at the BMB, you know, we talked a lot about how, you know, we all don't have a choice.

00:02:00:02 - 00:02:22:20
Unknown
You have to interact with B and B several times a year. And the system that you interact with is massive ERP. It processes. So like I had some stats here through five different mediums. This is just a little taste of the job and what really hooked me in the interview. You can do your transaction person online at a self-service kiosk, which is some really cool high tech stuff, and mail over the phone.

00:02:22:20 - 00:02:51:22
Unknown
We do 50,000 transactions a day. We process 20,000 doc and stay pieces of paper, have to get, saved and stored for ten years. By law, all this massive, technology going behind it. It's mostly the three transactions that you're probably all familiar with registrations, titles and credentials, but there's 185 other kinds of transactions, which gives you a taste of how complex this ERP system really is.

00:02:52:00 - 00:03:11:23
Unknown
And there's a lot of money flowing through the agency. And prior to come to the B, I have worked for most as small a startup software companies that were struggling to get that $20 million. Nothing in Mark. I mean, we were just with the same company that was working for start scale up companies and here sort to walk into $150 million offering budget.

00:03:11:23 - 00:03:31:06
Unknown
My IT budget was bigger than in revenue for, you know, any company that I've ever worked for. That all was really appealing to me. And I was also going to get inherits a contract and state's combination of contract and state staff, about 100 people. So I thought, this is really great. The table is set, can do a really good things here.

00:03:31:06 - 00:03:58:04
Unknown
And this is a massive challenge. I can never say, well, one more piece that also appealed to me. There were all these integrations with other systems with the feds to check on your social Security number, bring a passport, you we have to check the passport. All these different data interactions to talk to partner in a child. Services say please pull you over to talking to us all these principles into the system to had maintained the nurture.

00:03:58:06 - 00:04:20:21
Unknown
So really cool use case is what I thought in the interview was kind of went through it all is this also was something they shared in the interview. You know, all that said, as soon all the stuff that people genuinely really like us and I'm not taking a Pullman or my notepad as a know, I saw what I hear, I hear from all my relatives all the time what they every time they go, any new, any transaction.

00:04:20:23 - 00:04:44:11
Unknown
But this isn't just saying this is not the data center. This is not you need to be a speedy edge. But if you, know much about the state of Indiana, all of our a lot of our systems are still hosted in the government center. The fifth floor of the Government Center don't count. So I was interviewing on the fourth floor, and all this technology was running on the fifth floor, and I did just see it in the room.

00:04:44:13 - 00:05:03:02
Unknown
Jason Sinema may have been in the room. I can't remember if you were there, but there were, you know, 6 or 8 of us in a big conference room. And I could just tell something really bad had happened. People were checking their phones off working. And the reality was this system started going down during an interview, and you could just see the panicked fear.

00:05:03:02 - 00:05:22:18
Unknown
The stress just. Yeah, it's just terrifying. But I can never say I didn't know what I was walking into, because that became clear that they were dealing with a pattern of unplanned outages, and we're struggling with how to figure out how to get their arms around it. So the table was said the hack in offering budget a big, important mission.

00:05:22:20 - 00:05:34:15
Unknown
But they were really struggling to have stability or stable operations. Were you in that room, Jason? I thought you were you shrimp and Citrix a big problem?

00:05:34:17 - 00:05:58:04
Unknown
No. So let me I do remember this, Jason. So I did take the job once for a short outage, and I took the job. I thought, well, there's a lot of this is a target rich environment is what I told myself, let's do this. And I remember Jason falling inside very early on and saying, hey, you know, you come from the private side, which you might not realize is every time you have this separate, there's fault or there's an outage or anything.

00:05:58:09 - 00:06:20:18
Unknown
The press is like right there waiting to read, waiting to write about it. Just can't wait to put things about to be in me and the papers and just this whole other element of, you know what? You don't have a board of directors right now. You have a governor, his cabinet and General Assembly who can legislate and tell you what to do and dictate your map, and they will do so at least once a year.

00:06:20:20 - 00:06:47:23
Unknown
All these other elements came into it, and it was parks like Jason. The different people have Kobe inside, he said. So know what you walked into. It's not like what you came from. And it's never good to be there. And I really in the beginning I remember thinking, is he serious? Is he really serious? Absolutely. That's Google, the BMV, really the place they love to write about many problems that had and government problems.

00:06:48:01 - 00:07:05:19
Unknown
So first sunny days, like when I look back some of the lessons that I want to share, lots of books about this, I read some I don't remember any other, I'm just going to go from what actually worked for me and what we did and the first sunny days. I really focus in this war on people, process and technology less on technology.

00:07:05:19 - 00:07:30:23
Unknown
Most people process and I did a couple things that really works and I want to share, we created talent cards. I got this idea from my husband who worked at Roche long ago, and something they do there. So if any of you had worked for senior subprocesses and I, my own interpretation of what they did, what I did is I sat down with every person on my team and I said, out of the gate, look, I know you expect you make a bunch of changes.

00:07:30:23 - 00:07:50:18
Unknown
I'm not for the first few months or not. I want to get to know each of you I serve. And one was that's all in here to do that calm people down a little bit. Give me a little bit of goodwill, a little bit of time. But it takes a while to earn people's trust. It's so what I would then spend time at them doing is say, let me develop this tower card with you.

00:07:50:18 - 00:08:07:08
Unknown
Let me help you do this. Let's take your resume and let's look at, you know, all the steps you took. By the way, can you tell me what your what what was your raise and how did you feel about that raise? Why does she have that title change? Did you seek it for yourself or did someone just see it needed you?

00:08:07:08 - 00:08:25:01
Unknown
You applied or you know how do you get there? And having these conversations with people along the way and giving them an asset or giving them something as a PowerPoint slide. But it's a visual representation of what you've achieved and you know where you want to go. It's a way of letting me get to know people really well was an important part of this.

00:08:25:02 - 00:08:47:14
Unknown
I would ask them the questions and it would take a, you know, several different one on ones with each person to go through this process. But I think it was a really important part of getting to know the team that was already there and where they were, where their hearts and minds were. The one question that I would ask people that I still use this day, I think it's about me, is I would say, you have a home, you don't have to be working the night.

00:08:47:14 - 00:09:06:18
Unknown
But let's say you do want to work on something. And nine you open your laptop up and you just want to do some work. What do you work? Where are you doing? What is your are you cupping Python? Are you? Pre-entry. Inbox and find emails you can learn a lot about first. And when they enter that question they tell you, yeah, this is one I don't have to work.

00:09:06:18 - 00:09:23:18
Unknown
This is why I like to work on. And then you can start realizing people may or may not be in the right role in may or may not be in the right position. And the more you get to know them and the more they can get them to open up to you about what really makes them tick, the better you can put together a team that makes sense.

00:09:23:22 - 00:09:45:20
Unknown
So I think this was a really important part of what we did. We mapped all our processes. This was painful, but we went through, all of our processes and make sure you guys have been through exercises like we at all times are. Many times this was the development process for us, a lot of our outages. The next slide shows it, this is me scoring it.

00:09:45:20 - 00:10:05:16
Unknown
Not only did we cross delete Apple processes, but we scored it. So we kind of what I reflected, I would get everyone else's kind of like how are you see, we are on this maturity scale when it comes to how we deploy our code. For example, that was one that got one of the really low scores was cork code to deployment.

00:10:05:18 - 00:10:27:13
Unknown
So just getting each other to like each other on the AI knowledge that were really weak in that area. And talk about how the why was it important step making this pivot or making this turn. I also analyzed a lot of data. Oops. Each one there analyzed a lot of data. May get some. Don't worry about it. Analyzed a lot of data.

00:10:27:15 - 00:10:48:03
Unknown
And there was a there were a lot of information sitting there ready for me to understand where the outages were coming from. We just weren't looking at the data. We weren't aggravating in any way. You weren't really looking at the lessons to be learned from wise. Outages were happening. So that was a great piece of it. So in the first few days or the first 90 days, we developed these committed outcomes.

00:10:48:03 - 00:11:01:01
Unknown
This is what we're going to do. This is where we think the biggest problems are. And I'm not going to go over every piece of it. So if anybody wants me to share the side and talk to you instead, what I'm going to do is I'm going to share the top five things that I think mattered most about all the stuff that we did.

00:11:01:03 - 00:11:21:01
Unknown
They committed outcomes. Developing this was really about getting the executive leadership around me, for what I was before, what I was about to do. I was ready to make changes. I was ready to do some big shuffling of a deck. But did everybody on around me to be on board with what I was about to do? And getting is committed outcomes agree to at a very high level is my way.

00:11:21:02 - 00:11:46:15
Unknown
My department. Here's what I wanted to add, and permitting two was an important step. Getting everybody ready for this changes that we were going to make. First thing and the most important thing, the number one thing you did was to establish a leader since you didn't really have one. It was confusing and chaotic and people were performing roles, but they didn't have the accountability, in their lives, in their area.

00:11:46:15 - 00:12:08:21
Unknown
The one thing I would add that was unique, BMB is I was working with state and contracts and bodies, so I leaned on Jason, Sandy, Jeff Roy, other executives of other, companies in specific disciplines to have the conversations about who we needed to have, in what roles and what people brought to the table. Because at this point, I had observed everybody.

00:12:08:21 - 00:12:37:04
Unknown
I had really thought about it, and I shuffled the deck, and the worship team, the recreate it started meeting weekly, bonding weekly, making decisions together, weekly, thinking not just in their lives, but about each other and what we were all doing. This was a key part of it. He part of transformation. And I would say the most important thing to getting the rest of it right was getting people in these roles and establishing this leadership core.

00:12:37:06 - 00:13:08:23
Unknown
The second thing is that this is also weary mostly about people. We developed a change management process and a code red process. Now the code red process, Denise's. It didn't existed in that people would see the email. It starts to sound and start to sweat bullets Creek out. But you had people that you know, we're just the ones that always love to wear in a superhero cape, and it's always going to be me, no matter when is I'm going to be the one to fix it, and I'm going to board the insulation is a town wired into how the state's taught me.

00:13:08:23 - 00:13:28:00
Unknown
And, to it, it has value because I'm the only one that can fix it when it goes down. That is a horrible way to operate. And everyone really knew it, but we had to put a code red process in place to make it clear that we work and operate that way. Even more so, the code red process was a lot of different names.

00:13:28:00 - 00:13:47:13
Unknown
It was assigning someone to be that the brute that communicates you're not the problem solver, but you're going to be the only one that sent the emails. You're going to communicate people agency status. So every 15 minutes it hit that not you not you're not, you're not you're not me. I'm a no go on calling my petitioner. That's my roll call.

00:13:47:13 - 00:14:07:14
Unknown
My boss. Keep him updated. You know, just establishing those key roles, giving everybody cover fire or language. Four stars is down. There's a core group of people that have now have a free pass of every meeting. They are in a war room, or they're on a teams call or a teams channel. It's just the basic stuff that, as you guessing, you all know when you're all doing it.

00:14:07:14 - 00:14:25:05
Unknown
We were doing this and that was a key piece, moving past this cycle that we're in. The other one was change your address. I mean, when they talk, there's a pile on. The first thing you say is we'll change the book change. On that date, you couldn't answer back. There was no change management. That was very, very hard to do.

00:14:25:06 - 00:14:47:14
Unknown
What we did is we used a sure dev box and we created custom work item types. For what I tried to make a cute name is because something a cute name people will it will at the time was the idea and called production update notices pants as any change that is in our production environment has to go to this process has to be red, not approved.

00:14:47:14 - 00:15:02:15
Unknown
You have to back out. Clan just has a really simple but every change will go into Azure DevOps to get approved MLP. Not only that, but we're going to make sure when we're planning a stage, the people would actually have to do the work or cool is working on some of right. Because that was not a thing either.

00:15:02:15 - 00:15:18:12
Unknown
It's always the same superheroes that when you were at a hate and we just expected them to work every Saturday or Sunday, but nobody wants to work that way for long. So let's have that be a process where we're kind of watching and making sure people accept and acknowledge that they want to do that one on that Sunday morning.

00:15:18:14 - 00:15:41:10
Unknown
So many things were part of that change management process that got us through all that, all that chaos. That's a screenshot of it. That's just a screenshot of it. That's that's the DevOps. This is our change management born in Azure DevOps. You can see the completed patterns in the last few days. Whatever you track them, the last piece.

00:15:41:10 - 00:16:06:20
Unknown
This is also how you're technically being here a people problem. We automated our deployment process. We created Utah Tech savvy, we moved our source code controls to the cloud file. Diane, the government center, created, pipelines for deployments and automated the deployment. What that meant was, every time you do these high risk deployment activities, it usually meant people worked until two in the morning.

00:16:06:22 - 00:16:27:11
Unknown
And it was, you know, my first deployment that I know went state by online piece. I had no else. You had to do your help. But I wanted to understand what's happening. And they showed me the deployment spreadsheet and it was over 300 line. The individual tasks that individual people had to execute, and then they would put a lot of them.

00:16:27:11 - 00:16:45:22
Unknown
It was like high risk they'd have a QA assigned to. And I looked at that and said by were like, how many times do you get this wrong? Oh, every time the next morning you have to come in. Every time. Okay. Right. That's part of the process, part of the expectations, I said. But everyone, you know what? We're all human.

00:16:45:23 - 00:17:06:17
Unknown
You you're going to make mistakes. This is never going to be a fluid process if we don't automate it. So the automation was a really important piece of creating seamless deployments. And they're really we call them nothing burger sound anymore used to be a real high risk scare activity. It's mostly automated now it's just night and day night day it.

00:17:06:17 - 00:17:25:09
Unknown
But this wasn't that long ago, so I thought, you want I'll share these lessons because maybe others are an estate broker. We work. Oops. The next one we did adopt agile say, and I think this is was an important piece of, this is our first. And we know this is important to the way it's easy on back.

00:17:25:11 - 00:17:40:13
Unknown
This is what was most important about it is, you know what I interviewed that first interview. I think one of the things that was said is we keep starting projects and we never finish them. We need to stop starting. Start finishing. It's like a thing that kind of came up in the interview that really appealed to me.

00:17:40:13 - 00:18:07:17
Unknown
I would say implementing agile say is the way that we get everybody on the same page to use some kind of criteria to think analytically about what we were going to build next, what had the best, highest value for the bits. And Sybil next year could be next. And we would see t shirt size every opportunity and go through and argue or to offer make a use case, a business case among the executive team for what we were going to do next.

00:18:07:19 - 00:18:26:21
Unknown
And that got us from having like 20 things in flight to having er, 8 or 10, that any time that we were committed to and that we could deliver with confidence on time and on budget. I think those were other big, big fundamental things that were wrong that we had to fix to get to this place of stability.

00:18:26:23 - 00:18:31:05
Unknown
So to wrap it up.

00:18:31:07 - 00:18:47:02
Unknown
Is about the roadmap. Once we had it got operation stabilized. They don't have unplanned outages like that anymore. These we ones and I know.

00:18:47:04 - 00:19:07:03
Unknown
We developed drum that would another big piece of it. It's one thing to sit down and talk in any given time. And these little tired prairie projects of them in a building. But you need to be looking further out and having that strategic look. These are things that you all know. But I think when I look back and I'm at this, transition in my career, this phase where I'm trying to decide, do I want to continue being a CIO, where am I kind of Donald the CC?

00:19:07:03 - 00:19:22:16
Unknown
I've been doing it for a long time. There's a big piece of me that has really done the best, which ironically, your talk really spoke to you today. Greg really was the one give person.

00:19:22:17 - 00:19:43:06
Unknown
This is and this is my last slide at the BMV. They're looking at a new governor coming in and trying to say, all right, are we going to modernize this system that I told you about? That very first slide is this over 20 years old wheels. And I'm sure they really come out of it when they go in 2006, the monolithic architecture that everybody did in 2006, I'm sure that was that cool.

00:19:43:08 - 00:20:08:18
Unknown
That the latest and greatest. Yeah. Are for your second concrete now at the smallest. And we need to modernize it and to modernize a system like that would be, you know, ailing in a more and be a, you know, 3 or 10 year journey. Do I want to say for that or not? I'm not sure I'm having a lot of those same thoughts, but I have really, really enjoyed the ride, the stabilization and what we've done these last five years.

00:20:08:18 - 00:20:21:09
Unknown
That would be so I hope some of that spoke to you or helped in some way. I hope we can tie tech tighter. Our talks together, because I can't agree with you or I mean, we're very, very related. That's it. So I have my.

Creators and Guests

Rachel Clark
Guest
Rachel Clark
Chief IT and Security Executive @ Indiana BMV
What Could Go Wrong? Lessons From System Outages to Streamlined Operations ft. Rachel Clark
Broadcast by