Krome Cast: Tech-IT-Out

Krome Cast TECH-IT-OUT: Ransomware Recovery & The Evolution of Ransomware

Krome Technologies Season 1 Episode 14

In this episode of Krome Cast: Tech-it-Out, we are discussing ransomware, reviewing how it has evolved over the years and looking at the measures you can put in place to help protect your organisation from Ransomware attacks.

With the rise of ransomware attacks increasing daily, more and more businesses are falling victim to ransomware attacks, in this podcast we talk about preventing ransomware attacks and successfully recovering from a ransomware attack, along with some suggested Ransomware recovery steps.

This podcast features Krome’s Commercial Director, Sam Mager, along with Krome’s Technical Director Ben Randall, sharing their insights of the evolution of ransomware and discussing a recent example, where we have helped an organisation to recover from a REvil ransomware attack, in which the sodinokibi Ransomware-as-a-Service (RaaS) operation was used.

► ABOUT KROME: Krome Technologies is a technically strong, people-centric technology consultancy, focused on delivering end-to-end infrastructure and security solutions that solve business challenges and protect critical data. We work collaboratively with clients, forming long-term business partnerships, applying knowledge, experience and the resources our clients need to solve problems, design solutions and co-create agile, efficient and scalable IT services.

► KROME WEBSITE: https://www.krome.co.uk/

► SOCIAL MEDIA
• YouTube: https://www.youtube.com/@krometechnologies
• Linkedin: https://www.linkedin.com/company/krome-technologies-ltd
• Instagram: https://www.instagram.com/krometechnologies/
• Twitter: https://twitter.com/KromeTech
• Facebook: https://www.facebook.com/KromeTechnologies/

► CONTACT
• Telephone: 01932 232345
• Email: info@krome.co.uk

SPEAKERS: Sam Mager - Commercial Director Krome Technologies, Ben Randall - Technical Director Krome Technologies

 

Sam Mager  00:00

Welcome to Krome Cast. Tech IT Out.  I'm Sam Mager, Commercial Director for Krome Technologies  I'm joined once again by Technical Director, Ben Randall. 

Ben Randall  00:08

Hi, Sam. Thanks for having me. 

Sam Mager  00:09

No problem at all.  The topic for today's podcast is talking about ransomware recovery,  mitigation, how to protect yourself against ransomware.  But ultimately, what is ransomware? And how has it evolved over the years?  So pretty heavy, yet interesting topic.  And one, that's certainly front and centre, in a lot of news at the moment,  we've seen some very high level, and very, very costly ransoms,  which I think we can dig into in a bit.  But if we just peel this back a bit, because ransomware is not new,  you know, it's under various different guises and names.  And if we can just kind of step back, maybe 5-10 years,  I think kind of, it has been a while, but kind of what is ransomware?  You know, in your, in your learned opinion? 

Ben Randall  00:53

Yeah. 

Sam Mager  00:54

And kind of how did it start? And I guess, how is it evolved to what we're,  I'd like to get deep into what we're seeing today because some of it is really interesting. 

Ben Randall  01:00

Yeah.

Sam Mager  01:01

Let's peel back the onion a little bit as to where did it start?

Ben Randall  01:04

Yeah. So fundamentally, ransomware is where malware,  a bad player has gained access to your computer or to your data rather and encrypted it. They have a private key, which you don't have.  And they, I mean, back in the early days, what they initially did was offer,  was request a small ransom in return for the private key so you could decrypt your files.  More recently that has progressed to encrypting your files,  and also making a copy of your data exfiltrating it and threatening to expose it, expose that on the web,  publicly on the web, If you don't pay the ransom.

Sam Mager  01:44

So it's kind of now a double threat.

Ben Randall  01:46

Yeah, absolutely. I mean, it covers them for the situation where,  well, it's okay, I've got a backup, I can restore my data.  Yeah, but, ah we've got a copy of the data so we can spill it as well.  So it covers them in two ways really. 

Sam Mager  01:58

Double done.

Ben Randall  01:59

Yeah. 

Sam Mager  02:00

Okay. So we're going back to I guess, the early days,  the first things I remember around ransomware was Cryptolocker. 

Ben Randall  02:05

Yes.

Sam Mager  02:06

And excuse my ignorance on some of this, as obviously not the technical one here.  But was that the same as it is now? Obviously, you get infected on a system,  would that propagate and do it in the same way it does now?  I guess, how's that changed? 

Ben Randall  02:18

I think things have got more sophisticated,  I mean, back then what would happen is that typically,  it would be someone would open up a, an infected file,  like a PDF, your classic invoice, with something in there.  Yeah. And there, one machine would encrypt all the data it could find via shares, and so on.  Still pretty effective. To be honest.  A lot of users have access to a lot of important, important data that they work with day to day.  More recently, they've become more sophisticated,  I believe there's elements of AI in there, that machine learning so they can find their way through a network and get onto servers,  spread you know, within the network in a more intelligent fashion.  But fundamentally, the same thing is really happening.  It's just the encryption of data.  You know, that's, 

Sam Mager  03:06

Yeah.

Ben Randall  03:07

There's also an element of attacking your infrastructure,  which may provide a way out like so if you've got internal backups,  you might try and target those backup servers and remove, delete, reformat drives, that sort of thing if you don't have immutable backups.

Sam Mager  03:24

Interesting. Actually, that's a term, that I should be aware of,  having sold storage and virtualisation for many years.  But immutable backup, not that it's a new term, but it was new to me.  And actually understanding that's a real kind of air gap,  or firebreak, whatever you want to call it,  to being able to resolve some of these problems?

Ben Randall  03:44

Yes, yes. So essentially, that's where a backup isn't just a file that's stored somewhere which you can just go in and delete it or encrypt it.  Fundamentally, it's, it's a, it's a tape sitting on a shelf,  you can't erase that unless you literally put it in the drive and erase it.  Or it's in storage, which has, well they call the term, immutability  so that yeah you can see it, but it's kind of,  it's sort of a write-once read, read-only 

Sam Mager  04:10

Like the old worm

Ben Randall  04:12

Worm, that's the word, the worm drive, essentially.  So you can have a time limit on the ability to delete that,  I mean, that can be done in the cloud, on-prem, but it also that is a really good idea.  So it means you can't delete the backups that are only a month old or something like that.

Sam Mager  04:28

Yeah, that I suppose that helps you in recovery position.  But as far as someone actually getting access to the data,  and we talked about the exfiltration of data, and the copy,  doesn't resolve that issue, right? 

Ben Randall  04:38

No, no. I mean, ultimately, with all these things,  prevention is better than cure.  So it's all-around security. User education will be a real

Sam Mager  04:50

Does it always seem to come back to the problem in the chair?

Ben Randall  04:54

Yes Sam, Yes. [Laughter] But yeah, so the bottom line is, is It's phishing awareness.  It's people knowing to use good passwords.  Additionally, there's a technological side to it,  we can look at multi-factor authentication.  So even if someone's password is known, they haven't got the other factor.  Yeah. So that sort of thing.  You've got patching of systems to make sure that there aren't just known vulnerabilities. I'm thinking about the exchange vulnerability that was caught earlier this year,  which gave remote access.  Also, having systems which are fundamentally insecure,  I mean, things like Remote Desktop servers, which aren't really in use,  but still on the internet, everyone's forgotten about,  you know, you've got leaked credentials, someone logs into that you've given away the keys to the castle, 

Sam Mager  05:43

Web servers, that sort of stuff will be particularly vulnerable. 

Ben Randall  05:46

All these things, again, you have to stay on top of that, really.  And it's an ongoing process. It's you know, It's iterative, you need to keep on it regularly and be aware 

Ben Randall  05:56

of where you're standing is in terms of those updates.

Sam Mager  05:59

Because it is, I mean, it is becoming particularly scary,  looking at some of the data and like to cheat and look at some of the notes  I've made, because some of the numbers are too big for me to remember.  But, I find this incredible, 48% of businesses (UK) have been affected by ransomware in 12 months, and that's a quote from Mimecast.  So I'm quite happy to quote that on the podcast and stand by it,  that is not just a Sam Mager fact.  And 50% of those organisations have paid. 

Ben Randall  06:24

Yes. 

Sam Mager  06:25

Now when you think about the average cost of a ransomware attack,  I think has gone from five years ago, of $10,000.  And we're now seeing what's the US oil pipeline one?  Was that, 70 million?

Ben Randall  06:37

I think the Colonial pipeline, I think they actually paid $5 million.  You've got the JBS, the meat suppliers, I think they paid $11 million.  And most recently, we've got Kaseya. Who, 

Sam Mager  06:52

Of course, that's the one I am thinking of. 

Ben Randall  06:53

Which is the, which is an altogether more, that's the next level of evolution.  We'll go into that in a second, actually.  But certainly, their main initial ransom demand was the $70 million.  I believe that's reduced to 50 million now.  But, so how nice of them, it's a bargain! [Laughter]  But yeah, I mean, that really brings us on to the Kaseya thing,  it's what we call a supply chain attack,  so that is where you've got a tool, Kaseya, used by MSP's to support multiple clients.  And they use that to roll out patches and that sort of thing,  and so Kaseya getting breached meant that they got all the people further down in the supply chain.  I believe that the, the player involved in that, REvil, they,

Sam Mager  07:41

Now this again, you and I discussed this off camera. It's quite terrifying.  From a from a commercial perspective, sounds like a great business. Right?  But, but I mean, it's terrifying these organisations are there.  So it'd be for those that don't know, and I was kind of one of them.  Can you just talk us through the business model they've put together because it's fascinating?

Ben Randall  08:02

It is quite professional, these aren't just hackers sitting in a bedroom somewhere.  What we're looking at is that REvil, with their, ah pronunciation sodinokibi  or sodinokibi, I'm not quite sure how you pronounce it.  But their product, they've come up with some effective code and communications platform, basically.  And what they're doing is they're selling this as Ransomware-as-a-Service.  So they're franchising the, the, you know, they've obviously done their own initial attacks, and I guess they get more manpower,  it makes more sense then to franchise it out.  So people can pay a cut of the make, use those tools and pay REvil a cut.

Sam Mager  08:45

Just like franchise. So you know,

Ben Randall  08:47

Pay a 10% of the profits or something like that.  And when the ransoms are so big, that's significant. 

Sam Mager  08:52

Well yeah, like you've just talked about being a 70, but kindly reduced to 50 million.  10% of that, is a handsome payday. 

Ben Randall  08:59

Yes Yes, I believe in that case, with the Kaseya attack,  they're breaking it down that individual machines underneath that,  which have been affected below that.  You know, if they pay the 50 million, they get the keys to the whole lot.  If you pay just, you can pay on an individual basis. It's only $45,000.

Sam Mager  09:18

So if I want my accounts server back, I can have that back?

Ben Randall  09:20

Yeah, you can get that one back for that.  that it gets more likely to be paid.  And you can see that, there's, there's because they've gone a little bit smaller  You know, if you're someone a million dollars or something,  it's probably not going to happen, unless they're very large. 

Sam Mager  09:33

Depending on who they are.

Ben Randall  09:34

Yeah, exactly. But a small business might go you know what,  we can afford $45,000 and it just increases the temptation to pay. 

Sam Mager  09:41

I read, and I can't remember which it was,  it was a local Borough or County Council in England, they became, had a ransomware attack.  And the demand was something extortionate and then we went back,  we don't have that money.  We can raise this, and it went from being like a million pounds to they could raise £13,000.  But okay, but the problem with that is obviously you paid,  and obviously all the advice is, you should never pay these ransomware attackers,  because ultimately, A, alright you might get the encryption key back,  there's no guarantee they haven't exfiltrated your data,  there is, you're not necessarily in a position where it just won't happen again.

Ben Randall  10:15

Yeah, it's almost a guarantee, actually,  I mean, I believe that in a very large number of cases don't quote me on the number, but where the ransom has been paid.  And, you know, presumably, they've decrypted their data, to some degree of success.  There's more on that later, as well.  The problem with paying is you're putting a target on your back,  you know, they know, you're a soft target,  they're just going to come back for more later, it's very common.

Sam Mager  10:40

I totally get that. And obviously, there's always the risk that you pay a ransom,  you don't get your encryption keys,  or that they've passed on the information they've managed to gather to somebody else. There's so many ways that that door can be left open.  And actually, unfortunately, we've had to help a few people  to try and resolve, recover, and claw their way back. 

Ben Randall  11:01

Yes. 

Sam Mager  11:01

If you like, and I hate to use the word interesting, right? Because, 

Ben Randall  11:05

Yeah.

Sam Mager  11:05

It's a pretty dire situation when this happens, but 

Ben Randall  11:07

Absolutely 

Sam Mager  11:08

looking at the process that these people use,  there's an awful lot of thought that goes into how they do it.  It's probably, if you want, I know, you've been very hands-on with some 

Ben Randall  11:16

Yes, I mean, back in the early days, several years ago,  must be about six or seven years ago, now we dealt with a cryptolocker attack.  And that was, it was fairly basic, it had come from one machine,  we were able to help the affected company,  purely by restoring from backups.

Sam Mager  11:34

Yeah. 

Ben Randall  11:35

Fortunately, they had 

Sam Mager  11:36

And was it in those days, was it literally just, disconnect that machine.  at the time, and see what damage had been done,  and recover from backups. And you're kind of there right?  

Ben Randall  11:43

Pretty much, as it was, the, this is another part of it, actually,  as it was, it started before the weekend.  Okay, nobody had alerted anyone, and ground on throughout the whole weekend. And pretty much everything was done by by the Monday or Tuesday when we were contacted.  So, you know, the data was done.  Fortunately, they had a decent offsite backup,  we, they were using a Asigra at the time, and it was possible to run a recovery of all their data. And, you know, they may have lost a little bit of time,  but only talking a couple of days worth of data,  which was, for them, was was a good save really. 

Sam Mager  12:22

If you extrapolate that out, bring it forward to more modern companies,  or modern businesses,

Ben Randall  12:27

Yes, 

Sam Mager  12:28

And look at the size of the data, just to unpick some of that, so, I forget,  you were telling me about that, the company that paid their ransomware  and then restored from their backup anyway,  because they realised to unencrypt, the encryption was going to take a ridiculous amount of time.

Ben Randall  12:42

This was actually, I believe it was the colonial pipeline issue.  They paid the ransom, they got the keys.  But the decryption was incredibly slow.  I guess that it takes quite a while to encrypt the files. And I've not watched it in action. 

 Sam Mager  12:55

AIl the time you're not pushing oils, probably costing a little bit.

Ben Randall  12:58

So, the, as it happened, they had an effective backup system in place.  So they were able to restore. So ultimately, it was, 

Sam Mager  13:06

Why did they pay? I guess, is the question.

 Ben Randall  13:08

It's a good question. 

 Sam Mager  13:09

I mean, I don't know the specifics of that attack.  But I know one's we've seen where things like, the time that they'll pick,  knowing the encryption takes a while. They'll pick holidays,  they'll pick times they know people are out the office for an extended amount of time. So, the malware, ransomware, etc  has to be there for a while, they've got a way in right. And they're waiting.

 Ben Randall  13:27

I think it's quite clear, from, certainly the most recent incidents that we had,  which I believe was the REvil, or someone operating the REvil service.  If you can call it that. They, they got, we were contacted after the Easter weekend. 

 Sam Mager  13:46

Yeah, there you go.

 Ben Randall  13:47

There had been, there had been, I basically think they'd been in there  for some time before that, and they 

 Sam Mager  13:52

just waiting, 

 Ben Randall  13:53

They go, right, okay, let's start things going, when everyone's away,  their backs turned it gives it time to grind through it.  You know, in that case, companies have a lot of data, it takes a long time to encrypt it.

 Sam Mager  13:55

That's the thing. I guess nowadays,  we actually have an exponential growth of data.  So everyone, even smaller businesses, large business have huge amounts of data. But I suppose that is one,  not a blessing, that's the wrong word,  but well think how long it takes to encrypt. And I know, 

 Ben Randall  14:17

certainly, it can 

 Sam Mager  14:18

there is a particular case where we managed to get a lot of stuff back because it was taking so long to grind through the encryption, right?

 Ben Randall  14:24

As it happened. I mean, you know, they were unlucky in the fact that there were problems with their backups, and also their on-site backups were actually wiped by the by the, by the players.  So, but as luck would have it, they had so much data that not all of it was encrypted.  There was I think it was about 80% was recoverable purely because it hadn't been encrypted yet.  Yeah, rather than any other measures they had in place really.

 Sam Mager  14:50

Yeah. That's really interesting. And actually, I'd like to get into don't mind.  Because I know you we're kind of really hands-on with this,  but we were obviously approached by a company that had suffered an attack,  as we've, you know, we've been through and funnily enough it was over an extended period of time,  when they knew those people had been away from the office.  And, it would just be great if you could talk through some of the,  I know, it was a pretty bad one.  And we're going to help them recover an awful lot of data and get them back up and going. But, there are some pretty interesting things,  that if they just had have paid more attention, it wouldn't have been as bad.  It'd be interesting, I think to share with the audience,  because some people watch this, and will go, "I'm good" and I think that's great.  And others might think, ah there's things that,  potentially off the back of this, we should go and address.

Ben Randall  15:33

Yeah, absolutely.  In that particular instance, there were multiple failures,  which led, you know, I think if, if probably, if any one of the things if they hadn't all gone wrong together, they probably would have been okay.  I mean, they'd recently started doing Phishing, 

 Sam Mager  15:53

Assessments?

 Ben Randall  15:54

Phishing assessment tests, that's the word I'm looking for.  And a high percentage of their users failed those tests.  And, in fact, I think I believe one or two users even admitted to having actually clicked on genuine ransomware.  You know, phishing links, we don't know for sure exactly which one caused the problem.  But it could have been a combination of them.  There were things like, as I mentioned earlier, 

 Sam Mager  16:19

Wow

 Ben Randall  16:19

about a remote desktop server publicly available, which actually wasn't in use.  A lot of users had administration accounts, which wasn't strictly necessary.  So, so a high percentage of users could be, could give a widespread breach.  Then their backup, they didn't have an off-site backup.  So they, and the backup they had was not immutable.  As I say, the malware players if you like, they have actually,  I'm sure that of a human had been involved.  Because the actual drive where the backups are stored was being low-level formatted when we found it, so that took some intervention. It wasn't just software that did that. Then you've got the issue with their storage, had got quite full,  and they didn't have any, they didn't have any real snapshots, on their SAN storage.  So even though they had replicas to the other site,  the most recent snapshot, the most recent copy they had was encrypted.  So it's just a catalogue of unfortunate incidents.  You know, if any one of these it had an immutable backup off-site  or onsite, they'd have been okay.

 Sam Mager  17:30

If they didn't have domain passwords everywhere  this that and the other, like you say, a collection of 

 Ben Randall  17:34

Yeah,

 Sam Mager  17:34

unfortunate events.

 Ben Randall  17:35

Yes. It was striking the professionalism of the actual web page. 

 Sam Mager  17:41

I was going to say, I want to touch on that, because you showed me  I keep using the word interesting, it's probably the wrong word to use,  if we're talking about unfortunate things, but it is,  to see how professional, I didn't expect it to look like that.

 Ben Randall  17:53

Yeah, yeah, it was the web page, you had, they had a private key to access their key for help.  Where you could, they gave their address for their monero,  for the ransom to be paid.  I believe it was half a million dollars, if you responded within seven days after that, it was a million.  So it's like your parking fine if you pay it within? [Laughter] Very, very much like that.  But yes, there was, you know, once you put it in, they gave all those details, there was a 'chat to support' for assistance with it,  and they'll give you a, prove who they were, by decrypting one file for free,  one or two files I suspect, I think it was for free.  Just to prove that that was what it was. 

 Sam Mager  18:32

And we think, that's potentially part of the REvil,  I don't know what we call it?

 Ben Randall  18:36

Yes, it was absolutely, it was was REvil sodinokibi,  I can never pronounce that word.  But yes, it was that as far as I can tell.

 Sam Mager  18:46

Wow, and I guess and the other thing we mentioned on that we saw,  is obviously I guess what's escalating the ease of this now is  Bitcoin or Ethereum, or whatever it might be to 

 Ben Randall  18:55

Cryptocurrencies 

 Sam Mager  18:56

Cryptocurrencies, because now it's not, you know,  transfer to this bank account, oh what now we can find you.  It goes off into the ether. And it's just kind of driving this 

 Ben Randall  19:03

certainly very much harder, at least.  It's facilitated that international transfers with very low cost and really no accountability.

 Sam Mager  19:13

Again, it goes back, like you said, having the right technologies,  the right training this that and the other,  because you leave the door open these people will find a way in,  I guess, looking at some of the, some of the suggestions you'll see,  will be you know, have your Cyber Essentials. 

 Ben Randall  19:25

Yes 

 Sam Mager  19:26

Personal opinion, but I think it's worth the paper it's written on because it's self-certification. Right?  Cyber Essentials Plus is definitely a step up, having someone come in and rubber stamping that you've got the right technologies, processes.  One up from that will be ISO 27001, again, personal opinion,

 Ben Randall  19:39

Each time, well, each time that they're a fairly substantial ramp up. 

 Sam Mager  19:43

Yeah.

 Ben Randall  19:43

And if you've done those properly, 

 Sam Mager  19:46

Yeah

 Ben Randall  19:46

ISO 27001, Cyber Essentials Plus, it means your users will have awareness and this is really, you know, as we mentioned earlier,  the weakest link, unfortunately, is the people really,  but if people are on their guard, you're much less likely to get someone click on that link or open that file.

 Sam Mager  20:03

The, I guess the thing with, with those certifications, whilst as you say,  it's very important that you've analysed and understood the right technologies in place, a process in place, people are educated.  But it's a point in time, once you've got your rubber stamp,  I'm CE plus, or I'm ISO 27001. If people go, right the auditors have gone,  we all go back to what we were doing,  you've got a big problem on your hands.  So, I think it's very important that obviously MDR services are out there.  We do it for customers. And it's just, you know, I think it's good practice,  especially nowadays, because this isn't going to slow down. 

 Ben Randall  20:38

No No. 

 Sam Mager  20:38

Ransomware is getting more and more aggressive,  we're seeing more and more attacks, the stats I pulled out earlier,  people are paying, rightly or wrongly,  it's your business, you've got to do something. 

 Ben Randall  20:47

Yes. 

 Sam Mager  20:47

So obviously, the prevention is far less costly,  Then the cure, right. So I think we need to potentially dig in a bit 

 Ben Randall  20:51

Absolutely 

 Sam Mager  20:54

to what we can do around MDR and how we can help our clients.

 Ben Randall  20:58

As you say, MDR, managed detection response.  So that will be a service where you've got, got a service,  which is keeping an eye on the data from your systems in general. So that could be literally everything,  they're looking at all the logs and so on,  when they start noticing those IOC, as in indicators of compromise. They'll be on the phone to your IT department to let you know,  I mean, depending on the level of service, and who we're talking about,  some will actually automatically lock out those machines,  let's say it's a laptop or whatever, isolate that machine from the receiving network.  But certainly, there'll be, you know, over the Easter weekend,  you'll get the worrying phone call, but at least you've got the call before they've had three days of attack. 

 Sam Mager  21:42

We know it takes a while to grind through right.  So if you see that, behaviour, the wrong behaviour happening. 

 Ben Randall  21:48

Yeah. 

 Sam Mager  21:48

And using AI, and using people response, and so on,  to actually go. Is that right? 

 Ben Randall  21:52

Yes 

 Sam Mager  21:52

And someone will go, it doesn't seem right. Let's stop that. 

 Ben Randall  21:55

You can have systems that will alert you, but unless your staff are really keyed up on everything it's doing,  you know, you can get alert fatigue. So yeah, get 500 alerts a day, 

 Sam Mager  22:06

We've seen with other systems we use, with the PRTG stuff  and customers that just want to ignore  the flashing red thing because it keeps flashing.  But it's because it's something important.  And yeah, when it comes to this stuff, it's even more important,  but that's why we're now seeing, we provide it for our customers,  and see more and more people coming,  to actually have our managed services team,  we'll suffer the red light fatigue,  and we'll look at what's going on, and we can advise our customers as to what can happen, and that prevention is definitely,  definitely better than cure.

 Ben Randall  22:34

Yeah. And the early response to a detection of something,  you know, the quicker you react, the better you're going to be.

 Sam Mager  22:41

Okay, so if we were to wrap up,  and I kind of guess maybe the three key takeaways.  I get, I mean, we know the first one right, it's education. 

 Ben Randall  22:50

Yep. Yeah, 100 percent. 

 Sam Mager  22:51

I guess you'd probably say, having the right technologies was clear. I might even go for four here Ben. 

 Ben Randall  22:58

I was gonna say, well, you're saying technologies,  we've got your kind of your user login security technologies,  call it your MFA, your good passwords? That kind of thing. 

 Sam Mager  23:07

Yeah, yeah, yeah.  Immutable backup?

 Ben Randall  23:09

Yeah. 100%. If you've, if you've got a backup,  if you assume it's going to be a not if, but when,  and if you if you've got a solid backup, that you can recover from,  then you're in a much better position than then you would be.

 Sam Mager  23:21

Yeah. And then you know, I guess lastly, if you do get that fatigue or it's something, you know, you kind of want to outsource the problem.  Working with a good company that can help you with that.

 Ben Randall  23:30

Yeah. 

 Sam Mager  23:30

And almost yeah take that, not only we can,  there's no perfect solution, these things will happen.  But you said the key thing. The fastest response is the best. 

 Ben Randall  23:40

Yeah, absolutely. 

 Sam Mager  23:41

We can help our customers do that then fantastic. 

 Ben Randall  23:43

Yeah, yeah. 

 Sam Mager  23:44

Thank you, Ben, I've actually, I've actually learned stuff today,  which is great. As always. [Laughter] Thanks for joining me. 

 Ben Randall  23:49

Yeah. Good to see you. 

 Sam Mager  23:51

And thank you for joining us once again on Krome Cast. Tech IT Out.  Please remember to like, subscribe, and share,  If there's anything you'd like to discuss in future episodes do leave that in the comments below.