[00:00.000 --> 00:15.500]  This talk is basically for you guys because what we do as IT professionals is we're really good at the tech stuff and we can understand what's going on with our network, but we're really sucky at explaining that to managers and other people and explaining that in a way that is defensible.
[00:16.240 --> 00:29.500]  So while we can say that, yeah, you should probably update your SSL certs and you should probably plug these patches, we can't really say that in a way that makes them understand it to the dollar line.
[00:30.000 --> 00:38.920]  So hopefully with this presentation, I'll give you guys a framework to explain that risk to managers and have them understand it.
[00:39.820 --> 00:41.360]  So just a quick warning.
[00:42.660 --> 00:45.940]  A risk analysis is an intellectual process.
[00:46.060 --> 00:51.160]  And if you're not intellectually rigorous, the product you make is a pile of crap.
[00:51.760 --> 00:55.180]  And I've seen a lot of really bad risk analysis, risk analyses.
[00:55.780 --> 01:08.840]  So just be careful when you're doing this that you need to make sure that you aren't pigeonholing yourself to one specific threat or just focusing on one set of ideas, but you're including all of them.
[01:09.140 --> 01:12.240]  All the ideas that... we'll get to that in a minute.
[01:12.980 --> 01:18.160]  The other notice... I work for confidential stuff, so I have to put this in here so the FBI doesn't come arrest me.
[01:18.820 --> 01:20.900]  None of this is classified, so we're cool.
[01:22.320 --> 01:23.760]  So let's start with a little story.
[01:24.400 --> 01:25.260]  I have a friend named Nate.
[01:25.600 --> 01:27.020]  Nate works for an IT company.
[01:27.660 --> 01:33.700]  Nate was recently visited by his boss, who if you've ever read The Bastard Operator from Hell, that's his boss.
[01:33.700 --> 01:40.140]  So he trundles in and asks him... So we read this thing last night about hackers that are attacking our network.
[01:40.280 --> 01:43.940]  I want to know what the risk is posed to our network by hackers.
[01:44.200 --> 01:44.700]  And then walks out.
[01:45.320 --> 01:48.240]  So Nate's sitting there and he's like, Welp.
[01:49.080 --> 01:55.680]  So we'll use that sort of as the framework for talking about how we look at risk in this situation.
[01:55.960 --> 02:02.480]  Because we all intrinsically, inherently, know what's risky and what the risks are.
[02:02.480 --> 02:04.440]  But we're not really good at explaining it, like I said.
[02:04.640 --> 02:07.920]  So, what is risk?
[02:08.840 --> 02:09.860]  Can anyone here?
[02:10.040 --> 02:10.520]  Shout it out.
[02:10.680 --> 02:12.200]  Please explain to me what risk is.
[02:12.300 --> 02:13.480]  What do you think risk is?
[02:14.600 --> 02:15.120]  Sorry?
[02:15.520 --> 02:16.360]  Board game.
[02:16.660 --> 02:17.860]  It's an excellent board game.
[02:18.140 --> 02:20.040]  The probability of an adverse event.
[02:20.200 --> 02:21.400]  Probability of an adverse event.
[02:21.740 --> 02:22.660]  End users.
[02:22.920 --> 02:24.120]  End users are a risk.
[02:24.280 --> 02:24.560]  Brilliant.
[02:25.080 --> 02:25.860]  Waking up.
[02:26.060 --> 02:27.180]  Waking up is a risk.
[02:29.080 --> 02:30.920]  So, things that need to be managed.
[02:30.940 --> 02:32.360]  Things that need to be managed.
[02:32.480 --> 02:32.600]  Alright.
[02:33.580 --> 02:39.940]  So, a risk, as we're explaining it here, seems to be things that might be adverse in the future.
[02:40.120 --> 02:50.100]  But, the definition of risk that's used by at least the intelligence community and used by the academic community is that risk involves only uncertainty about the future.
[02:50.540 --> 02:53.040]  So, it doesn't weight it bad or good.
[02:53.700 --> 02:55.240]  Good things can happen in the future, too.
[02:55.440 --> 02:56.760]  And that's a risk that we take, right?
[02:57.300 --> 03:03.880]  So, the return of an investment could be less than we want, but it's still a positive investment.
[03:04.300 --> 03:06.680]  So, there's a risk that it will come back less.
[03:07.340 --> 03:14.400]  So, if we look at the standard distribution curve, which is over here, the green is the good stuff and the red is the bad stuff.
[03:14.840 --> 03:18.340]  So, in security, we generally only look at the bad stuff.
[03:18.860 --> 03:23.320]  So, we accept that of the definition of risk that we've given, we're only looking at the bad stuff.
[03:26.230 --> 03:32.930]  Now, when your manager is asking you something about risk, it's generally one of these six questions.
[03:34.150 --> 03:34.930]  What can happen?
[03:34.990 --> 03:35.850]  How likely is it to happen?
[03:35.930 --> 03:36.630]  What are the consequences?
[03:37.150 --> 03:37.990]  What can be done?
[03:38.150 --> 03:39.710]  What are the benefits and costs of each option?
[03:39.890 --> 03:40.610]  And what are the impacts?
[03:41.230 --> 03:46.810]  So, everything that's ever questioned about a risk, a possibility of the future, boils down to one of these six questions.
[03:46.810 --> 03:49.430]  So, if you prepare to these six questions, you'll be fine.
[03:51.190 --> 03:54.970]  And today, we're going to talk about these three over here, the risk assessment trio.
[03:58.170 --> 03:58.970]  So, stop.
[03:59.650 --> 04:00.050]  Don't.
[04:00.090 --> 04:00.390]  No.
[04:00.750 --> 04:01.910]  I see it in your eyes.
[04:02.030 --> 04:02.650]  You're glazing over.
[04:02.750 --> 04:03.750]  You're thinking this is actual math.
[04:03.770 --> 04:04.130]  You're wrong.
[04:05.090 --> 04:06.070]  This is not math.
[04:06.210 --> 04:08.470]  This is a sentence explained in fancy terms.
[04:10.250 --> 04:17.170]  So, how do we explain the three things, the three statements we just said about risk assessment?
[04:17.670 --> 04:19.030]  It's in that.
[04:19.410 --> 04:20.630]  So, I'll go through it slowly.
[04:21.570 --> 04:27.530]  Risk, which is the big R, is the combination of, which is the sigma, the probability of an event.
[04:28.070 --> 04:29.330]  So, it's an initiating event.
[04:29.490 --> 04:31.150]  So, nuke goes off in Times Square.
[04:31.350 --> 04:32.690]  So, that's the initiating event.
[04:33.730 --> 04:35.530]  Probably of an outcome given that event.
[04:35.690 --> 04:36.670]  Nuke goes off in Times Square.
[04:36.970 --> 04:37.450]  Casualties.
[04:38.870 --> 04:42.990]  The valuation of that outcome, of that event and outcome pair.
[04:43.870 --> 04:44.690]  E comma O.
[04:45.410 --> 04:49.090]  So, given that a nuke went off, and there are casualties, how many casualties are there?
[04:50.950 --> 04:52.550]  For every event and outcome.
[04:53.370 --> 04:55.690]  This is generally where we get tripped up a little bit.
[04:55.850 --> 04:57.750]  Because notice where it says every?
[04:58.070 --> 05:01.830]  That includes like zombies and space aliens and zombie George Bush.
[05:02.390 --> 05:08.910]  So, you know, it gets kind of big, the set.
[05:09.330 --> 05:11.810]  Slides are going to be online later, so you can get everything you want.
[05:11.910 --> 05:13.370]  I'll have a big slide with the website.
[05:15.610 --> 05:17.430]  So, how do you scale that down?
[05:17.770 --> 05:25.930]  And that gets to be one of the biggest sticking points, is how do you make that gigantic list of things that could possibly happen into something that's manageable that you can actually look at?
[05:27.450 --> 05:28.130]  It's kind of big.
[05:28.690 --> 05:29.430]  So, you scope.
[05:29.650 --> 05:31.590]  And it's not an equation.
[05:31.590 --> 05:32.270]  Don't do that.
[05:32.810 --> 05:34.070]  So, scope.
[05:34.370 --> 05:44.230]  The scenario that we're looking for, the scope, is the set of, the brackets are set, a protector, a threat, and an asset.
[05:44.870 --> 05:46.650]  So, let's look at it from the backwards here.
[05:46.830 --> 05:47.430]  An asset.
[05:47.550 --> 05:55.650]  An asset is something that either you derive profit from as a person, or a company, or an entity, or you are in charge of safekeeping.
[05:55.830 --> 05:59.250]  So, this could be a database, if you're a company.
[05:59.370 --> 06:01.230]  This could be people, if you're the police department.
[06:01.390 --> 06:03.510]  This could be your hose, if you're a pimp.
[06:03.710 --> 06:05.230]  This could be anything...
[06:05.730 --> 06:09.510]  literally anything that you derive profit from, or you're in charge of the safekeeping for.
[06:11.070 --> 06:19.670]  So, you would be the protector, the P. So, you can even scale that down inside of company to, okay, this might be the network operations center.
[06:19.750 --> 06:21.030]  This could be the IT staff.
[06:21.170 --> 06:22.110]  This could be the managers.
[06:22.110 --> 06:24.710]  This could be the finance department that we're looking at.
[06:24.910 --> 06:28.190]  Specific departments within a company, or the company as a whole, or the United States.
[06:28.490 --> 06:34.650]  So, if the threat is to the security of the nation, the United States is the protector, not just specific people.
[06:34.950 --> 06:45.150]  And the threat is anything that's trying to lessen the amount of profit you get from the asset to tarnish the asset somehow, or destroy the asset.
[06:45.970 --> 06:47.470]  So, this is basically what I just said.
[06:49.390 --> 06:49.830]  Yep.
[06:50.750 --> 06:51.190]  Yep.
[06:51.970 --> 06:59.610]  So, for Nate and Cliff, if we go back, the boss asked what our risk was from hackers.
[06:59.870 --> 07:03.270]  So, we are the protector, Nate and the knock, because we're the only people that can stop it.
[07:03.710 --> 07:11.410]  So, the finance department isn't going to step in and start running NMAP scans on the network and set up a snort box.
[07:11.950 --> 07:12.750]  It's the knock.
[07:12.990 --> 07:15.070]  So, they're the ones that are ultimately protecting the network.
[07:15.530 --> 07:17.810]  The threat is from hackers.
[07:19.030 --> 07:20.330]  Not you guys, the bad ones.
[07:21.770 --> 07:24.910]  And the asset, according to the boss, is the company information.
[07:25.330 --> 07:28.230]  So, at the end of the day, he doesn't really care about the networks.
[07:28.310 --> 07:29.310]  He cares about the information.
[07:29.490 --> 07:30.850]  It's where he gets his dollars from.
[07:31.330 --> 07:31.770]  All right.
[07:33.970 --> 07:34.970]  So, back to the equation.
[07:35.770 --> 07:36.790]  I mentioned probability.
[07:37.030 --> 07:37.770]  What the hell is probability?
[07:37.770 --> 07:46.290]  So, probability is best described as, of all the things that can happen, how likely is each one?
[07:46.970 --> 07:52.090]  So, thinking more abstractly, think of the universe as a big box.
[07:52.590 --> 07:55.410]  A big two-dimensional blue box, such as the one up here.
[07:56.030 --> 07:58.450]  So, let's say we're wanting to flip a coin.
[07:58.730 --> 07:59.110]  All right.
[07:59.570 --> 08:03.550]  And if coin flip, you flip it, it can either come up heads or it can come up tails, right?
[08:03.790 --> 08:05.770]  So, there's only two solutions.
[08:05.770 --> 08:07.290]  Or there's only two things that could happen.
[08:07.450 --> 08:12.610]  So, our universe, if we flip that coin, gets divided in two, right?
[08:12.810 --> 08:14.370]  So, either it comes up heads or it comes up tails.
[08:14.930 --> 08:18.310]  So, given that it comes up heads, what's the probability that it's going to come up heads again?
[08:19.470 --> 08:21.410]  You divide that into another half.
[08:21.830 --> 08:29.470]  So, if you want to think about this visual concept, if you're thinking of this as a dartboard, it's a lot easier just to hit the tails than to hit the heads and the heads.
[08:29.990 --> 08:30.910]  Or the heads and the tails.
[08:31.630 --> 08:32.150]  Right?
[08:32.570 --> 08:36.150]  So, and the size of each box represents its relative probability.
[08:39.700 --> 08:40.520]  What I just said.
[08:41.600 --> 08:43.380]  And you always want to strive for MISI.
[08:43.500 --> 08:46.660]  MISI means mutually exclusive and collectively exhaustive.
[08:46.800 --> 08:48.500]  So, no events overlap.
[08:48.940 --> 08:56.780]  So, there's no... In this example, you can't have a coin be simultaneously on heads and tails, technically.
[08:57.980 --> 09:01.460]  And also, you have to account for other things that might happen, like the coin rolling away.
[09:02.780 --> 09:06.920]  So, you're trying to make the box as encompassing as possible.
[09:07.040 --> 09:08.160]  You want to fill the entire box.
[09:08.300 --> 09:11.420]  You want to make sure that your universe is as complete as possible given the scope.
[09:11.980 --> 09:12.400]  Right?
[09:14.580 --> 09:16.600]  This is a great quote from Isaac Asimov.
[09:16.980 --> 09:19.000]  And his second foundation book.
[09:20.060 --> 09:22.240]  You must not say never.
[09:22.640 --> 09:25.180]  As an actual... As a lazy slurring over the facts.
[09:25.480 --> 09:25.920]  Actually...
[09:25.920 --> 09:28.700]  And his word was the psychohistory.
[09:28.840 --> 09:33.300]  But risk analysis, which is kind of the same thing if you think about it, predicts only probabilities.
[09:33.580 --> 09:35.480]  A particular event may be infinitesimally probable.
[09:35.580 --> 09:37.740]  So, massively unprobable.
[09:37.860 --> 09:39.560]  But the probability is always greater than zero.
[09:39.840 --> 09:41.080]  The world could end tomorrow.
[09:41.640 --> 09:45.740]  That should probably be in your universe if you're thinking about, you know, a large encompassing universe.
[09:46.320 --> 09:47.740]  Zombie Hitler could come back tomorrow.
[09:48.060 --> 09:50.720]  Not probable, but possible.
[09:52.120 --> 09:55.800]  So, when you're calculating probability, the best thing to do is to get raw data.
[09:56.080 --> 09:57.500]  Raw data is beautiful.
[09:59.300 --> 10:05.460]  So, generally, it's the number of successful attacks per period divided by the number of attempts per period.
[10:05.700 --> 10:10.620]  Which will give you, in that period, how often you have a successful attack.
[10:10.880 --> 10:11.320]  Right?
[10:11.600 --> 10:19.720]  So, if you have three successful attacks out of 30,000 attempts in a month, say, then you have a probability of .0001.
[10:20.020 --> 10:21.720]  That's a one hundredth of a percent.
[10:23.660 --> 10:25.420]  And that's a lot of numbers, though.
[10:25.580 --> 10:28.220]  So, especially if you don't have raw data.
[10:28.380 --> 10:34.680]  Like, we don't really have good data on terrorist attacks, because there aren't a lot of terrorist attacks in the United States.
[10:34.800 --> 10:38.100]  So, and terrorists don't talk to us about how they do their attacks and such.
[10:38.540 --> 10:41.200]  So, we don't really know what's going on with the terrorist attacks.
[10:41.640 --> 10:46.460]  So, what we use mostly is called subject matter elicitations.
[10:46.620 --> 10:53.360]  So, we talk to experts who are experts in terrorist attacks, and they tell us, okay, the risk of this is low, medium, high, whatever.
[10:53.940 --> 10:54.040]  Right?
[10:54.760 --> 11:02.200]  So, they're putting their gut feeling, which is what we all feel about, you know, networks and such, into one of these three bins, low, medium, high.
[11:03.220 --> 11:13.760]  And while you can't do math, necessarily, on low, medium, high bins, it's really good for showing managers when you don't have actual data to do.
[11:15.480 --> 11:24.360]  So, remember that the probability for this must be calculated for both the event and the outcome given the event, because those were the two separate probability parts and our big equation.
[11:25.160 --> 11:26.060]  That's not an equation.
[11:26.780 --> 11:29.180]  So, make sure you calculate both.
[11:30.880 --> 11:32.240]  So, why does valuation matter?
[11:32.340 --> 11:33.940]  It's that little part that's tacked on at the end.
[11:34.060 --> 11:36.380]  It's the V, E, comma, O.
[11:37.800 --> 11:44.580]  And the reason is, think about it, a lot of people here had to come from very long distances.
[11:44.740 --> 11:47.380]  Would you rather have driven or taken an airplane?
[11:47.840 --> 11:57.100]  A lot of people will avoid taking airplanes and taking said trains or cars or buses or whatever, because they fear dying in a plane crash so much.
[11:57.340 --> 12:01.660]  So, they value their death in a plane crash higher than their death in a car accident.
[12:01.660 --> 12:12.160]  So, when you're talking about people's response, and you're using that as a valuation metric, the way to go is to understand that the valuation matters.
[12:12.400 --> 12:17.600]  So, the risk might be higher in death in a car accident, but the valuation is lower.
[12:17.740 --> 12:20.360]  So, the overall risk that people perceive is lower.
[12:20.860 --> 12:21.860]  So, valuation matters.
[12:22.420 --> 12:25.080]  Well, there's another slide later that I'll just explain a little bit more.
[12:26.060 --> 12:29.360]  But the valuation can be based on money.
[12:29.840 --> 12:31.900]  So, attack A happens.
[12:32.020 --> 12:32.840]  How much money do you lose?
[12:33.180 --> 12:36.500]  It can be based on time.
[12:36.760 --> 12:37.740]  How much time do you lose?
[12:37.840 --> 12:38.220]  Goodwill.
[12:38.860 --> 12:42.900]  If your iPhone is a crappy product, how many people will be pissed off at you?
[12:43.800 --> 12:44.780]  Whatever's of most concern.
[12:45.020 --> 12:53.720]  Where you start to get into a problem is where you get to the sort of statistical value of life stuff, where you start assigning numbers to people's lives, like you're worth 6.4 million dollars, whatever.
[12:54.440 --> 12:54.640]  Yeah.
[12:55.200 --> 13:00.680]  There's a moral dilemma with that, because then you're assigning monetary values to people's lives, and people don't like that.
[13:01.060 --> 13:03.900]  So, I'd try to avoid that, but some people like it.
[13:04.000 --> 13:04.180]  I don't know.
[13:05.360 --> 13:06.760]  So, here's the process in general.
[13:07.060 --> 13:08.600]  Now, this is a very basic process.
[13:09.400 --> 13:11.080]  And you can do this pretty quickly.
[13:11.300 --> 13:14.420]  It might take a couple weeks to get the data, and then you just slap it together and it's done.
[13:15.000 --> 13:17.540]  So, along the top, you're going to basically make a big chart.
[13:18.240 --> 13:25.100]  And along the top, you're going to put things like, you know, no attack, unsuccessful attack.
[13:25.380 --> 13:28.000]  So, basically, Nate came up with three categories.
[13:28.360 --> 13:30.460]  No attack, unsuccessful attack, successful attack.
[13:30.640 --> 13:35.400]  He split the last category into internal and external because he could, and he wanted to look at it.
[13:36.520 --> 13:40.140]  And along the other side, you're going to put the outcomes of concern.
[13:40.260 --> 13:45.220]  So, the initiating events around the top that we're And the outcomes of concern are along the left.
[13:45.880 --> 13:58.040]  So he was looking at data loss, data exfiltration, which is someone came in, grabbed a database, and now has all your customer data, or data corruption, which is someone went in and altered the last number of every credit card to be plus one or something.
[14:00.340 --> 14:07.240]  So if we look at it, and we think about it, the probability that no attack is ever going to happen is very, very low.
[14:07.240 --> 14:10.220]  So, we're all being attacked all the time.
[14:10.420 --> 14:13.060]  So you're never going to have a day where no attack happens.
[14:13.200 --> 14:14.680]  So the probability of that happening is low.
[14:15.020 --> 14:20.040]  Given that you have no attack happen, what's the probability you can have data loss resulting from that no attack?
[14:20.820 --> 14:24.520]  Unless you're an idiot and you start deleting all your tables yourself, it's probably not going to happen.
[14:24.900 --> 14:26.920]  So that probability is very low.
[14:27.340 --> 14:28.700]  How much do we care about that?
[14:28.820 --> 14:32.920]  We're going to use the valuation of how much do we care for this particular example.
[14:32.920 --> 14:39.960]  So, how much do we care that, you know, because of user error there was data loss?
[14:40.060 --> 14:40.620]  We don't really care.
[14:41.020 --> 14:43.480]  It happens, it's a fact of life, we move on.
[14:45.480 --> 14:46.980]  Given that there's an unsuccessful attack.
[14:47.160 --> 14:54.960]  Now, me and Nate defined unsuccessful attack as someone got through the first layer of defense but didn't achieve their goal.
[14:55.220 --> 15:00.780]  So, they may have, you know, gotten through the border routers but didn't get to their final destination.
[15:01.280 --> 15:06.320]  So, the probability that someone unsuccessfully attacks is pretty high.
[15:06.580 --> 15:08.080]  We all have unsuccessful attacks all the time.
[15:08.220 --> 15:12.920]  My website, which is a pretty crappy website that no one goes to, is probably getting attacked right now.
[15:13.600 --> 15:14.940]  And it just happens.
[15:15.640 --> 15:25.420]  So, given that an unsuccessful attack happens, so someone's bumbling around your network, what's the probability they're going to do something stupid and drop a table?
[15:27.080 --> 15:27.900]  It's pretty high.
[15:28.060 --> 15:30.200]  It's not massively high, but it's pretty high.
[15:30.200 --> 15:31.700]  So, we rate that as a medium.
[15:31.880 --> 15:35.300]  And then we value that kind of low because they didn't succeed.
[15:36.440 --> 15:38.060]  They still attacked, but they didn't succeed.
[15:38.200 --> 15:38.580]  So, we don't care.
[15:39.760 --> 15:42.060]  And we keep doing this for the rest of the table.
[15:42.860 --> 15:51.000]  So, we do judgments based on what we think in addition to raw data that we have.
[15:51.160 --> 15:54.340]  And we come up with table.
[15:55.200 --> 15:59.880]  And this is eventually what you're going to present to your managers in this format because it's more colorful.
[16:02.740 --> 16:08.120]  And if you look at it, there's kind of a pattern to the risk analysis, how it came out.
[16:09.400 --> 16:13.760]  And when you're presenting data to managers, it's very important to have a pattern to your data.
[16:13.920 --> 16:16.700]  So, they can instantly recognize, okay, this is the hot spot.
[16:16.820 --> 16:20.860]  This is where we have the most risk based on the risk analysis we did.
[16:21.220 --> 16:22.560]  This is where we should focus.
[16:24.080 --> 16:31.320]  Because, generally, people, decision makers won't look at a graph and say, oh, you know, we can reduce more risk here.
[16:31.440 --> 16:33.620]  They'll say, oh, this is where the most risk is.
[16:33.740 --> 16:35.180]  This is where we should put all of our money.
[16:36.160 --> 16:41.360]  So, you want to be able to give them that instant gratification of knowing this is where we should funnel all of our dollars.
[16:42.580 --> 16:43.960]  This is not a risk matrix.
[16:44.480 --> 16:47.100]  If you've ever heard the term risk matrix, it's horrible.
[16:47.340 --> 16:48.040]  Put it out of your mind.
[16:48.280 --> 16:48.840]  Forget about it.
[16:49.140 --> 16:49.800]  This is better.
[16:52.260 --> 16:52.680]  Yeah.
[16:53.580 --> 17:03.400]  So, let's say that, for example, we took the last graph and only did actual hard data in terms of the probabilities that we did.
[17:03.540 --> 17:06.280]  So, we came out with actual numbers that we could use, right?
[17:06.280 --> 17:20.680]  So, if we have statistics on attacks and how successful they are, we can actually have a probabilistic chart such as this that shows the probabilities of each specific event and outcome combination.
[17:22.400 --> 17:27.300]  Now, alongside... along the outcomes and concern side, you may notice that there are some values.
[17:28.060 --> 17:28.880]  There are some dollar values.
[17:29.020 --> 17:36.160]  That's the value, the supposed value that I pulled out of my ass, of a company recovering from such and such outcome.
[17:36.860 --> 17:51.640]  Now that we have numbers that we can work with, actual hard data, and a number that we can work with in terms of a value for that data, we can revalue all of the risk assessment that we just did using money instead of how managers feel.
[17:52.600 --> 17:53.920]  And it comes out something like this.
[17:55.500 --> 17:59.540]  And notice, it shifts.
[18:00.160 --> 18:00.260]  Right?
[18:01.180 --> 18:07.600]  The valuation we put on the risk assessment vastly changes the outcome that we give.
[18:08.160 --> 18:12.800]  So, when you're doing a risk assessment, always make sure to identify what they're looking at.
[18:12.880 --> 18:17.040]  So, if they're looking at how much money they can save, use a monetary-based valuation.
[18:17.220 --> 18:19.360]  If they're looking at live saves, use a lives-based valuation.
[18:19.360 --> 18:23.060]  If they're looking at how much they get pissed off, use how much they pissed off as a valuation.
[18:23.780 --> 18:25.180]  So whatever makes most sense.
[18:25.300 --> 18:28.120]  Because that's, at the end of the day, what you want to be giving them.
[18:28.920 --> 18:33.880]  You want to be showing them, in terms that they want to see, a risk assessment.
[18:38.230 --> 18:39.030]  So, yep?
[18:40.950 --> 18:44.410]  No, that was something called annualized loss expectancy.
[18:44.750 --> 18:48.590]  So given that, for example, the probabilities were within a year.
[18:48.590 --> 18:53.410]  So let's say the three successful attacks, 30,000 unsuccessful attacks, are for a year.
[18:54.050 --> 18:57.250]  And the money that you lose is within a year.
[18:57.470 --> 19:00.030]  You can do for that year, so annualized, every year.
[19:00.290 --> 19:04.250]  How much money do you expect to lose from a specific event not compare?
[19:04.990 --> 19:09.730]  So you took a sum probability, 60%, so you multiplied by an amount.
[19:12.400 --> 19:13.580]  That's the loss that we did.
[19:13.840 --> 19:18.460]  His question was, so we get the probability from adding...
[19:18.460 --> 19:21.200]  It's actually, we multiply the probabilities together.
[19:21.340 --> 19:23.920]  Adding the probabilities together is never a very good idea.
[19:24.060 --> 19:28.400]  But multiplying the probabilities together gives us a better idea of the actual combination of the probabilities.
[19:30.420 --> 19:35.700]  That, with the money, gives us a good idea of how much we should expect to lose per year.
[19:36.060 --> 19:38.780]  So over a given period, what's the expected loss?
[19:39.520 --> 19:47.880]  And it's gonna be less than the full event, but it accounts for the probability that this event is gonna happen over a certain number of years.
[19:48.320 --> 19:49.960]  So eventually it'll happen.
[19:50.220 --> 19:54.600]  And when it happens, you'll lose all the money, but it'll be like you lost all the money all the years.
[19:55.680 --> 20:01.200]  So it's more like, if you shovel this amount of cash towards me now, I can fix it and we won't have to worry about it later.
[20:03.160 --> 20:05.680]  So given that, that takes a pretty lengthy process.
[20:05.880 --> 20:08.400]  There are some shortcuts and some methodologies we can use.
[20:08.480 --> 20:09.440]  And I will show you a couple.
[20:09.880 --> 20:11.440]  But you should be forewarned.
[20:11.900 --> 20:14.100]  These are what we call factor-based models.
[20:14.280 --> 20:20.980]  And factor-based models are great for doing an instant risk assessment of a number of items.
[20:21.020 --> 20:24.780]  But you have to be aware that it only gives you a relative ranking.
[20:24.960 --> 20:29.920]  It will not give you a definite final probability for each event.
[20:30.560 --> 20:31.080]  All right.
[20:31.660 --> 20:34.660]  So we'll go through one and you'll see how it works in a couple seconds.
[20:34.880 --> 20:35.860]  So just bear with me here.
[20:37.040 --> 20:39.460]  So first, you're gonna have a factor-based model.
[20:39.520 --> 20:40.480]  It's gonna have a number of factors.
[20:40.680 --> 20:43.300]  So criticality, accessibility, vulnerability, whatever.
[20:44.920 --> 20:46.920]  And you're gonna assign a range of numbers to those factors.
[20:47.140 --> 20:48.340]  So like one through four.
[20:48.980 --> 20:50.660]  Try to use even numbers.
[20:50.820 --> 20:55.520]  Using even numbers, as humans, we tend to use one through five scales.
[20:55.620 --> 20:57.460]  We use the number three a whole hell of a lot.
[20:57.460 --> 21:01.040]  And that generally doesn't tell us very much about the risk.
[21:01.300 --> 21:07.760]  So forcing yourself to use either a higher or a lower number will generally give you a better idea of what you're looking at.
[21:07.860 --> 21:11.500]  And will force you to categorize the risk as higher or lower.
[21:11.660 --> 21:19.000]  And also generally provides an impetus for figuring out if it's higher or lower.
[21:19.000 --> 21:23.560]  So it forces you to be more mentally prepared for that question.
[21:26.780 --> 21:32.140]  The higher the number is, it should point more towards the... whatever the issue at hand is.
[21:32.220 --> 21:40.000]  So if you're looking at... one of the examples is how much a community contributes to the crime rate.
[21:40.000 --> 21:48.620]  So the... the feelings of a community contribute... or the... situation of a community contributes to the crime within that community.
[21:49.140 --> 21:52.760]  So the more it contributes, the higher the number should be.
[21:53.020 --> 21:56.000]  So it shouldn't be that the first four numbers all represent, you know...
[21:56.900 --> 21:59.480]  one is... it doesn't contribute, four it contributes.
[22:05.040 --> 22:06.640]  We will look at that in a second.
[22:08.000 --> 22:09.300]  Evaluate each factor using that range.
[22:09.740 --> 22:11.000]  And then you add up the combined score.
[22:12.500 --> 22:13.940]  So this is my favorite one.
[22:14.000 --> 22:16.060]  This is one the Navy SEALs use for a long time.
[22:16.520 --> 22:17.060]  It's called Carver.
[22:17.300 --> 22:18.500]  It's for target selection.
[22:18.780 --> 22:28.920]  So if you land on a beach in the middle of the night and you have to take something out in order to achieve your goal, and you have a list of things, you can just use this tool and it will give you a pretty good idea of where you should attack.
[22:29.660 --> 22:36.240]  So criticality is how critical the system or item is to whatever you're trying to infiltrate.
[22:36.380 --> 22:40.820]  So if we're talking about, you know, you could just picture it.
[22:40.880 --> 22:42.400]  Navy SEALs on a beach in the middle of the night.
[22:42.520 --> 22:46.060]  There's a hacienda in the distance with the lights on and guards around it with AK-47s.
[22:46.140 --> 22:50.860]  And you and your scrappy team of SEALs have to get in there and rescue some prisoner.
[22:51.760 --> 22:54.340]  And you have to first thing you do is turn off the lights.
[22:54.580 --> 22:55.840]  So criticality.
[22:55.940 --> 23:02.800]  How critical is a power station, power lines or a D cell battery sitting on the desk.
[23:03.980 --> 23:08.820]  The power station is going to be massively critical because without it you generally can't get your power.
[23:09.520 --> 23:13.260]  The power lines are kind of critical because, you know, there could be alternate power lines.
[23:16.000 --> 23:19.940]  The D cell battery is not very critical to the house being lit.
[23:21.420 --> 23:21.940]  Accessibility.
[23:22.780 --> 23:25.440]  The power station will not generally be very accessible.
[23:25.440 --> 23:27.420]  The power lines will be massively accessible.
[23:28.220 --> 23:32.040]  The D cell battery will probably be sitting next to some guy named Pokey who's watching TV.
[23:33.340 --> 23:33.700]  Recoverability.
[23:33.860 --> 23:35.640]  The power station will take a long time to fix.
[23:35.920 --> 23:37.860]  Power lines, they just string new ones up.
[23:38.080 --> 23:39.260]  D cell battery, no one really cares.
[23:40.280 --> 23:40.640]  Vulnerability.
[23:41.560 --> 23:43.180]  How vulnerable is a power station?
[23:43.300 --> 23:45.800]  It's probably well defended, you know, it's got a lot of stuff going for it.
[23:46.340 --> 23:48.380]  Power lines, you can just go up and cut them.
[23:48.800 --> 23:52.440]  D cell battery, you can shoot it from a while away, no one really cares.
[23:54.560 --> 23:54.960]  Effect.
[23:54.960 --> 23:57.380]  Power station, power goes down, done.
[23:59.800 --> 24:02.500]  The power lines, cut them, power goes down, done.
[24:02.700 --> 24:04.120]  D cell battery, eh.
[24:04.980 --> 24:06.220]  So you get the idea.
[24:06.340 --> 24:10.900]  You go through and you evaluate each single one for whatever you're looking at.
[24:11.120 --> 24:12.880]  And you can eventually come up with a number.
[24:13.140 --> 24:14.720]  And that number will give you a relative ranking.
[24:14.840 --> 24:17.840]  So let's do this for us, the next hope.
[24:19.100 --> 24:21.900]  So given that, here's the scope I use.
[24:22.000 --> 24:23.200]  This is just a little exercise.
[24:24.960 --> 24:25.920]  Protectors of HOPE staff.
[24:26.320 --> 24:28.320]  The asset is the enjoyment of the attendees.
[24:28.640 --> 24:30.660]  Yes, the enjoyment of the attendees can be an asset.
[24:31.240 --> 24:34.760]  And the threat is some rogue attendee who, I don't know, was jilted in somehow.
[24:36.340 --> 24:38.000]  So we're going to use a scale of one to six.
[24:38.740 --> 24:42.140]  Six being contributes highly to the attack success probability.
[24:42.140 --> 24:44.620]  One being does not contribute in the slightest bit.
[24:45.280 --> 24:45.660]  All right.
[24:46.380 --> 24:51.300]  And let's just put an array of possible targets up there.
[24:51.540 --> 24:53.720]  So we can attack the knock and take down the network.
[24:53.920 --> 24:56.240]  We can attack the elevator and immobilize it.
[24:56.320 --> 24:58.880]  We can attack the projectors and knock them out.
[24:59.000 --> 25:00.400]  We can attack the Segways.
[25:00.540 --> 25:01.240]  Do we even have Segways?
[25:02.000 --> 25:02.400]  Excellent.
[25:02.880 --> 25:05.220]  We can attack the Segways and knock those out.
[25:05.320 --> 25:06.260]  Or we can kidnap a manual.
[25:08.860 --> 25:15.500]  So we evaluate each one for all the items.
[25:15.740 --> 25:20.200]  And surprisingly, at least for this analysis, the elevator will actually cause the most consternation.
[25:20.440 --> 25:21.680]  And if you look at it...
[25:22.660 --> 25:24.680]  I'm going to use my fancy pointer here.
[25:25.040 --> 25:29.440]  If you look at it, this is accessibility.
[25:29.800 --> 25:32.940]  So how likely is it that you'll be able to get to the thing?
[25:33.600 --> 25:35.700]  For the elevator, you can just walk up and walk in.
[25:35.840 --> 25:36.820]  And that's kind of the point of it.
[25:37.140 --> 25:39.820]  But I haven't seen a manual yet today.
[25:40.540 --> 25:42.300]  So he's not really available.
[25:42.580 --> 25:44.020]  And that's where it loses the most points.
[25:45.220 --> 25:52.280]  So based on all the factors, you can come up with a relative ranking of which ones will accomplish your goal.
[25:53.480 --> 25:54.520]  Everyone with me so far?
[25:55.760 --> 25:56.160]  Excellent.
[25:57.300 --> 25:58.120]  Here's a couple more.
[25:58.380 --> 25:59.680]  I'm just going to throw them at you pretty quick.
[26:00.140 --> 26:01.540]  Evil Done is for target selection.
[26:01.700 --> 26:02.220]  It's like Carver.
[26:02.800 --> 26:03.760]  But different.
[26:05.260 --> 26:07.580]  D-Sharp is another one for target selection.
[26:10.000 --> 26:11.460]  It's a little bit more on the history.
[26:12.840 --> 26:14.480]  Craved is for attractiveness of assets.
[26:14.600 --> 26:15.500]  We're talking like burglars.
[26:15.820 --> 26:19.160]  So if someone breaks into your home, which things are they going to steal?
[26:20.080 --> 26:22.500]  The concealable and removable are really big in this.
[26:22.500 --> 26:28.500]  Because, you know, walking out with a flat screen TV is not as easy as walking out with some diamond rings.
[26:31.560 --> 26:32.840]  Murderous is for weapon selection.
[26:33.240 --> 26:36.220]  You can tell that the names kind of fit with what they're talking about.
[26:36.360 --> 26:37.160]  That's kind of the point.
[26:37.760 --> 26:44.520]  That's like 90% of getting your factor based model agreed to by the rest of the community is having a great name.
[26:45.860 --> 26:48.260]  So murderous is for weapon selection.
[26:48.460 --> 26:50.360]  So Al-Qaeda wants to kill some people.
[26:50.480 --> 26:51.380]  What weapon do they use?
[26:51.740 --> 26:52.200]  Murderous.
[26:54.020 --> 26:54.980]  Easier is for...
[26:55.310 --> 26:58.280]  This is the one I was talking about for facilitation of crime for different communities.
[26:59.100 --> 27:03.440]  So if it's easy, safe, excusable, enticing and rewarding to do crime, you'll do crime.
[27:04.520 --> 27:06.140]  This one I cooked up this afternoon.
[27:08.220 --> 27:10.580]  So how easy is it to social engineer someone?
[27:11.060 --> 27:12.240]  What hour of the day is it?
[27:12.240 --> 27:14.180]  Is it midnight and they are kind of sleepy?
[27:14.200 --> 27:16.380]  Or is it high noon and they're wide awake?
[27:16.880 --> 27:18.220]  Are they overseen by a manager?
[27:18.440 --> 27:20.700]  Do they have some oversight where they can't really say anything?
[27:20.880 --> 27:22.600]  Or are they allowed to say whatever the hell they want?
[27:23.100 --> 27:23.460]  Pressure.
[27:23.680 --> 27:24.880]  How much do you pressure them?
[27:25.240 --> 27:25.600]  Encouragement.
[27:25.900 --> 27:27.960]  That's kind of like the carrot and the stick thing right there.
[27:28.620 --> 27:31.080]  So, factor based model for social engineering.
[27:33.580 --> 27:39.320]  Now, when we're talking about factor based models and other things in general, we want to talk a little bit about scales as well.
[27:39.320 --> 27:46.260]  So, what scale we use determines how we can do math on the specific item we're talking about.
[27:48.260 --> 27:53.100]  So, I was telling you before, you know, about maintaining the polarity of scales.
[27:53.460 --> 27:54.660]  Sorry about the big word.
[27:55.500 --> 28:03.100]  Making sure that higher means that it's contributing more towards the success and lower means it's contributing less.
[28:04.300 --> 28:06.720]  So, what about the range of the scale?
[28:07.420 --> 28:09.740]  So, let's assume a factor based model of A plus B plus C plus D.
[28:10.380 --> 28:12.060]  A is just a one through four vulnerability.
[28:12.460 --> 28:13.700]  B is dollars and damages.
[28:14.000 --> 28:15.760]  C is time to return to operation in seconds.
[28:15.900 --> 28:16.760]  And D is lives lost.
[28:17.200 --> 28:19.100]  For a server, this might make sense.
[28:20.080 --> 28:24.740]  Because seconds out of operation could be, you know, 50 seconds while it's rebooting and other stuff.
[28:24.740 --> 28:26.380]  But what about for a cruise ship?
[28:27.200 --> 28:29.640]  It's going to take a long ass time to make a new cruise ship.
[28:30.000 --> 28:34.680]  And that in seconds is going to way outweigh the one through four vulnerability assessment.
[28:36.220 --> 28:44.400]  So, you see how choosing the right scale and making sure they mesh together makes for basically all of the risk assessment.
[28:44.400 --> 28:48.420]  You can sway a risk assessment however you want using the right scale.
[28:48.740 --> 28:53.540]  But having a intellectually appropriate scale is massive.
[28:54.900 --> 29:03.560]  And it'll maintain your intellectual integrity, give you a answer that is actually defensible and is the key to a good risk assessment.
[29:04.820 --> 29:06.400]  So, use the right scale.
[29:07.220 --> 29:08.540]  So, here's some types of scales.
[29:09.080 --> 29:11.700]  There's nominal scales which you bin things.
[29:11.700 --> 29:13.580]  There's no order, no hierarchy.
[29:13.760 --> 29:14.960]  Apples, oranges, pears.
[29:15.120 --> 29:16.840]  There's no hierarchy.
[29:17.120 --> 29:17.460]  Right?
[29:17.660 --> 29:18.660]  It's just different types.
[29:19.500 --> 29:20.200]  Then there's ordinal.
[29:21.580 --> 29:23.180]  Which is like high, medium, low.
[29:23.260 --> 29:31.620]  If you've seen the Department of Homeland Security has their red, orange, green, purple, future, whatever, bars for risk assessment.
[29:31.800 --> 29:33.180]  That is an ordinal scale.
[29:33.560 --> 29:34.020]  Right?
[29:35.280 --> 29:40.900]  Because higher doesn't, or elevated doesn't necessarily mean that the risk is two times better than bad.
[29:40.900 --> 29:41.920]  or whatever.
[29:42.320 --> 29:42.780]  Right?
[29:43.020 --> 29:45.180]  It's not telling you anything quantifiable about the risk.
[29:45.300 --> 29:47.000]  It's just telling you that it's higher.
[29:48.120 --> 29:53.920]  So, while there's a hierarchy involved, you can't do any real math on it and expect to get a meaningful answer.
[29:54.300 --> 29:54.760]  Right?
[29:55.980 --> 29:59.020]  In interval scale, however, you can do both hierarchy and calculations.
[29:59.440 --> 30:11.160]  So, a threat that's a 16, or a threat that happens 16 times a day, happens twice as often as a threat that happens 8 times a day.
[30:11.620 --> 30:13.840]  Happens twice as much as a threat that happens 4 times a day.
[30:14.180 --> 30:14.580]  Right?
[30:14.700 --> 30:15.840]  So, you can do actual math on it.
[30:17.240 --> 30:19.620]  And then a natural scale is just countable items.
[30:19.960 --> 30:22.640]  So, dollars, human lives, stuff like that.
[30:22.640 --> 30:25.000]  So, you can also do calculations on that.
[30:27.160 --> 30:28.200]  So, let's bring this all together.
[30:28.420 --> 30:31.620]  I'm going to show you Nate's presentation that he gave to his company at the end.
[30:32.260 --> 30:34.940]  And it brings together all the factors that we've been talking about.
[30:35.160 --> 30:37.240]  And you can see how it all sort of blends together.
[30:38.000 --> 30:39.340]  This is his actual presentation.
[30:39.340 --> 30:40.860]  So, I'm sorry for the theme.
[30:43.660 --> 30:45.340]  So, the problem at issue...
[30:45.860 --> 30:46.900]  He outlines it.
[30:47.080 --> 30:49.780]  So, you get the scope in the first slide.
[30:49.980 --> 30:51.180]  So, this is what we're talking about.
[30:51.700 --> 30:55.180]  So, attackers, attempting to penetrate our network to steal, destroy our altercorp data.
[30:55.380 --> 30:55.740]  Right there.
[30:56.160 --> 30:59.140]  Tells you the threat, the protector, and the asset.
[31:00.180 --> 31:00.520]  Right?
[31:02.680 --> 31:05.880]  So, getting that out on the first slide is monumental.
[31:06.700 --> 31:09.620]  You want to make sure that your audience understands what you're going to be talking about.
[31:09.620 --> 31:12.360]  And the scope of the assessment that you've done.
[31:14.420 --> 31:18.480]  Then you want to give them an idea of what the threat...
[31:20.300 --> 31:21.980]  The threat environment looks like.
[31:23.440 --> 31:23.840]  So...
[31:23.840 --> 31:25.520]  I made this shit up.
[31:25.820 --> 31:26.780]  So, given that...
[31:27.860 --> 31:32.320]  Given what you're looking at, what's the probability that it's going to happen?
[31:32.420 --> 31:34.140]  So, how likely is it to happen?
[31:34.320 --> 31:34.940]  What can happen?
[31:35.000 --> 31:35.960]  How likely is it to happen?
[31:36.260 --> 31:39.340]  You see, we're going back to those six questions of risk we talked about in the beginning.
[31:41.980 --> 31:45.560]  Then you want to give them something about, you know, this has happened to other people and here's what's happened.
[31:45.740 --> 31:46.400]  So, what can happen?
[31:47.080 --> 31:48.100]  How likely is it to happen?
[31:48.200 --> 31:48.680]  What can happen?
[31:49.120 --> 31:49.540]  You know?
[31:50.020 --> 31:51.640]  So, give them something about the outcomes.
[31:51.840 --> 31:52.960]  Something that's happened to other people.
[31:55.340 --> 31:56.860]  And then you plug in your risk analysis.
[31:58.020 --> 31:59.640]  Here's the probability of everything that will happen.
[31:59.640 --> 32:01.480]  Here's probably what will happen to us.
[32:01.580 --> 32:02.440]  Here's how screwed we are.
[32:03.180 --> 32:03.280]  You know?
[32:06.610 --> 32:07.710]  And that's basically it.
[32:08.830 --> 32:09.350]  Yes.
[32:09.510 --> 32:10.490]  It's very...
[32:11.070 --> 32:12.350]  What was that Shakespeare play?
[32:13.110 --> 32:13.630]  Hamlet.
[32:13.730 --> 32:14.130]  Hamlet.
[32:14.230 --> 32:14.790]  It's very like Hamlet.
[32:15.750 --> 32:16.270]  So...
[32:16.850 --> 32:17.750]  Six questions.
[32:18.870 --> 32:19.390]  Right?
[32:19.550 --> 32:20.830]  We talked about these in the beginning.
[32:21.030 --> 32:24.810]  He answered all three questions in a short number of slides.
[32:26.490 --> 32:28.410]  So, in those slides, he answered what can happen.
[32:28.690 --> 32:30.570]  He gave the threats that he came up with.
[32:31.090 --> 32:32.070]  How likely is it to happen?
[32:32.190 --> 32:32.770]  He gave the probabilities.
[32:32.930 --> 32:34.190]  What are the consequences if it happens?
[32:34.490 --> 32:39.830]  He gave us the probabilistic table with the annualized loss expectancy.
[32:40.130 --> 32:43.070]  So, over a number of years, this is how much, on average, we expect to lose.
[32:46.810 --> 32:47.130]  So...
[32:47.130 --> 32:48.410]  Well, that was pretty quick.
[32:49.510 --> 32:52.170]  In doing a risk assessment, there are a number of things to remember.
[32:53.050 --> 32:55.170]  Use common sense is the biggest one.
[32:55.370 --> 32:57.150]  If it looks wrong, it usually is.
[32:59.550 --> 33:02.190]  Make sure you check all of your slides.
[33:02.450 --> 33:04.130]  Make sure you check all of your calculations.
[33:04.330 --> 33:05.950]  Make sure someone else double checks for you.
[33:06.030 --> 33:10.810]  I've had a number of times where I've had the wrong calculation on a sheet, and it's thrown everything off.
[33:11.090 --> 33:12.790]  Make sure that you check it.
[33:13.170 --> 33:18.890]  Make sure you scope the question, because you don't want to be sitting in your cubicle for the next three years thinking about a threat that's a year out of date.
[33:19.570 --> 33:22.010]  So, you want to make sure you come up with an assessment that's timely.
[33:22.630 --> 33:25.270]  You want to make sure that your assessment is complete.
[33:25.590 --> 33:29.050]  And you want to make sure that you're not biting off more than you can chew.
[33:29.850 --> 33:32.750]  So, scoping the question is definitely something to remember.
[33:32.990 --> 33:36.230]  Using proper scales, like we talked about, that'll make a break in assessment.
[33:37.530 --> 33:38.750]  Remembering the six questions of risk.
[33:38.830 --> 33:43.910]  If you prepare for the six questions of risk, you will almost guaranteed have no problems in a presentation.
[33:43.910 --> 33:47.910]  You'll be prepared for all the questions they could probably ask you.
[33:49.250 --> 33:56.130]  Factor-based models are quick and easy, but remember, they're kind of dangerous in that, you know, they don't give an absolute probability.
[33:56.130 --> 34:00.850]  They only give a ranking of risky scenarios.
[34:01.870 --> 34:03.110]  And check your work.
[34:03.390 --> 34:05.850]  Academic integrity before making managers happy.
[34:05.850 --> 34:13.530]  So, it happens more often than we like to think that managers will just ask for something.
[34:14.010 --> 34:19.590]  Like, I want to see a risk assessment where X threat is the biggest threat.
[34:20.470 --> 34:22.070]  And they'll ask you to produce that.
[34:22.390 --> 34:24.310]  And personally, I just refuse to do that.
[34:24.390 --> 34:27.670]  Because my academic integrity is higher than getting paid.
[34:29.170 --> 34:31.430]  And that's something that's completely immoral.
[34:31.470 --> 34:36.490]  And we shouldn't, as risk assessments individuals and IT professionals, be stooping to that level.
[34:37.950 --> 34:41.930]  We should be producing risk assessments that are based on probabilities.
[34:41.930 --> 34:45.630]  They're based on data that are fair and explain all the risks.
[34:46.570 --> 34:49.230]  And it shouldn't be about, you know, making a certain manager happy.
[34:50.650 --> 34:52.390]  And that is basically it.
[34:52.450 --> 34:54.490]  All my slides are up on that website.
[34:54.970 --> 34:56.610]  If anyone has any questions, I'll be here.
[34:56.610 --> 34:57.970]  Thank you.
[35:04.720 --> 35:05.220]  Yeah.
[35:21.660 --> 35:22.240]  So...
[35:25.400 --> 35:31.320]  He's asking about if you've already used a risk matrix in your workplace.
[35:31.360 --> 35:34.880]  What's a good way to transition from the risk matrix to something like this?
[35:35.020 --> 35:41.040]  And using this chart is actually pretty visually similar to some of the risk matrices I've seen.
[35:41.300 --> 35:48.420]  So, visually, moving them from the risk matrix to the probabilistic risk assessment is just as easy as changing the database.
[35:48.440 --> 35:48.940]  behind the chart.
[35:49.220 --> 35:53.380]  Because visually, the managers are only going to care about the slides with the graphs on them.
[35:53.520 --> 35:58.740]  I've literally seen briefings where you walk in and they flip to the first page with the graph on it.
[35:59.160 --> 35:59.480]  Boom.
[35:59.680 --> 36:00.860]  That's their answer they want to see.
[36:00.960 --> 36:02.260]  They don't care about the rest of the analysis.
[36:02.460 --> 36:03.460]  They just want to see a graph.
[36:03.460 --> 36:07.960]  So, moving away from risk matrices should be pretty easy.
[36:08.100 --> 36:24.220]  In terms of getting the managers to sign off on the probabilistic risk assessment, there's a lot of work I think that's been done by SRA, their professional organization for risk assessment.
[36:25.920 --> 36:34.640]  They do really good white papers and such that should have a lot of information on how do you support probabilistic risk assessments over risk matrices.
[36:37.680 --> 36:38.360]  Any other questions?
[36:38.480 --> 36:38.620]  Yeah.
[36:38.760 --> 36:39.560]  Where do you get data from?
[36:39.720 --> 36:40.700]  There's no hackers all the time.
[36:41.700 --> 36:44.180]  And that is the problem that we run into most times.
[36:44.180 --> 36:50.040]  So, in that scenario, his question is what do we do when there's no data?
[36:50.920 --> 36:58.320]  So, a good thing to do is if you have the ability to put like a snort box on your system or on your network and just watch the data coming through.
[36:58.540 --> 37:03.060]  So, you can identify, okay, these are attacks that came in that weren't successful.
[37:03.080 --> 37:04.040]  These are successful attacks.
[37:04.200 --> 37:04.740]  Get some data.
[37:05.000 --> 37:10.340]  But if you can't get data using the subject matter expert elicitations.
[37:10.340 --> 37:17.880]  So, you talk to a whole bunch of hackers about what is, in your opinion, the most, where is the most risk in terms of attacks.
[37:18.140 --> 37:20.520]  And then you average out what they say.
[37:20.960 --> 37:26.940]  So, five people say, I don't know, insert attack X here.
[37:27.340 --> 37:29.720]  Four people say insert attack Y here.
[37:29.960 --> 37:31.440]  Three people say insert attack Z.
[37:31.700 --> 37:32.780]  It's X, Y, Z.
[37:33.200 --> 37:33.620]  Right.
[37:34.040 --> 37:38.200]  So, instead of using details about previous attacks.
[37:38.240 --> 37:38.720]  Right.
[37:38.720 --> 37:40.840]  So, instead you're using the binning your gut method.
[37:41.060 --> 37:44.920]  So, you're saying okay, this attack, we believe that it's a high probability that it'll happen.
[37:45.120 --> 37:46.860]  Or there's a low probability that it'll happen.
[37:47.060 --> 37:51.140]  And you're just using the subject matter experts, the hackers, the people who know it.
[37:51.480 --> 37:53.020]  And you're using their gut.
[37:53.440 --> 37:57.940]  And the more people you get in that group, the more accurate your results will be.
[37:58.200 --> 38:02.100]  Because then the more samples you'll have of people saying this is the highest, this is the highest.
[38:02.340 --> 38:02.380]  Right.
[38:03.040 --> 38:03.700]  That answer your question?
[38:04.140 --> 38:04.580]  Cool.
[38:05.200 --> 38:05.640]  Yes.
[38:06.180 --> 38:10.520]  I've heard it said that if you don't audit your predictions, then you get the predictions you deserve.
[38:11.000 --> 38:11.280]  Mm-hmm.
[38:11.600 --> 38:22.360]  Can you speak at all to how you do or convince other people to do audits and post-mortems after the period is over and you can find out whether your prediction is being true?
[38:23.860 --> 38:26.440]  I personally deal only with risk assessment.
[38:26.860 --> 38:27.860]  That is what I do.
[38:27.980 --> 38:28.820]  That is my trade.
[38:28.980 --> 38:29.500]  That is what I do.
[38:29.620 --> 38:31.200]  I don't do the follow-up stuff.
[38:31.220 --> 38:32.640]  I have no idea how to do that.
[38:32.800 --> 38:34.020]  I haven't figured that out yet.
[38:34.020 --> 38:42.140]  So, her question was, you know, if you've done the risk assessment, how do you go back and follow-up to see if what you've done is correct or on track or whatever?
[38:42.520 --> 38:52.100]  And generally, in terms of the government, they'll go back eventually, eventually, and they'll do another risk assessment and they'll see, okay, we tracked this risk properly.
[38:52.100 --> 38:53.840]  This is where the scene is shifting.
[38:53.960 --> 38:54.700]  This is what's going on.
[38:55.020 --> 38:57.840]  But in companies, especially, that's going to be really hard.
[38:58.220 --> 39:04.860]  Because managers want to do a one-time risk assessment that gets them that check mark on the certification for whatever they're doing.
[39:04.860 --> 39:08.900]  So, PCI, I think, has a risk assessment requirement of some sort.
[39:09.040 --> 39:10.400]  And they just want that check mark.
[39:10.520 --> 39:11.500]  They don't want to follow-up at all.
[39:11.680 --> 39:15.340]  So, you have to hound it into them that this is a risk assessment snapshot.
[39:15.840 --> 39:16.200]  Right?
[39:16.900 --> 39:20.100]  So that it's only for this specific period in time and it's going to change.
[39:21.600 --> 39:23.220]  And they're probably not going to agree with you.
[39:23.460 --> 39:24.740]  I haven't figured that part out yet.
[39:24.920 --> 39:25.800]  But that's the best I got.
[39:25.940 --> 39:26.020]  Yeah?
[39:45.720 --> 39:46.080]  Right.
[39:46.220 --> 39:51.480]  And that's going into risk management, which is the next three questions, which might be another talk some other year.
[39:52.500 --> 39:54.880]  And it goes into how much have I reduced my risk.
[39:54.980 --> 39:56.200]  I have put in this amount of dollars.
[39:56.420 --> 39:58.020]  I have reduced my risk this percent.
[39:58.020 --> 40:03.380]  And that's why you use the probabilistic risk assessments instead of the factor-based models.
[40:03.400 --> 40:05.860]  Because you have a number you can come back to later.
[40:06.380 --> 40:06.440]  Right?
[40:08.460 --> 40:09.020]  Yep.
[40:11.220 --> 40:12.420]  I'm just curious.
[40:12.700 --> 40:16.460]  The relationship between risk assessment and penetration testing.
[40:17.400 --> 40:25.360]  Do we see risk assessments done and then immediate follow-ups with penetration tests to see how accurate the risk assessments were?
[40:25.440 --> 40:26.540]  Is there a relationship there?
[40:26.540 --> 40:27.720]  I would personally...
[40:27.720 --> 40:32.680]  I think there should be a greater meshing between a risk assessment and penetration test.
[40:32.840 --> 40:37.600]  I think a penetration test by itself is not necessarily an actionable item for a corporation.
[40:37.880 --> 40:44.020]  But a risk assessment based on that penetration test will give you a lot more depth, a lot more insight on what happened.
[40:44.600 --> 40:46.860]  And assessing more where...
[40:46.860 --> 40:47.960]  Okay, so these are your risks.
[40:48.160 --> 40:49.520]  This is how it affects your business.
[40:49.800 --> 40:50.760]  This is your bottom line.
[40:50.760 --> 40:51.340]  Right.
[40:52.440 --> 40:58.400]  So, I think a penetration test first and then a risk assessment based on the penetration test will give you great data.
[40:58.640 --> 41:00.320]  Okay, we fired X amount of attacks.
[41:01.140 --> 41:04.020]  We see X amount of attacks in real life from these types of attacks.
[41:04.180 --> 41:06.120]  These are how... this is how vulnerable you are.
[41:06.480 --> 41:06.920]  Right?
[41:06.920 --> 41:07.660]  So...
[41:07.660 --> 41:16.040]  And would there be a conflict of interest or do you think it would be possible for either the same individual, the leader of the penetration testing team, to then do the reassess...
[41:16.040 --> 41:16.920]  the risk assessment?
[41:17.120 --> 41:20.360]  Or would you really need two different parties moving those two different arrangements?
[41:21.000 --> 41:32.100]  Personally, I think I would use two different teams, but in reality with, you know, budgets and such, the same person could do a pretty good risk assessment given that they also did the penetration test.
[41:32.360 --> 41:33.920]  Because they're the expert in the matter.
[41:34.260 --> 41:35.040]  They've done it.
[41:35.100 --> 41:35.760]  They have the information.
[41:35.880 --> 41:36.300]  They have the knowledge.
[41:36.560 --> 41:42.440]  They're probably the best suited to do it in terms of, you know, knowledgeable about the matter.
[41:42.840 --> 41:43.240]  Okay?
[42:09.590 --> 42:11.410]  Yeah, I deal with the government.
[42:14.630 --> 42:17.510]  You know, you do a risk assessment, you do a penetration test.
[42:17.610 --> 42:17.970]  Thank you very much.
[42:19.210 --> 42:22.430]  And you come back later and reassess it and do they care?
[42:24.090 --> 42:27.130]  This comes from a Coast Guard buddy of mine.
[42:27.250 --> 42:34.710]  He walked into a briefing and someone had done a risk assessment, a full probabilistic risk assessment, and it laid it out on the table.
[42:34.710 --> 42:39.950]  And at the end of the risk assessment, one of the people in the briefing stands up and says, this is beast crap.
[42:40.150 --> 42:41.650]  This doesn't support what we want to do.
[42:41.930 --> 42:43.010]  We're going to do something else.
[42:43.770 --> 42:46.590]  And so, you know, in some parts of the government, it works like that.
[42:46.750 --> 42:48.590]  In some companies, it'll work like that.
[42:48.870 --> 42:50.050]  But it shouldn't.
[42:50.070 --> 42:55.890]  And the more we do real probabilistic risk assessments, the more that it might hammer it into their brains that, you know, they're idiots.
[42:57.990 --> 42:58.350]  Yeah?
[43:04.780 --> 43:05.140]  Yep.
[43:05.140 --> 43:05.180]  Yeah.
[43:05.740 --> 43:09.320]  You mentioned that your graduating class was a Star Trek.
[43:09.540 --> 43:09.780]  Yes.
[43:09.980 --> 43:10.360]  Yes, it was.
[43:10.580 --> 43:11.220]  Which series?
[43:11.480 --> 43:13.520]  It was the next generation actually.
[43:13.740 --> 43:14.120]  It was a...
[43:14.120 --> 43:14.700]  Thank you.
[43:15.360 --> 43:21.980]  It was a local horn quartet that had come in and they did, you know, the standard pomp and circumstance and all that stuff.
[43:22.080 --> 43:28.460]  And so as I'm walking down the aisle, and it's a big auditorium, it's a Penn State, the Eisenhower auditorium is huge.
[43:28.460 --> 43:33.780]  And so we're walking down the aisle and it switches and it's like, I turn to the person next to me, is that Star Trek?
[43:34.020 --> 43:35.440]  They're like, damn straight.
[43:37.120 --> 43:38.160]  It was pretty pimp.
[43:39.760 --> 43:40.160]  Yeah?
[43:40.860 --> 43:47.000]  You mentioned also earlier that, you know, throwing up the snort box and getting some of your information on Larry.
[43:47.180 --> 43:55.440]  Are there tools currently available that can take some pretty standard pieces of data and turn it into a probabilistic risk assessment?
[43:55.440 --> 43:56.920]  I have not seen any.
[43:57.000 --> 44:04.940]  His question is if there are any semi-automated processes or programs that will take like snort data and give you a probabilistic risk assessment.
[44:05.300 --> 44:09.740]  And as far as I've seen, it's got to be more human interaction.
[44:09.740 --> 44:18.820]  Because as we go back to the first slide, the first couple slides, it's not only the data that you have, but it's a process that involves your imagination.
[44:18.820 --> 44:29.180]  And so the more expert you are in the field and the more your imagination encompasses more of that big blue box that is the universe, the better your risk assessment will be.
[44:29.300 --> 44:33.600]  And a snort box is only going to give you the attacks that you've already seen.
[44:33.780 --> 44:38.120]  What if the next Super Mongo attack comes and it only happens once, but it's really effective?
[44:40.180 --> 44:41.000]  Yeah, in the back.
[44:41.000 --> 44:58.060]  What if either wants to remain blind or thinks they don't think, if they say we don't really care about the risks, we don't have to take a great time, how do you convince them to do a risk assessment assessment, which should all turn convince them that they should do a risk assessment?
[44:59.300 --> 45:03.420]  If they basically think we know we're already, so we're not going to do that.
[45:03.940 --> 45:13.140]  Generally, the way the government's worked in the past is a bunch of guys sat down in a table, like three or four guys, and one of the guys says, I think we should do X.
[45:13.140 --> 45:15.840]  And the other guys are like, well, you pay my salary, so I agree.
[45:17.260 --> 45:23.040]  And it's been a sort of beating them over the head from the top effect.
[45:25.720 --> 45:27.840]  So, you see, this is the part of my job that sucks.
[45:27.900 --> 45:33.200]  I can't really tell you about all the things I do, so I can't tell you specifically in answer to your question.
[45:33.480 --> 45:42.480]  But in general, beating them over the head from the top, so convincing someone higher than them that they're an idiot, and forcing them to use the probabilistic risk assessments generally works.
[45:43.000 --> 45:52.920]  So, the people higher than them, if you can get to them and show them, listen, the managers have been doing it this way, but if we do it this way, it'll save money, money is huge, and it'll be more effective.
[45:53.800 --> 45:54.520]  That's big.
[45:54.820 --> 45:57.900]  And once you get them bought on the idea, you're golden.
[45:57.900 --> 45:58.840]  Yep.
[45:59.220 --> 46:01.000]  Okay, this is not a question.
[46:01.220 --> 46:07.420]  This is a, I got a master's in tech management specifically to help deal with shit like this.
[46:07.520 --> 46:16.540]  Where you have some technical information, you need to communicate up to you to managers, so they understand how that will make the company more effective and make everybody's lives easier.
[46:16.540 --> 46:26.340]  So, I know about a bunch of tools, aphorisms, practices, techniques, and so on, to help you communicate, to be the GeekSoup interface that you know you can be.
[46:26.760 --> 46:27.680]  So, come talk to me after.
[46:29.480 --> 46:30.160]  Very nice.
[46:31.400 --> 46:31.880]  Yep.
[46:44.170 --> 46:46.890]  And the black swan raises its ugly head.
[46:47.570 --> 46:48.410]  Not you personally.
[46:48.590 --> 46:53.050]  This is a theme that's been going in especially intelligence communities for a while.
[46:53.250 --> 46:54.310]  There was a general...
[46:56.610 --> 47:00.310]  The question was, how do you account for situations that you don't envision?
[47:01.010 --> 47:02.250]  So, things that...
[47:09.110 --> 47:10.210]  So, right.
[47:10.330 --> 47:14.210]  So, you don't have any empirical evidence to support the probability that this will happen.
[47:14.450 --> 47:22.290]  And the black swan, the reason I bring it up is, if you've only seen white swans in your life, you can't imagine a world where there's a black swan.
[47:22.290 --> 47:25.150]  And so, you have no data to back up that there might be a black swan.
[47:25.270 --> 47:34.850]  And there's a book by Nassam Taleb that a general just had stacks of it in his office and would give out to every single person that came into office and said, we look for the black swan.
[47:35.230 --> 47:37.410]  And just sort of did it nonchalantly and didn't actually.
[47:38.590 --> 47:38.950]  So...
[47:40.050 --> 47:40.770]  Unknown unknowns.
[47:41.070 --> 47:42.050]  Unknown unknowns.
[47:43.230 --> 47:45.070]  So, in general, we...
[47:45.830 --> 47:47.950]  That's where the imagination comes in, right?
[47:47.990 --> 47:48.690]  So, you can't...
[47:48.690 --> 47:51.850]  Like he was asking, you can't just take the output dump from a snort box.
[47:51.850 --> 47:54.530]  Run it through a program and come up with a risk assessment.
[47:54.750 --> 48:00.450]  You have to sort of imagine a world where a super mongo virus will come along and kill you all.
[48:00.930 --> 48:01.370]  Right?
[48:01.610 --> 48:05.390]  So, you have to imagine that and what your vulnerability would be from that.
[48:05.830 --> 48:08.590]  So, it involves sort of imagining.
[48:08.770 --> 48:09.630]  And you have to...
[48:09.630 --> 48:14.730]  When you do something like that, if you explain it in your methodologies, right?
[48:14.870 --> 48:25.010]  When you're presenting to managers or whatever, how you came up with this risk assessment, the first couple pages of your report should be, okay, this is what we did in detail.
[48:25.190 --> 48:26.510]  This is how we got to these results.
[48:26.790 --> 48:32.630]  This is, you know, we made these assumptions, leap to these conclusions, and we thought that these might happen.
[48:33.910 --> 48:43.090]  And if you explain it upfront and account for that in your imagination, that will, I think, cover the black swan.
[48:44.310 --> 48:49.630]  But, as you know, September 11th and all the other crap that's happened, we can't envision everything.
[48:49.630 --> 48:51.130]  So, shit just happens.
[48:52.010 --> 48:52.450]  Yeah?
[49:08.090 --> 49:14.370]  I mean, every spare battery holds as much power as a hand grenade and I can't take it in.
[49:15.250 --> 49:18.790]  I'm sorry, but I cannot discuss this topic due to national security reasons.
[49:21.490 --> 49:22.370]  No, I...
[49:23.830 --> 49:25.030]  I hate my job sometimes.
[49:25.550 --> 49:27.190]  I seriously can't talk about that.
[49:27.770 --> 49:29.510]  Like, I just straight up can't talk about that.
[49:29.630 --> 49:32.470]  So, I would love to answer your question, but I can't.
[49:32.470 --> 49:32.810]  Yes.
[49:37.120 --> 49:37.780]  So...
[49:37.780 --> 49:38.200]  Welcome.
[49:38.360 --> 49:38.980]  I'm...
[49:44.100 --> 49:45.100]  Apology accepted.
[49:45.620 --> 49:47.060]  I don't think the conclusion is a little bit.
[49:47.140 --> 49:47.660]  Please jump.
[49:47.800 --> 50:06.630]  I think what he said is significant in that, like, for example, Schneier, maybe a couple of months ago, said that that sort of, like, ripple threats came here.
[50:08.630 --> 50:13.330]  And you're suggesting that that's not the case?
[50:13.330 --> 50:17.910]  I'm suggesting that in doing a risk assessment, you should...
[50:17.910 --> 50:23.050]  You might consider including situations that have not occurred in the past.
[50:23.630 --> 50:25.350]  If your risk assessment...
[50:25.350 --> 50:26.670]  If you scope it...
[50:26.670 --> 50:28.890]  So, we were talking before about scoping a question.
[50:29.430 --> 50:29.710]  So...
[50:29.710 --> 50:30.090]  That's fine.
[50:30.590 --> 50:31.490]  Slides are online.
[50:31.590 --> 50:31.910]  Have fun.
[50:31.910 --> 50:34.150]  We were talking before about scoping a question.
[50:34.370 --> 50:42.630]  So, if within the scope of your question, it comes up that you might want to include situations that might happen in the future that haven't happened in the past.
[50:42.830 --> 50:43.930]  These black swan events.
[50:44.090 --> 50:45.590]  September 11th of the world and such.
[50:46.530 --> 50:46.970]  Then...
[50:46.970 --> 50:48.750]  And you think that might be helpful.
[50:48.750 --> 50:52.010]  This is where being a human comes into play.
[50:52.150 --> 50:56.210]  And if you think that might be helpful in the risk analysis, you should probably include that.
[50:56.750 --> 51:07.150]  But being that black swan events are so infinitesimally probable, so minuscule in their probability, that they almost never happen, sometimes washes them out of the entire analysis.
[51:07.150 --> 51:08.150]  So, then...
[51:09.450 --> 51:16.190]  Doesn't try and take into account black swan events actually serve to increase...
[51:25.300 --> 51:31.840]  Because if you plan for black swan event A1, you get black swan event B250.
[51:32.400 --> 51:34.100]  Again, this is where I...
[51:34.100 --> 51:38.540]  He's talking about, like, if you plan for one black swan event and another one happens, what...
[51:38.540 --> 51:39.320]  How screwed are you?
[51:39.700 --> 51:40.100]  And...
[51:40.100 --> 51:42.820]  This is, again, where being a human comes into play.
[51:42.820 --> 51:51.680]  So, if when you're thinking about it, you include these black swan events, you're including events that you're thinking up out of your mind, right?
[51:51.920 --> 51:56.120]  So, sometimes the decision makers will actually ask you not to include them.
[51:56.340 --> 51:59.580]  So, depending on what the analysis asks for and what the analysis calls for...
[51:59.580 --> 52:10.220]  So, if you're looking at just a risk assessment based on current threats, sometimes they'll add in the current threats thing, or based on threats perceived, there'll be specific lists of threats that you're looking at.
[52:10.220 --> 52:13.340]  And you'll just stick to those lists, and you'll just keep going.
[52:13.480 --> 52:15.560]  But the black swan involves a paradox.
[52:15.860 --> 52:19.480]  So if we look at black swan events, we're just become massively paranoid.
[52:19.720 --> 52:22.100]  And we become completely immobilized.
[52:22.140 --> 52:27.240]  But if we don't look at black swan events, then, you know, September 11's happened and people die.
[52:27.400 --> 52:28.860]  But those are going to happen anyway.
[52:29.380 --> 52:39.140]  So what I'm getting at is maybe a posture of simple readiness.
[52:40.100 --> 52:51.960]  Rather than trying to plan for this particular black swan event, you know, and driving the alarmism that seems to drive so much of the security theater in our...
[52:51.960 --> 52:55.280]  And that's definitely another discussion for another time.
[52:55.860 --> 52:57.940]  I'm sorry to have to do that to you, but there's other people.
[52:58.540 --> 53:00.100]  I'd love to discuss with you more.
[53:00.380 --> 53:01.460]  Please, come talk to me.
[53:01.540 --> 53:02.380]  I'd love to discuss with you more.
[53:02.580 --> 53:02.720]  Yeah.
[53:06.030 --> 53:06.690]  Go for it.
[53:07.870 --> 53:09.570]  and do the black swan concept.
[53:09.770 --> 53:10.270]  Two seconds.
[53:10.430 --> 53:18.430]  There's a difference between movie plots, which is what Bruce Schneier is talking about, and system brittleness.
[53:19.090 --> 53:24.810]  So that's one of the arguments about the black swan idea, is that you actually have a system that bends and actually accommodates.
[53:24.810 --> 53:30.690]  So one way to deal with black swan events is to have a reinforced cockpit that Bruce Schneier argued for.
[53:30.850 --> 53:37.590]  Then all of a sudden you turn around and you do movie plots, it's like snakes on a plane, 2.9 ounces of liquid on a plane, or whatever.
[53:37.850 --> 53:38.870]  That's the contrast.
[53:39.090 --> 53:40.570]  Movie plots versus gorillas.
[53:41.130 --> 53:42.730]  That's where we're kind of talking.
[53:43.190 --> 53:44.170]  Any other questions?
[53:44.950 --> 53:45.470]  Yep.
[53:54.250 --> 53:56.070]  How do you...
[54:02.380 --> 54:09.660]  And we're running short on time so I'm going to have to cut you short, but he's talking about, so what if the posture you take creates more risks?
[54:09.980 --> 54:13.980]  And how do we account for deterrence?
[54:14.040 --> 54:18.480]  How do we account for the fact that just doing something changes people's ideas and what they're going to do?
[54:18.540 --> 54:20.880]  How do we account for half the things that people do?
[54:20.980 --> 54:24.020]  How do we model the human mind in a way that we can understand what it's going to do?
[54:24.020 --> 54:24.940]  And we have no idea.
[54:26.160 --> 54:28.960]  We're just getting to the point where we can use probability properly.
[54:29.140 --> 54:30.680]  We're like taking baby steps here.
[54:31.730 --> 54:33.200]  So that's something we have to look at.
[54:33.940 --> 54:34.300]  Yeah?
[54:35.400 --> 54:41.960]  In risk assessment, do you have any lookbacks where basically you have what you measure, then you come back and you modify the model?
[54:41.960 --> 54:45.940]  In theory, risk analysis should be baked into everything we do.
[54:46.300 --> 54:49.740]  So every step we take should have a risk analysis at the beginning and the middle and the end.
[54:49.940 --> 54:50.260]  Right?
[54:50.400 --> 54:51.420]  At the very least.
[54:51.820 --> 54:58.460]  So we can understand, okay, here are the risks, we've mitigated the risks, we've gotten rid of the risks, and we're fine.
[54:59.060 --> 55:01.380]  So in theory, everything should have a risk assessment.
[55:01.800 --> 55:05.740]  But then again, you know, it lacks streamlining.
[55:06.080 --> 55:06.740]  Am I done?
[55:07.720 --> 55:08.680]  Yeah, I have a minute.
[55:09.300 --> 55:09.740]  Yeah.
[55:09.880 --> 55:11.680]  So this is for Dougie's, but if you want to go further?
[55:11.780 --> 55:15.780]  If we want to go further, take college courses in statistics.
[55:16.220 --> 55:19.600]  Really, college courses in statistics are a huge step on this.
[55:21.100 --> 55:30.300]  SARMA, the something-something for risk analysis, is a professional organization that can help you get more training in probabilistic risk assessment and things of that nature.
[55:30.820 --> 55:36.380]  There are risk assessment courses, there are risk assessment classes, there are things online.
[55:36.660 --> 55:37.460]  There's a whole bunch of stuff.
[55:37.520 --> 55:38.540]  You just have to go and look for it.
[55:38.680 --> 55:42.240]  And I'd be happy to point you in the right direction offline because I'm out of time.
[55:42.440 --> 55:43.180]  So thank you, everyone.
[55:44.700 --> 55:45.520]  I hope you had fun.