[00:01.590 --> 00:04.530]  Saturday, quick announcements to start us off with.
[00:04.870 --> 00:07.870]  Remember to check out amd.hope.net to explore your badge.
[00:07.950 --> 00:13.150]  You're not going to be able to see all the live stuff after you leave here, so make sure you do it before you go.
[00:14.190 --> 00:18.610]  I took a look at the name of this talk and I'm reminded back at H2K.
[00:18.850 --> 00:27.370]  We used to joke that you could see the random number generators with how far off the schedule you actually had your talks.
[00:28.590 --> 00:29.890]  Things have gotten a lot better since then.
[00:30.050 --> 00:33.430]  We were only one hour off in this track and one hour off in Lovelace.
[00:33.710 --> 00:40.570]  However, we are starting the Robert Steele talk still at midnight instead of moving it an hour later.
[00:40.710 --> 00:43.370]  We're starting in midnight in Tesla, not in Lovelace.
[00:43.810 --> 00:47.550]  Lovelace will get the Hacker Cinema and that will start at 1 o'clock in the morning.
[00:47.930 --> 00:53.670]  So, a lot of late stuff today and I hope you check them out because, well, you don't get that chance every day.
[00:54.190 --> 00:56.610]  So, much ado about randomness.
[01:03.590 --> 01:04.230]  Thank you.
[01:05.090 --> 01:07.070]  So, first of all, thank you everyone for coming.
[01:07.630 --> 01:10.950]  So, the goal of this talk is to talk about what is randomness.
[01:11.170 --> 01:17.150]  It seems like all of us know what it is, but at the same time nobody can really explain what exactly it is.
[01:17.330 --> 01:25.570]  So, this talk is going to be a collection of information about what is randomness from theoretical perspective as well as from practical perspective.
[01:25.890 --> 01:28.430]  Where's the bridge between theory and practice?
[01:28.430 --> 01:36.450]  So, first of all, nobody can define what exactly randomness is as evidenced by this cartoon.
[01:36.690 --> 01:39.550]  Here, the random number generator keeps generating numbers nine.
[01:40.430 --> 01:44.770]  So, the key point is that randomness is very easy to get incorrect.
[01:45.990 --> 01:47.230]  So, if you look...
[01:49.070 --> 01:49.970]  Is this better?
[01:50.190 --> 01:50.350]  Yeah.
[01:50.550 --> 01:51.130]  Okay, great.
[01:52.490 --> 01:54.890]  So, in theory, everything is great.
[01:55.070 --> 02:00.350]  If you look at theoretical cryptographic literature, why do we need random numbers?
[02:07.260 --> 02:08.460]  So, thank you.
[02:09.140 --> 02:11.680]  So, the random numbers are used to generate session IDs.
[02:11.880 --> 02:13.500]  They're used to generate cryptographic keys.
[02:13.640 --> 02:15.920]  They're used to generate gift certificate numbers.
[02:16.260 --> 02:19.660]  And the key assumption in a lot of cryptographic talks...
[02:19.660 --> 02:20.540]  Hi, Stuart.
[02:21.120 --> 02:23.300]  Is that secret keys must be random.
[02:23.300 --> 02:28.540]  And a very common assumption in theory is a random oracle model.
[02:28.840 --> 02:33.000]  Where you assume that all parties have access to this oracle.
[02:33.160 --> 02:34.560]  Which is a perfect random source.
[02:34.700 --> 02:36.020]  You send it an input x.
[02:36.240 --> 02:40.920]  And it's going to reply with a perfectly uniformly distributed h of x.
[02:41.720 --> 02:46.460]  And under this assumption, you can prove a lot of interesting theorems.
[02:47.260 --> 02:49.060]  You can prove that zero-knowledge proofs exist.
[02:49.280 --> 02:51.980]  You can prove that different types of signatures are possible.
[02:51.980 --> 02:53.460]  So, in theory, everything is great.
[02:53.600 --> 02:56.000]  And I come from a theoretical crypto community.
[02:56.980 --> 03:00.440]  And in practice, are things as good as in theory.
[03:00.740 --> 03:03.700]  So, in practice, we see a sad smiley face.
[03:03.900 --> 03:07.000]  And we keep hearing about different types of exploits.
[03:07.000 --> 03:12.620]  Which are actually related to the fact that bad random numbers have been used.
[03:12.840 --> 03:18.520]  So, the Kaminsky bug relied on the predictability of DNS session IDs to poison DNS caches.
[03:18.520 --> 03:27.800]  Recently, people discovered that versions of Debian open SSL implementations generated weak SSL keys.
[03:28.480 --> 03:31.700]  Because of a change in code to correct the purify comments.
[03:33.240 --> 03:36.380]  Some major DOMA mailing list versions were susceptible.
[03:36.700 --> 03:42.180]  Because, you know, when you subscribe to a mailing list, it's going to reply with an email with a random number.
[03:44.840 --> 03:51.120]  So, some earlier versions of major DOMA, which have now been patched, had a predictable ID.
[03:51.460 --> 03:56.780]  So, as a result, some people decided to prank... to play pranks on different... on their different friends.
[03:56.920 --> 04:00.300]  And subscribe them to thousands upon thousands of mailing lists.
[04:01.280 --> 04:05.240]  Kerber's four secret keys, they had a bug.
[04:05.540 --> 04:07.860]  They did mem set, seed zero.
[04:08.320 --> 04:10.900]  And the entire seed was set to zero.
[04:10.980 --> 04:14.020]  As a result, the secret keys of Kerber's four were guessed in seconds.
[04:14.680 --> 04:18.860]  So, you can... you can enumerate many such examples.
[04:19.180 --> 04:20.560]  So, in theory, things are good.
[04:20.660 --> 04:21.740]  In practice, things are bad.
[04:21.920 --> 04:27.980]  And the reason that things are bad in practice is because most of the time we make mistakes in how we generate the numbers.
[04:27.980 --> 04:30.220]  So, here's example number one.
[04:30.440 --> 04:32.300]  And it's a question to the audience.
[04:32.680 --> 04:35.480]  What is wrong with this C program?
[04:35.800 --> 04:42.200]  Do you see anything wrong if the magic cookie is supposed to be a secure... a secure number?
[04:45.120 --> 04:48.420]  And the answer is, it uses... yes?
[04:48.720 --> 04:50.120]  It's time for the input.
[04:51.420 --> 04:52.800]  Okay, anything else?
[04:55.650 --> 04:56.250]  Yes.
[04:56.250 --> 05:00.350]  So, it uses RAND, which is a linear congruential generator.
[05:00.650 --> 05:08.870]  Now, interestingly, this is somewhat similar to the code used by Xwindows magic cookie generation, which was guessable in X11 R6.
[05:09.030 --> 05:11.630]  And it used a weak linear congruential generator.
[05:12.410 --> 05:13.790]  Here's example number two.
[05:14.790 --> 05:18.590]  It's a snippet of a Java program.
[05:18.590 --> 05:21.910]  And you see that random is equal to new Java util random.
[05:22.250 --> 05:28.230]  Then you do some gobbledygook to set the seed and, you know, use that for generating the sessions.
[05:28.410 --> 05:30.110]  Do you see anything wrong with this?
[05:41.750 --> 05:53.130]  So, besides the fact that it's Java, the interesting part is an author of the program is using Java util random for an operation to generate random numbers, which are used in the session ID context.
[05:53.130 --> 05:55.790]  And it's actually code of Jetty.
[05:56.190 --> 06:04.350]  So, Jetty was vulnerable and had weak session IDs prior to version 4.2.26.
[06:05.330 --> 06:06.850]  Here's a third example.
[06:07.630 --> 06:08.670]  It's pseudocode.
[06:08.770 --> 06:09.710]  It's not actual code.
[06:09.710 --> 06:17.830]  But, I mean, the way it looks is, you know, you say, you try to extract seconds and microseconds out of the time of day.
[06:18.030 --> 06:22.190]  Then you figure what the process ID and the parent process ID are.
[06:22.610 --> 06:24.730]  Again, do some bit shifts.
[06:24.890 --> 06:28.750]  Compute the MD5 function of the microseconds and the process IDs.
[06:28.970 --> 06:33.750]  And do some simple scrambling and use that as a session ID.
[06:34.070 --> 06:36.210]  So, is there anything wrong with it?
[06:40.510 --> 06:42.430]  So, a couple of problems.
[06:44.570 --> 06:45.590]  I couldn't hear.
[06:45.790 --> 06:45.990]  Sorry.
[06:46.130 --> 06:52.770]  So, assuming you have access onto the system IDs already, you know what the system is.
[06:52.970 --> 06:55.150]  So, you can easily figure this out.
[06:55.230 --> 06:55.630]  Exactly.
[06:56.310 --> 07:01.370]  Now, the interesting part which made this difficult is the source code to the program wasn't available.
[07:01.490 --> 07:06.670]  But the two students back in the time in 1996 in UC Berkeley reverse engineered the code.
[07:06.670 --> 07:11.130]  One of them was David Wagner, who is now a pretty well-known computer security figure.
[07:11.350 --> 07:13.690]  And the pseudocode is for Netscape 1.1.
[07:14.470 --> 07:14.910]  So...
[07:17.990 --> 07:18.430]  Yeah.
[07:19.010 --> 07:21.830]  So, Netscape 1.1 was vulnerable to this as well.
[07:22.510 --> 07:23.670]  So, given those examples...
[07:24.450 --> 07:25.290]  And we can...
[07:25.290 --> 07:28.590]  And you know, if you poke around the Internet, you can find many more examples like this.
[07:28.590 --> 07:30.630]  But the lessons learned are...
[07:30.630 --> 07:31.710]  There were two mistakes made.
[07:32.510 --> 07:38.410]  I mean, in general, if you want to summarize it, the numbers which were used to derive session IDs and keys weren't truly random.
[07:39.390 --> 07:43.890]  But the two lessons that must be learned are, number one, the seeds must be unpredictable.
[07:43.970 --> 07:52.450]  So, the initial seed, which was used to seed the pseudorandom generator, had to be, you know, well dispersed, uniformly random.
[07:52.450 --> 07:54.870]  All possibilities needed to be equally likely.
[07:55.310 --> 07:59.410]  And the pseudorandom generator function must be secure.
[07:59.650 --> 08:04.030]  So, for example, in the Java JETI case, the Java Util Random is not secure.
[08:04.330 --> 08:10.810]  In the linear congruential generator example in C, the random number generator was not secure as well.
[08:11.390 --> 08:16.030]  The Netscape 1.1 used an MD5 to get a stream of random data.
[08:16.270 --> 08:17.650]  It's not a horrible idea.
[08:18.370 --> 08:21.110]  But they used the predictable session ID.
[08:21.110 --> 08:27.010]  So, they used the process ID and the time of day as a session ID and as a result that could be guessed.
[08:27.350 --> 08:29.590]  So, these are the two lessons.
[08:29.850 --> 08:30.970]  Seeds must be unpredictable.
[08:30.970 --> 08:33.110]  And the random number generator must be secure.
[08:33.390 --> 08:44.790]  And consequently, if you reverse it and get the contrapositive, the two methods of attack are guessing what the seed are and trying to break the state of the pseudorandom generator because it's weak.
[08:46.290 --> 08:49.650]  So, just a little bit of a foray into theory.
[08:50.450 --> 08:53.150]  In general, there are two types of random number generators.
[08:53.330 --> 08:59.830]  One of them is truly random, like, you know, physical, such as radioactive decay, if you do a fair coin flip.
[09:00.710 --> 09:08.130]  Some people might argue that on some operating systems, disk IO threat scheduling is kind of random.
[09:08.390 --> 09:14.950]  So, for example, Java makes this assumption to generate the seeds for its secure random numbers by looking at the threat scheduling.
[09:15.870 --> 09:18.590]  Interestingly, on some operating systems, it's truly random.
[09:18.750 --> 09:20.050]  On others, it's not so random.
[09:21.090 --> 09:27.090]  And the pseudorandom number generator takes the initial seed and then it stretches it out into a longer sequence.
[09:27.390 --> 09:31.410]  And, in fact, in practice, we usually use pseudorandom number generators.
[09:31.410 --> 09:48.170]  So, if you look at how a good pseudorandom number generator on a computer works is it tries to create a seed from, you know, disk IO operations from threat scheduling and then it applies one of the secure or insecure constructions to stretch the seed into a multitude of random numbers,
[09:48.170 --> 09:51.730]  which are used to create session IDs, cryptographic keys, et cetera.
[09:54.390 --> 10:02.970]  So, in theory, right, what I just described is you have a random seed, let's say 1, 0, 0, 1, just four bits, for example.
[10:03.910 --> 10:09.010]  You feed it to some kind of a mathematical transformation called a pseudorandom number generator.
[10:09.590 --> 10:15.950]  And it outputs a much longer pseudorandom string, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1.
[10:15.950 --> 10:29.610]  And, you know, from theoretical perspective, if you want to define what exactly does it mean for the string to be pseudorandom, it means that an observer cannot tell the pseudorandom string apart from a truly random string.
[10:29.830 --> 10:39.330]  So, the formal definition by Bloom, Mikali, and Yao is that a pseudorandom generator is a function and you can play a game, you can play an experiment.
[10:40.030 --> 10:41.610]  I have two strings.
[10:41.610 --> 10:45.450]  One of them is a pseudorandom string, which is an output of a function.
[10:46.230 --> 10:49.190]  And this generated from a truly random seed.
[10:49.370 --> 10:54.670]  And the other string is a really truly random string generated from physical sources of randomness.
[10:55.070 --> 11:00.670]  And a pseudorandom generator is defined to be secure if you cannot tell apart between the two.
[11:00.890 --> 11:02.350]  You can define it formally.
[11:02.750 --> 11:08.350]  In mathematics, you just say that no probabilistic polynomial time algorithm can tell apart the two distributions.
[11:08.350 --> 11:13.470]  And we consider only polynomial time algorithms as efficient.
[11:13.730 --> 11:19.170]  So, you know, if no algorithm can tell it apart and you can prove it formally, then the function is good.
[11:19.410 --> 11:28.970]  An example of, you know, an example of a good random number generator is not linear congruential, but, for example, SHA-1 pseudorandom number generator.
[11:29.490 --> 11:38.010]  So, the output of a function SHA-1, you know, the property of SHA-1 is that if I give you the output of SHA-1, you cannot compute what the input is.
[11:38.150 --> 11:44.950]  You can try to compute the values of the function for different types of inputs and guess what the output corresponded to.
[11:45.110 --> 11:47.830]  But, you know, computationally it's hard to do one from the other.
[11:48.010 --> 11:49.970]  So SHA-1 scrambles the bits pretty well.
[11:50.170 --> 11:52.110]  And that's what a lot of programs use.
[11:52.270 --> 11:54.270]  I mean, I'm kind of lying here a little bit.
[11:54.270 --> 11:56.190]  You know, they use SHA-1 with a few tweaks.
[11:56.390 --> 11:57.810]  But, essentially, that's what they do.
[11:58.350 --> 12:04.030]  So, the methods of attack on weak pseudorandom number generators are as follows.
[12:04.370 --> 12:08.870]  So, I mean, if you typically think about how do you break into a system, you know, there are clear-cut stages.
[12:09.530 --> 12:12.190]  Reconnaissance, you try to find out as much about the system.
[12:12.330 --> 12:14.610]  You enumerate different ports, services in a system.
[12:14.770 --> 12:17.110]  You fingerprint them to see if anything is vulnerable.
[12:17.470 --> 12:19.230]  You try to break in and maintain access.
[12:19.430 --> 12:23.730]  So, you can apply the same methodology for weak pseudorandom number generators.
[12:23.730 --> 12:26.790]  The first step is to do reconnaissance.
[12:26.930 --> 12:29.790]  To find programs with weak pseudorandom number generators.
[12:30.730 --> 12:33.350]  And then, you know, you analyze the program.
[12:33.470 --> 12:35.750]  You look at them somehow in some magical way.
[12:35.930 --> 12:40.430]  And you guess either the initial seed of a pseudorandom number generator.
[12:40.430 --> 12:45.050]  Or you try to guess the state of a pseudorandom number generator because the function is bad.
[12:45.230 --> 13:00.130]  And then you break in, you know, and by break in I mean usually the goal of breaking a random number generator is to compromise session IDs, to guess what the cryptographic keys are so that you can forge a signature, so that you can forge a certificate,
[13:00.410 --> 13:06.010]  or just login into a program by guessing what the next session ID is.
[13:06.830 --> 13:08.450]  So here's the schema.
[13:08.710 --> 13:13.590]  So the first question is, well, how do you find programs with weak pseudo-random number generators?
[13:14.510 --> 13:16.890]  And it's actually not very difficult.
[13:16.910 --> 13:18.950]  It's actually probably the easiest part.
[13:19.890 --> 13:22.250]  The difficult part is figuring out how to break them.
[13:23.010 --> 13:24.430]  So what are the options, right?
[13:24.710 --> 13:29.430]  You know, if you look at the decision tree, if the program is open source, that's pretty easy.
[13:29.490 --> 13:34.450]  You can look at its source and, you know, analyze it all you want and figure out if it uses weak API calls.
[13:34.610 --> 13:38.410]  And I'll tell you what the weak API calls are in a couple of slides.
[13:41.190 --> 13:45.850]  If you have a binary, and a binary most of the time for programs is not obfuscated.
[13:46.030 --> 13:47.930]  You know, most of the time programs are not obfuscated.
[13:48.030 --> 13:49.690]  Very few programs are really obfuscated.
[13:50.130 --> 13:53.570]  But if a binary is available, you can also reverse engineer the program.
[13:53.750 --> 13:59.010]  And by reverse engineering, I don't mean you need to do something as complicated as attaching a debugger and so on.
[13:59.210 --> 14:03.110]  Sometimes you do, as an example with Netscape 1.1.
[14:03.530 --> 14:06.650]  But, you know, often you can just grab the binary for weak system calls.
[14:07.630 --> 14:09.590]  I'll show you how you do it on the next slide.
[14:10.670 --> 14:20.670]  If you have just the output of a program, you can also use tools such as Stumpy or Entropy, ENT, SessionID to analyze the output's randomness quality.
[14:20.890 --> 14:34.030]  And for web-based programs, the pretty well-known penetration proxies, Burp Suite and WebScarab, which people typically use to break into applications, they actually have functionality to analyze the SessionID randomness.
[14:34.130 --> 14:37.390]  And you can also use Google hacking to find some weak SessionIDs.
[14:37.390 --> 14:38.930]  I'll give you a couple of examples.
[14:40.930 --> 14:58.890]  So, I mean, if you look at commonalities between programs with good randomness and weak randomness, you know, in a good program, you would see usage of Java secure random, RNG crypto service provider, reading from DevRandom, using some type of hardware security module.
[15:00.290 --> 15:08.910]  And in weak, low entropy programs, typically what they do is they use something funky, like use time and date as a SessionID.
[15:09.210 --> 15:12.950]  They use some kind of a random static string inside the source code.
[15:13.290 --> 15:17.790]  They use bad API calls, like Java util random, RAND, and so on.
[15:18.590 --> 15:25.830]  Or they would take low entropy seed and use MD5, for example, to try to garble up the low entropy source.
[15:26.070 --> 15:28.950]  Of course, we need to have a good seed and a good function.
[15:29.130 --> 15:32.090]  You know, if one of the elements is broken, you lose.
[15:32.950 --> 15:34.330]  So, here are the weak API.
[15:36.830 --> 15:40.110]  So, in the left column, they're the weak API for programming languages.
[15:40.290 --> 15:42.050]  In the right column, they're the strong API.
[15:42.590 --> 15:44.230]  So, in Java, Java util random is bad.
[15:44.810 --> 15:52.190]  Typically, the weak API use LCG, linear shift feedback registers, Mersin twisters, and so on.
[15:52.510 --> 15:58.370]  The strong APIs may use SHA-1 based PRNG, DES, in theory, Bloom Bloom Shroop, for example.
[15:59.750 --> 16:02.510]  So, typically, in Java, you use secure random.
[16:02.750 --> 16:07.170]  In UNIX, C, C++, a lot of people are using RAND, RANDOM, RANDOMIZE.
[16:07.370 --> 16:12.830]  The correct method is to actually read from slash dev random or slash dev urandom.
[16:13.010 --> 16:14.690]  There are slight differences between the two.
[16:15.150 --> 16:23.230]  Dev urandom doesn't block, so it's possible that dev urandom is going to give you a source of bad entropy, although it usually doesn't happen.
[16:23.670 --> 16:31.470]  Dev random blocks, but always tries to give you a good randomness, but there are also some exhaustion attacks against dev random.
[16:32.290 --> 16:37.770]  And, you know, so for each language, there are weak API and strong API, and it's very easy to find which one to use.
[16:38.210 --> 16:41.730]  And still, you know, from what I'm seeing, developers usually don't know them.
[16:41.790 --> 16:48.670]  If you ask a regular developer, probably eight out of ten would not know the difference between Java util random and secure random.
[16:49.250 --> 17:01.240]  So language providers like, you know, for Java and C sharp, don't just substitute the insecure random calls for secure random calls?
[17:01.360 --> 17:02.360]  It's a great question.
[17:02.660 --> 17:06.490]  And three slides later, I asked myself the same question and did an experiment.
[17:06.510 --> 17:08.140]  So why don't we do it?
[17:08.180 --> 17:09.250]  So I'll actually get to it.
[17:09.940 --> 17:14.340]  So remember I talked about finding programs with bad randomness.
[17:15.800 --> 17:21.120]  So, you know, there's a Java profiler, and it does reverse bytecode engineering.
[17:21.440 --> 17:25.310]  So you can just do... So bad random is a program I wrote, which uses Java util random.
[17:25.600 --> 17:29.640]  You do Java P minus C, bad random, grab random in UNIX, and there you go.
[17:29.770 --> 17:31.380]  You get the Java dot util random.
[17:31.810 --> 17:42.140]  It's very easy to take the simple command line in UNIX and extend it into a script, which is just going to unpack the jar files, iterate over every single class in a jar file, and look for it.
[17:42.380 --> 17:51.470]  I've been having some fun with some commercial programs, which I'm not going to name, but, you know, it's very easy to extend it, and, you know, there are a lot of programs which get this wrong.
[17:52.360 --> 17:57.440]  In C, C++, you know, rand, random, randomized, these are all libc functions.
[17:57.660 --> 18:02.730]  So, you know, you can do a variety of different tricks in UNIX just to analyze the binary.
[18:02.970 --> 18:05.880]  You can do an M, and then just grab for random.
[18:06.180 --> 18:11.550]  And, you know, if there are any rand or srand calls, you know, you're going to see a simple reference right there.
[18:11.770 --> 18:18.830]  You can do strings, bad random, and again, see that, you know, they're using srand, time, printf.
[18:18.940 --> 18:21.860]  I didn't even bother attaching the debugger to this program.
[18:22.270 --> 18:24.660]  It's like a one-second exercise to check.
[18:27.790 --> 18:29.270]  So analyzing the output.
[18:29.750 --> 18:33.810]  So I keep talking about, you know, bad randomness, good randomness, but what does it really mean?
[18:33.860 --> 18:39.810]  How do you check and practice that these theoretical definitions, which I put in the first slide, actually apply?
[18:40.810 --> 18:42.120]  So there are various tests.
[18:42.400 --> 18:49.880]  So, for example, NIST came up with FIPS 142 statistical tests, and they have a battery of different tests.
[18:50.050 --> 18:55.210]  So, for example, a very simple test is you count the number of ones in the bit representation and the number of zeros.
[18:55.510 --> 18:59.940]  If an output is random, you're going to have approximately the same number of ones as the number of zeros.
[19:01.600 --> 19:04.490]  You can try to compress the sequence.
[19:04.750 --> 19:14.360]  If the sequence is truly random, it's not going to compress very well because, you know, there are no, you know, there are no redundancies that could be compressed.
[19:14.880 --> 19:19.860]  If a system, if a sequence is not very, is not very random, then it would be better compressed.
[19:20.120 --> 19:26.710]  So if you had a huge sequence, you can just take, you know, zip or rar and just compress it and look at what the output is.
[19:28.920 --> 19:34.070]  So, speaking of Java Util Random, you know, everybody keeps saying about why is it bad.
[19:34.270 --> 19:36.550]  So, it is a linear congruential generator.
[19:36.640 --> 19:41.750]  So, it uses a predictable formula, Xn plus 1 equal to Xn plus B.
[19:42.010 --> 19:47.180]  So, you know, if you look at the source code, it's going to be just a bunch of multiplications and bit shifts.
[19:47.440 --> 19:49.210]  So, in theory, this is horrible.
[19:49.880 --> 19:58.920]  You know, and there are papers published by John Boyer, for example, a professor in Denmark, which show how, you know, linear congruential generators can be predicted.
[19:59.160 --> 20:04.730]  In practice, I tried to find really hard implementations of where people tried to guess the Java Util Random.
[20:05.600 --> 20:09.120]  It wasn't easy, but there are some people who did that and actually used their paper.
[20:09.360 --> 20:11.010]  I'm not posting a link to that, though.
[20:12.640 --> 20:17.600]  So, but just geometrically, it's very easy to understand why Java Util Random is bad.
[20:17.750 --> 20:23.730]  So, if you plot the output of Java Util Random on a single line from 0 to 1, it looks kind of random.
[20:24.620 --> 20:35.460]  If you plot it in two-dimensional, and what I mean is you generate a sequence of numbers, and you just take two consecutive numbers, and you treat the first one as an X coordinate, the second as a Y coordinate.
[20:35.680 --> 20:36.830]  It also looks pretty random.
[20:36.970 --> 20:38.640]  So, just by looking at this graph, you can tell.
[20:39.790 --> 20:48.200]  Now, there was an ingenious person who said, why don't we try plotting it in 3D, where we take the consecutive three numbers and use the first one as X, Y, and Z.
[20:48.770 --> 20:51.570]  There were some manipulations he did, but essentially that's what he did.
[20:51.660 --> 20:54.360]  Just take three consecutive numbers in the output.
[20:54.620 --> 20:55.920]  And here's what you got.
[20:57.100 --> 21:04.030]  So, it turns out that output of linear congruential generators fall on linear planes if you plot it in 3D.
[21:04.230 --> 21:09.200]  If this was a truly random number generator, then, you know, the entire cube would be filled.
[21:09.490 --> 21:16.030]  And there are actually programs which use this type of flaw in the Java Util Random to predict the state.
[21:16.210 --> 21:17.660]  They're not very easy to get, though.
[21:18.400 --> 21:26.360]  So, in practice, you know, in practice, honestly, in all honesty, this is noticeable only if you keep running your program over and over.
[21:26.530 --> 21:42.160]  If you just need to generate, let's say, 20,000 samples using Java Util Random and compute the entropy of 20,000 samples, you get almost the same entropy for Java Util Random and Java Secure Random using a simple script that I hacked up.
[21:42.290 --> 21:46.330]  They also pass all of the statistical tests by FIPS for 20,000 samples.
[21:46.470 --> 21:50.510]  If you do it for 200,000 samples, you're going to see a divergence in entropy.
[21:53.020 --> 21:53.980]  I don't remember.
[21:54.380 --> 21:56.290]  It's a standard Java Util Random.
[21:58.420 --> 21:59.500]  No, just integers.
[21:59.700 --> 22:02.860]  Just generate a bunch of integers and then analyze their randomness.
[22:04.740 --> 22:08.940]  And, I mean, as you generate more samples, kind of predictably, you're going to see a divergence.
[22:09.240 --> 22:15.760]  So, for Java Security Secure Random, you get 17.6096 bits of entropy.
[22:15.880 --> 22:16.780]  And the more, the better.
[22:17.020 --> 22:19.310]  And for Java Util Random, you get slightly less.
[22:19.720 --> 22:23.140]  But still, you know, to exploit this type of stuff, you have to be very, very sophisticated.
[22:23.260 --> 22:25.790]  But there are some programs which allow you to do it.
[22:26.160 --> 22:34.760]  So, going back to your question, first of all, why don't people just always use Secure Random?
[22:35.140 --> 22:38.780]  And I'll just skip that slide and then go back to it.
[22:38.920 --> 22:43.760]  So, what I did is there's a framework called ASM bytecode manipulation framework.
[22:44.440 --> 22:50.380]  And what that framework allows you to do is just to hack the class loader and just change the bytecode on the fly.
[22:50.620 --> 22:57.480]  And what I did is I put a simple stub method which is going to change every invocation of Java Util Random to Java Security Secure Random.
[22:57.760 --> 23:02.420]  And because Secure Random inherits from Java Util Random, you're not going to really see the difference.
[23:03.550 --> 23:05.360]  So, why don't people always do it?
[23:05.520 --> 23:11.260]  And the folklore says, I mean, if you just Google and see, people say that Secure Random is horribly slow.
[23:11.640 --> 23:15.100]  People say that it's 68 times slower than Random.
[23:15.540 --> 23:19.640]  Of course, you know, Random takes much less than a second.
[23:19.810 --> 23:22.020]  So, you're not really going to notice the difference.
[23:22.280 --> 23:26.040]  So, my approach is, you know, I would just use Secure Random all the time.
[23:26.260 --> 23:34.810]  But, you know, I read these news groups which say Secure Random is horrible and I ask myself, well, is it really 68, 70 times slower in reality?
[23:35.480 --> 23:39.830]  And interestingly, it's actually dependent on an operating system.
[23:40.020 --> 23:44.180]  So, on Open Solaris, it's 67.9 times slower.
[23:44.620 --> 23:48.540]  On Windows XP, it's 64.5 times slower.
[23:48.790 --> 23:51.880]  On Windows 7, it's 24.5 times slower.
[23:52.020 --> 23:53.540]  So, Windows 7 was an improvement.
[23:53.810 --> 23:57.160]  And on Mac OS, 25 times slower.
[23:57.330 --> 23:57.960]  So, Macs are good.
[23:58.980 --> 24:01.790]  But Windows, not so good until Windows 7.
[24:02.540 --> 24:07.460]  But, I mean, honestly, in all honesty, I still think people should just use Secure Random all the time.
[24:07.810 --> 24:11.540]  Unless it's some kind of a trading application where you keep generating the numbers.
[24:11.540 --> 24:13.780]  I don't know why you would use Java Util Random.
[24:14.000 --> 24:17.900]  I would just use Secure Random all the time and not... And the same for Random.
[24:18.070 --> 24:20.520]  I would just read from slash Dev Random.
[24:20.810 --> 24:22.330]  But you have to ask yourself a question.
[24:22.460 --> 24:23.570]  What do you use a number for?
[24:23.880 --> 24:33.310]  So, if you use a random number to decide what color of a background to display to a user when he visits the page, who cares, right?
[24:33.500 --> 24:33.780]  So, okay.
[24:33.880 --> 24:37.620]  So, somebody can tweak the page to be cyan instead of magenta.
[24:38.500 --> 24:51.160]  But, you know, if you use a random number to generate session IDs, maybe it's still okay to use Java Util Random if it's, you know, if it's not for shopping, but just for keeping track of pages you visited.
[24:51.440 --> 24:57.980]  But if you use it for session ID on an e-commerce site, or if you use it for cryptographic keys, I mean, there's no question about it.
[24:58.020 --> 25:00.400]  You need to know what the Secure API are and use them.
[25:00.570 --> 25:02.180]  And a lot of people don't do it.
[25:03.440 --> 25:04.240]  Google hacking.
[25:04.480 --> 25:06.900]  It seems like Google hacking is all the rage.
[25:07.240 --> 25:10.780]  You can actually use Google as your friend for some of the interesting things.
[25:11.400 --> 25:18.020]  And, you know, typically, know the common session IDs, J session ID, session ID, PHP session ID, and so on.
[25:18.790 --> 25:26.550]  So, you know... So, the first step is let's look for web pages which have session ID in the URL.
[25:26.810 --> 25:28.440]  So, session ID is used to track session.
[25:28.700 --> 25:33.880]  So, some sites don't trust cookies, and they want to put it in a query string.
[25:34.020 --> 25:35.160]  Not necessarily a good idea.
[25:35.830 --> 25:42.260]  So, in the URL, question mark session ID is going to give you sites which use session ID in the query string.
[25:42.900 --> 25:47.380]  Then, try to play with some predictable non-random sequences.
[25:47.620 --> 25:48.940]  Like, what session ID is predictable?
[25:49.260 --> 25:59.380]  Well, you know, if a session ID starts with six, six, six, or six, six, six, or, you know, twenty-six hundred, that's a pretty bad session ID generator.
[25:59.960 --> 26:02.440]  Because a truly random sequence would not be like that.
[26:02.600 --> 26:05.830]  So, you're going to get a lot of churn if you just try that query on Google.
[26:06.040 --> 26:12.360]  But if you go look through the results, I've been able to find quite a bit of interesting sites for further exploration.
[26:13.160 --> 26:15.720]  You know, you can also search on Google for programs.
[26:16.810 --> 26:21.380]  So, you know, you can just search language Java, Java Util Random session.
[26:21.660 --> 26:24.360]  And, you know, again, a lot of noise to filter through.
[26:24.520 --> 26:31.240]  But, you know, I found quite a bit of interesting open source software which uses Java Util Random for sessions, just, you know, last week.
[26:32.160 --> 26:37.940]  So, to analyze randomness of client programs, as I mentioned, I don't have much time to go into it.
[26:38.160 --> 26:41.640]  But Formulab has a suite for entropy tests.
[26:41.900 --> 26:52.740]  So, instead of writing a script to analyze the output of a program, you can just download the script, run it, and it's going to tell you, you know, is this output random or not random.
[26:52.960 --> 26:54.070]  There's also Stumpy.
[26:54.920 --> 26:56.040]  So, I tried it.
[26:56.070 --> 26:56.700]  I played with it.
[26:56.790 --> 26:58.220]  But it seems to be a little bit optimistic.
[26:58.400 --> 27:02.180]  For some reason, it says that too many random number generators are good.
[27:02.380 --> 27:04.830]  I tried it with Java Util Random and thought it's good.
[27:04.960 --> 27:08.160]  I tried it with Linear Congruential Generator and thought it's good.
[27:08.640 --> 27:11.260]  So, my guess is I just didn't try enough numbers.
[27:11.420 --> 27:14.550]  But I had slightly strange results with Stumpy.
[27:14.830 --> 27:16.360]  So, this is a picture of Fiji.
[27:16.550 --> 27:18.260]  I figured it's a really technical talk.
[27:18.380 --> 27:19.310]  So, I'll let you look at it.
[27:19.440 --> 27:20.220]  Enjoy for a second.
[27:22.070 --> 27:23.050]  Take a deep breath.
[27:23.940 --> 27:34.260]  Now, the interesting part, by the way, about Fiji, besides that it's a happy island, they don't have, interestingly, much laws and regulations about hacking and computer security.
[27:34.500 --> 27:38.900]  So, there was a Russian hacker who police was pursuing and he ended up in Fiji.
[27:39.140 --> 27:48.140]  And here's a quote from the Fiji police cyber crime unit surgeon that they couldn't charge him because technically, you know, he didn't commit anything criminal.
[27:48.460 --> 27:50.000]  So, Fiji is a good place.
[27:50.160 --> 27:52.780]  In New York, we're a little bit more distrustful of people.
[27:54.260 --> 27:56.720]  And the Fiji picture is from Amazon.
[27:57.020 --> 28:04.240]  So, the next part of the talk, just a couple more slides, is about web programs and session ideas and web programs.
[28:05.220 --> 28:10.540]  And I didn't want to pick on a website which uses bad randomness because that's too easy.
[28:10.540 --> 28:15.900]  There are many examples like that, such as 123greetings.com was hacked before it got fixed.
[28:16.220 --> 28:18.880]  But they used horrible session IDs for greeting cards.
[28:19.880 --> 28:21.020]  So, I picked on Amazon.
[28:21.220 --> 28:22.260]  Amazon is a big company.
[28:22.810 --> 28:24.640]  And they have different layers of security.
[28:24.680 --> 28:26.000]  It's a picture of the Fiji book.
[28:26.100 --> 28:27.400]  I'm not endorsing it.
[28:28.370 --> 28:30.860]  But Amazon has different levels of security.
[28:30.860 --> 28:43.680]  If you just want to keep track of, you know, what books you clicked on, what you've done, they use a cookie called a session ID, which is a 17-digit random number, a persistent cookie that expires after seven days.
[28:43.860 --> 28:45.860]  The first time you reach Amazon, it's set.
[28:45.980 --> 28:48.040]  And the value usually doesn't change after you log in.
[28:48.620 --> 28:52.400]  So, to be completely fair, it's a fairly low-level security cookie.
[28:52.540 --> 28:56.740]  So, if you want to purchase something or log into your account, you're going to have to need another cookie.
[28:56.740 --> 28:59.810]  So, this is not going to be enough to get you anything else.
[29:00.570 --> 29:06.140]  And I Googled, looked on some forums, and everybody says that Amazon is great, session IDs are great.
[29:06.310 --> 29:07.830]  So, I decided to do a little bit experiment.
[29:09.730 --> 29:10.700]  So, it's okay.
[29:10.830 --> 29:13.120]  I didn't break it, but I found some interesting things.
[29:13.330 --> 29:17.790]  So, to analyze session IDs, there are a number of GUI tools that you can use.
[29:18.020 --> 29:21.160]  The ones that I like the most are WebScarp and BurpsWid.
[29:21.160 --> 29:38.960]  And they, essentially, what they do is you point them to a site, you capture a couple of requests from a site, they figure out what the session ID is, what's the name of a cookie in which the session ID is stored, and then they essentially apply a battery of different statistical tests to analyze the cookie for randomness,
[29:39.310 --> 29:40.620]  and display nice graphs.
[29:41.040 --> 29:45.400]  So, they do essentially everything I talked about, but on Web programs.
[29:46.660 --> 29:53.980]  This is not Amazon, but if you had a predictable session ID, you would see a line such as this in this graph.
[29:54.480 --> 30:00.330]  So, you know, the interpretation is, you know, to the left is, so, on the x-axis is date and time.
[30:00.500 --> 30:03.740]  As you generate more cookies, they follow a predictable trend.
[30:03.810 --> 30:12.040]  So, if you found out what the cookie was at 6 o'clock, you could deduce what the cookie is at 7 o'clock because it's predictable.
[30:12.280 --> 30:15.540]  So, I ran the test on Amazon, and I wasn't very successful.
[30:16.430 --> 30:17.720]  But, here's what I saw.
[30:18.260 --> 30:22.920]  It looks kind of random for Amazon.com session IDs, but...
[30:23.910 --> 30:26.280]  So, you know, at that point, I thought, you know what, it's bad.
[30:26.480 --> 30:28.790]  So, I decided to run a couple more tests.
[30:29.070 --> 30:30.000]  This is WebScarab.
[30:30.180 --> 30:31.520]  So, I tried BurpsWid.
[30:31.760 --> 30:42.600]  So, in BurpsWid, I collected the session IDs, and the session IDs have the form of, you know, 180-3029497-6970862.
[30:42.680 --> 30:47.100]  So, there are two dashes in the middle, and the rest are numbers from 0 to 9.
[30:48.760 --> 30:56.000]  And, BurpsWid did a bunch of different tests, computed the difference between the cookies based on the time, and here's the histogram it produced.
[30:56.180 --> 30:57.550]  And the histogram was very interesting.
[30:57.980 --> 31:03.160]  So, positions 3 and 11, which are red, we don't care about them because these are dashes.
[31:03.310 --> 31:06.940]  So, obviously, you know, the tool said that 3 and 11 were bad.
[31:06.940 --> 31:11.100]  Now, the interesting part ended up being in positions 0 and 1.
[31:12.330 --> 31:20.830]  So, it turned out, if you look at the Amazon session IDs, usually in a position number 1, you're almost always going to see digit number 1.
[31:20.980 --> 31:25.180]  In a position 2, you're usually going to see 7, 8, or 9.
[31:25.520 --> 31:30.420]  So, in other words, for position 0 and 1, you have fairly predictable numbers.
[31:31.910 --> 31:40.140]  So, it's not necessarily terrible because you have, what, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15 digits afterwards.
[31:40.810 --> 31:43.420]  But, it makes the brute force analysis that much easier.
[31:43.960 --> 31:48.180]  I don't know why they do it, but they should have really used something different.
[31:48.720 --> 31:53.740]  So, I mean, just the fact that they have it, you know, kind of makes me wonder about what algorithm they use.
[31:53.900 --> 31:58.200]  And this was done in 10 minutes of analysis without looking at the code or anything.
[31:58.940 --> 32:02.180]  So, this is another histogram on Amazon.
[32:02.550 --> 32:07.660]  So, the overall quality of randomness within the sample was estimated to be poor.
[32:07.900 --> 32:11.700]  And again, it's just a session ID used to track the persistent logins.
[32:12.240 --> 32:16.660]  But the reason I'm showing to you these tools is that you can use them for all kinds of sites.
[32:17.040 --> 32:18.960]  And Amazon is a tough cookie to crack.
[32:19.140 --> 32:24.740]  But, you know, if you look carefully, there are a lot of different sites which just screw up and use horrible session IDs.
[32:25.200 --> 32:27.420]  You know, greetings one, two, three, I mentioned.
[32:27.780 --> 32:29.860]  There were many other sites which used that.
[32:30.650 --> 32:35.160]  Facebook, it used bad numbers to identify the photos on Facebook.
[32:35.330 --> 32:40.780]  So, you could look at people's photos by guessing what the numbers are.
[32:40.920 --> 32:41.500]  They fixed it.
[32:41.640 --> 32:46.420]  But, you know, it was a pretty well-known exploit in the community until recently.
[32:47.140 --> 32:48.160]  So, this is it.
[32:49.330 --> 32:52.200]  I didn't want to burden you guys too much with theory.
[32:52.200 --> 32:57.380]  But the conclusion is, you know, know what the common types of attacks are.
[32:57.780 --> 33:01.550]  So, you know, breaking the seed, guessing the state of a pseudorandom number generator.
[33:01.880 --> 33:04.550]  The moral of a story is use good API.
[33:05.240 --> 33:08.050]  Use good seeds and strong pseudorandom number generator.
[33:08.260 --> 33:09.360]  And just Google for those tools.
[33:09.570 --> 33:11.240]  Stump and WebScare, BurpSuite.
[33:11.830 --> 33:12.780]  And happy hacking.
[33:13.070 --> 33:17.120]  So, now if you have any questions, I'll be glad to take them.
[33:26.970 --> 33:27.450]  Yes.
[33:29.270 --> 33:33.550]  I'm wondering if you have any comment on the state of the random number generator in the Linux kernel.
[33:33.690 --> 33:43.690]  I know that there have been a couple of repeated requests to shift it over to more standard algorithms, more inference from the disk or each piece from inputs.
[33:45.250 --> 33:46.650]  There's a really good paper.
[33:46.650 --> 33:48.890]  I believe it was by Tzvi Guterman.
[33:49.210 --> 33:50.270]  If you just Google...
[33:51.090 --> 33:52.150]  And Benny Pincus.
[33:52.510 --> 33:55.550]  And if you just Google for that, Linux random number generator.
[33:55.770 --> 33:59.830]  So, they outline a number of problems in the Linux random number generator.
[34:00.010 --> 34:01.850]  And different types of attacks you can use.
[34:02.050 --> 34:05.550]  So, in particular, the algorithm they used to define... to derive it.
[34:05.710 --> 34:06.850]  I don't know...
[34:06.850 --> 34:11.010]  I don't think they fixed it in the latest Linux kernel versions as far as I remember.
[34:11.450 --> 34:12.730]  So, take a look at that paper.
[34:12.890 --> 34:13.970]  Just Google for that.
[34:14.050 --> 34:15.270]  It's a very good overview.
[34:20.600 --> 34:21.060]  Anybody?
[34:21.440 --> 34:21.720]  Yes?
[34:21.980 --> 34:22.360]  And
[34:26.150 --> 34:33.930]  hardware solutions like some of the IBM boxes that they offer that are supposed to just give you great random numbers?
[34:35.190 --> 34:37.370]  So, I mean, I looked at a couple of them.
[34:37.850 --> 34:39.310]  I know a lot of them are good.
[34:39.310 --> 34:40.570]  But I don't...
[34:40.570 --> 34:43.730]  I mean, I don't exclude the possibility that there could be some bad ones.
[34:43.930 --> 34:46.710]  I haven't looked at it in very much detail, though.
[34:47.110 --> 34:47.830]  But hardware...
[34:47.830 --> 34:51.030]  You know, hardware modules for generating random numbers are usually...
[34:52.050 --> 34:52.950]  they're pretty good.
[34:53.150 --> 34:55.110]  But it's a dangerous assumption to make.
[34:55.690 --> 35:00.150]  You know, if you look at electronic voting systems, for example.
[35:00.570 --> 35:03.090]  That was a hardware machine as well.
[35:03.390 --> 35:06.130]  And then a group of researchers showed that it's horrible.
[35:06.430 --> 35:09.410]  And that's why we don't trust electronic voting so much.
[35:10.530 --> 35:11.050]  Yes?
[35:22.100 --> 35:24.300]  But what application do you want to use it for?
[35:24.400 --> 35:25.060]  That's the question.
[35:25.740 --> 35:27.080]  Well, for reasonable entropy.
[35:27.420 --> 35:44.260]  So I think for reasonable entropy, I mean, I really think that, you know, using even a programmatic solution where you just harvest the initial seed from, you know, threat scheduling or disk IO that would be sufficient if you use a good algorithm.
[35:44.580 --> 35:52.740]  You know, I think the moral I'm trying to convey is that the reason a lot of these things get broken is not because people didn't use expensive hardware to generate random numbers.
[35:52.860 --> 35:54.520]  I know all the vendors are going to hate me.
[35:54.600 --> 35:56.160]  But it's because people are being stupid.
[35:56.460 --> 35:58.960]  You know, they don't know better.
[35:59.760 --> 36:01.340]  And they're not supposed to do better.
[36:01.800 --> 36:04.220]  Developers don't necessarily need to know about security.
[36:04.220 --> 36:08.160]  And especially about this type of area, which is very esoteric.
[36:08.480 --> 36:11.940]  You know, very few people really use this type of exploits.
[36:12.040 --> 36:17.100]  People are typically not going to try to guess what the session ID number is, although some researchers do.
[36:17.320 --> 36:25.860]  Usually they're going to try to steal the session ID using, I mean, CSERF, for example, as the gentleman talked in the talk above.
[36:26.040 --> 36:30.340]  So, you know, if you look at the attacks, generally randomness is not very high up on the list.
[36:30.400 --> 36:31.180]  It's esoteric.
[36:31.180 --> 36:36.260]  The top attacks are weak passwords, SQL injection, you know, cross-site scripting, CSERF.
[36:36.700 --> 36:38.780]  And this is going to be down the list.
[36:38.920 --> 36:43.500]  But nonetheless, it is an important area, especially for security community and hackers to know about.
[36:44.800 --> 36:45.180]  Yes?
[36:45.460 --> 36:46.240]  As far as
[36:49.460 --> 36:54.660]  addressing these problems and doing the best thing, the United States and China, we have our tips.
[36:55.020 --> 36:57.840]  But on the other hand, we all want each other stable at the same time.
[36:58.100 --> 36:58.420]  Yeah.
[36:59.340 --> 37:03.960]  So, man, I know that there's a lot of research being done on random number generators.
[37:03.960 --> 37:17.420]  So, one of my co-authors and one of my papers actually invited me to come to NIST about five years ago to give a talk about this type of stuff, random number generators, at NIST to see how, you know, what's the state.
[37:17.620 --> 37:18.980]  So, I mean, there's a lot of interest.
[37:19.200 --> 37:27.580]  I'm not privy to exactly what the governments are doing, but I know from, you know, research community, like, you know, public standards and so on, people are very interested in this.
[37:31.610 --> 37:32.050]  Yes?
[37:32.250 --> 37:33.110]  Can you do any research
[37:37.960 --> 37:38.340]  generation?
[37:38.700 --> 37:39.760]  That's a good question.
[37:40.000 --> 37:52.220]  So, allegedly, that's, I mean, I remember, so I'm not an expert on quantum computing, but I remember being a paper where you can use quantum computing to generate, like, you know, a really random number.
[37:54.480 --> 37:57.020]  Do you have anything to add, or...
[37:58.040 --> 37:58.480]  Skepticism.
[37:58.740 --> 37:59.180]  Skepticism?
[38:00.900 --> 38:01.780]  Yeah, so...
[38:01.780 --> 38:02.140]  Hmm?
[38:03.280 --> 38:03.720]  Yeah.
[38:05.920 --> 38:15.720]  So, quantum computing allows you all types of cool stuff, so, you know, a lot of the cryptographic algorithms need to be completely reworked, but, you know, I don't claim expertise in quantum computing.
[38:15.720 --> 38:25.280]  So far as the random number generation goes, if you get quantum computers working, it's going to be far, far after we get quantum number generation working.
[38:25.480 --> 38:34.460]  I mean, at this point, we're looking at megabits in the 10 to the 100s just from using light with beam speeders and doing post-continues to generate random number streams.
[38:34.880 --> 38:38.900]  I mean, quantum computing is far further away than having a very, very small box.
[38:40.100 --> 38:43.000]  And a lot of different algorithms are going to have to be used.
[38:43.000 --> 38:45.360]  You're not going to necessarily use RSA encryption.
[38:45.800 --> 38:47.360]  You're going to use quantum encryption.
[38:47.600 --> 38:53.600]  So, you know, there's a lot of research in academic community, but in practice, I think we're extremely far out.
[38:53.760 --> 38:54.900]  They're very small successes.
[38:58.100 --> 38:58.620]  Okay.
[38:59.300 --> 39:00.480]  Well, thank you very much.