[00:02.110 --> 00:04.930]  Thank you for having me here at HOPE.
[00:05.490 --> 00:08.510]  And yeah, I'm happy to be talking about homomorphic encryption today.
[00:09.550 --> 00:11.910]  So I'm going to start a little bit about myself.
[00:12.490 --> 00:14.210]  My name is Vikram Saraf.
[00:14.830 --> 00:20.030]  I'm a software engineer at the Johns Hopkins University Applied Physics Lab, or APL4 short.
[00:20.710 --> 00:26.130]  I have a PhD in computer science, and I have degrees in math and computer science from Notre Dame.
[00:27.110 --> 00:39.650]  I've been in this field of homomorphic encryption, researching both homomorphic encryption itself as well as applications of it for about two years at the lab, although I also work on several other things as well.
[00:41.170 --> 00:44.510]  So this is a little outline of the talk.
[00:44.830 --> 00:52.010]  First, I'm going to be talking about what is homomorphic encryption with like a really basic illustrated toy example, just to illustrate the concept.
[00:52.810 --> 01:00.230]  Then I'll be getting into the history of homomorphic encryption and some of the recent breakthroughs and why it is that people are interested in it today.
[01:01.130 --> 01:10.430]  Then I'll be talking about this concept called arithmetic circuits, which lets us reason about how it is that we can apply homomorphic encryption to problems that we want to solve.
[01:12.110 --> 01:19.470]  Then I'll be talking about some practical or software implementations for how it is that we can actually use homomorphic encryption in practice.
[01:20.150 --> 01:32.450]  And then I'll specifically be building up to an application where we can use homomorphic encryption to privately detect strokes among x-ray image data that we might have available.
[01:32.970 --> 01:39.990]  And then I'll be talking about some of the limitations in this field and some of the ongoing research and some of the future work that still has to happen.
[01:41.750 --> 01:51.550]  So if you go to Wikipedia and you look at, you know, what is homomorphic encryption, you'll see something like homomorphic encryption, is a privacy enhancing technology that enables direct computation on encrypted data.
[01:51.810 --> 02:04.330]  What this really means is that if you're an individual homomorphic encryption, it lets you encrypt your data in such a way that you can hand it off to a third-party service provider and let them do some sort of useful computation on it.
[02:05.670 --> 02:13.390]  And then let them hand that result back to you without them ever having to have seen that data to begin with.
[02:14.590 --> 02:23.550]  So let me just go through an example, which is a really quick toy example of what this concept actually means.
[02:23.930 --> 02:26.390]  So you know, this is cryptography.
[02:26.610 --> 02:32.210]  So we have Alice and Bob, and I'm going to illustrate a toy example using Alice and Bob.
[02:32.390 --> 02:44.770]  So for example, say Alice has these two numbers, two and three, and she wants them added, but say she doesn't have the compute capacity, say, to add these two numbers.
[02:45.970 --> 02:54.390]  So she wants to hand these two numbers off to Bob without Bob actually knowing what the numbers are, but she wants Bob to be able to add these two numbers.
[02:54.390 --> 02:55.470]  So she has two and three.
[02:55.750 --> 02:57.030]  She starts with two and three.
[02:58.510 --> 03:05.190]  Then she homomorphically encrypts two and three, and then she's able to send this data off to Bob.
[03:06.290 --> 03:07.890]  Bob is the service provider.
[03:08.170 --> 03:10.090]  He receives these two numbers.
[03:10.390 --> 03:20.330]  He doesn't know what these numbers two and three are, but he has the capability to add these numbers two and three, and he gets a number five back.
[03:20.510 --> 03:23.170]  He's not able to see that five because it's still encrypted.
[03:23.570 --> 03:28.850]  Of course, we as the omniscient viewer can see that it's five.
[03:29.830 --> 03:35.990]  So once he's computed that number five, he can take that encrypted five and set it back to Alice.
[03:36.450 --> 03:40.830]  And of course, Alice has the key, so she can decrypt that number and see the result.
[03:40.990 --> 03:42.310]  And it's five, of course.
[03:44.110 --> 03:47.610]  And so in practice, you know, we're not going to have a...
[03:47.610 --> 03:51.690]  we're not going to use homomorphic encryption to do something silly like this, where you just add two numbers.
[03:52.610 --> 04:00.730]  In practice, Alice is some sort of client or customer, some sort of user, some sort of third-party service provider.
[04:00.990 --> 04:09.130]  And yeah, Bob is a third-party service provider that has a lot of compute and the capacity to do complicated computations.
[04:10.750 --> 04:17.690]  So the general paradigm is that Alice, your user, your customer, they have sensitive data.
[04:18.850 --> 04:27.430]  They want some computation, some sort of useful computation to be done on that data without it being revealed to the service provider.
[04:27.730 --> 04:34.230]  The service provider has lots of compute available to them, but they don't want...
[04:35.330 --> 04:39.310]  the end user doesn't want them to see, you know, what's in that data.
[04:40.170 --> 04:52.350]  The example we'll be building up to is Alice having some sort of CT scan, and we want to determine whether there is a stroke in that scan privately without revealing what is actually in that scan.
[04:53.250 --> 04:57.350]  So I'm going to be covering now some of the history of homomorphic encryption.
[04:58.390 --> 05:01.230]  Why... like, some of the, I guess, key milestones.
[05:04.190 --> 05:12.170]  So this is a video by Craig Gentry, who was the first to develop a technique called fully homomorphic encryption.
[05:12.830 --> 05:20.030]  And I just want to play this video, too, where he describes in his own words what homomorphic encryption is and what it lets you do.
[05:31.060 --> 05:36.440]  Okay, I'm not sure we have audio here, so let me just explain what it is that he says in the video.
[05:37.180 --> 05:47.620]  So basically, he describes homomorphic encryption using this metaphor of like a glove box, like an opaque glove box, where it lets you manipulate...
[05:47.620 --> 05:55.260]  homomorphic encryption lets you manipulate what's inside the glove box without actually being able to see what's inside the glove box.
[05:55.380 --> 05:59.940]  So you can manipulate but not see what's inside unless you have the key, in which case you can open the glove box.
[06:04.100 --> 06:12.080]  So, yeah, so as I had said before, homomorphic encryption, it lets one add or multiply numbers directly.
[06:13.400 --> 06:15.800]  So homo means same, of course.
[06:16.020 --> 06:18.560]  Morphic is structure, so same structure.
[06:18.900 --> 06:29.020]  The reason why it's called homomorphic encryption is because these types of encryption schemes, they let you basically do additions and multiplication operations.
[06:30.540 --> 06:36.960]  on encrypted data without having to know what it is that you're adding or multiplying.
[06:38.060 --> 06:41.420]  And this concept was actually introduced several decades ago.
[06:41.780 --> 06:47.200]  It was introduced in a paper by Rivest, Edelman, and Dertuzos.
[06:49.900 --> 06:51.820]  Yes, so it was introduced in this paper.
[06:51.820 --> 06:54.680]  It was on data banks and privacy homomorphisms.
[06:56.080 --> 07:08.660]  And the paper describes, like, a loan company and how it is that they have, like, say you have a commercial time-sharing service and how it is that you could use homomorphic encryption in that kind of context.
[07:09.940 --> 07:16.020]  I thought that that actually looked a lot like what a cloud service provider might do today and what it is that they do.
[07:16.020 --> 07:19.520]  That's one potential application of homomorphic encryption.
[07:21.020 --> 07:35.560]  So, if you look at the Wikipedia page, they'll break down the different types of homomorphic encryption from weakest to strongest in terms of partially homomorphic encryption somewhat and then fully homomorphic encryption, where fully homomorphic encryption was,
[07:35.720 --> 07:40.160]  I guess, one of the major breakthroughs about 15 years ago now.
[07:40.160 --> 07:45.140]  So, I'm going to start with partially homomorphic encryption and just briefly talk about that concept.
[07:47.400 --> 07:52.000]  So, RSA is an example, actually, of partially homomorphic encryption.
[07:53.540 --> 07:59.320]  And textbook RSA lets you multiply ciphertext directly.
[08:00.120 --> 08:08.760]  The example that I showed before was adding them, but this one only lets you multiply encrypted numbers directly but not add them.
[08:08.760 --> 08:14.540]  And, for this reason, it's called partially homomorphic encryption, because you can only do one operation but not the other.
[08:15.220 --> 08:22.780]  This was published in 1978, soon after the paper that introduced homomorphic encryption was introduced.
[08:25.160 --> 08:30.260]  Another example of a partially homomorphic encryption scheme is called PILIA.
[08:30.500 --> 08:33.020]  It was published in 1999.
[08:33.020 --> 08:37.600]  It's also an example of a partially homomorphic encryption scheme.
[08:38.170 --> 08:43.100]  This one, in contrast, lets you add encrypted numbers, but it does not let you multiply them.
[08:43.810 --> 08:56.200]  So having these two examples of partially homomorphic encryption schemes, you know, it's a natural question to wonder whether there are encryption schemes that let you do both additions and multiplications.
[08:57.320 --> 09:08.120]  So there are what's called somewhat homomorphic encryption schemes, which do, in fact, you both add and multiply numbers or ciphertext directly.
[09:08.800 --> 09:15.360]  The reason that it's called somewhat is because there's still limitations on what it is that you can add and multiply.
[09:15.980 --> 09:32.620]  Namely, a lot of these encryption schemes, if you do too many multiplication operations, you build up... since multiplications tend to be more complex than addition operations, you build up what's called or, you know, what's described as noise, like in the literature.
[09:34.000 --> 09:40.060]  And the more multiplication operations you do, the more noise you accumulate in your ciphertext.
[09:40.740 --> 09:46.240]  Eventually so much so that the ciphertext becomes useless and you can't really decrypt it because there's just too much noise.
[09:47.100 --> 09:53.280]  And so somewhat homomorphic encryption schemes, they do let you add or multiply numbers directly with limits.
[09:53.520 --> 10:01.200]  So of, course the next question is, are there encryption schemes that let you do arbitrary addition and multiplication operations?
[10:01.200 --> 10:04.380]  And yes, the answer to that question is, in fact, yes.
[10:06.200 --> 10:08.400]  That's what's called fully homomorphic encryption.
[10:09.740 --> 10:16.020]  And fully homomorphic encryption is This is an encryption scheme that lets you both add and multiply numbers directly.
[10:17.380 --> 10:27.700]  This work was the result of Craig Gentry's thesis, which was... PhD thesis, which was published in 2009.
[10:29.060 --> 10:35.700]  And has been... and is the reason why there's been a lot of interest since 2009 in this field.
[10:35.700 --> 10:42.500]  So yeah, there are no limits on adding and multiplying these ciphertext numbers.
[10:43.800 --> 10:49.480]  And the key novelty that he introduced in his dissertation, it was called bootstrapping.
[10:49.960 --> 10:56.820]  This is the key technique that he introduced in his dissertation and called a fully homomorphic encryption scheme.
[10:57.340 --> 11:03.500]  And this more or less showed that fully homomorphic encryption is even possible to begin with.
[11:05.020 --> 11:10.780]  Prior to that, it was an open question that had been, you know, whose answer had been unknown for a long time.
[11:13.540 --> 11:27.460]  So bootstrapping, what it lets you do is it lets you take this ciphertext after what you've possibly accumulated noise through doing lots of multiplication operations.
[11:27.880 --> 11:29.600]  And it lets you reset that noise.
[11:30.420 --> 11:39.560]  And so having this bootstrapping operation enables you to multiply numbers arbitrarily.
[11:39.740 --> 11:50.900]  Because if you say accumulate too much noise and you have all these multiply operations, then you can just apply this bootstrapping operation that Craig Gentry had proved exists in his dissertation.
[11:50.900 --> 11:58.520]  And so it's this technique that he introduced in his dissertation that proved fully homomorphic encryption is even possible.
[11:59.560 --> 12:07.880]  And so just to, you know, really hammer home with a point, you can start with a somewhat homomorphic encryption scheme like he did in his thesis.
[12:08.060 --> 12:14.360]  You can add this bootstrapping operation and you get a fully homomorphic encryption scheme.
[12:14.360 --> 12:36.080]  And since he published his dissertation, I actually have this screenshot of, you know, me looking up the search term homomorphic encryption on Google Trends to really just illustrate how much interest there has been and research activity there has been since he published his dissertation.
[12:36.080 --> 12:45.600]  This is due to people, researchers, once realizing that fully homomorphic encryption is possible.
[12:46.540 --> 12:50.940]  They wanted to figure out how it is that they can make it better and more usable.
[12:52.000 --> 13:13.280]  So next, I'm going to be talking about a little bit about arithmetic circuits, which is a concept that lets you sort of reason about how to apply homomorphic encryption and think about addition and multiplication operations in a more complex, for more complex applications beyond just, you know, adding two numbers.
[13:16.720 --> 13:22.780]  So homomorphic encryption, as I've said a few times now, it lets you add or multiply numbers, encrypted numbers directly.
[13:23.620 --> 13:32.660]  We can think of these things as building blocks to build up more complex algorithms rather than just doing, you know, two plus three or two times three.
[13:34.260 --> 13:40.180]  And if you're familiar with, you know, Boolean circuits, arithmetic circuits are, you know, an analogous concept.
[13:40.180 --> 13:51.620]  With Boolean circuits, you have, you know, bits and you have ANDs and OR gates, and you combine bits by sending them through ANDs and OR gates.
[13:51.800 --> 13:59.300]  And you can build up more complex circuits by combining and wiring up Boolean gates to get Boolean circuits.
[14:00.160 --> 14:04.360]  Similarly, with arithmetic circuits, you have numbers and you want to combine them in some way.
[14:04.620 --> 14:06.980]  Our operations are addition and multiplication.
[14:06.980 --> 14:14.640]  And visually, you can illustrate your, you know, Boolean, your additions and multiplications as gates.
[14:15.060 --> 14:21.360]  So, you know, you'll have, like, a plus operation or multiplication operation like this.
[14:21.500 --> 14:24.320]  And you feed two and three to your gate, you get AND gate.
[14:24.580 --> 14:24.740]  Sorry.
[14:25.320 --> 14:27.000]  Plus gate, you get a five.
[14:28.000 --> 14:31.740]  You feed two and three through your multiplication gate, you get a six.
[14:33.420 --> 14:37.900]  And, of course, as with Boolean circuits, you can make more complex arithmetic circuits.
[14:38.200 --> 14:52.140]  So, in this example, we're feeding in two and three as before through our add gate and our multiply gate, and then feeding the results of those two circuits into another gate, multiplication gate.
[14:52.140 --> 14:54.980]  And we see that the, you know, number that we get back is 30.
[14:57.260 --> 15:07.980]  And you can combine gates in more complex ways to be able to have, you know, more complex algorithms or operations.
[15:08.720 --> 15:15.980]  And more, you probably want a deeper, wider circuit, the more complex thing it is that you want to compute.
[15:18.120 --> 15:29.280]  And next, I'll be talking about the, some of the implementations out there that exist that let you use and apply homomorphic encryption.
[15:30.820 --> 15:46.480]  So, as I said previously, once Craig Gentry proved that homomorphic encryption is possible in his thesis, proving this 30-year-old hypothesis that homomorphic encryption is even possible.
[15:47.080 --> 15:54.340]  The next question is, how is it that we can make this thing fast?
[15:55.960 --> 15:56.520]  Because...
[15:59.520 --> 16:08.380]  It was 100 trillion times slower than just doing, you know, normal arithmetic on, you know, regular numbers, unencrypted computation.
[16:09.760 --> 16:23.340]  And, you know, in most applications, even though this is a nice theoretical result to have, it's not a very useful one because 100 trillion times factor, like an order of magnitude, is just prohibitive.
[16:24.360 --> 16:41.320]  So, yeah, after this dissertation was approved, the reason that there was a lot of research activity afterwards is because there was a lot of effort in, towards making homomorphic encryption and its applications fast, so that it can actually be usable,
[16:41.620 --> 16:45.960]  for example, applications that I had, you know, already illustrated.
[16:45.960 --> 16:48.260]  And so...
[16:50.980 --> 16:56.980]  These are a few different open-source libraries out there that actually implement homomorphic encryption.
[16:57.800 --> 17:05.460]  HELib is by IBM, and I believe was worked on when Craig Gentry still worked at IBM.
[17:06.000 --> 17:08.680]  There's also SEAL by Microsoft.
[17:10.360 --> 17:15.720]  OpenFHE is actually, I think, distinct among these ones because they're kind of a mix of...
[17:15.720 --> 17:18.320]  They have a kind of mix of open-source contributors.
[17:18.760 --> 17:22.260]  One of the main contributors is Duality Labs, which is some startup.
[17:24.340 --> 17:29.480]  And there's TFHE by ZAMA AI, which I believe is another, I think, a European startup.
[17:30.400 --> 17:33.880]  And I think that implementation is in Rust.
[17:34.080 --> 17:35.740]  The others, I think, are in C or C++.
[17:36.340 --> 17:39.540]  There are definitely implementations out here that I've not captured on this slide.
[17:39.680 --> 17:47.000]  There's a lot of, you know, open-source projects that try and implement homomorphic encryption efficiently right now.
[17:47.000 --> 17:48.620]  And a lot of this, of course, is on GitHub.
[17:49.040 --> 17:56.860]  You can, you know, Google search, look up the implementations, play with them yourselves, try and contribute if that's, you know, something that you do.
[17:58.520 --> 18:11.080]  And so one of the key concepts in making anything fast, I guess, at the hardware level is this concept of, you know, SIMD, you have single instruction.
[18:12.920 --> 18:29.020]  It's this idea of single instruction, multiple data, where you want to apply single instruction to, say, a vector or multiple elements or numbers all at once rather than individually applying this operation one at a time to each element.
[18:29.900 --> 18:40.540]  Since here we have, you know, addition and multiplication operations, and we want to apply these operations typically on vectors of data, SIMD is a natural way.
[18:40.540 --> 18:50.960]  Of accelerating this type of computation since we typically want to combine vectors of numbers rather than just single numbers at a time.
[18:51.260 --> 19:06.120]  This concept is, you know, it's available at the hardware level typically and then exposed via hardware level instructions and are supported by, you know, most modern CPUs these days.
[19:07.960 --> 19:15.500]  And so FHE libraries today leverage this type of computation to really make it fast.
[19:17.220 --> 19:26.420]  Just, you know, to illustrate the concept of SIMD, we have, say we have vectors, a vector of ones and twos and a vector of threes and fours.
[19:42.020 --> 19:50.720]  And you multiply them with a single instruction and you'll get back a vector of threes and eights.
[19:51.660 --> 20:06.900]  And one other useful operation that you wouldn't think about if you have single elements that you want to combine but have vectors of elements instead is the rotation operation that is made available through a lot of these libraries.
[20:08.360 --> 20:17.420]  So this operation also ends up being useful and what it lets you do is take a vector of ones and twos and rotate that vector.
[20:17.600 --> 20:22.860]  So in this case, you know, you're starting with a vector of ones and twos and you have your twos on the side.
[20:22.860 --> 20:26.660]  If you rotate, cyclically shift the whole thing by two.
[20:28.220 --> 20:35.760]  So the literature that applies homomorphic encryption, you'll call these encrypted vectors, ciphertext vectors.
[20:36.300 --> 20:50.100]  And when building more complex things, more complex operations on top of homomorphic encryption, these are really the fundamental building block that we think of when it is that we want to do more complex things on encrypted data.
[20:50.100 --> 21:02.280]  And it is what libraries like OpenFHE provide you with when working with homomorphically encrypted data.
[21:03.500 --> 21:18.580]  And so I took a screenshot of an example that OpenFHE has on their documentation page, just to show you how relatively simple it is to work with homomorphic encryption libraries.
[21:18.580 --> 21:33.280]  So in this example, you can instantiate a vector of doubles, two vectors of doubles, and then you can instantiate an object that lets you encrypt these vectors.
[21:33.640 --> 21:42.760]  And once they're encrypted, it's a matter of just calling, you know, for addition, it's cc.evaladd of your two encrypted vectors.
[21:42.760 --> 21:46.000]  And then for multiplication, it's just eval mult.
[21:46.240 --> 21:57.540]  There are, of course, other useful functions in these libraries, like eval sub, which is just a, you know, combination of addition and multiplication by negative one.
[21:58.560 --> 22:14.100]  You can also take a vector and multiply it by an unencrypted number, which ends up being a useful operation when you want to build applications with homomorphic encryption.
[22:14.100 --> 22:27.040]  So next, I'll be talking about a hypothetical application of homomorphic encryption, provided that this work can be made faster.
[22:28.700 --> 22:36.420]  So what I'll be talking about is being able to use homomorphic encryption to privately detect strokes that are in the brain.
[22:37.920 --> 22:52.680]  So I'll be talking about another paper that was published in 2018, which was on detecting these strokes, these types of strokes called emergent large vessel occlusions or elbows.
[22:53.380 --> 22:56.540]  It's a type of sudden blockage in the brain.
[22:56.540 --> 22:59.640]  It's pretty severe and requires immediate medical attention.
[23:02.160 --> 23:16.940]  As with this or with, you know, other strokes, the patient will typically receive a CT scan, and then some sort of expert will take the result of that scan and try and analyze that scan to determine what exactly it is that is wrong with that patient.
[23:18.260 --> 23:44.720]  And so in 2018, researchers at Brown, in collaboration with Rhode Island Hospital, who have some of this data, they applied image processing techniques in order to determine the presence of these types of... type of blockages in the brain in order to automatically determine whether these types of
[23:44.720 --> 23:46.840]  blockages were present.
[23:47.420 --> 23:56.220]  And, you know, this is six years... this work is six years old now, and so the state of the art has probably gotten better since then.
[23:56.220 --> 24:06.240]  But in this paper, they were able to automatically predict blockages in the brain with an 86.3% accuracy.
[24:07.000 --> 24:28.740]  But the point that I really want to highlight here is that the same techniques that they used in this paper can actually be applied homomorphically in order to privately determine whether there exists a blockage in the brain just by looking at encrypted x-ray images rather than looking at the x-ray images directly.
[24:30.360 --> 24:43.680]  So just going back to this paradigm where we had Alice and Bob, the hypothetical application that we have here is you have a patient, and they're wondering about whether it is that they have a stroke in the brain.
[24:44.340 --> 24:50.840]  And they have an x-ray image, CT scans that they received from, like, a radiologist.
[24:51.100 --> 24:54.760]  And they want to provide these scans to a third-party.
[24:54.760 --> 24:56.140]  In this case, it's a hospital.
[24:56.140 --> 25:07.620]  And say they have the compute capacity and the algorithms to determine whether it is that there is a blockage in the brain that's present.
[25:07.960 --> 25:20.080]  So homomorphic encryption applied correctly will let you determine whether there is a blockage in the brain privately.
[25:20.980 --> 25:40.440]  So the algorithm, the technique that was used in the 2018 paper that was used to automatically determine whether there are blockages in the brain, it's a version of convolutional neural networks.
[25:40.800 --> 25:47.140]  It's ResNet 50 for anyone that's familiar with, you know, neural networks or machine learning.
[25:48.260 --> 25:49.820]  These things are built up.
[25:50.340 --> 25:52.500]  This is a type of convolutional neural network.
[25:52.760 --> 25:56.660]  And these things are built up from more primitive operations called convolutions.
[25:57.640 --> 25:59.800]  It's a common technique in image processing.
[26:00.000 --> 26:00.880]  It's been around for decades.
[26:01.200 --> 26:11.820]  And it lets you basically extract useful features from images in order to do more complex processing, I guess, downstream.
[26:12.700 --> 26:32.160]  So just to illustrate briefly what a convolution, like a single convolution operation actually does, if you imagine your input image that you want to process as just like a matrix of, say, pixels, and you want to extract some sort of feature from that,
[26:32.400 --> 26:56.840]  the way you do it is by taking the smaller matrix highlighted in red here called a kernel and basically sliding that kernel over your pixels and applying addition and multiplication operations to more or less summarize what is in that small red region and get a sort of smaller shrunken down version
[26:56.840 --> 27:04.000]  of the original image that is, that captures some sort of features, some sort of desirable features from the original image.
[27:04.000 --> 27:12.120]  And to make this more concrete, one thing that you can do with convolutions is edge detection.
[27:12.540 --> 27:16.040]  So I think this is actually a common picture that's used to...
[27:17.120 --> 27:23.060]  in the computer vision community to illustrate the concept of edge detection.
[27:23.240 --> 27:42.460]  But what it lets you do is it lets you take an image and with the right kernel, that little red matrix that I showed you in the previous slide, if you slide that kernel over the image, you're going to get back a picture of the edges that were like picked out from the original picture.
[27:42.740 --> 27:56.280]  So here you see, you know, a man with a tripod and the output of edge detection is just, you know, his background and the background of the camera and some of the stuff that's in the background itself.
[27:58.100 --> 28:16.140]  And so the algorithm, the unencrypted algorithm, that was used to detect blockages in the brain, it consists of these building blocks of convolutions that I just showed you in the previous slide.
[28:16.380 --> 28:24.020]  So they apply lots and lots of these operations together with some other operations that are common in neural networks.
[28:24.020 --> 28:29.940]  And the key idea here is that each of these individual operations can be done homomorphically.
[28:30.100 --> 28:45.520]  So we can take this ResNet 50, this neural network, this complex image processing algorithm, and turn it into one giant arithmetic circuit in order to think or reason about that algorithm as an arithmetic circuit.
[28:47.000 --> 29:09.040]  And so the, the, one of the difficulties that we encounter when trying to do image processing on homomorphically encrypted images, in this case, is that images are, you know, three-dimensional things.
[29:09.300 --> 29:12.560]  They can be two or three-dimensional depending on what images that you have.
[29:12.560 --> 29:20.180]  But since images can have multiple channels, typically red, green, and blue, they can be three-dimensional as well.
[29:20.460 --> 29:28.200]  However, I did tell you that fully homomorphic encryption libraries, they typically operate on vectors of data.
[29:28.400 --> 29:44.900]  So the question becomes, if you want to do image processing on homomorphically encrypted data, what's the right way to like structure your images within these, you know, these ciphertexts, vectors of ciphertexts that you have to work with since these things are one-dimensional.
[29:46.900 --> 30:12.540]  So, the key idea which, that me and my collaborators at APL developed, is an efficient technique to be able to apply these convolution operations on encrypted images, provided that they're formatted correctly within these ciphertexts.
[30:12.700 --> 30:32.060]  The idea is that you just take your image, which has multiple channels, you slice it up into these ciphertexts of a very specifically chosen size, and then you basically pack the slices that you have into just a bunch of these ciphertexts vectors.
[30:32.340 --> 30:50.660]  And then you think of your image as, you know, ciphertexts, the set of ciphertexts vectors, rather than just the original image to begin with, and then you build your operation, you build your arithmetic circuit by working with these ciphertexts vectors.
[30:50.660 --> 31:04.500]  And so, convolution can be implemented on encrypted images that are represented using these sets of encrypted vectors.
[31:05.120 --> 31:09.600]  This is what we did in the paper.
[31:14.570 --> 31:23.890]  And once you have these convolution operations, you can build up the rest of that ResNet 50 image processing algorithm.
[31:24.790 --> 31:32.410]  So, you might ask, why is it that we can't just take a single image and just pack it into one giant, you know, long vector?
[31:32.670 --> 31:49.470]  The problem with that currently is that the libraries that implement homomorphic encryption, they don't scale linearly as you increase the size of the vector that you work with.
[31:49.710 --> 31:56.470]  So, for performance reasons, you do actually want to slice up this image, the images that you have, not just, you know, stuff it into one giant vector.
[31:58.170 --> 32:03.570]  On the flip side of that is, why don't you just, like, you know, work with individual elements at a time?
[32:04.490 --> 32:16.850]  There are, again, performance reasons for that as well, because you want to, one, leverage the performance that vector operations provide you with, but it also compromises security.
[32:16.870 --> 32:30.950]  If you look at how these algorithms work, they actually, if you, if your ciphertexts are too small, then you're, you're, you're losing the security that fully homomorphic encryption gives you.
[32:31.390 --> 32:33.070]  And so, there's a sweet spot, right?
[32:33.090 --> 32:35.570]  You don't want your ciphertext to be too large or too small.
[32:35.570 --> 32:41.010]  And that's, you can look for more details about that in the paper that we have.
[32:42.050 --> 32:55.390]  So, yeah, the idea is that you can take ResNet-50 and you can turn it into, like, a homomorphic, like, an HE-friendly version of the original algorithm.
[32:55.550 --> 33:11.930]  And then you can take that FHE version of the algorithm and apply that instead to encrypted images rather than applying the original algorithm to unencrypted images.
[33:13.790 --> 33:25.910]  So, the 2018 paper, which applied ResNet-50 to X-ray images, that was a different dataset, but it was the same technique that we used in our paper.
[33:26.190 --> 33:40.470]  We used what's called ImageNet, which is a typical dataset that's used in the computer vision community to evaluate convolutional neural networks, generally speaking.
[33:41.950 --> 33:49.710]  But it's just a matter of just changing the dataset in order to apply it to a different application, in this case.
[33:50.510 --> 34:01.070]  So, just to illustrate very clearly how this would work is, you would take ResNet-50, which is the original X-ray image processing algorithm.
[34:01.070 --> 34:07.730]  You would turn it into a giant arithmetic circuit, which is what we did in our paper.
[34:08.590 --> 34:18.730]  And once you have the circuit, you can take your encrypted image, you know, you being the person who has, like, a CT scan.
[34:20.370 --> 34:25.410]  And once you encrypt that image, you can send it through this arithmetic circuit.
[34:25.410 --> 34:31.590]  And the output will be, there is a blockage in the brain, yes or no, there's not a blockage in the brain.
[34:31.910 --> 34:43.490]  And the result can be sent back to the user without the third-party ever having directly looked at the image.
[34:45.190 --> 34:53.730]  So, I'll be ending with just a little bit about some of the limitations and future work that remains in this field.
[34:55.450 --> 35:01.590]  So, if it hasn't been clear, homomorphic encryption, it's still very expensive to apply in practice.
[35:01.590 --> 35:08.210]  This is still a relatively new field that, for which the state of the art needs to be advanced.
[35:08.810 --> 35:20.770]  It will always be a trade-off between the privacy that you get and the computational resources that are required in order to give you that privacy.
[35:21.470 --> 35:29.170]  The research that needs to be done is minimizing the cost required in order to give you the privacy that homomorphic encryption gives you.
[35:30.470 --> 35:47.290]  If you look at the specific case that I had discussed, encrypted ResNet results from our paper, they're 50,000 times slower, as measured by time, to run a single evaluation compared with unencrypted ResNet 50.
[35:47.290 --> 35:55.270]  I think if you look at the raw numbers, it's something like some number of milliseconds versus minutes, which could be acceptable depending on your use case.
[35:55.410 --> 35:59.370]  It's not the original, you know, 100 trillion times slower order of magnitude.
[36:00.050 --> 36:09.810]  50,000 could be reasonable depending on what it is that you're applying, you know, what your application is.
[36:10.130 --> 36:16.310]  This is the state of the art as 2023, so it's possible that it may have gone a little bit faster since then.
[36:16.310 --> 36:23.030]  One thing that I didn't really talk about or touch upon at all yet is there's also a space trade-off.
[36:24.010 --> 36:27.310]  Homomorphic encryption, it requires a lot of RAM.
[36:27.790 --> 36:34.930]  The results that we obtained in our paper, they required an 800-gigabyte cluster...
[36:35.750 --> 36:41.270]  sorry, cluster with 800 gigabytes of RAM in order to run encrypted ResNet 50.
[36:42.050 --> 36:53.290]  If you look at the original ResNet 50, in an unencrypted setting, it only requires 50 megabytes of RAM, which can fit on, you know, commodity hardware.
[36:53.630 --> 36:56.750]  You don't need a whole cluster to throw at this problem.
[36:57.370 --> 37:11.450]  And so if you look at the difference here, it's, you know, 16,000 times more RAM required to process encrypted data in this case, which again could be acceptable depending on your use case and what trade-offs you have available to you.
[37:13.070 --> 37:18.210]  And so, yeah, there's a lot of work to be done in this field still.
[37:19.690 --> 37:29.870]  Some of my opinions on what needs to be done are the... there's still work to be done on improving the usability of the implementations that are out there.
[37:29.870 --> 37:34.530]  A lot of these are written in systems languages like C++ or Rust.
[37:34.730 --> 37:39.010]  And it makes sense because, you know, you want the performance that a systems language will give you.
[37:40.270 --> 37:45.590]  But they could also have, you know, higher level language interfaces like Python.
[37:46.590 --> 37:54.570]  And some of the code that we have written for the work that we did at APL, we used Python...
[37:54.570 --> 38:04.370]  We created Python bindings for OpenFHE to make it easier to write neural networks via Python rather than just pure C++.
[38:05.270 --> 38:19.090]  There are hardware accelerator designs and papers about those to improve the... or accelerate the processing of the specific operations that are used in homomorphic encryption.
[38:20.430 --> 38:31.130]  I'm not aware of how... what the state of is for creating... like manufacturing hardware based on those designs.
[38:31.130 --> 38:34.950]  But it is also one way of, you know, making this stuff faster.
[38:35.290 --> 38:41.510]  You can also use GPUs to accelerate this stuff since, you know, a lot of the operations that I talked about...
[38:41.510 --> 38:43.070]  They're vector operations.
[38:43.290 --> 38:47.850]  So it's just a matter of turning some of the C++ code or Rust code...
[38:48.690 --> 38:53.130]  That's already been written into CUDA kernels and then invoking those kernels instead.
[38:53.990 --> 39:00.150]  And there's work that can be done in automating the process of taking an algorithm...
[39:00.910 --> 39:10.650]  that operates on unencrypted data and turning it into the corresponding algorithm that is FHE-friendly or will work on encrypted data.
[39:11.010 --> 39:16.010]  In our paper, we, of course, we did this all manually.
[39:16.010 --> 39:27.490]  But you can think of, you know, like some sort of compiler that would take one algorithm and turn it into an encrypted one.
[39:27.490 --> 39:35.570]  And I think... so Google has this project, which I think has been worked on recently, called HEIR.
[39:35.810 --> 39:37.230]  HE stands for homomorphic encryption.
[39:37.410 --> 39:39.530]  IR stands for intermediate representation.
[39:39.550 --> 39:45.790]  And I think that effort is actually focused on automating some of this stuff.
[39:46.590 --> 39:47.830]  So I'll leave it there.
[39:48.150 --> 39:49.350]  And I'll take any questions.
[39:52.550 --> 39:57.710]  Yeah, so the question is what it is that we mean by noise in the context of homomorphic encryption.
[39:57.710 --> 40:13.770]  So I believe noise is critical to how the security that homomorphic encryption gives you.
[40:14.010 --> 40:20.970]  And that noise has to be bounded in some sort of sense as you apply more operations.
[40:22.730 --> 40:28.730]  And I think if you, again, apply too many multiplication operations, then you exceed that bound.
[40:29.370 --> 40:31.650]  And then you lose whatever guarantees that you had.
[40:32.830 --> 40:35.610]  But I should... we can talk offline.
[40:35.750 --> 40:39.530]  I can point you to papers about that specifically.
[40:39.550 --> 40:40.510]  If you'd like to know more.
[40:44.700 --> 40:54.780]  Yeah, so the question is why is it that the algorithms, the implementations, don't let you operate on matrices directly rather than vectors?
[40:55.400 --> 40:57.780]  I think it's just because the field is very new.
[40:57.980 --> 40:59.980]  The stuff still has to be built.
[41:00.180 --> 41:07.820]  There's probably more demand in general to work with vectors rather than matrices.
[41:08.500 --> 41:14.840]  The example that I talked about is, you know, using convolutions.
[41:15.160 --> 41:18.240]  And so I don't... I think it's just time.
[41:18.460 --> 41:19.100]  I think that's it.
[41:23.770 --> 41:32.710]  So that is a separate paper that was done by researchers at Brown that I wasn't involved with.
[41:32.710 --> 41:41.910]  And the reason that I picked out that example is because they used the same algorithm that we used in our paper at APL.
[41:42.350 --> 41:44.070]  So that was just a hypothetical case.
[41:44.190 --> 41:45.130]  That was all unencrypted.
[41:46.450 --> 41:57.770]  But if you combine what we developed in our paper with their application, then you could, you know, do this privacy-preserving detection of blockages.
[41:58.010 --> 41:58.890]  Yeah, the result too.
[41:59.110 --> 41:59.190]  Yeah.
[42:04.350 --> 42:06.410]  The images in our experiments.
[42:06.750 --> 42:11.190]  So the question was, how big were the images that we used in our experiments?
[42:11.430 --> 42:18.130]  And those were, I believe, the size of ImageNet pictures, which I think...
[42:18.130 --> 42:20.410]  Yeah, I think it was like 256 by 256.
[42:20.930 --> 42:22.190]  But I'll have to double check that.
[42:27.510 --> 42:30.290]  That's outside my domain of knowledge.
[42:30.410 --> 42:43.470]  But if you want my opinion, I think that we need this type of legislation in order to, you know, enforce third parties to use this sort of stuff because the technology can be available.
[42:43.470 --> 42:52.210]  But, you know, someone has to make companies, authority, or I'm sorry, organizations use this.
[42:53.570 --> 42:54.090]  So...
[42:57.720 --> 43:03.400]  You're talking about like the... So the question is about Intel SGX and how it relates to that.
[43:04.660 --> 43:08.380]  I don't think I know enough about SGX to comment.
[43:10.400 --> 43:16.660]  As far as I know, it's part of the CPU that has some sort of secure enclave in it.
[43:16.660 --> 43:21.980]  But I don't... I think it's an orthogonal technology to homomorphic encryption.
[43:23.040 --> 43:26.580]  But I'm... It'd be interesting to combine the two technologies.
[43:27.220 --> 43:27.300]  Yeah.
[43:32.610 --> 43:36.650]  So bootstrapping, it itself is actually... So the question is...
[43:37.470 --> 43:43.450]  The question is what exactly is involved with bootstrapping and how does it affect your ciphertext?
[43:43.450 --> 43:47.750]  It itself, if you actually look at it, is an arithmetic circuit itself.
[43:47.950 --> 43:53.450]  And it's just you evaluating an arithmetic circuit on the original ciphertext.
[43:57.290 --> 44:05.650]  I don't think you're actually changing the original size of the ciphertext.
[44:05.770 --> 44:09.810]  But I think I'll have to point you... If we can talk offline about that.
[44:13.270 --> 44:14.070]  Yeah.
[44:14.510 --> 44:18.690]  So the question is why is the... Why are the RAM requirements so large?
[44:18.910 --> 44:29.350]  I think this is... Relates to the security guarantees that you need to prevent, you know, a...
[44:31.630 --> 44:36.030]  Your adversary from breaking the encryption scheme.
[44:36.950 --> 44:44.690]  You would have to look at the abstract algebra that's involved in the actual implementations of the homomorphic encryption algorithms.
[44:46.330 --> 44:51.410]  I think these... End up being just like giant vectors...
[44:51.410 --> 45:00.210]  That are required in order to, you know, guarantee the security.
[45:06.680 --> 45:07.540]  Yeah.
[45:07.820 --> 45:09.460]  So I...
[45:09.460 --> 45:12.120]  These end up being...
[45:12.120 --> 45:16.450]  Your plaintext space and your ciphertext space, they are...
[45:16.450 --> 45:17.630]  Are...
[45:18.150 --> 45:19.450]  If I remember correctly...
[45:19.970 --> 45:21.570]  Certain types of quotient rings...
[45:24.410 --> 45:24.810]  That...
[45:25.390 --> 45:27.310]  You work with...
[45:27.310 --> 45:29.510]  Quotient rings of...
[45:29.510 --> 45:31.450]  I think it's...
[45:31.990 --> 45:33.450]  Like the ring of integers...
[45:33.450 --> 45:34.270]  Um...
[45:34.990 --> 45:35.690]  And...
[45:35.690 --> 45:37.890]  How it is that this works...
[45:37.890 --> 45:38.050]  I mean...
[45:38.050 --> 45:39.290]  There are proofs out there...
[45:39.290 --> 45:40.810]  There are papers out there...
[45:41.450 --> 45:42.090]  Um...
[45:42.090 --> 45:43.070]  That...
[45:43.070 --> 45:44.330]  Um...
[45:44.330 --> 45:45.310]  Let you...
[45:45.310 --> 45:46.130]  Show...
[45:46.130 --> 45:47.430]  Um...
[45:47.430 --> 45:48.070]  That...
[45:48.070 --> 45:49.910]  This stuff is secure...
[45:49.910 --> 45:51.710]  I think the...
[45:51.710 --> 45:52.450]  The...
[45:53.750 --> 45:54.450]  Um...
[45:55.110 --> 45:55.630]  The...
[45:55.630 --> 45:56.670]  Corresponding hard...
[45:56.670 --> 45:57.170]  It's...
[45:57.170 --> 45:58.190]  It's called ring learning with errors.
[45:58.370 --> 45:59.330]  If you want something to...
[45:59.330 --> 45:59.830]  To look up.
[46:00.210 --> 46:03.290]  That is a hard problem that is difficult to break...
[46:03.830 --> 46:04.350]  Um...
[46:04.350 --> 46:05.070]  When...
[46:05.070 --> 46:07.270]  You want to apply, you know, homomorphic encryption.
[46:08.410 --> 46:08.510]  Yeah.
[46:10.310 --> 46:10.830]  Um...
[46:10.830 --> 46:11.030]  Yes.
[46:14.730 --> 46:16.470]  I don't think it's a hardware thing.
[46:16.590 --> 46:18.810]  I think it's a purely a theoretical thing.
[46:19.150 --> 46:19.610]  So like...
[46:19.610 --> 46:22.150]  More RAM is not gonna, you know...
[46:22.150 --> 46:22.650]  Uh...
[46:22.650 --> 46:23.170]  Change the...
[46:23.170 --> 46:24.070]  Um...
[46:24.070 --> 46:25.070]  The security department there.
[46:25.070 --> 46:26.070]  Yep.
[46:27.010 --> 46:27.030]  Yep.
[46:28.490 --> 46:29.890]  Let's all thank Vikram.
[46:30.770 --> 46:31.260]  And...
[46:31.750 --> 46:32.310]  Wonderful presentation.
[46:32.650 --> 46:33.410]  Thank you.