[00:00.000 --> 00:01.600]  All right, hello, I'm Suha.
[00:01.760 --> 00:07.940]  I'm really excited to be at HOPE today, and I'm especially excited that I get to be presenting in Queens where I grew up.
[00:12.050 --> 00:14.870]  So today I want to talk to you all about ML security.
[00:15.110 --> 00:23.470]  Specifically, I want to talk about this new class of exploits I identified called incubated ML exploits that combine back doors and input handling bugs.
[00:23.710 --> 00:26.790]  So don't worry if you don't know much about ML or ML security.
[00:27.030 --> 00:29.310]  I'll explain all the important stuff as we go along.
[00:30.930 --> 00:32.410]  So who am I?
[00:32.550 --> 00:34.670]  Why am I even talking to you today?
[00:35.270 --> 00:39.030]  I'm an engineer at Trail of Bits, where I focus on AI ML security.
[00:39.370 --> 00:41.170]  I graduated from Georgia Tech.
[00:41.330 --> 00:46.010]  I've been working in ML security for a few years now, and I'm originally from Queens, as I said.
[00:46.510 --> 00:55.570]  Outside of work, I really like Brazilian Jiu Jitsu, trying new restaurants, making things, especially at Hack Manhattan, and an obscure card game called Q-Birds.
[00:57.770 --> 01:01.910]  All right, so it's... Y'all can hear me in the back, right?
[01:02.250 --> 01:02.690]  Okay.
[01:03.270 --> 01:12.210]  So it's becoming pretty clear that with ML and AI popping up everywhere, people are figuring out how to trick the system based on how these models work internally.
[01:12.210 --> 01:17.690]  So maybe you've seen someone use prompt injection to convince a chat bot to give them a refund.
[01:17.930 --> 01:25.210]  Or maybe you've seen this example up here of the story of protesters tricking self-driving cars using traffic cones.
[01:25.450 --> 01:31.590]  So notice the fact that that trick is rooted in an understanding of the training data for these models.
[01:33.010 --> 01:38.110]  So how can we actually construct our own useful exploits against ML systems?
[01:39.930 --> 01:42.530]  So let's play a game of pretend real quick.
[01:42.750 --> 01:47.410]  You're a college student, and you really, really want the prize money for a robotics team.
[01:47.530 --> 01:50.290]  So naturally, you decide to sabotage the other team.
[01:51.010 --> 01:52.930]  Side note, I don't recommend this.
[01:52.990 --> 01:54.270]  I don't condone this behavior.
[01:54.270 --> 01:55.430]  I've never done it myself.
[01:57.970 --> 02:02.770]  So the competition requires teams to build this tiny autonomous vehicle.
[02:03.010 --> 02:05.670]  And this vehicle has to stop at stop signs.
[02:05.710 --> 02:09.190]  And it also has to use a specific pre-trained model that they provide.
[02:09.590 --> 02:14.730]  So you find out that on the competition site, some of these stop signs have stickers on them.
[02:14.890 --> 02:18.710]  And you also find some flaws with how they've stored and distributed the model.
[02:19.050 --> 02:21.410]  Which, by the way, isn't out of the question.
[02:21.410 --> 02:28.350]  Machine learning model artifacts are often shared widely without any useful trust mechanisms.
[02:29.010 --> 02:37.010]  So you decide to go and grab that file and inject a model back door in it using a file format RCE of some kind.
[02:37.970 --> 02:45.910]  And then, on the day of the competition, you just sit back and you watch as your competitor's vehicle just plows through all the stock stickers.
[02:47.610 --> 02:52.830]  But you executed an incubated ML exploit, which is what my talk is all about.
[02:54.930 --> 02:59.290]  So obviously, the stakes of this story is just a lost competition.
[03:00.350 --> 03:08.310]  But the idea of attacking a real autonomous vehicle is a hallmark of model back door research, as you can see on the image on the left.
[03:08.770 --> 03:11.870]  So I'll let you use your imagination to raise the stakes.
[03:13.130 --> 03:19.630]  So in this talk, I'm first going to tell you about this framework I've been using to bridge the gap between model and system security.
[03:20.270 --> 03:24.970]  Because I also want to convince you that we can't continue to treat models as standalone objects.
[03:25.650 --> 03:31.170]  Next, I'll tell you about these input handling bugs I found in model serialization and connect them to back doors.
[03:31.530 --> 03:37.150]  And I'll be doing that by taking a page out of this subfield called LangSec, or Language Theoretic Security.
[03:37.150 --> 03:43.950]  So essentially, I'm going to be going through a bunch of examples of incubated ML exploits and use LangSec to organize them.
[03:45.770 --> 03:48.030]  But first up, I need to explain some stuff.
[03:48.250 --> 03:51.670]  What even is a model vulnerability or an ML back door?
[03:53.590 --> 04:02.590]  So, super briefly, you can think of ML models like these squishy, flexible sequences of linear algebra ops that are trained on tons and tons of data.
[04:03.230 --> 04:11.790]  And you might have heard the saying before that all models are wrong, but some are useful, which is just a way to say that these models aren't perfect.
[04:12.090 --> 04:17.890]  There are many different ways that models can mess up or get tripped up by something that may be unexpected to us.
[04:17.990 --> 04:20.590]  And that's the basis of such model vulnerabilities.
[04:20.990 --> 04:29.250]  While some examples of model vulnerabilities include model inversion and membership inference, we're going to zoom in on one very specific type, backdoors.
[04:30.430 --> 04:32.710]  So, let's get a little precise about this.
[04:32.910 --> 04:40.970]  A backdoor attack allows a malicious actor to force an ML model to produce specific outputs given inputs with an attacker chosen trigger.
[04:41.590 --> 04:45.510]  So, think back to the example of the stop sign with the stickers on it.
[04:46.050 --> 04:52.070]  So, there's a couple of things that I think makes backdoors really, really interesting to study.
[04:52.070 --> 04:58.290]  The first is that model backdoors are primitives for other types of model vulnerabilities.
[04:58.570 --> 05:02.990]  So, you can use a backdoor to conduct membership inference and some of the other kinds of attacks.
[05:03.270 --> 05:07.350]  You can also go into the model and identify pre-existing backdoors.
[05:07.570 --> 05:10.150]  And that line of research is called natural backdoors.
[05:10.450 --> 05:16.590]  There's also some pretty strong evidence that suggests that model backdoors are an inherent threat to machine learning models.
[05:18.570 --> 05:29.670]  So, while there's some awesome research on attacking ML models, they can actually be a bit hard to exploit in the real world, with some exceptions, of course.
[05:29.950 --> 05:32.270]  And there's multiple reasons for that.
[05:32.270 --> 05:38.950]  But one thing that really, really strikes out to me is that there's this huge gap between research and the real world.
[05:40.290 --> 05:46.750]  So, for the most part, many attacks and attack frameworks and tools restrict their analysis to this formulation.
[05:47.370 --> 05:48.890]  There is an ML model.
[05:49.070 --> 05:51.670]  It gets an input, produces an output.
[05:51.850 --> 05:56.490]  But that's not an accurate representation of what an ML system actually looks like.
[05:56.630 --> 05:58.790]  There's so much more going on in practice.
[06:00.090 --> 06:05.250]  So, here's a software architecture diagram for an ML system reviewed by Trillibits recently.
[06:05.250 --> 06:08.330]  And I've circled where the model actually is in this photo.
[06:08.770 --> 06:10.690]  So, you see what I mean here, right?
[06:10.930 --> 06:13.170]  We need to be looking at all of this holistically.
[06:13.550 --> 06:18.870]  There's this large and evolving landscape of tools being used for and in ML systems.
[06:18.870 --> 06:20.910]  And that brings me to my exploit framework.
[06:21.650 --> 06:23.090]  What does it mean hybrid?
[06:24.350 --> 06:25.550]  I'll get to that right now.
[06:28.230 --> 06:33.150]  So, the title of my talk obviously references an incubated ML exploit.
[06:33.150 --> 06:38.430]  But there's a larger category of exploits called hybrid ML exploits that I want to talk about first.
[06:39.790 --> 06:44.690]  Specifically, a hybrid ML exploit chains a system security issue with a model vulnerability.
[06:45.110 --> 06:49.270]  So, if you look at the diagram here, the arrow is actually bidirectional.
[06:49.630 --> 06:53.130]  A model vulnerability can expose a system security issue.
[06:53.390 --> 06:57.430]  Or, a system security issue could be used to exploit a model vulnerability.
[06:59.030 --> 07:01.130]  So, this part's really important here.
[07:01.330 --> 07:08.430]  The big issue, in my opinion, with a lot of ML security works is that model security and system security are treated separately.
[07:08.750 --> 07:14.670]  But what I need you to understand is that if you're only covering model security, you're missing a big piece.
[07:14.750 --> 07:18.770]  And if you're only covering system security, you're still missing a big piece.
[07:18.770 --> 07:22.110]  But these two aren't directly composable.
[07:22.250 --> 07:24.070]  They cannot just be tacked on with each other.
[07:24.210 --> 07:27.850]  Because then you'll be ignoring the potential for hybrid ML exploits.
[07:28.130 --> 07:30.130]  Your model is embedded in a system.
[07:30.310 --> 07:35.570]  And it's going to interact with all of the components of your system in new and exploitable ways.
[07:36.810 --> 07:41.810]  So, one thing you'll notice is that there's a lot of screenshots of paper titles on this slide.
[07:41.810 --> 07:48.050]  That's because there have been specific instances of hybrid ML exploits in the literature and in practice.
[07:48.550 --> 07:50.330]  They're just not called that explicitly.
[07:51.150 --> 07:54.150]  So, exploitable software gadgets have been used for backdoors.
[07:54.410 --> 08:00.250]  That summoning demons paper at the very top chained memory corruption vulnerabilities with model invasion.
[08:00.790 --> 08:09.870]  And the paper next to it, Learned System Security, includes this example of a poisoning attack being used to cause an exponential memory blowup in an index structure.
[08:10.770 --> 08:18.010]  So, the ML security literature frameworks tools, at least the ones that I'm familiar with, are exposed to just that though.
[08:18.410 --> 08:20.470]  Specific instances or implications.
[08:20.830 --> 08:24.890]  And what I'm trying to do is treat this explicitly and systematically.
[08:24.890 --> 08:26.410]  Which is why I made this framework.
[08:27.930 --> 08:32.010]  So, one kind of system security issue is an input handling bug.
[08:32.270 --> 08:35.230]  And one kind of model vulnerability is a backdoor.
[08:37.890 --> 08:41.550]  So, put that together and you get an incubated ML exploit.
[08:41.850 --> 08:43.990]  Which is a type of hybrid ML exploit.
[08:44.270 --> 08:47.710]  Where the attacker uses an input handling bug to inject a backdoor.
[08:48.310 --> 08:51.510]  So, I made this diagram to make this distinction a lot clearer.
[08:52.810 --> 08:54.050]  And here it is again.
[08:54.370 --> 08:56.590]  I'm going to leave the framework here for now.
[08:56.590 --> 09:02.850]  We did end up defining a more formal model of exploitation that included a schema for incubated ML exploits.
[09:03.250 --> 09:05.330]  But we will return to this notion later.
[09:06.750 --> 09:13.150]  So, to backdoor a pre-existing model, the attacker should be able to change the parameters or the architecture.
[09:13.930 --> 09:19.570]  And at the level of attraction we're dealing with, we can put aside input and component manipulation.
[09:20.690 --> 09:23.730]  But how this actually plays out can vary a lot.
[09:24.550 --> 09:29.410]  So, for example, sometimes an attacker has control over some element of the training process.
[09:29.970 --> 09:35.930]  And if they do, they can sneak in some manipulated data that will change the model parameters.
[09:36.150 --> 09:38.030]  And that's usually called data poisoning.
[09:38.830 --> 09:43.910]  Another example is if an attacker somehow gets access to the source code.
[09:44.050 --> 09:46.190]  And then they go in and they change the architecture.
[09:48.310 --> 09:54.970]  So, before we dive into exploit proof concepts, I'm going to explain a few things about input handling bugs.
[09:56.250 --> 09:58.970]  So, an ML model is stored as a file, right?
[09:59.270 --> 10:01.790]  And to process these models, you need parsers.
[10:01.990 --> 10:06.430]  And parsing these files into objects and back is serialization and deserialization.
[10:07.810 --> 10:10.610]  But wait, I'm going to quote Ange Albertini here.
[10:10.850 --> 10:13.310]  A file has no intrinsic meaning.
[10:13.310 --> 10:15.510]  It's the meaning of a file.
[10:15.630 --> 10:16.170]  It's type.
[10:16.330 --> 10:16.970]  It's validity.
[10:17.230 --> 10:20.430]  It's contents can be different for each parser or interpreter.
[10:20.930 --> 10:29.210]  So, this is the reason we can make really cool, potentially malicious file artifacts, like polyglots and ambiguous files, which I'll talk more about later.
[10:30.450 --> 10:36.130]  So, this project is focused very specifically on bugs that occur when you parse ML model files.
[10:36.470 --> 10:41.970]  There's also cool bugs in other parts of the pipeline, but I'm picking ML model files for a couple of reasons.
[10:41.970 --> 10:45.350]  The first is very obviously the most important.
[10:45.610 --> 10:46.530]  I think it's fun.
[10:47.250 --> 10:53.690]  But the security... more seriously, the security of ML file formats have become increasingly important.
[10:54.430 --> 10:58.030]  ML has a culture of sharing these artifacts without sufficient validation.
[10:58.710 --> 11:03.070]  Real malicious models have been found on the Hugging Face Hub, for example.
[11:03.070 --> 11:07.690]  And there's just an absolute ton of file formats being used in and for ML.
[11:07.930 --> 11:11.650]  So, I've tried to list and organize them in that repository in the middle there.
[11:11.810 --> 11:18.310]  But what's important for you to take away is that there's a large set of possibilities for exploits and just fun hacks with these formats.
[11:18.310 --> 11:21.710]  And there's a lot of great work already as shown on this slide.
[11:22.830 --> 11:30.690]  So, file format tricks are within the realm of LangSec, but LangSec as a field thinks about inputs as a general class.
[11:30.830 --> 11:32.430]  So, they think about this more abstractly.
[11:32.710 --> 11:36.370]  So, what it does, it applies formal language theory to security problems.
[11:36.370 --> 11:44.130]  It focuses on exploring input handling bugs, or as they say, parser problems, as this big root cause of many security issues.
[11:44.310 --> 11:45.450]  And it makes sense, right?
[11:45.550 --> 11:50.110]  A lot of impactful vulnerabilities like Heartbleed and Android Master Key have been parser bugs.
[11:50.890 --> 11:57.650]  So, I really like formal language theory, but this isn't theoretical computer science 101, unfortunately.
[11:57.970 --> 12:04.810]  So, just know that fundamentally what LangSec is saying is, hey, let's treat all inputs as a specific language.
[12:04.810 --> 12:09.690]  And then make our code just capable enough to understand that language properly.
[12:11.070 --> 12:16.070]  So, our work is centered around a very specific taxonomy of input handling bugs.
[12:16.390 --> 12:19.850]  These are all the bug classes that are listed in this taxonomy.
[12:20.050 --> 12:21.490]  So, there are eight different types.
[12:21.870 --> 12:25.990]  Now, one thing that's important to note, they're actually not completely distinct from each other.
[12:26.170 --> 12:29.110]  You pick the category by performing a root cause analysis.
[12:29.430 --> 12:32.910]  So, they're not completely separate.
[12:32.910 --> 12:41.390]  So, with the exception of one, I'm going to show you multiple examples of each in ML tools and use them to construct an ML backdoor.
[12:42.670 --> 13:01.130]  So, to repeat myself a bit, in order to prove that input handling bugs are a vector for backdoors, I'm going to identify and exploit ML model serialization issues across these different LangSec bug classes and then use that to construct ML backdoors.
[13:02.210 --> 13:05.650]  So, now we can actually dive into the most fun part, the exploits.
[13:05.890 --> 13:13.550]  So, quick note, these actually vary a bit in the fidelity of the exploit, but I hope you'll find them useful nonetheless.
[13:14.710 --> 13:19.910]  So, first, let me introduce some characters that play important roles in the ML ecosystem.
[13:21.650 --> 13:23.350]  First up, we have Alice.
[13:23.570 --> 13:24.910]  Alice distributes models.
[13:25.110 --> 13:26.570]  She takes open-source LLMs.
[13:26.750 --> 13:28.150]  She fine tunes them.
[13:28.290 --> 13:30.610]  Maybe applies a couple of other modifications.
[13:31.030 --> 13:35.510]  But the model she distributes is what everyone else in the story is going to be using.
[13:37.070 --> 13:42.330]  Bob is this frontline user who's taking Alice's models and directly using it.
[13:42.390 --> 13:44.930]  Maybe through some sort of chat interface, right?
[13:46.490 --> 13:48.010]  Next up, we have Dave.
[13:48.190 --> 13:53.030]  Dave's an engineer who's trying to take Alice's models and use them into actual products.
[13:53.690 --> 13:56.450]  And Frank is someone who uses Dave's products.
[13:56.650 --> 14:00.790]  Frank might not even know that there are ML models being used behind the scenes, right?
[14:02.670 --> 14:06.070]  Lastly, but not... Last but not least, we have Chuck.
[14:06.270 --> 14:07.170]  Chuck is the attacker.
[14:07.370 --> 14:11.990]  He wants to exploit vulnerabilities in these systems and disrupt everyone's work.
[14:11.990 --> 14:18.190]  So, our focus will mainly be here... mainly be on Bob and Dave and how they're impacted by the exploits.
[14:18.750 --> 14:26.870]  And I'll show some exploits involving the pickle, ONNX, TorchScript, PyTorch, and SafeTensors formats.
[14:27.870 --> 14:31.830]  So, the first category is called non-minimalist input handling code.
[14:32.190 --> 14:39.590]  So, it sounds a little fancy, but all it means is that the code that's being used to check and parse these inputs is too complex.
[14:39.590 --> 14:43.670]  So, an attacker can potentially grab the necessary gadgets for their exploits.
[14:45.550 --> 14:48.230]  So, this example is quite popular.
[14:48.730 --> 14:52.470]  So, pickling is a serialization method that allows you to save arbitrary objects.
[14:52.670 --> 14:55.170]  It's very, very common in the ML ecosystem.
[14:55.550 --> 15:03.250]  So, recently, my coworker, Boyan Milanov, led the development of Sleepy Pickle, which is an incubated ML exploit.
[15:03.250 --> 15:07.170]  And what it does is it chains a Pickle RCE with model backdoors.
[15:07.590 --> 15:12.070]  So, on the right, you can see an LLM that has been backdoored to Phish users.
[15:12.570 --> 15:19.070]  In his blog post, he also points to some examples of an LLM being used to spread misinformation and even steal user data.
[15:19.250 --> 15:22.950]  But what's really cool about this exploit is that it can happen on the fly.
[15:23.110 --> 15:28.650]  So, there's far more room and possibilities for an attacker here than just uploading a malicious model.
[15:29.810 --> 15:33.050]  So, what do I even mean by Pickle RCE, right?
[15:33.510 --> 15:40.410]  So, Python Pickles are compiled programs that run in this unique virtual machine called the Pickle Machine, or the PM for short.
[15:40.590 --> 15:49.130]  And what the PM does is that it takes the sequence of opcodes in a pickle file and constructs this arbitrarily complex Python object.
[15:49.370 --> 15:57.990]  But the thing about the PM is that it has two opcodes, global and reduce, that can execute arbitrary code outside of the PM.
[15:58.210 --> 16:03.010]  So, you can end up constructing malicious pickle data that'll execute arbitrary code.
[16:03.370 --> 16:12.250]  And the underlying reason, of course, is that the PM is just far, far more complex than something that's only parsing ML models should be.
[16:13.570 --> 16:19.530]  So, way, way back in 2021, we released something called Fickling.
[16:19.530 --> 16:22.650]  So, this project was led by Evan Soltanik.
[16:22.790 --> 16:27.590]  And to our knowledge, Fickling was the first pickle security tool tailored for ML use cases.
[16:28.890 --> 16:34.230]  So, Fickling is a decompiler, static analyzer, and bytecode rewriter for Python Pickle.
[16:34.430 --> 16:39.230]  So, what you can do with it is you can detect, analyze, or even create malicious pickle files.
[16:39.430 --> 16:46.430]  So, the reason it's safe to run on potentially malicious pickle files is because it has its own implementation of the PM.
[16:46.650 --> 16:49.430]  And it uses that to symbolically execute code.
[16:50.150 --> 16:58.630]  I also very recently added a PyTorch module to it, so you can statically analyze and inject code into PyTorch files.
[16:58.990 --> 17:02.390]  But, clearly, pickles are an issue for Bob.
[17:02.610 --> 17:09.830]  If Alice is distributing models as pickle files or PyTorch files, it makes it that much easier for Chuck to go in and backdoor them.
[17:11.490 --> 17:12.670]  So, on to the next class.
[17:12.870 --> 17:14.730]  Once again, bit of a fancy term here.
[17:14.930 --> 17:19.490]  But, all this is saying is that you shouldn't be trying to correct invalid input.
[17:19.610 --> 17:20.990]  You should just reject it altogether.
[17:21.730 --> 17:25.950]  This can be called the anti-robustness principle.
[17:27.050 --> 17:32.690]  So, to mitigate issues with Pickling, some developers write these things called restricted unpicklers.
[17:32.690 --> 17:38.350]  So, these are subclasses of unpickler that tries to enforce an allow list or a block list.
[17:38.550 --> 17:41.870]  And it does that by overriding the final class method.
[17:42.450 --> 17:43.930]  Now, that sounds...
[17:43.930 --> 17:47.210]  The thing is, they're actually not that hard to bypass.
[17:47.230 --> 17:49.390]  It's a pretty common CTF challenge to do that.
[17:50.050 --> 17:55.970]  And there's this methodology called pain pickle that tries to automatically bypass restricted unpicklers.
[17:56.330 --> 18:02.630]  It does this by saying there are eight different types of unpicklers and you can use three general strategies to do so.
[18:03.410 --> 18:12.070]  But, much like pickle was a problem for Bob, restricted unpickling bypasses is bad for Dave because his product might be relying on them in some fashion.
[18:13.770 --> 18:17.770]  So, now we can talk about one of my favorite exploits.
[18:18.390 --> 18:19.430]  Parser differentials.
[18:19.770 --> 18:26.350]  So, parser differentials occur when different parsers in a system read the same input but interpret it differently.
[18:26.910 --> 18:33.670]  So, when two parsers interpret the same file in different ways, that file can be called an ambiguous file.
[18:34.750 --> 18:37.530]  So, this is a pretty common exploit technique.
[18:37.710 --> 18:39.550]  It's really, really good for bypasses.
[18:39.850 --> 18:49.190]  But, it means you can create an ML model file that's okay, that's benign for one system or system component, but backdoored for another.
[18:49.490 --> 18:55.930]  There's some pretty big implications for ML system exploitation as a whole here that we'll talk a bit more about later.
[18:56.630 --> 19:00.530]  But, remember, whether or not this is impactful all depends on your system.
[19:00.750 --> 19:01.870]  So, threat modeling is key.
[19:03.190 --> 19:08.070]  So, we were able to create two differential proof of concepts with TorchScript.
[19:08.330 --> 19:11.710]  So, TorchScript is a popular format for storing ML models in.
[19:11.890 --> 19:13.590]  Used a lot for edge devices, by the way.
[19:14.070 --> 19:15.610]  It's pretty good.
[19:16.290 --> 19:20.450]  The reason it's pretty common is related to performance and portability.
[19:20.450 --> 19:25.850]  But, the point is you can make a parser differential using it and chain it to an architectural backdoor.
[19:26.170 --> 19:31.930]  This is because the way you turn a PyTorch model into a TorchScript one is through tracing or scripting.
[19:32.530 --> 19:36.490]  And tracing doesn't effectively represent dynamic control flow.
[19:36.770 --> 19:44.750]  So, all you have to do is take that malicious component for an architectural backdoor and represent it with dynamic control flow.
[19:44.750 --> 19:50.010]  And, boom, you have a TorchScript incubated ML exploit.
[19:52.150 --> 20:01.150]  So, this second example was found in the YOLO v7 codebase, which is a popular open-source codebase for computer vision.
[20:01.450 --> 20:07.450]  So, what they did is they released standard versions of the model as well as TorchScript versions for deployment.
[20:07.990 --> 20:14.230]  But, in the source code, we noticed that there are many cases where tracing didn't accurately capture the model.
[20:14.590 --> 20:18.070]  And they did a one-hop of serialization and deserialization.
[20:18.290 --> 20:23.410]  So, a lot of key information was lost and the usual PyTorch warnings didn't show up.
[20:23.990 --> 20:30.970]  So, to actually spot this differential, we use these tools, the TorchScript automatic trace checker, TorchFX, and the TorchScript IR.
[20:31.370 --> 20:37.090]  But, with what we found, we were able to create this input that made the two versions of the model behave differently.
[20:37.810 --> 20:39.330]  Essentially, a backdoor attack.
[20:39.330 --> 20:41.850]  So, once again, big problem for Bob here.
[20:41.990 --> 20:47.630]  He's getting a fundamentally different model than the one Alice trained, which breaks any pre-existing promises.
[20:49.750 --> 20:53.310]  So, we also identified a parser differential with SafeTensors.
[20:53.530 --> 20:56.710]  So, SafeTensors is another file format for NL models.
[20:57.050 --> 21:00.590]  It was actually developed in response to the insecurity of pickling.
[21:00.970 --> 21:10.190]  So, last year, I was in an audit of the SafeTensors library where we pointed out that the fact that this file format includes JSON as a big source of parser differentials.
[21:11.250 --> 21:15.570]  So, JSON is pretty well known to be an under-specified format.
[21:15.790 --> 21:19.650]  There's a bunch of exploits, especially in the web security realm, that leverage this.
[21:20.430 --> 21:25.450]  But the thing is, this reference SafeTensors implementation uses the serveJSON parser.
[21:25.630 --> 21:26.870]  That's a great parser.
[21:26.970 --> 21:27.830]  It's in Rust.
[21:28.770 --> 21:32.450]  But point being, it's very strict, and it rejects duplicate keys.
[21:32.710 --> 21:43.490]  But a lot of external tools build their own SafeTensors parsers using the PyTorch built-in JSON parser, which is quite permissive and accepts the second duplicate key.
[21:43.650 --> 21:51.210]  So, that gives an attacker an opportunity to create a duplicate key for the offsets inside of the SafeTensors metadata.
[21:51.210 --> 22:03.570]  So, essentially, they can append backdoored waits, which leads to corrupted SafeTensors file that is rejected by the reference SafeTensors implementation, but accepted by external parsers.
[22:03.730 --> 22:06.910]  And those external parsers will end up loading those backdoored waits.
[22:07.590 --> 22:20.250]  One thing that's cool is that it has to be a waits-based backdoor, because SafeTensors forces you to separate the waits in architecture storage, which is a good minimization of the attack surface.
[22:20.250 --> 22:26.590]  But there's some more details and caveats, but the point is the SafeTensors parser differential is impactful for Dave.
[22:26.890 --> 22:34.490]  He needs to be making sure that all the parsers in his product agree, because if his tools is using a more permissive parser, then maybe he can get a backdoor.
[22:36.410 --> 22:42.050]  So, one big part of my research is analyzing previous literature and noticing trends.
[22:42.410 --> 22:44.730]  I don't want to get too into the weeds here.
[22:44.890 --> 22:47.470]  I'll save that for accompanying materials.
[22:47.470 --> 22:55.070]  But one thing that became pretty clear is that there's something larger than parser differentials going on here with ML system exploitation.
[22:55.570 --> 22:57.230]  There's model differentials.
[22:57.370 --> 23:00.110]  Instances where the same model is interpreted differently.
[23:00.110 --> 23:06.130]  So, the attack here is dependent on the supply chain component and the life cycle stage, as per usual.
[23:06.630 --> 23:12.190]  But more specifically, in an ML system, you can have inputs that are pre-processed.
[23:12.230 --> 23:17.950]  Or you can apply model transformations to the model, such as quantization and compilation.
[23:17.950 --> 23:23.290]  And the thing is, a lot of these parser differentials at the pre-processing stage change the weights.
[23:23.630 --> 23:26.290]  Things like image scaling or Unicode parsing.
[23:26.510 --> 23:31.930]  And a lot of these model transformation backdoors change the architecture.
[23:32.430 --> 23:42.170]  So, I think it's very possible that model transformations that can be encoded into the loss function results and exploitable backdoors as a rule.
[23:42.170 --> 23:46.090]  But, moving forward, we're on to this next class.
[23:46.290 --> 23:47.910]  This is called shotgun parsing.
[23:48.090 --> 23:53.510]  This is just what happens when you don't fully and properly check your input before beginning to process it.
[23:54.810 --> 23:57.110]  So, let's talk about polyglot files.
[23:57.490 --> 24:03.250]  So, polyglot files are files that can be validly interpreted as two or more file formats.
[24:03.630 --> 24:11.190]  So, they've been used to distribute malware, bypass code signing checks, and just enable other malicious behavior.
[24:11.390 --> 24:13.250]  So, you can have a PDF that's also a zip.
[24:13.830 --> 24:22.110]  But, for ML model files specifically, you can take these polyglot files and put it in model hubs to confuse downstream consumers.
[24:22.290 --> 24:25.450]  And you can also use it to mess up pipelines.
[24:25.670 --> 24:29.470]  So, two different ML pipelines can interpret the same file as two different models.
[24:29.790 --> 24:33.050]  So, you can smuggle in a backdoored model with the benign one.
[24:34.090 --> 24:39.110]  So, during our audit of the safe tensors library, we are able to make multiple polyglots.
[24:39.230 --> 24:42.850]  This includes zip, PDF, Keras native, and later on PyTorchmar.
[24:43.090 --> 24:46.970]  And the report that we delivered itself was a PDF zip polyglot.
[24:47.090 --> 24:51.250]  And that zip contained all the safe tensors polyglots we made during the audit.
[24:51.650 --> 24:59.230]  So, you can just slap on a weights-based backboard model in one of these formats to a benign model in a safe tensors file.
[24:59.690 --> 25:02.330]  So, open it up with safe tensors, everything's fine.
[25:02.510 --> 25:06.630]  Load it up with like the PyTorchmar parser and boom, there's your backdoor.
[25:06.630 --> 25:15.430]  This is a big problem for folks like Dave who's depending on these models because now you've got malicious models that are just sneaking in with the benign ones.
[25:16.050 --> 25:21.330]  So, the underlying reason for this polyglottery is because of a missing check.
[25:21.850 --> 25:30.190]  Specifically, the program didn't check whether the start and end offsets corresponded with the tensor size.
[25:30.190 --> 25:33.570]  So, you could just append arbitrary data to the file.
[25:33.730 --> 25:41.110]  And then, once that you combine that with the ability to change the header size, you just have a whole number of polyglots you can make.
[25:41.390 --> 25:46.370]  This issue has since been fixed in safe tensors, by the way.
[25:47.130 --> 25:50.250]  So, next category, incomplete protocol specification.
[25:51.190 --> 25:52.550]  Once again, fancy term.
[25:52.790 --> 25:54.750]  Just think of it as under specification here.
[25:54.750 --> 26:01.830]  There's a lot of examples of this out there, but for the sake of time, I'm just going to focus on PyTorch polyglots.
[26:02.550 --> 26:06.990]  So, many people are unaware that PyTorch actually supports many file formats.
[26:07.470 --> 26:11.170]  Some are deprecated, but are still supported by external tools.
[26:11.910 --> 26:15.190]  One big issue is that there's a lack of consistent versioning.
[26:15.190 --> 26:22.470]  And that means it's pretty easy to create polyglots of different files that are validly interpreted as different types of PyTorch file formats.
[26:22.650 --> 26:29.890]  So, if you add three files to the PyTorch version 1.3 file, you can get the TorchScript version 1.4 polymocked.
[26:30.710 --> 26:33.930]  Another big issue is the reliance on zip and pickle here.
[26:34.370 --> 26:38.970]  So, pickle is a streaming file format that ends once it reaches the stop code.
[26:39.150 --> 26:41.510]  Any data... the stop opcode.
[26:41.990 --> 26:44.730]  So, any data after that is fair game.
[26:45.130 --> 26:49.970]  But, most zip parsers, like the PyTorch MAR, don't enforce their magic at the start.
[26:50.150 --> 26:54.970]  So, what you can do is you can take a zip file, append it to a pickle file, and boom.
[26:55.370 --> 26:56.550]  Zip pickle polyglot.
[26:56.650 --> 26:59.270]  And you can use that to create a bunch of PyTorch polyglots.
[27:00.010 --> 27:07.470]  Fickling now has a polyglot module, so you can differentiate, identify, and create polyglots for the different PyTorch file formats.
[27:08.810 --> 27:10.350]  So, on to the next class.
[27:10.350 --> 27:15.410]  This one just means that your input should actually be simple and well-defined, so you can check it thoroughly.
[27:16.810 --> 27:17.910]  And take ONNX.
[27:18.630 --> 27:21.970]  ONNX is this protobuf-based way to store ML models.
[27:22.130 --> 27:28.050]  And my colleague, Adelyn Travers, discovered a really neat hack for it that he packaged into a tool called Lobotomy.
[27:28.510 --> 27:33.790]  So, ML runtimes and frameworks often let you add custom operators to a model on the fly.
[27:35.290 --> 27:36.570]  Emphasis on on the fly.
[27:37.310 --> 27:42.230]  And the language that's being used for the ONNX runtime custom op was pretty complex.
[27:42.510 --> 27:53.670]  So, even though the specification officially disallowed side effects, you could just take arbitrary code, encapsulate it into a custom op, and then use that to launch an architectural backdoor.
[27:53.670 --> 27:56.230]  Just like Pickle, bad news for Bob.
[27:57.470 --> 27:58.450]  So, to recap.
[27:58.770 --> 28:03.710]  Bob, our direct consumer, was affected by our exploits using Pickle, ONNX, and TorchScript.
[28:04.130 --> 28:08.810]  Dave, on the other hand, affected by PyTorch, safe tensors, and restricted unpickling issues.
[28:08.810 --> 28:17.130]  Now, one thing that a lot of people miss about ML security is just how complex and how important the ML stack is.
[28:17.330 --> 28:20.890]  The model you choose changes the technologies in the stack.
[28:21.190 --> 28:29.590]  So, whenever I'm assessing a system or doing some kind of vulnerability research, I'm always trying to think about what layer of the ML stack I'm dealing with.
[28:29.590 --> 28:39.950]  So, the layers I have listed here, from bottom to top, is hardware, infrastructure, low-level, compiler, high-level, framework, model, and knowledge.
[28:40.410 --> 28:49.930]  So, at the framework level, the exploits that are exposed and impactful are the restricted unpickler, ONNX, one-time, and pickle-proof concepts.
[28:50.650 --> 28:58.430]  The TorchScript differential corresponds to the compiler level, and the safe tensors and PyTorch polyglot issues are impactful at the infrastructure level.
[28:59.030 --> 29:01.770]  So, this is really just a starting point.
[29:02.070 --> 29:06.250]  There's going to be exploits up and down the stack that impact all these ML systems.
[29:06.530 --> 29:08.890]  So, are you really good at breaking hardware?
[29:09.110 --> 29:11.410]  Go take a stab at the TPU.
[29:11.710 --> 29:14.890]  Do you happen to know a lot about distributed system security?
[29:15.110 --> 29:18.390]  Go write some hybrid ML exploits at the infrastructure level.
[29:19.170 --> 29:22.910]  So, I made this schema for incubated ML exploits.
[29:23.190 --> 29:26.350]  This is just one piece of a formal model of exploitation.
[29:26.350 --> 29:31.070]  But I'm going to talk about this at a very high level to shed some light on the terrain here.
[29:31.410 --> 29:37.230]  At the core, you, of course, need either a write primitive for the weights or a write primitive for the architecture.
[29:37.530 --> 29:42.590]  But the proof of concepts that we looked at point to some additional capabilities that help.
[29:43.590 --> 29:46.290]  So, of course, you probably want read primitives as well.
[29:47.130 --> 29:53.310]  But with the SafeTensors parser differential, you saw that access to the metadata can really facilitate an attacker.
[29:53.710 --> 29:59.710]  Specifically, if you have read and write primitives to the metadata of a model file format, that can enable both kinds of backdoors.
[30:00.670 --> 30:04.350]  Model transformations and model differentials are both also really useful.
[30:04.550 --> 30:10.190]  They can happen at different stages of the pipeline and be stealthy by exploiting existing procedures.
[30:10.910 --> 30:15.350]  Differentials are pretty broadly useful, and they're localized to the stage they're applied to.
[30:16.070 --> 30:27.350]  And with ONNX, it became pretty obvious that custom operators in serialization formats, and potentially in places like compiler dialects, can help with incubated ML exploitation.
[30:27.350 --> 30:31.050]  Lots more details on this in the accompanying materials.
[30:31.370 --> 30:35.010]  But I do want to make some more explicit recommendations.
[30:35.390 --> 30:37.530]  Apologies for how busy the slide is.
[30:37.730 --> 30:41.590]  But I think model files and metadata should be properly validated.
[30:41.950 --> 30:43.850]  We want good trust mechanisms.
[30:43.870 --> 30:45.950]  We want good parsers.
[30:46.190 --> 30:48.730]  And we also really want to minimize complexity.
[30:48.730 --> 30:54.110]  So we should be avoiding custom operators and separating the storage of architectures and weights.
[30:54.110 --> 30:58.990]  There's also a bunch of best practices for file formats that should be more commonly practiced.
[30:59.210 --> 31:02.090]  Like, you should have versions and checksums and magic signatures.
[31:02.370 --> 31:07.830]  You should enforce a signature at offset zero so you can, like, reject appended or prepended data.
[31:07.970 --> 31:13.790]  And we really need robust specifications and with no overloading fields.
[31:13.950 --> 31:17.710]  And we need to just reject invalid input instead of trying to correct it.
[31:18.570 --> 31:25.250]  So I'm really hoping we can see hybrid ML exploits and incubated ML exploits addressed by more frameworks and tools.
[31:25.490 --> 31:31.170]  I'd love to see this framework evolve and be applied to specific ML tools and contexts.
[31:31.610 --> 31:34.870]  I want to see it applied to more bug classes, more model vulnerabilities.
[31:35.170 --> 31:40.730]  And I think there's a lot of work to be done on exploit persistence, reliability, mitigations, and defenses.
[31:40.730 --> 31:51.090]  I also think there's just a lot more interesting work to be done in ML infrastructure security as a whole, with differentials and file formats, specification improvements, and just reverse engineering.
[31:52.550 --> 32:00.190]  But before we finish, what really helps me identify and make progress on ML security problems is understanding the two root causes.
[32:00.650 --> 32:03.390]  First of all, we're building all these new systems for ML, right?
[32:03.590 --> 32:07.870]  New hardware, new programming languages, new frameworks, new file formats, new compilers.
[32:07.870 --> 32:13.490]  And there's conferences dedicated just to new and interesting ways to design ML infrastructure.
[32:13.810 --> 32:18.570]  And that means all of these new systems are introducing new attack surface.
[32:19.010 --> 32:26.210]  And it's also, at the same time, becoming increasingly clear that the stack and supply chain isn't subject to sufficient review.
[32:26.450 --> 32:28.070]  That's why we're seeing pickles everywhere.
[32:28.830 --> 32:40.650]  But the second issue is that simply placing an ML model into a program introduces all these new vulnerabilities that stem from how the model interacts with different components.
[32:41.190 --> 32:42.630]  Resilience is .
[32:44.390 --> 32:46.430]  ML should be a quick add-on.
[32:46.570 --> 32:49.910]  It's something that can fundamentally change your system security posture.
[32:51.230 --> 32:57.910]  So, I hope you leave this talk knowing that we need to concurrently and holistically think about model security and system security.
[32:57.910 --> 33:05.390]  I do really recommend checking out the full audit reports for safe tensors in YOLO, the Fickling blog post, and the file formats repo.
[33:05.650 --> 33:09.430]  I'll post more details on the framework and exploit proof of concepts soon.
[33:09.430 --> 33:12.070]  We'll also be releasing an academic paper on the topic.
[33:12.730 --> 33:17.050]  You can find my contact info on my website or just send me a message on Twitter.
[33:17.250 --> 33:18.770]  But thank you all for coming.
[33:18.970 --> 33:19.890]  Thank you all for listening.
[33:19.890 --> 33:20.930]  Do you have any questions?
[33:22.110 --> 33:23.230]  Are your slides available?
[33:27.310 --> 33:27.550]  Yeah.
[33:27.950 --> 33:29.930]  But you can contact me and I'll send it to you.
[33:30.050 --> 33:30.270]  Okay.
[33:30.410 --> 33:30.730]  Thank you.
[33:31.570 --> 33:33.030]  You may get a lot of emails from me.
[33:36.250 --> 33:37.870]  What's the etymology of pickle?
[33:38.110 --> 33:39.150]  Where does it come from?
[33:41.090 --> 33:41.810]  I don't know.
[33:41.990 --> 33:44.330]  You've got to ask the Python devs for pickle.
[33:45.150 --> 33:45.570]  Sorry.
[33:45.870 --> 33:46.290]  Mike.
[33:46.610 --> 33:46.830]  Yeah.
[33:46.930 --> 33:47.310]  I don't know.
[33:47.390 --> 33:50.730]  You've got to ask the Python devs about why they call it pickle specifically.
[33:50.730 --> 33:54.310]  I guess pickling, serialization, it kind of makes sense.
[33:54.750 --> 33:56.810]  But it's a nice little metaphor.
[33:57.470 --> 34:08.490]  So it seems that you listed restricted unpicklers sort of as not the right kind of avenue, like sort of a dead end in terms of security.
[34:08.850 --> 34:13.990]  Is that because even restricted pickle is to open up a format or...?
[34:14.750 --> 34:15.850]  Yeah, exactly.
[34:15.850 --> 34:17.290]  It's essentially...
[34:18.610 --> 34:22.510]  It's essentially like saying, you have something that's potentially malicious.
[34:22.890 --> 34:24.490]  Let me try to fix it up.
[34:24.850 --> 34:28.010]  Instead of just saying, hey, I don't feel like...
[34:28.010 --> 34:28.990]  I don't think this is good.
[34:29.150 --> 34:30.370]  Let's just reject it altogether.
[34:32.470 --> 34:32.990]  There's...
[34:32.990 --> 34:37.770]  The paper goes into a lot of just really cool exploits of...
[34:37.770 --> 34:48.470]  They built this entire framework and they were like, here's this way that we're going to actually search for a read primitive, a write primitive within the parts that are available.
[34:50.890 --> 34:57.750]  Can you tell us how a backdoor attack algorithm works in more detail?
[34:57.750 --> 34:58.750]  All right, let's see.
[34:58.850 --> 35:00.230]  I think I have a slide for that.
[35:02.290 --> 35:03.050]  I do.
[35:03.930 --> 35:07.610]  So this is my favorite kind of backdoor attack.
[35:08.990 --> 35:11.110]  It's called a handcrafted backdoor.
[35:11.870 --> 35:15.650]  And it allows you to directly edit the parameters of a model.
[35:16.050 --> 35:19.650]  So I'm going to explain this by an analogy.
[35:19.650 --> 35:21.170]  So you have a building, right?
[35:22.550 --> 35:23.670]  And you...
[35:23.670 --> 35:28.330]  It's general knowledge that not every wall in a building is load-bearing.
[35:28.550 --> 35:33.030]  So let's say you have someone who wants to create this hidden passageway in a building.
[35:33.090 --> 35:39.810]  So he goes in and he just tears down a bunch of non-load-bearing walls to create this hidden pathway.
[35:40.010 --> 35:44.910]  You can essentially do that with a neural net to do a wait space backdoor.
[35:44.910 --> 35:49.510]  And what's cool is this is different from just poisoning the training data.
[35:49.630 --> 35:51.130]  You're directly editing the weights.
[35:51.310 --> 36:00.170]  And as we spoke about, the ML ecosystem, by sharing artifacts without sufficient validation, really opens up the possibility for something like this.
[36:00.530 --> 36:02.670]  Now, this is a weights-based backdoor.
[36:03.930 --> 36:06.630]  But there's also architectural backdoors.
[36:06.910 --> 36:09.630]  And I don't have a slide for that one.
[36:09.770 --> 36:17.270]  But fundamentally, there's this thing called the ReLU operation that are using neural network architectures.
[36:17.430 --> 36:20.310]  You can actually turn ReLU into a NAND gate.
[36:21.010 --> 36:23.390]  Ilya Shumalov has a really great paper on this.
[36:23.650 --> 36:29.710]  And you can use the NAND gate to essentially just compose whatever malicious operations you want.
[36:29.850 --> 36:37.230]  So in the TorchScript differential, the simplest version of that proof-of-concept, it was an image classifier.
[36:37.390 --> 36:41.130]  And I just added in, if you see the image inside...
[36:41.130 --> 36:50.250]  If you see the sticker or this particular pattern in the image, then you send out this particular output.
[36:50.570 --> 36:51.970]  That's the simplest version.
[36:52.090 --> 36:56.890]  There's obviously a lot more complex versions involving, such as the ReLU NAND gate one.
[36:56.890 --> 37:03.790]  But yeah, that's two versions of a ML backdoor attack from like an algorithmic level.
[37:05.530 --> 37:11.290]  Beyond trail of bits resources, are there any resources that you recommend for learning more about these topics?
[37:14.070 --> 37:17.310]  For Langsec, you can just go to Langsec.org.
[37:17.310 --> 37:21.530]  And you'll have like this huge host of Langsec papers.
[37:22.630 --> 37:24.510]  I have... let's see.
[37:24.850 --> 37:28.050]  I do have some other references.
[37:30.070 --> 37:32.490]  Yeah, this paper is really good.
[37:35.110 --> 37:36.550]  And then...
[37:38.110 --> 37:41.910]  Yeah, the ones in the middle of the slide are Trillibit's resources.
[37:42.230 --> 37:47.070]  The ones surrounding it are non-Trillibit's resources.
[37:47.070 --> 37:49.370]  I really like the blog post at the top.
[37:50.470 --> 37:51.830]  GGUF, the long way around.
[37:51.990 --> 37:53.030]  It's from Vicky Boykus.
[37:53.310 --> 37:57.690]  And it's like a history of the evolution of ML file formats.
[37:58.890 --> 37:59.470]  And then...
[38:00.450 --> 38:03.050]  Anjal Bertini's work is really fantastic.
[38:03.310 --> 38:04.750]  It's all about file format tricks.
[38:05.370 --> 38:09.610]  The citation there is proof of concept or get the F out.
[38:09.730 --> 38:11.470]  A very great hacker journal.
[38:12.650 --> 38:13.610]  And then...
[38:15.850 --> 38:21.850]  Yeah, this slide has just a bunch of papers with hybrid ML exploits that are non-Trillibit's resources.
[38:26.600 --> 38:27.080]  Yeah.
[38:27.720 --> 38:41.200]  So based on what you guys are seeing in the wild and the research that you're doing, how accurate do you think the OWASP top 10 for AIML is in terms of actually being the top 10?
[38:43.220 --> 38:44.560]  So I actually...
[38:44.560 --> 38:46.540]  I know they've made updates to it recently.
[38:46.540 --> 38:53.460]  I think I saw like the initial version, but I haven't seen the most recent version they put out.
[38:53.460 --> 38:54.920]  So I'd have to look back at that.
[38:55.120 --> 38:56.680]  But I do remember...
[38:56.680 --> 38:58.260]  Do they have pickle on it?
[38:58.320 --> 38:59.560]  Do you know off the top of your head?
[38:59.860 --> 39:00.120]  No.
[39:00.300 --> 39:00.580]  So not.
[39:01.520 --> 39:05.180]  I know they have prompt injection, which is something that we've reported.
[39:06.780 --> 39:08.420]  But yeah, there's...
[39:08.920 --> 39:12.460]  I do remember they also had like over-reliance on LLMs.
[39:12.720 --> 39:13.700]  Or maybe I'm...
[39:13.700 --> 39:14.020]  Sorry.
[39:14.200 --> 39:15.580]  I feel like everyone...
[39:15.580 --> 39:20.880]  There's a new AI security top 10 list or vulnerability framework every week.
[39:20.880 --> 39:24.700]  And it's hard to keep up with which ones are on which list.
[39:24.860 --> 39:29.260]  But at least one of them had like over-reliance on LLMs as like a...
[39:29.260 --> 39:32.780]  Like underlying issue, which I think is like something you definitely see.
[39:33.020 --> 39:36.420]  Like sometimes you do have to say, yeah, decommission your ML model.
[39:36.580 --> 39:37.580]  You're not using it right.
[39:37.680 --> 39:39.340]  You're not using it in a secure fashion.
[39:39.560 --> 39:40.560]  It shouldn't be here altogether.
[39:41.240 --> 39:42.280]  Did I answer your question?
[39:42.740 --> 39:43.180]  Ish.
[39:43.440 --> 39:44.180]  But yeah, that's fine.
[39:44.660 --> 39:44.820]  Okay.
[39:48.100 --> 39:49.960]  So in the...
[39:49.960 --> 39:51.840]  Sort of...
[39:52.680 --> 39:56.140]  Scenario that you described with Alice and Bob and Chuck.
[39:56.800 --> 40:01.200]  How exactly does like an attacker like Chuck attack these models?
[40:01.360 --> 40:07.220]  Are they redistributing modified versions of the model that Alice has created?
[40:07.660 --> 40:08.420]  Or...
[40:08.420 --> 40:10.620]  Like how do they sort of...
[40:10.620 --> 40:13.040]  Like create the...
[40:16.040 --> 40:17.160]  So the...
[40:17.160 --> 40:19.320]  Let me find the sleepy pickle diagram.
[40:25.160 --> 40:26.460]  Sorry, lots of slides.
[40:27.660 --> 40:27.960]  Okay.
[40:29.020 --> 40:29.580]  Yeah.
[40:29.800 --> 40:34.820]  So one way is of course to just like directly upload a malicious model to Hugging Face Hub.
[40:35.000 --> 40:39.340]  But sometimes you can chain exploits and get access to it some other way.
[40:39.340 --> 40:41.160]  Like if you have a man in the middle there.
[40:41.340 --> 40:43.400]  And you know that pickle file is being transferred.
[40:43.440 --> 40:45.820]  So you're like alright, let me just dynamically change it up.
[40:47.020 --> 40:49.500]  So yeah, I guess my answer really is...
[40:50.660 --> 40:52.240]  It all depends on your system.
[40:52.920 --> 40:55.340]  What are the capabilities of an attacker?
[40:55.340 --> 40:56.400]  What the vectors are?
[40:56.400 --> 40:57.000]  Yeah.
[40:57.260 --> 41:02.740]  Are the models that are uploaded to Hugging Face, are they like generally signed by their authors?
[41:03.120 --> 41:03.440]  Or...
[41:03.440 --> 41:04.360]  Um...
[41:04.360 --> 41:04.840]  No.
[41:05.560 --> 41:06.060]  Yeah.
[41:06.560 --> 41:07.260]  The...
[41:08.180 --> 41:16.600]  Yeah, the trust mechanism ecosystem for ML is very nascent.
[41:17.000 --> 41:20.320]  So like, there's a SIG store I think.
[41:20.320 --> 41:25.060]  There's a SIG store for ML GitHub repo.
[41:25.300 --> 41:27.300]  And I think they don't cover safe tensors at all.
[41:27.840 --> 41:30.540]  Like I think they're limited to a handful of file formats.
[41:30.740 --> 41:33.500]  And there are many, many ML file formats out there.
[41:33.680 --> 41:39.200]  So like I think the repository here has like 50 ML file formats.
[41:39.680 --> 41:47.220]  So for signing, like really robust effective signing mechanisms versus like 50 file formats.
[41:50.310 --> 41:50.870]  Yeah.
[41:51.370 --> 41:52.230]  Any other...
[41:52.230 --> 41:52.370]  Yeah.
[41:52.370 --> 41:52.710]  Oh, yeah.
[41:53.530 --> 41:59.330]  Why is the problem of signing ML formats different than just generally signing the file?
[41:59.590 --> 42:03.370]  Do you just take a signature of the file in general for all formats?
[42:05.610 --> 42:06.170]  Uh...
[42:06.170 --> 42:09.590]  Yeah, I'd have to think about this one a bit more.
[42:09.590 --> 42:10.170]  Um...
[42:10.170 --> 42:10.930]  There's some...
[42:12.210 --> 42:12.690]  Uh...
[42:12.690 --> 42:14.630]  There's some limitations, uh...
[42:15.330 --> 42:16.590]  Out there for like...
[42:17.590 --> 42:19.610]  Well, a direct sign is a direct sign.
[42:19.850 --> 42:22.310]  But for the cryptographic, uh...
[42:22.310 --> 42:26.370]  Or rather the guarantees that are provided by SIG store specifically, it's different.
[42:26.550 --> 42:28.550]  And of course signing doesn't, uh...
[42:28.550 --> 42:31.970]  Signing is not a comprehensive solution, right?
[42:32.050 --> 42:34.010]  To every single type of attacker out there.
[42:39.260 --> 42:40.220]  Hey, um...
[42:40.220 --> 42:42.020]  So you mentioned attack chaining.
[42:42.280 --> 42:46.460]  And I was wondering if in your research that includes things like, uh...
[42:46.460 --> 42:49.600]  A little bit higher level than, you know, looking at the model.
[42:49.760 --> 42:56.880]  But when you talk about infrastructure, does that include the million people who want to, like, host their own chat AI now these days?
[42:56.880 --> 42:58.580]  Or the, you know, uh...
[42:59.140 --> 43:02.440]  OpenAI's got this, you know, GPT function calling thing.
[43:02.620 --> 43:04.620]  Where now, you can kind of do prompt engineering.
[43:04.780 --> 43:06.440]  But it's also coding in the background.
[43:07.000 --> 43:07.640]  Is that...
[43:07.640 --> 43:10.700]  Is there any interesting attack chains that you've seen with those two things?
[43:10.840 --> 43:13.760]  Like the OpenAI, API, and also web apps?
[43:15.040 --> 43:15.520]  Uh...
[43:15.520 --> 43:15.660]  Yeah.
[43:15.920 --> 43:16.640]  So, um...
[43:16.640 --> 43:17.100]  Let's see.
[43:19.500 --> 43:23.880]  Do you consider RAG applications to be one of those?
[43:24.300 --> 43:24.780]  Um...
[43:25.520 --> 43:26.000]  So...
[43:27.720 --> 43:28.160]  Yeah.
[43:29.480 --> 43:29.920]  Uh...
[43:29.920 --> 43:30.340]  Sorry.
[43:30.460 --> 43:31.020]  Lots of slides.
[43:31.180 --> 43:32.120]  I'm trying to find the diagram.
[43:32.480 --> 43:32.920]  Uh...
[43:32.920 --> 43:33.000]  Yeah.
[43:33.080 --> 43:36.700]  So for this one, it had a poisoning attack, uh...
[43:36.700 --> 43:38.060]  Where there was like a, um...
[43:38.880 --> 43:39.140]  Uh...
[43:39.140 --> 43:40.420]  I can't remember off the top of my head.
[43:40.580 --> 43:41.120]  But there were...
[43:41.120 --> 43:43.660]  It enabled, like, front running and split view poisoning.
[43:43.720 --> 43:47.760]  So you could essentially use RAG, chain it with, like, a software security issue.
[43:47.840 --> 43:51.240]  And you do a really comprehensive data poisoning attack through that.
[43:51.760 --> 43:52.200]  Um...
[43:52.200 --> 43:53.740]  So this is with, um...
[43:53.740 --> 43:55.740]  The Ask Astro LLM app.
[43:55.740 --> 43:57.500]  It's by, um...
[43:57.500 --> 43:58.200]  It's...
[43:58.620 --> 44:00.040]  It's essentially a RAG application.
[44:03.370 --> 44:04.010]  Uh...
[44:04.010 --> 44:04.250]  Yeah.
[44:04.530 --> 44:05.070]  So, uh...
[44:05.070 --> 44:07.290]  It's a specific, um...
[44:07.290 --> 44:07.750]  Uh...
[44:07.750 --> 44:12.450]  It's a specific way to poison the training, uh...
[44:12.450 --> 44:12.710]  Sorry.
[44:12.850 --> 44:15.170]  Poison the application in and of itself.
[44:15.170 --> 44:17.190]  I don't want to butcher the, uh...
[44:17.930 --> 44:20.110]  Like, precise definition that's in the blog post.
[44:20.450 --> 44:20.710]  Uh...
[44:20.710 --> 44:21.210]  But, yeah.
[44:21.290 --> 44:22.730]  I can send you a link to this blog.
[44:22.830 --> 44:23.830]  It's also right here.
[44:23.970 --> 44:24.250]  Uh...
[44:24.250 --> 44:24.890]  And I'll pop up.
[44:25.690 --> 44:26.370]  Oh, thank you.
[44:30.530 --> 44:30.930]  Uh...
[44:30.930 --> 44:31.150]  Uh...
[44:31.150 --> 44:32.830]  If you go back to the slide of the stack.
[44:33.510 --> 44:33.910]  Uh...
[44:33.910 --> 44:34.670]  You have the stack.
[44:35.710 --> 44:36.110]  Uh...
[44:36.110 --> 44:36.510]  Oh, yeah.
[44:36.810 --> 44:38.930]  Well, why is there nothing next to the...
[44:38.930 --> 44:39.490]  The...
[44:39.490 --> 44:40.630]  The model itself there?
[44:40.790 --> 44:41.090]  Or what...
[44:41.090 --> 44:42.550]  What goes there, in your opinion?
[44:44.710 --> 44:45.510]  Oh, yeah.
[44:45.730 --> 44:50.290]  This was just organizing the proof of concepts I showed you, uh...
[44:50.290 --> 44:52.550]  I showed in this presentation according to the stack.
[44:53.250 --> 44:53.490]  Uh...
[44:53.490 --> 44:53.910]  So...
[44:54.450 --> 44:55.250]  This is...
[44:55.250 --> 44:58.550]  I organize the exploits based on where is this most...
[44:59.070 --> 45:03.030]  What part of the stack is this exposed at and or most impactful?
[45:08.550 --> 45:09.110]  Uh...
[45:09.110 --> 45:10.110]  No, but stay tuned.
[45:14.120 --> 45:14.680]  Uh...
[45:14.680 --> 45:15.320]  Any other questions?
[45:22.170 --> 45:22.490]  Any other questions?
[45:22.490 --> 45:22.650]  Okay.
[45:23.370 --> 45:23.690]  Uh...
[45:23.690 --> 45:24.570]  Thank you all for being here.