[00:00.000 --> 00:01.400] So, thank you for coming. [00:01.620 --> 00:06.840] So, about a year ago, I wrote this web app to scan Chrome extensions. [00:07.740 --> 00:10.580] And I'll talk about why I cared about Chrome extensions. [00:11.540 --> 00:25.140] But I also, by writing like a sort of scanner, I realized there are challenges to just making your own security tools in how you present the findings. [00:26.220 --> 00:31.000] And I will also talk about those and I would like to hear your opinions. [00:36.880 --> 00:41.420] So, the web app is online. [00:41.700 --> 00:45.080] It's at crxaminer.tech. [00:45.640 --> 00:47.040] I'll show the URL at the end. [00:47.120 --> 00:47.780] You can use it. [00:47.860 --> 00:49.680] It's been up for a few months now. [00:51.800 --> 01:01.640] So, before going into the details of how the web app works and how it scans, I want to talk about why I got interested about Chrome extensions in the first place. [01:05.260 --> 01:13.440] So, I started my career as a pen tester and one of the high severity findings we used to report was cross-site scripting. [01:13.440 --> 01:17.540] And that means you can run JavaScript in the context of a browser. [01:18.200 --> 01:24.940] And we consider that high severity because once you can run JavaScript in someone's browser, you can take a lot of actions. [01:25.060 --> 01:27.580] You can change the content of a web page. [01:27.720 --> 01:32.320] You can show them other stuff, redirect them. [01:32.580 --> 01:34.060] So, it's very powerful. [01:34.280 --> 01:39.800] And so, fast forward a few years later, I was talking to a developer. [01:41.640 --> 01:44.340] And they... that was a few years ago. [01:44.440 --> 01:45.260] That was like almost 10 years ago. [01:45.360 --> 01:48.120] And they showed me that they were using GitHub in dark mode. [01:48.680 --> 01:50.480] There was no dark mode in GitHub at the time. [01:50.580 --> 01:52.900] I was like, huh, like how would you have GitHub in dark mode? [01:53.040 --> 01:54.340] And they were like, Chrome extension. [01:54.720 --> 01:56.340] And I thought like, oh, that's pretty cool. [01:56.400 --> 02:02.200] Then I was like, wait, that means there's a Chrome extension out there where they can read everything you have in GitHub. [02:02.380 --> 02:03.860] And I thought, huh, that's kind of interesting. [02:05.600 --> 02:09.300] And it's still out on the Chrome Web Store. [02:09.520 --> 02:14.840] There's built-in dark mode to GitHub now, but there's still a Chrome extension for GitHub dark mode, which is sketchy. [02:15.060 --> 02:15.960] 20,000 users. [02:19.220 --> 02:26.480] And around late last year, there was an extension for Honey, which helps you find discounts. [02:26.780 --> 02:31.460] And they were doing a sketchy thing where people found out they were swapping out the... [02:31.460 --> 02:34.620] or they call the referral codes or the... [02:35.480 --> 02:39.860] So they would take commission instead of someone else, which is unethical. [02:41.140 --> 02:48.120] But people realize like, wait, this Chrome extension has the ability to change form data on other websites. [02:48.420 --> 02:49.100] That's weird. [02:51.220 --> 02:56.520] And to me, the most interesting one happened around Christmas, last Christmas. [02:56.740 --> 02:59.080] So there's a company called Cyberhaven. [02:59.280 --> 03:00.760] It's DLP, Data Loss Prevention. [03:02.220 --> 03:05.120] DLP basically means corporate spyware. [03:05.600 --> 03:11.320] But basically, it's like, how do you make sure your employees aren't like sending like 20,000 SSNs to their personal email? [03:11.440 --> 03:15.560] So you want something to monitor the traffic on their machines. [03:15.560 --> 03:30.220] And how do you monitor traffic is by like gaining access to all websites and seeing what they're doing, including like breaking TLS and really like God mode on their machine. [03:30.220 --> 03:33.300] In this case, Cyberhaven has a browser extension. [03:33.500 --> 03:37.980] So if you don't want to install an agent on the machine, you can just push a browser extension to Chrome. [03:38.180 --> 03:41.540] And it can see everything on like necessarily for it to work. [03:41.660 --> 03:43.880] It has to see everything you're doing on the browser. [03:44.900 --> 03:48.360] So the extension got compromised in a pretty interesting way. [03:48.480 --> 03:49.100] It was phishing. [03:49.240 --> 03:53.040] So they got this phishing email, I think like December 24th or 25th. [03:53.040 --> 03:55.020] And it looks pretty good. [03:55.160 --> 03:57.060] It's like, hey, you need to take this action. [03:57.200 --> 03:59.640] Otherwise, we're going to remove your Chrome extension from the store. [03:59.780 --> 04:00.540] They're a business. [04:00.980 --> 04:03.440] They can't afford to lose their Chrome extension. [04:03.720 --> 04:07.400] So they clicked on it and it took them to this page. [04:08.480 --> 04:19.280] I guess it's a bit hard to see on the screen, but this is just the Google's page to allow an application access to your account. [04:20.500 --> 04:27.480] It was a legitimate page hosted on Google, but the application itself was a fake one. [04:27.760 --> 04:32.280] Someone made a malicious fake application called privacy policy extension. [04:33.680 --> 04:37.860] And so it's hard to spot if you're like panicking like, oh, they're going to remove my extension. [04:37.860 --> 04:39.000] I need to act now. [04:39.140 --> 04:39.640] You see this. [04:39.760 --> 04:44.200] It's not immediately obvious that this is a malicious application. [04:45.060 --> 04:46.240] So they allowed it. [04:46.240 --> 04:50.000] And so the attackers gained access to the Cyber Haven Chrome extension. [04:50.220 --> 04:53.880] And so which is basically access to all data on a browser. [04:54.300 --> 05:00.440] So in your case, if you had access to all data on a browser, what would you decide to do? [05:05.100 --> 05:06.240] Look at the data. [05:06.780 --> 05:07.980] Look at all of the data. [05:08.120 --> 05:08.980] That's a good one. [05:09.080 --> 05:09.180] Yeah. [05:09.340 --> 05:09.400] Yeah. [05:09.400 --> 05:12.480] You can also copy it. [05:12.620 --> 05:12.880] This is great. [05:13.040 --> 05:13.140] Yeah. [05:13.320 --> 05:13.420] Yeah. [05:13.720 --> 05:14.680] You can send it to places. [05:14.840 --> 05:15.200] Yeah, exactly. [05:15.760 --> 05:24.020] And so weirdly enough, they they decided specifically to steal Facebook, Facebook business account cookies. [05:26.040 --> 05:28.940] I think I think it's for like ad fraud. [05:29.380 --> 05:36.180] I think there's like ad fraud rings in Vietnam and stuff where they they they like resell ad space or something. [05:36.380 --> 05:42.540] But I suspect they chose to scope it only to Facebook cookies just to make it harder to detect maybe. [05:43.300 --> 05:45.320] But they could have stolen all cookies. [05:45.660 --> 05:47.060] They could have stolen so many other things. [05:50.300 --> 05:52.760] Even more interesting, there was an info seal or malware. [05:53.080 --> 05:53.160] Okay. [05:53.360 --> 05:57.960] Malware where like you basically go to a phishing site, download malware and run it. [05:58.120 --> 05:58.180] Okay. [05:58.200 --> 05:59.820] Someone can run malware on your machine. [05:59.920 --> 06:01.660] They can run anything they want on your machine. [06:01.800 --> 06:05.920] And what they chose to do was to silo the Chrome extension into your browser. [06:05.920 --> 06:10.960] Like given that they can do anything, they were like, no, it's actually even better to have a Chrome extension on a browser. [06:11.160 --> 06:12.100] It gives us even more power. [06:13.720 --> 06:18.420] And there was a list of all the permissions they had that that extension had on your browser. [06:19.560 --> 06:25.200] Now, an interesting thing about Chrome extensions is you might think like, oh, like there's some browser security mechanisms like on cookies. [06:25.340 --> 06:26.580] You have like HTTP only. [06:27.320 --> 06:27.720] Right. [06:27.880 --> 06:30.120] And so like, oh, JavaScript can't manipulate the cookie. [06:30.240 --> 06:34.200] But like browsers have browser extensions have a God mode on everything in your browser. [06:34.200 --> 06:36.120] They can just grab any cookie they want. [06:38.240 --> 06:42.020] If that extension is scoped to that website, which many are, we'll see. [06:42.960 --> 06:47.580] And of course, there's a systemd butts, which replaces systemd to the word butts. [06:48.500 --> 06:50.080] That had a bunch of users as well. [06:50.160 --> 06:51.660] Now, this one was made by Jesse Frazell. [06:51.780 --> 06:52.680] We love Jesse Frazell. [06:53.220 --> 06:54.420] It's not malicious. [06:54.640 --> 06:57.040] But for example, this was last updated 10 years ago. [06:57.220 --> 06:58.640] It was still on the web store. [06:59.260 --> 07:03.080] Although they might have removed it recently because they were cleaning up ones with manifest V2. [07:03.860 --> 07:05.280] So what is a Chrome extension? [07:06.800 --> 07:12.120] The Chrome extension itself is just a file that's in the format of CRX. [07:12.240 --> 07:14.380] CRX3 is a specific format now. [07:14.760 --> 07:18.620] And the CRX3 file format is just a zip archive. [07:19.180 --> 07:21.560] It's just a zip file with some extra headers. [07:21.580 --> 07:23.780] But you can basically unzip it and we'll see. [07:24.400 --> 07:28.080] So they always start with CR24 as the magic number. [07:28.760 --> 07:32.200] And essentially then it's just a zip archive. [07:33.400 --> 07:38.540] This is some information from the header, like the file format itself, which you can... [07:38.540 --> 07:41.860] I found these, I think it was on the Chromium source code. [07:41.980 --> 07:46.140] You can find all of the details of how it handles CRX files. [07:47.580 --> 07:49.360] One thing we care about is... [07:49.360 --> 07:54.620] So we want to fetch any Chrome extension and download it to analyze it, right? [07:54.780 --> 07:57.160] So the way you do it is on the Chrome web store. [07:58.400 --> 08:00.420] Every extension has an extension ID. [08:00.620 --> 08:02.700] And when you download an extension, it saves it. [08:02.860 --> 08:04.760] And the folder name is that extension ID. [08:04.760 --> 08:06.820] So you can find those locally as well. [08:08.320 --> 08:09.620] So this is... [08:09.620 --> 08:11.280] Each extension has a unique extension ID. [08:11.420 --> 08:13.180] And it's this long string you see here. [08:14.100 --> 08:17.280] You can find it in the URL after you search for an extension. [08:18.460 --> 08:24.380] So if you want to download any extension you want, you can pull it through by just curling this. [08:24.560 --> 08:25.560] You give it the extension ID. [08:26.080 --> 08:28.140] One thing you want to give it is actually... [08:28.140 --> 08:33.880] The prod version here I found out is just a version of Google Chrome or like a relatively recent version of Google Chrome. [08:34.560 --> 08:37.620] If it's too old, the website just denies you. [08:37.680 --> 08:38.320] It doesn't say anything. [08:38.440 --> 08:40.060] It just doesn't return the Chrome extension. [08:40.340 --> 08:42.080] So you want to keep this relatively up to date. [08:44.780 --> 08:46.940] So let's say we download the Chrome extension. [08:47.660 --> 08:48.800] We unzip it. [08:49.000 --> 08:49.760] What's inside? [08:50.440 --> 08:52.000] This is basically the contents. [08:53.000 --> 08:56.380] You have some locales for like translation and language stuff. [08:56.980 --> 08:58.240] There's the data itself. [08:58.460 --> 09:01.660] The biggest stuff is JavaScript libraries that are inside. [09:04.420 --> 09:09.800] The thing we're going to talk about a lot is that last file which is the manifest.json. [09:10.220 --> 09:21.260] So the manifest.json is very interesting because it's one file that states all of the permissions that we're granting to that Chrome extension. [09:22.320 --> 09:24.000] And we're going to see what it looks like. [09:24.000 --> 09:25.220] So this is one example. [09:25.380 --> 09:28.760] This is for a Chrome extension called Cloud to Butt Plus. [09:29.560 --> 09:31.980] It replaces the word cloud with butts. [09:32.200 --> 09:38.040] So because it's an extension that wants to replace a word by another, it wants to work on all websites. [09:38.040 --> 09:45.460] And so you can see here, it says matches and the protocol is star and then star slash star. [09:45.540 --> 09:47.240] So it works on all websites. [09:49.020 --> 09:50.060] It's hard to see. [09:50.300 --> 09:50.800] Yeah. [09:52.540 --> 09:54.260] I'll skip in a few slides. [09:54.460 --> 09:55.560] There's a zoomed in version. [10:03.520 --> 10:11.160] So other than what it runs on which website, there is just a list of permissions here. [10:12.120 --> 10:14.680] And for example, this has cookie permissions. [10:14.680 --> 10:18.380] So it can just access any cookie on the websites that we allow it. [10:19.900 --> 10:27.320] And this is just a zoomed in version of like the JavaScript that it injects and then where it's allowed to run. [10:27.480 --> 10:29.480] It's allowed to run on all websites here. [10:30.200 --> 10:32.160] Star slash star slash star. [10:33.520 --> 10:35.300] So you can inject JavaScript. [10:35.320 --> 10:36.740] You can run it on any website. [10:38.120 --> 10:40.120] That's pretty dangerous in itself. [10:41.000 --> 10:43.280] Now, depends on the nature of the Chrome extensions. [10:43.580 --> 10:45.580] Some need this privilege. [10:45.580 --> 10:49.840] But the problem I started seeing is a lot of them don't actually need this. [10:51.480 --> 10:59.280] So I learned that once you mention Chrome extensions, this person just shows up and starts talking to you about Chrome extensions. [10:59.480 --> 11:02.240] Yeah, he's actually a very cool person that I met, John Tuckner. [11:03.320 --> 11:07.060] If you ever want to talk about extensions, just hit him up on Twitter. [11:07.260 --> 11:07.960] He's very friendly. [11:08.540 --> 11:13.660] But a very interesting thing is we started talking and he showed me this. [11:14.320 --> 11:19.020] So there was a what looked like a marketplace for a Chrome extension. [11:19.280 --> 11:21.360] So like someone can be like, hey, like I can just sell you. [11:21.600 --> 11:23.520] I can sell my Chrome extension to anyone else. [11:23.980 --> 11:28.440] And like I can imagine like some like 16 year old makes the Chrome extension as a joke. [11:28.560 --> 11:32.320] Then like five years pass and they have like 100,000 users, but they don't really care about it. [11:32.360 --> 11:33.980] Like I'll just sell it to someone else. [11:34.040 --> 11:34.220] Right. [11:34.900 --> 11:37.120] So he showed me this, this, this like marketplace. [11:37.180 --> 11:39.900] And I was like, okay, but does it, does it actually work? [11:40.000 --> 11:40.700] Like, is this a scam? [11:40.860 --> 11:42.600] Like, can you actually buy a Chrome extension? [11:43.320 --> 11:44.900] Um, so he tried to do that. [11:45.000 --> 11:46.840] He was like, okay, I'm going to buy a Chrome extension. [11:47.020 --> 11:48.940] He, he wrote a very interesting blog post about it. [11:48.940 --> 11:52.420] So he found a Chrome extension that had all permissions. [11:52.520 --> 11:55.400] Like, like what we saw allows you to do anything on any website. [11:56.000 --> 12:01.540] And he sent an email that the website, the, the, the marketplace allows you to email the, the, the Chrome extension owner. [12:01.560 --> 12:03.860] He was like, Hey, I want to buy this. [12:04.200 --> 12:06.680] And, and, and the, the person was like, sure. [12:06.760 --> 12:07.340] What's your budget? [12:07.380 --> 12:09.020] And there was some back and forth. [12:09.300 --> 12:10.700] He tried to pay him $50. [12:10.700 --> 12:13.820] The transaction failed, but then the guy was like, you know what? [12:13.840 --> 12:14.720] You can just have it. [12:15.260 --> 12:15.800] It's fine. [12:15.800 --> 12:16.820] I'll just give you my Chrome. [12:16.920 --> 12:17.380] I don't care. [12:17.540 --> 12:22.520] It had like 50 users, which is not, not much, but it's a proof of concept. [12:23.100 --> 12:27.880] Um, and so the, the guy transferred ownership to, to John. [12:28.100 --> 12:36.840] And so he got, uh, like, this is what it looks like on, on the, on the Google side where you're getting, um, ownership when you're the owner of a Chrome extension. [12:37.240 --> 12:40.460] Um, and he made a change. [12:40.700 --> 12:43.980] He, so, so John's website is called secure annex.com. [12:43.980 --> 12:46.960] And he made it redirect to this YouTube URL. [12:47.240 --> 12:48.520] Does anyone know what, what this is? [12:50.740 --> 12:51.220] Yeah. [12:51.380 --> 12:51.440] Yeah. [12:51.440 --> 12:51.800] Yeah. [12:51.800 --> 12:52.040] Yeah. [12:52.140 --> 12:52.200] Yeah. [12:52.320 --> 12:57.360] Many, many people have memorized the, it says it's a survival skill. [12:57.480 --> 13:01.840] And I recommend you memorize the, the last, uh, parts of the idea of the video. [13:01.960 --> 13:06.140] So, so when you visit his website, you, you, you, you just get Rick rolled. [13:06.320 --> 13:09.920] And, and, uh, so I tried that and it worked. [13:10.000 --> 13:10.720] They just recalled me. [13:10.840 --> 13:20.820] And, and so, so on his end, basically he gets an email and like, congrats, you're the owner of, of this Chrome extension, but on my end as the user. [13:20.940 --> 13:26.280] So I, while during the test, I had installed this Chrome extension before he bought it, before he, he became the owner. [13:27.040 --> 13:31.420] And there was no indication on my end that ownership transfer happened. [13:31.420 --> 13:36.660] Like I, I had this Chrome extension installed and like ownership happened that ownership transfer. [13:36.900 --> 13:38.160] Like he, he pushed the change. [13:38.400 --> 13:39.740] And on my end, nothing. [13:39.840 --> 13:40.860] Like my browser said nothing. [13:40.860 --> 13:43.600] Like, like I had no way to tell that anything changed. [13:45.260 --> 13:58.900] Um, I remember also, I, I remember seeing this tweet, which this, this person was proud, uh, proudly admitting that they bought a, a Chrome extension with 400,000 users. [13:59.900 --> 14:04.400] Um, now, I mean, if they're bragging about it on Twitter, they're probably not malicious. [14:05.520 --> 14:07.500] They, they, they could have not said anything. [14:07.500 --> 14:15.960] And, and so that, that, that, that's a bit worrying because that's, that's access, direct access to the browser of 400,000 people. [14:16.500 --> 14:23.120] Like, including, well, like your CEO, maybe, you know, and, um, this person chose to, to talk about it on Twitter. [14:23.120 --> 14:27.400] But like, like that means very easily someone can just buy a very popular extension and say nothing. [14:27.400 --> 14:30.200] And now they have like backdoor access to, to many browsers. [14:32.040 --> 14:32.480] Okay. [14:32.500 --> 14:34.220] So that's all the context around Chrome extensions. [14:35.160 --> 14:36.640] Now, what about my app? [14:36.640 --> 14:44.620] So, um, I, I wrote it in, in Ruby on rails and, uh, I hosted it on Heroku. [14:44.880 --> 14:47.300] I like Heroku cause you can just like link your GitHub repo. [14:47.460 --> 14:49.260] And then, you know, I have, I don't have to worry about infrastructure. [14:49.280 --> 14:52.120] I just pushed it to GitHub and it just deploys the app. [14:53.320 --> 15:05.580] Um, so the way it works, so I, I can do a live demo, but I had this video just as backup in case, like you just always have backups for, for like, oh, it's very hard to see here. [15:05.580 --> 15:10.740] Um, but basically I, I put the Chrome extension ID and then there's a loading bar. [15:11.520 --> 15:15.740] And then, uh, which on the backend it's downloading zip file. [15:15.860 --> 15:18.420] And then it basically shows me like a list of findings. [15:19.020 --> 15:22.000] Um, and then there's also the raw manifest itself. [15:22.400 --> 15:26.220] So we'll, we'll, we'll, we'll go over it slower, um, outside of the video. [15:28.380 --> 15:31.560] Uh, and, um, yeah, [15:35.070 --> 15:37.490] dark mode here is, is, is really hard to see. [15:37.970 --> 15:40.830] Um, dim the lights, no, no, it's okay. [15:41.930 --> 15:44.670] Um, but this is just like a search bar. [15:44.670 --> 15:52.830] If you go to the website, it's just a search bar where the only thing you can, you can input is the, the ID of the Chrome extension. [15:53.610 --> 16:01.410] Um, no, it's kind of hard to see, but, but here at the top, there's a statistics button and we'll, we'll, we'll, we'll show that a bit later. [16:01.910 --> 16:05.130] Um, but yeah, all it takes is you just paste the Chrome extension ID. [16:05.130 --> 16:07.370] Like, like the, I, I made it so it's like one by one. [16:07.870 --> 16:21.950] Um, and the, the first thing you see at the top, after you scan an extension and we'll, we'll do a live scan, but at the top, it just shows you the, the browser name and version and the icon. [16:22.270 --> 16:31.210] And then like the rating when it was last updated, the size number of users, uh, uh, this information I ha I had to scrape the Chrome web store to get. [16:31.330 --> 16:33.390] It's not like in the extension files itself. [16:34.670 --> 16:40.170] And then, so, um, to present the finding itself. [16:40.390 --> 16:43.050] And, and this is where I started thinking about like just security tools in general. [16:43.710 --> 16:54.090] Um, I realized like I, I can download the Chrome extension and, and look at the manifest and look at the permissions, but how do you actually assess the risk and how do you present it to the user? [16:55.430 --> 17:00.890] And so I'll explain in a bit, but there's actually two parts for it. [17:00.890 --> 17:07.010] The, the top part is, is called the context aware, uh, um, rate risk risk rating. [17:07.270 --> 17:16.210] And the bottom part is just like a, a series of, of findings that, um, my tool, um, surfaced. [17:18.310 --> 17:28.430] And my first thought was just using regular expressions and just looking through, through the, the extension itself, basically gripping for some things to, to find like, what are the permissions? [17:28.430 --> 17:33.350] And so for example, if it's asking for all permissions on all sites, that's, that's high severity. [17:34.270 --> 17:39.670] Um, then depending on, on, on the, the actual permissions that's asking and, and, and what's, what's in it. [17:39.670 --> 17:51.410] Um, like there's like a, like a bunch of like hard coded URLs or, or, or, or, are those fine, you know, looking at all of those and then trying to determine based on a number of findings, like overall risk rating. [17:53.810 --> 17:57.370] Uh, in a bit, I'll, I'll, I'll, I'll say what, what the challenge with, with that is. [17:57.850 --> 18:03.850] Now there's a separate tab where I, I just list the, the hard coded URLs in, in the Chrome extension. [18:04.310 --> 18:07.090] Um, just nice, nice to have. [18:07.290 --> 18:13.330] Uh, and then I, I see it's, yeah, it's not very visible, but you see the raw manifest itself. [18:13.430 --> 18:16.970] So it's just a JSON blob, uh, and you see all of the details in there. [18:17.410 --> 18:21.270] Um, so this is, uh, zoomed in. [18:21.270 --> 18:30.970] Um, this is zoomed in where you see like the, just a random example of a Chrome extension where it's showing, uh, that it has access to browsing data storage. [18:31.310 --> 18:36.850] It can make network requests on all websites, on all protocol protocols. [18:39.550 --> 18:45.590] So if you click on the statistics tabs right now, it just lists the last 10, 10 extensions scanned. [18:47.290 --> 18:50.750] And, and for, for fun, I made a, a, a Twitter bot. [18:51.250 --> 18:52.150] It's, it's useless. [18:52.270 --> 19:00.110] It's mostly just for me to, to, to, to see that it's still working, but, uh, you have a limit of up to 17, uh, posts per day. [19:00.110 --> 19:02.410] So sometimes it just posts something that it scanned. [19:03.170 --> 19:05.010] Now here's a problem that I ran into. [19:05.130 --> 19:07.450] And, and I think like a lot of security tools actually have that. [19:08.010 --> 19:13.550] Um, especially with something like Chrome extensions, where a lot of them do seem risky. [19:14.050 --> 19:20.730] And so the problem is the point of the tool was to, to, to tell you if, if an extension is dangerous to use or not. [19:21.750 --> 19:28.770] And so I found that there was actually an overwhelming number of, of, of extensions where it would say like, yeah, this is critical severity. [19:29.550 --> 19:34.630] But, but then if every single extension you scan is critical severity, that's not useful as a tool. [19:34.630 --> 19:38.550] You don't have to use it anymore because you can just assume like, Oh, that's going to be critical severity. [19:39.190 --> 19:39.370] Right. [19:41.370 --> 19:44.290] And that's something I'm, I'm completely open to feedback on, on how to solve. [19:44.570 --> 19:46.310] I, I, I tried one thing. [19:46.390 --> 19:52.390] I don't know if there there's better ways, but so how do we avoid making every single extension just return? [19:52.530 --> 19:53.590] Like, Hey, this is critical severity. [19:54.390 --> 20:00.510] So what I thought is as an example, a privacy badger is a Chrome extension that's made by the EFF. [20:01.290 --> 20:07.330] I trust the EFF at least more than any, any other, uh, Chrome extension, uh, publisher. [20:08.230 --> 20:16.190] And so by default, my tool was returning a critical severity because this wants, uh, basically excessive permissions. [20:17.630 --> 20:27.050] Now I thought the way to, to solve it is to give it more context on like, who, who is the publisher and what are they actually trying to achieve, uh, with this Chrome extension? [20:27.050 --> 20:29.130] And, and like, are they like a trusted publisher? [20:29.850 --> 20:36.210] And I thought like, instead of like a hard coding, a list of like trusted publisher, that's like a lot of work. [20:36.570 --> 20:39.970] Uh, I basically used, uh, Claude, I use a Claude API. [20:40.210 --> 20:44.490] And, and this, that, that, that's why the bottom part was what I was returning by default. [20:44.490 --> 20:54.290] And then I take my results and all of the details of the Chrome extensions, and I feed them into, into, uh, the Anthropic API. [20:54.570 --> 21:02.250] Uh, so it originally was using Claude 3.5, but they, they, um, decommissioned, deprecated that, that, that API. [21:02.250 --> 21:03.890] So now it's using Claude 4, I think. [21:04.690 --> 21:08.350] Um, and so it sends all of that information to Claude. [21:08.570 --> 21:12.210] And it's like, Hey, like you're, you're a security person who wants to analyze this Chrome extension. [21:12.490 --> 21:18.070] I want to see if it's safe to use, but I want you to also take into account, like, who is this publisher? [21:18.650 --> 21:21.190] Um, what is the Chrome extension trying to do? [21:21.330 --> 21:22.350] Is it the trusted publisher? [21:23.830 --> 21:33.470] And this is an example where for privacy badger by the EFF, my, my rating by default was critical, but, um, Claude actually returned low, low risk. [21:33.990 --> 21:41.290] Um, which I liked because, because it's, it's, it's more useful because like, if your CEO is using privacy badger against like no remove, it's actually probably fine. [21:41.910 --> 21:44.730] There is still a risk, but that's not like, there's risk in everything we do. [21:44.830 --> 21:48.430] Like Chrome extensions are inherently risky, but I don't want everything to stay critical. [21:48.590 --> 21:48.670] Right. [21:48.810 --> 21:51.890] And this, this seemed to address it, um, pretty well. [21:51.890 --> 21:57.030] And, and we'll see there, there's some, there's some, um, metrics here to, to compare actually. [21:57.130 --> 21:58.210] Yeah, this is the, this slide. [21:58.310 --> 21:59.930] So let's start with some numbers. [22:00.090 --> 22:03.990] Oh, actually, I'm very happy to announce that these slides are in light mode, not dark mode. [22:04.110 --> 22:06.230] So you'll, these might be the first slides you can see. [22:07.470 --> 22:13.750] So, uh, as of yesterday, there was 10,920 unique extensions. [22:14.010 --> 22:21.190] So I, I just published this, this website and I just, honestly, I don't use it that much. [22:21.190 --> 22:31.290] But, but a bunch of people started using it and organically it's accumulated about like almost 11,000, uh, unique, uh, extensions scanned. [22:32.530 --> 22:35.450] And so let's, let's take a look at the results. [22:35.650 --> 22:41.450] And so the, the first thing is, um, I was curious about the numbers of users that each extension has. [22:42.290 --> 22:48.130] Um, and, um, most extensions have less than 10,000. [22:48.470 --> 22:53.550] Um, but I was actually surprised that here, the biggest category is between one and 10,000 users. [22:54.110 --> 22:56.530] Um, which is a pretty decent number. [22:56.630 --> 22:59.230] Like if you compromise like 10,000 users, I like, I would be happy. [22:59.230 --> 23:06.350] Um, but there was only a very, very small number amount, uh, a small number of extensions. [23:06.610 --> 23:11.010] That's, um, over a million users, uh, or even over 200,000. [23:13.810 --> 23:24.950] And, um, I looked at the permissions and, uh, about a third of the extensions, um, had, um, access to all, everything on all websites. [23:24.950 --> 23:28.750] Um, which seems pretty excessive. [23:31.450 --> 23:43.230] And here is, um, the, the ratings, but here I'm comparing the, the original rating I had and then what the AI, like the AI adjusted rating was. [23:44.510 --> 23:47.810] And I was trying to answer of like, was it worth it? [23:47.990 --> 23:53.870] Like, did the AI actually make the ratings more meaningful? [23:54.950 --> 24:00.250] Um, what, what, what do you think at, at like, just glancing at this right now? [24:02.990 --> 24:03.430] Yeah. [24:03.790 --> 24:04.050] Yes. [24:05.070 --> 24:07.570] As in like the AI was, was, was better at ranking. [24:08.310 --> 24:08.750] Yeah. [24:12.210 --> 24:13.450] Yeah, exactly. [24:13.690 --> 24:16.090] So, so, so that's exactly what I thought. [24:16.230 --> 24:26.010] Like the main thing that I was doing is, if I want to simplify it, it's basically move, move the bunch from critical to high. [24:26.430 --> 24:28.890] Um, that's fine. [24:29.050 --> 24:30.030] Like it, it's pretty good. [24:30.110 --> 24:31.430] It's better than everything being critical. [24:31.850 --> 24:34.470] Uh, it, it significantly reduced the number of, of critical. [24:35.010 --> 24:37.650] Um, now a lot, a lot of them are high. [24:38.070 --> 24:43.330] Um, for some reason it, it reduced the number of minimal rated severities. [24:44.170 --> 24:45.950] Uh, that's weird. [24:46.090 --> 24:56.510] I'm not sure why, but, um, I think it's good that there's less, um, critical ones, but this also made me think of security tools in general. [24:56.830 --> 25:04.290] Like, I feel like I want some more, actually, I started thinking like, is this even the right way of, of, of ranking things? [25:04.290 --> 25:12.970] Like critical, high, medium, you know, um, most of the extensions here you scan are going to be critical or high. [25:13.750 --> 25:24.130] Um, I guess that means like, if you're using the tool, it's still up to the analyst or whoever's using the tool to, to make the, the call of like, do I want to allow this? [25:24.190 --> 25:24.890] Is this worth it? [25:25.470 --> 25:27.790] Um, it depends on the context. [25:27.970 --> 25:28.790] Like, why are you using it? [25:29.290 --> 25:35.250] Um, what I like about the AI output is it also gives some additional, um, recommendations. [25:35.630 --> 25:42.690] Like if this is, let's say high severity, but you still want to use it, then you can run it in a separate Chrome profile. [25:42.790 --> 25:45.690] So you can create a separate Chrome profile and, and run it there. [25:45.690 --> 25:53.550] And that means that the, the, um, um, what the, what the browser can manipulate is, is constrained within that profile. [25:53.570 --> 25:57.730] And so like, if you have all of your sessions and everything in a different Chrome profile, they're, they're separate. [25:57.730 --> 26:02.210] And I thought that that was a good way, but that's still kind of like up to the user to, to, to, to make that call. [26:05.370 --> 26:06.090] So, yeah. [26:06.350 --> 26:08.670] Um, was it worth it? [26:08.750 --> 26:12.890] Like, I, I think it's slightly better, but I, I think it's so like, like security tools. [26:13.110 --> 26:21.470] So we still need to find a way to give more meaningful output and results to, to the users to make it, to take meaningful action. [26:22.450 --> 26:26.350] Um, so we'll, we'll do a quick live demos, but this is the, the URL. [26:26.350 --> 26:29.390] Um, CRX examiner dot tech. [26:30.210 --> 26:31.770] Um, you can, you can use it. [26:31.830 --> 26:33.150] It's been, it's been up for awhile. [26:33.950 --> 26:41.370] Um, if you run into any issues, I guess you can, you can reach out, you can open a GitHub issue or it's, it's all open-source on GitHub also. [26:41.490 --> 26:41.950] You can fork it. [26:42.510 --> 26:46.850] Um, so we can, we can try a live demo. [26:46.850 --> 26:50.210] Um, so it's going to be slightly hard to see though. [26:50.690 --> 26:58.070] So this is the actual Chrome extension ID and, and I can go to the Chrome web store, but, uh, does anyone want to suggest the Chrome extension to scan? [27:00.350 --> 27:01.030] Yeah, sure. [27:01.190 --> 27:01.410] Yeah. [27:01.850 --> 27:02.030] Yeah. [27:02.090 --> 27:02.610] That's a good one. [27:03.110 --> 27:07.570] That's actually a very good one because it does have, um, a lot of, uh, permissions. [27:08.490 --> 27:14.330] Um, actually when I was testing, this was actually the one I was using a lot for testing. [27:14.510 --> 27:18.070] So, so this is, we go to the Chrome web store and we want the Chrome extension ID. [27:18.250 --> 27:20.250] And, and so it's in the URL here. [27:20.350 --> 27:21.850] It's, it's, that's the Chrome extension ID. [27:21.850 --> 27:27.790] So I'm just going to copy that and I'm going to paste it here and I'm going to click scan. [27:27.910 --> 27:37.090] But one thing is it immediately, uh, it immediately returned the results, but you can see here that it was last scanned seven months ago. [27:37.330 --> 27:40.570] Uh, so by default, if you scan something I was already scanned, it just loads. [27:40.730 --> 27:48.630] But if it was scanned, uh, it was, if it was last scanned over either 24 or 48 hours, I forgot what I set it to. [27:48.630 --> 27:52.550] It shows you a forestry scan button, uh, just to get updated results. [27:52.710 --> 27:55.350] So we're going to do that where I'm just going to click on, on forestry scan. [27:56.570 --> 28:02.950] And, um, so here, when, when you scan it, it just does what we talked about in that, like it, it fetches it from the Chrome store. [28:03.430 --> 28:04.690] It unzips the file. [28:04.870 --> 28:08.610] It looks at the contents, uh, and then it sends them to the cloud API. [28:08.930 --> 28:10.970] And, and now we have the updated results. [28:11.130 --> 28:12.790] So that last scanned less than a minute ago. [28:13.630 --> 28:18.570] And so first, uh, well, yeah, the 5 million users, three, three stocks. [28:18.630 --> 28:18.890] Of course. [28:19.650 --> 28:20.190] Sure. [28:20.350 --> 28:21.670] Many people don't, don't like it. [28:22.050 --> 28:30.970] Um, but, um, my, my original risk rating was, was critical because, uh, well, it has a lot of permissions. [28:31.670 --> 28:35.850] Um, it, it can, it even have, has unsafe wasm. [28:38.350 --> 28:45.170] Um, so, so by default, the, by the way, the, the, the, the, the algorithm to, to rank the ratings is, is very dumb. [28:45.330 --> 28:49.490] It's like, if you have more than like five high rating high, then it's, then it's critical. [28:49.730 --> 28:51.990] It's, it kind of just counts the number of findings. [28:52.690 --> 28:56.790] Um, and here it, it ranks it as medium. [28:58.750 --> 28:59.770] I, I don't know about that. [28:59.850 --> 29:05.590] I think like maybe, maybe low would have been better maybe, but, but technically like, like, like, like it is risky. [29:05.590 --> 29:07.470] It is a risky, uh, extension. [29:08.010 --> 29:13.430] Um, and the, the, the, the wasm unsafe eval is, is, is pretty spicy as well. [29:14.370 --> 29:19.550] Um, now if you click on details, uh, it lists all of the permissions it has. [29:20.030 --> 29:27.550] Um, all URLs, um, the CSP and, uh, 670 hard coded URLs in there. [29:28.050 --> 29:30.650] Um, so you can keep scrolling through the pages. [29:31.330 --> 29:35.570] Um, the, the, this is just grepping for yourself. [29:35.590 --> 29:39.570] URLs, by the way, it's, it's not like, it's not necessarily, um, meaningful. [29:39.690 --> 29:46.310] Uh, some of them might be just like URLs and license or something, but, um, yeah, it just reps for, for URLs in case you want them. [29:47.090 --> 29:48.850] And then, uh, raw manifests. [29:49.250 --> 29:56.150] Um, so, um, so maybe, I don't know. [29:56.210 --> 29:57.030] I've never tried that. [30:00.050 --> 30:04.770] Um, there might be a way to switch to live mode. [30:04.770 --> 30:10.190] But I don't know if, I forgot if the, I don't have a button for it. [30:10.350 --> 30:21.450] So, um, yeah, it's hard to see, but, but by default, many extensions have, uh, this MSG extension name in, in the manifest itself. [30:22.470 --> 30:36.270] And then, um, and then you can see all of the permissions here and you can see that it, it, it works on all, uh, uh, all URLs, uh, including local hosts. [30:37.310 --> 30:37.790] Sure. [30:39.430 --> 30:42.870] Um, yeah. [30:43.010 --> 30:45.010] And then it loads a bunch of resources as well. [30:45.650 --> 30:47.170] Um, including JavaScript. [30:48.130 --> 30:54.030] Um, and then I had, I wanted to implement, uh, secure annex is John Tuckner's thing. [30:54.150 --> 30:58.890] So actually he made his own version of, of, of this as well, but it's more enterprise. [30:58.890 --> 31:00.710] Like, like he has like live monitoring. [31:01.670 --> 31:06.810] So when, when, when I was talking, uh, I, I wanted to pull from his API, but I just got too lazy and didn't do it. [31:06.870 --> 31:12.570] But when I was talking to him about like, how do we know if, um, if a Chrome extension, um, changed ownership? [31:12.570 --> 31:16.750] Like my tool has no way of flagging it because you're, you're doing point in time scans. [31:17.330 --> 31:19.530] So he does, uh, like live monitoring. [31:19.530 --> 31:27.490] And, and if, uh, if the owner of a Chrome extension changes, um, he can send you on a Slack alert, um, which is pretty useful. [31:27.670 --> 31:31.370] But then, then as with all Slack alerts, now you have to like monitor those. [31:32.690 --> 31:34.430] Um, cool. [31:34.610 --> 31:34.670] Yeah. [31:34.790 --> 31:38.790] And then at the top, you, you, I, I linked the GitHub in case you want to see the source code. [31:39.190 --> 31:40.090] Um, yeah. [31:40.090 --> 31:43.810] Does anyone have, uh, any other Chrome extensions to scan? [31:44.350 --> 31:45.310] Um, keep it. [31:46.230 --> 31:47.510] Oh, you want to scan keeper? [31:51.430 --> 31:52.770] It's probably going to be the same. [31:52.970 --> 31:53.090] Um, [31:56.940 --> 31:57.440] this one, right? [31:57.680 --> 31:57.800] Yeah. [31:59.120 --> 32:00.520] I feel like it's going to be the same. [32:00.700 --> 32:01.020] Let's see. [32:01.160 --> 32:03.100] So it was scanned before eight months ago. [32:03.320 --> 32:04.500] We can force a rescan. [32:05.600 --> 32:09.360] Also, I'm curious, like the, the, the old scan for the AI was a cloud 3.5. [32:09.520 --> 32:11.120] And the new scan is going to be cloud four. [32:11.120 --> 32:13.320] And the old one said medium. [32:13.540 --> 32:14.920] Let's see if the new one says something else. [32:16.600 --> 32:17.520] Uh, it's the same. [32:18.800 --> 32:19.220] Yeah. [32:19.400 --> 32:20.260] So mine said critical. [32:20.540 --> 32:25.380] And then it's really the same, the same, uh, uh, ratings as, as one password. [32:25.660 --> 32:26.860] This has less findings. [32:27.440 --> 32:29.840] Um, it doesn't have wasm looks like. [32:33.940 --> 32:34.280] Yeah. [32:34.520 --> 32:34.540] Yeah. [32:34.540 --> 32:42.300] And so here the, the context, so where it was meant more, it gives some more context and gives some more like, like, yeah, you can like alternative recommend it. [32:42.340 --> 32:45.020] Like you can run it as a separate Chrome profile. [32:45.300 --> 32:47.700] Um, it explains a bit, some of the risk. [32:47.820 --> 32:51.480] And the idea here is like, you might have like an, like a junior analyst. [32:51.720 --> 32:55.140] That's supposed to like, sometimes in your company, like CEOs like, Hey, is this safe? [32:55.200 --> 32:55.940] Can I use this extension? [32:55.940 --> 32:59.620] And so like, you can send it, um, to a junior analyst and like, Hey, figure it out. [32:59.720 --> 33:03.740] And they can like run this tool and they can read this and they can like make their own call. [33:03.900 --> 33:09.220] I think that like risk is subjective depending on, on their risk appetite. [33:11.060 --> 33:11.980] Um, cool. [33:12.440 --> 33:12.800] Yeah. [33:12.960 --> 33:15.020] That's all I had today. [33:15.180 --> 33:19.780] So, so if you want to access it, this is the, the URL it's, it's live. [33:20.040 --> 33:25.260] And, um, you know, if you want to play around with more, more extensions, it can. [33:25.260 --> 33:27.240] But if you have any questions, I'm happy to answer. [33:27.360 --> 33:38.700] But also, I am very curious if you had any ideas or recommendations on, on, for this tool and for other tools, like how, how do we make output meaningful? [33:39.100 --> 33:46.320] Like how, if you're writing like a, like a scanner or any sort of security tool, how do we make sure that when you scan something, it's not just always saying critical? [33:48.820 --> 33:49.180] Yeah. [33:50.260 --> 33:53.020] I personally think you might be being a bit too humble about it. [33:53.020 --> 33:58.180] Like your qualification or quantification of the risk might not be wrong. [33:58.660 --> 33:59.700] It might not be wrong. [34:00.060 --> 34:04.000] So, so I, I've thought about that. [34:04.260 --> 34:10.580] And the, like the way I'm going to say is the problem is Chrome extensions themselves. [34:11.740 --> 34:12.840] The problem is Google. [34:13.560 --> 34:17.360] So, so, so, so, but I am biased, right? [34:17.360 --> 34:18.780] That, that, that's why I'm, I'm not sure. [34:18.920 --> 34:26.300] So, I think the way Chrome extension permissions are, are configured is kind of weird. [34:26.520 --> 34:29.080] Like, like it's often overly permissive. [34:30.320 --> 34:31.900] It's not granular enough. [34:33.300 --> 34:33.820] Okay. [34:33.940 --> 34:34.940] So, so here's another thing. [34:35.460 --> 34:43.300] Recently, we moved from the manifest version two to the manifest version three, and there was a lot of pushback. [34:43.300 --> 34:52.860] I don't know if you, you, you know, so, so one of the major, the cataclysm was that they, they removed the ad blocker, or at least one of them. [34:52.980 --> 34:54.960] There's a blocker light because it uses manifest v3. [34:55.060 --> 34:59.620] But one of the ad blockers use manifest v2, and they, they removed that. [34:59.740 --> 35:01.500] Everything that was using manifest v2 was removed. [35:01.500 --> 35:04.020] And so why did they do that? [35:06.260 --> 35:09.420] Technically manifest v3 has more granular permissions. [35:09.860 --> 35:17.780] So it actually allows the, the, the extension developers to, to, to, to specify with more detail what you're, what, what you're allowed to access. [35:18.960 --> 35:30.500] Um, now a lot of people criticized Google saying that it was a way to, to, uh, just make people not use ad blockers so that they get more ad revenue from, from YouTube or whatever. [35:30.500 --> 35:38.600] Um, I did speak to, uh, um, developer on the, on the Chrome team, um, at Hushcon remember. [35:38.780 --> 35:45.980] And, and, and so they said that the, the team that makes those changes has nothing to do with the ad revenue team. [35:46.100 --> 35:51.600] Like they were actually making those changes because they actually wanted Chrome extensions to be more secure. [35:52.280 --> 35:58.820] Um, so, so they wanted, uh, uh, like the manifest to, to, to be able to, to specify like more granular permissions. [35:59.920 --> 36:05.700] But for example, the, um, um, Gore Hill, I forgot his real name. [36:05.840 --> 36:14.300] Gore Hill, like the, the guy who made ad block, he has like a, like a giant thread on Twitter going on a rant on, on, on why he, he actually thinks that that was not the right move. [36:14.300 --> 36:22.360] And he, he, he gave a lot of really good, good reasons saying like, it's, it's not, it's kind of like not the right incentive. [36:22.520 --> 36:24.180] Like they're not really doing it for the right reasons. [36:24.220 --> 36:28.020] And they're, they're actually like, and he's like, I don't have a good way to re-implement. [36:28.020 --> 36:35.160] And, uh, cause I, I think ad block light still works, but the real one is like, I don't have a really good way to re-implement it in manifest v3. [36:35.340 --> 36:36.440] So I'm not even going to bother. [36:37.100 --> 36:44.300] Um, now there are still ways to, to, to, to use it to force, uh, force load, uh, uh, manifest v2 extension. [36:44.860 --> 36:50.000] Um, and obviously you can still use it on like brave, like the brave browser has, as can, uh, yeah. [36:50.380 --> 36:50.700] Yeah. [36:50.700 --> 36:59.860] My next question, since we know that the problem is Google, and, uh, I was wondering if there's anything that we use with brave. [37:00.580 --> 37:11.340] I know it's, of course, we've been saying VA engine, so it's unlikely, but any chance that there are other browsers that find a problem bigger around this in a way, or not for Google as brave? [37:11.700 --> 37:13.560] Yeah, that's, that's a very interesting question. [37:13.840 --> 37:15.200] Um, I'm not sure. [37:15.360 --> 37:18.680] So, so, so brave, big brave fan, by the way. [37:18.680 --> 37:21.160] I like, cause you don't need to use a, uh, ad blocker. [37:21.240 --> 37:21.580] It just does. [37:21.780 --> 37:25.980] So, so I like brave, but brave uses the, the same, uh, web store. [37:26.340 --> 37:26.780] Sorry. [37:26.940 --> 37:30.320] Like you're downloading the exact same Chrome extensions with the exact same permissions. [37:30.560 --> 37:33.540] So between, uh, brave and, and Chrome, there's no difference. [37:33.700 --> 37:35.200] Any Chromium and Chrome, there's no difference. [37:35.780 --> 37:43.400] Um, Firefox, I think Firefox and edge and Safari have their own web stores with their own extension format. [37:43.400 --> 37:44.340] So that's different. [37:45.000 --> 37:55.560] Um, but, um, I haven't looked at, at the different having like large scale statistically analyze the difference, uh, between like, do they actually do a better job? [37:55.640 --> 37:56.300] I don't know. [37:56.480 --> 38:14.140] Uh, I think part of it is, um, any, um, anytime you want to allow third parties to develop, um, extensions or mods or anything for, for your tools, you want them to have the ability to, to do a lot of things. [38:14.140 --> 38:20.100] And so by nature, like, like you want developers to be able to change colors on any website. [38:20.300 --> 38:20.380] Right. [38:20.880 --> 38:24.980] But then to change colors on any website, you have to access CSS on any website. [38:25.100 --> 38:30.300] And so you have to, so, so it's, it's by nature of Chrome extensions, like as a feature, right? [38:30.420 --> 38:39.520] Like it's hard to say like what's successive as, as a permission, uh, are they like, they can't afford to manually review every single extension to see if this is benign or not. [38:39.520 --> 38:45.300] Now, I think there are some enhancements you can make to, to protect the account developers themselves. [38:45.760 --> 38:53.140] Like, I think like, for example, when, when, when the ownership changed between one extension and another, I think that that's not something that should, that should change often. [38:53.260 --> 38:58.620] And I think when that does happen, Google should show me a pop-up saying like this extension changed ownership. [38:59.200 --> 39:00.420] Do you still want to allow it? [39:00.560 --> 39:00.640] Right. [39:00.640 --> 39:03.280] Because why, why would an extension change owner, like the email changed? [39:03.680 --> 39:15.700] Like, like there are maybe some small edge cases where like maybe a, like a corporate domain changed and they want to, you know, but, but normally you don't often change the owner of a Chrome extension. [39:15.920 --> 39:21.980] So I think they can do more in terms of alerting, in terms of like making sure people understand the permissions. [39:22.100 --> 39:27.680] And like, if they know, for example, many people are just asking like CSS, then make it access CSS, but not other things. [39:28.120 --> 39:28.480] Right. [39:28.620 --> 39:33.940] Like, like, like, so more, more granularly, um, more like being more strict on, on, on, on permissions of Chrome extensions. [39:34.660 --> 39:36.300] It's not an easy problem to solve. [39:36.460 --> 39:45.080] Like, I don't, I don't really know what, like if I was Google or if you were like how you, if you were hired as, as like, you know, fix Chrome extension permissions. [39:45.320 --> 39:46.740] Now, like, how would you do it? [39:47.240 --> 39:48.080] Uh, I don't know. [39:48.160 --> 39:48.580] It's not easy. [39:53.780 --> 39:54.260] Go ahead. [39:54.260 --> 39:54.360] All right. [39:55.040 --> 39:57.500] Uh, I have two questions, but I'll, I'll, I'll ask one. [39:57.760 --> 40:11.520] Um, the, uh, have you, do you think that this also applies to other things like, let's say, tamper monkey, where people would use that, like, uh, you know, in a, in a, in a other setting, like a corporate setting? [40:12.160 --> 40:17.260] Um, is, is there similar risks there or is that built in? [40:17.880 --> 40:19.520] Have you ever looked into that by any chance? [40:19.520 --> 40:23.860] Yeah, I think, I think, we can scan tamper monkey. [40:24.040 --> 40:25.560] It's, it's, it's still an extension, right? [40:25.700 --> 40:26.820] Was it on Firefox or? [40:26.900 --> 40:35.860] Yeah, it actually meant more like, because tamper monkey is the extension, but it also lets you kind of, you know, crowdsource other, other folks to build modules, right? [40:36.100 --> 40:39.800] So, in a sense, the people building the modules or the... [40:39.800 --> 40:41.740] Yeah, oh, that's, that's extremely risky. [40:41.960 --> 40:43.160] Yeah, that's extremely risky, right? [40:43.360 --> 40:45.680] But, but, it's extremely risky. [40:45.680 --> 40:52.800] Uh, you're, um, I don't know why, I think, an example I'm thinking of is in World of Warcraft. [40:53.120 --> 40:54.280] There's, uh, Weak Auras. [40:54.540 --> 41:00.060] It's a, uh, World of Warcraft extension, and it basically allows anyone, like, the Weak Auras you can download from other players and they pass around. [41:00.160 --> 41:01.180] They're basically Lua script. [41:01.340 --> 41:04.880] And so people are just, like, passing each other, like, Lua scripts running in their game. [41:05.000 --> 41:07.680] And, like, someone made them a malicious one that steals their gold when you open the mail. [41:07.800 --> 41:10.560] But, the idea is, like, if you make an extension that allows the... [41:10.560 --> 41:11.500] Okay, nice. [41:11.620 --> 41:12.980] So, tamper monkey is listed as critical. [41:12.980 --> 41:14.760] I think that, I think that's valid, right? [41:15.560 --> 41:18.100] Uh, tamper monkey is, is obviously very, very risky. [41:18.600 --> 41:20.620] Um, but, uh, very small tangent. [41:20.760 --> 41:21.680] You're mining, you're, like, other stuff. [41:21.800 --> 41:29.760] So, this is, it's not unique to Chrome extensions because what's a very, uh, what's another avenue where you can install extensions that's very, that's very spicy. [41:30.680 --> 41:31.520] Other than browsers. [41:35.130 --> 41:35.670] VS Code. [41:35.750 --> 41:36.810] VS Code, exactly. [41:36.930 --> 41:37.610] IDE extensions. [41:37.810 --> 41:38.790] It's the same, it's the same thing. [41:39.110 --> 41:39.550] It's the same thing. [41:39.750 --> 41:39.850] Docker as well. [41:40.230 --> 41:40.730] What was that? [41:40.910 --> 41:42.270] Docker and VS Code, yeah. [41:42.770 --> 41:43.170] Docker. [41:43.170 --> 41:44.310] Docker has extensions? [41:45.010 --> 41:45.130] Yeah. [41:45.750 --> 41:46.670] I've, I've never tried that. [41:46.790 --> 41:47.650] That's terrifying. [41:47.950 --> 41:48.370] That's terrifying. [41:48.690 --> 41:48.770] Yeah. [41:49.450 --> 41:49.970] So, exactly. [41:50.210 --> 41:51.070] Docker and VS Code. [41:51.150 --> 41:55.990] So, uh, actually, John Tuckner, in his tool, he started analyzing VS Code extension as well. [41:56.170 --> 42:00.750] And there's been a recent Twitter threads about people, like, trying to download, like, a Solidity VS Code extension. [42:01.650 --> 42:03.750] I don't know why you would do, but anyway, yeah. [42:03.750 --> 42:07.370] So there was, like, malicious ones and they stole, oh no, my crypto gone. [42:07.570 --> 42:08.130] You know, but... [42:08.130 --> 42:09.010] It's for Docker Desktop, sorry. [42:09.170 --> 42:09.830] It's for Docker Desktop. [42:09.950 --> 42:10.610] Docker Desktop, okay. [42:10.770 --> 42:11.050] Yeah, yeah, yeah. [42:11.130 --> 42:11.550] That makes sense. [42:11.690 --> 42:11.990] Yeah, yeah. [42:12.290 --> 42:17.790] I didn't even know Docker Desktop had extensions, but that's very interesting and risky. [42:17.950 --> 42:18.330] It's the same. [42:18.570 --> 42:19.050] Yeah, yeah. [42:19.170 --> 42:20.430] Sorry, what was your second question? [42:20.430 --> 42:23.310] The second question was about the context-aware version. [42:23.750 --> 42:24.670] First of all, great talk. [42:25.070 --> 42:30.330] The question I had was, how many times have you iterated on, like, the prompt itself? [42:30.710 --> 42:34.770] And have you seen any real changes in the rating itself? [42:35.070 --> 42:38.790] Like, you know, how small word changes could affect the way that you rate things? [42:39.490 --> 42:40.730] Or the way that... [42:40.730 --> 42:40.910] Yeah, yeah. [42:41.390 --> 42:42.470] That's an amazing point. [42:42.650 --> 42:44.410] And I think I can... [42:44.850 --> 42:46.470] We should definitely play around with it more. [42:46.650 --> 42:48.890] I think my prompt was very basic. [42:48.890 --> 42:52.330] It was basically like, here's all of the data I just downloaded. [42:52.370 --> 42:54.530] And here's all of my findings from my current tool. [42:54.730 --> 42:57.030] And I'll try to pull up the prompt now. [42:58.610 --> 42:59.850] It's in GitHub, so it's public. [43:00.270 --> 43:05.350] But I got lazy and I was like, I guess it's not going to get better than that. [43:05.430 --> 43:06.970] But I think that's a very good point. [43:07.110 --> 43:13.730] And you can definitely give it like a smarter prompt and get it to give more meaningful results. [43:14.810 --> 43:16.050] How do I find the prompt? [43:16.050 --> 43:19.370] I think if I do claw dash... [43:20.810 --> 43:22.130] Yeah, so it's in this file. [43:23.370 --> 43:25.250] And so the prompt is this... [43:25.250 --> 43:27.130] Oh, I forgot. [43:27.130 --> 43:28.250] It's all in one line. [43:28.430 --> 43:30.470] But it's also very hard to read. [43:30.630 --> 43:33.410] But I said, you're a security analysis tool called... [43:33.410 --> 43:34.210] I gave it the name. [43:34.370 --> 43:37.070] You don't have to give it a name, but just for like role playing. [43:37.850 --> 43:39.430] You're analyzing Chrome extensions. [43:40.310 --> 43:42.170] Given metrics that are visible to the user. [43:42.830 --> 43:43.850] Analyze the extension... [43:43.850 --> 43:45.950] The risk of the extension and provide a short summary. [43:46.170 --> 43:47.050] Under 200 words. [43:47.430 --> 43:47.970] Because I don't want... [43:47.970 --> 43:49.810] So the other thing about Cloud API is... [43:49.810 --> 43:50.230] It's expensive. [43:50.770 --> 43:52.050] So I wanted to keep it short. [43:52.050 --> 43:55.230] So it's not like costing me $10 per scan. [43:55.390 --> 43:55.490] You know? [43:56.610 --> 43:59.390] I think right now it's like one third of a cent. [43:59.690 --> 44:01.210] Or even less probably. [44:01.350 --> 44:02.730] But it's not too bad right now. [44:02.930 --> 44:04.550] But in general, Cloud APIs... [44:04.550 --> 44:06.350] It does not scale well at all. [44:08.070 --> 44:08.510] Anyway... [44:09.130 --> 44:09.570] So... [44:09.570 --> 44:10.390] Less than 200 words. [44:10.490 --> 44:13.090] And add context to the finding that the user can already see. [44:13.190 --> 44:14.590] Do not repeat the extension's permissions. [44:14.950 --> 44:15.390] Because... [44:15.390 --> 44:16.050] Waste of space. [44:17.090 --> 44:18.550] Some example recommendations include... [44:19.170 --> 44:19.330] If... [44:19.330 --> 44:20.650] So this thing... [44:20.650 --> 44:21.390] I gave it this example. [44:21.570 --> 44:23.810] And it includes it almost every single time. [44:23.810 --> 44:25.010] Which is kind of a waste. [44:25.130 --> 44:25.430] But... [44:25.430 --> 44:26.470] If an extension is very high risk. [44:26.530 --> 44:28.190] The user can run it in a separate Chrome profile. [44:28.630 --> 44:30.110] The response should be formatted as follows. [44:30.270 --> 44:31.530] Include your own risk level. [44:32.170 --> 44:35.110] And then include trust factors based on the extension's description. [44:35.330 --> 44:35.770] Downloads. [44:35.850 --> 44:36.650] Company reputation. [44:37.370 --> 44:38.590] Include a list of concerns. [44:39.150 --> 44:40.630] Including unnecessary permissions. [44:40.830 --> 44:41.130] I mean... [44:41.130 --> 44:42.430] To the best of its ability. [44:43.110 --> 44:43.910] Sometimes it's like... [44:43.910 --> 44:44.590] Hey, like... [44:44.590 --> 44:45.110] Like... [44:45.110 --> 44:46.650] If this is just playing Flappy Bird. [44:46.770 --> 44:47.790] Why does it need... [44:47.790 --> 44:48.570] You know... [44:48.570 --> 44:49.390] Cookie permission? [44:49.990 --> 44:50.230] You know? [44:51.670 --> 44:52.110] Um... [44:52.110 --> 44:53.390] Finally include some recommendations. [44:53.870 --> 44:54.090] Important. [44:54.230 --> 44:54.290] No. [44:54.370 --> 44:55.950] Do not format the response on markdown. [44:56.130 --> 44:56.930] Oh, there's just... [44:56.930 --> 44:57.550] Those are some... [44:57.550 --> 44:58.950] Some wacky things after updated. [44:59.170 --> 45:00.690] Cloud 4 was returning markdown. [45:00.850 --> 45:01.150] But... [45:01.150 --> 45:01.310] Anyway. [45:01.550 --> 45:01.930] Yeah. [45:02.070 --> 45:03.430] So the prompt is extremely basic. [45:05.650 --> 45:06.130] Um... [45:06.130 --> 45:08.610] You could potentially like add something like... [45:08.610 --> 45:10.830] Examples of things that you would consider good. [45:11.030 --> 45:13.210] Examples of things that you would consider like... [45:13.210 --> 45:13.650] You know... [45:13.650 --> 45:14.610] Would be really bad. [45:15.270 --> 45:15.670] Yeah. [45:15.810 --> 45:16.490] Might help it. [45:16.870 --> 45:17.270] But... [45:17.270 --> 45:17.930] Yeah. [45:18.010 --> 45:19.550] There's probably some little prompt engineering. [45:20.450 --> 45:20.850] Yeah. [45:21.050 --> 45:21.190] Definitely. [45:21.410 --> 45:21.610] You can... [45:21.610 --> 45:23.330] You can try a bunch of others and... [45:23.330 --> 45:24.810] And see which one is best. [45:25.410 --> 45:25.810] Um... [45:25.810 --> 45:26.890] But I guess like the... [45:26.890 --> 45:27.990] What is the ideal... [45:27.990 --> 45:29.710] Like what is the result I'm looking for that... [45:29.710 --> 45:31.030] That would make me... [45:31.430 --> 45:32.570] Be happy with it even more. [45:33.890 --> 45:34.290] Um... [45:34.290 --> 45:36.230] You can make it longer and say more details. [45:36.510 --> 45:38.910] But then if it's too long, then it's just annoying to read. [45:38.910 --> 45:39.410] Uh... [45:39.410 --> 45:39.930] And so... [45:39.930 --> 45:40.790] I don't know. [45:41.250 --> 45:41.670] Maybe... [45:42.830 --> 45:43.250] Um... [45:43.250 --> 45:46.430] Instead of like critical, high, medium, maybe you can rank it over 10. [45:46.930 --> 45:47.350] And... [45:47.350 --> 45:47.850] And it... [45:47.850 --> 45:48.810] It takes more things into... [45:48.810 --> 45:49.230] Into account. [45:49.370 --> 45:51.010] So it's like a more granular... [45:51.530 --> 45:51.950] Uh... [45:51.950 --> 45:52.490] I'm not sure. [45:53.390 --> 45:53.910] But yeah. [45:53.970 --> 45:54.670] Good question on the... [45:54.670 --> 45:55.410] On the prompt itself. [45:57.690 --> 45:58.110] Um... [45:58.110 --> 45:59.010] Does the Chrome... [45:59.750 --> 46:03.190] Extension API let you like download old versions or like... [46:03.190 --> 46:03.790] Um... [46:03.790 --> 46:05.910] How much data to give you about like the... [46:06.970 --> 46:07.390] The... [46:07.390 --> 46:08.090] What's it called? [46:08.090 --> 46:09.050] The company behind it. [46:09.350 --> 46:11.490] I'm curious if there's like more stuff you could... [46:12.210 --> 46:12.810] Analyze there. [46:13.270 --> 46:14.270] No, that's such a good one. [46:14.390 --> 46:14.830] I forget. [46:15.670 --> 46:18.490] I think some old versions, but... [46:18.490 --> 46:20.390] To a certain extent, I forgot... [46:20.390 --> 46:21.930] How old you can go. [46:22.090 --> 46:24.250] But what John Tuckner does in his app... [46:24.690 --> 46:26.590] He actually stores old versions. [46:27.230 --> 46:28.250] And he diffs... [46:28.250 --> 46:30.030] When there's an update, he diffs them. [46:30.310 --> 46:31.210] Which is very smart. [46:31.810 --> 46:32.250] And... [46:32.250 --> 46:33.230] And I think that... [46:33.230 --> 46:35.590] Gives you some more hints into... [46:35.590 --> 46:37.610] He actually found a bunch of malware that way. [46:38.430 --> 46:38.810] Um... [46:38.810 --> 46:39.890] Or like malicious Chrome extensions. [46:40.330 --> 46:40.550] Uh... [46:40.550 --> 46:41.110] Because... [46:42.010 --> 46:43.090] You know what they're adding. [46:43.250 --> 46:43.610] It's like... [46:43.610 --> 46:44.670] Why are they adding this? [46:44.770 --> 46:45.510] Why are they adding... [46:45.510 --> 46:46.650] Why are they, you know... [46:46.650 --> 46:48.610] Calling out to this random sketchy domain? [46:48.890 --> 46:49.010] You know? [46:50.130 --> 46:50.510] Um... [46:50.510 --> 46:51.810] But by default, what the... [46:51.810 --> 46:53.270] What the web store allows... [46:53.270 --> 46:53.470] Uh... [46:53.470 --> 46:53.850] I forget. [46:55.810 --> 46:56.190] Um... [46:56.190 --> 46:56.370] Okay. [46:56.430 --> 46:57.170] Maybe like one... [46:57.170 --> 46:57.630] One more minute. [46:58.310 --> 46:58.870] Yeah. [47:00.710 --> 47:01.270] Uh... [47:01.270 --> 47:02.110] Thank you for the talking to me. [47:02.210 --> 47:02.830] This amazing story. [47:03.070 --> 47:03.630] It's really... [47:04.290 --> 47:04.850] Um... [47:04.850 --> 47:06.070] Something I've definitely... [47:06.070 --> 47:06.650] Consider using. [47:07.190 --> 47:08.630] Have you ever considered... [47:09.370 --> 47:09.550] Um... [47:09.550 --> 47:10.230] Trying to... [47:27.050 --> 47:27.610] Uh... [47:30.010 --> 47:30.570] Yeah. [47:30.730 --> 47:30.790] No. [47:30.930 --> 47:31.290] That's... [47:31.290 --> 47:32.110] That's a very... [47:32.110 --> 47:32.430] Um... [47:32.430 --> 47:33.130] That's a very... [47:33.130 --> 47:34.150] Um... [47:34.150 --> 47:34.650] Good... [47:35.670 --> 47:36.230] Suggestion. [47:37.330 --> 47:37.890] Um... [47:37.890 --> 47:38.310] My... [47:38.310 --> 47:39.090] Actually would... [47:39.090 --> 47:40.270] Would be nice to add. [47:40.490 --> 47:40.750] For sure. [47:42.750 --> 47:43.310] Um... [47:43.310 --> 47:43.870] CVSS. [47:43.870 --> 47:44.450] I'm not sure. [47:44.590 --> 47:45.050] I... [47:45.050 --> 47:45.290] I... [47:45.290 --> 47:46.750] I don't know if I like CVSS. [47:46.750 --> 47:47.590] In general. [47:49.590 --> 47:49.870] Um... [47:49.870 --> 47:50.530] But... [47:50.530 --> 47:50.990] But... [47:50.990 --> 47:51.030] That... [47:51.030 --> 47:51.190] That... [47:51.190 --> 47:52.910] That would make the tool more mature for sure. [47:53.130 --> 47:53.350] So... [47:53.350 --> 47:54.010] That's a good idea. [47:58.460 --> 47:58.880] Um... [47:58.880 --> 47:59.440] Um... [47:59.440 --> 48:00.380] Maybe one more question. [48:05.200 --> 48:05.620] Okay. [48:06.340 --> 48:06.740] Oh. [48:06.900 --> 48:07.200] Hector? [48:13.780 --> 48:15.100] Do I feed the source code? [48:15.800 --> 48:16.220] Uh... [48:16.220 --> 48:16.940] I think I did. [48:17.140 --> 48:17.940] It's been like... [48:17.940 --> 48:18.600] Almost a year. [48:18.680 --> 48:19.160] So I forget. [48:19.420 --> 48:19.900] But I think... [48:19.900 --> 48:20.380] I think I... [48:20.380 --> 48:21.520] I send it... [48:22.080 --> 48:22.500] Um... [48:23.020 --> 48:26.380] I either send it all of my findings and the snippets that led to them. [48:26.620 --> 48:27.480] Or the entire... [48:27.480 --> 48:28.660] Or the entire... [48:30.440 --> 48:30.920] Um... [48:30.920 --> 48:31.120] Yeah. [48:31.220 --> 48:33.880] I forget if I send the manifest itself or the entire... [48:33.880 --> 48:34.480] Uh... [48:34.480 --> 48:34.740] File. [48:34.940 --> 48:35.140] That's... [48:35.140 --> 48:36.060] That's a very good... [48:36.640 --> 48:37.120] So... [48:37.120 --> 48:37.700] Actually... [48:37.700 --> 48:38.020] Wait. [48:38.120 --> 48:38.780] You just gave me an idea. [48:38.920 --> 48:41.460] Because Claude just released the security review... [48:41.460 --> 48:41.720] Uh... [48:42.300 --> 48:42.780] Ah. [48:43.320 --> 48:44.440] I saw this... [48:46.880 --> 48:49.200] This massive thing that Microsoft is in... [48:49.200 --> 48:50.440] I think. [48:50.680 --> 48:50.780] Yeah. [48:50.940 --> 48:52.200] And basically... [48:52.200 --> 48:52.220] They... [48:52.220 --> 48:53.300] They wanted you to... [48:53.300 --> 48:55.820] Every binder that you can run is gonna... [48:55.820 --> 48:57.000] Go through a model... [48:57.000 --> 48:59.160] That is gonna try to explain what it's doing. [48:59.360 --> 49:00.200] What it's capable of doing. [49:00.500 --> 49:00.700] Oh. [49:01.640 --> 49:02.100] Uh... [49:02.100 --> 49:02.280] And... [49:02.280 --> 49:03.420] Do a little guide assessment. [49:03.740 --> 49:04.580] And here... [49:04.580 --> 49:04.640] Okay. [49:04.740 --> 49:05.520] If you do it... [49:06.020 --> 49:07.820] To the intention. [49:08.840 --> 49:10.000] Or maybe you can... [49:10.900 --> 49:12.440] Have permission to imagine... [49:12.440 --> 49:13.560] And then we call it just... [49:13.980 --> 49:14.260] You know... [49:14.260 --> 49:15.300] A little footprint... [49:15.300 --> 49:16.360] That is additional... [49:16.360 --> 49:17.080] If you want it. [49:17.140 --> 49:17.460] Wow. [49:17.980 --> 49:18.380] So... [49:18.860 --> 49:19.260] Microsoft... [49:19.260 --> 49:21.080] Announced that as a project that they want to do? [49:21.220 --> 49:22.600] They are really working on that. [49:23.100 --> 49:23.500] I... [49:24.460 --> 49:24.860] May... [49:24.860 --> 49:26.260] I doubt very much that they... [49:28.680 --> 49:30.520] But there's no website that you can share. [49:30.760 --> 49:31.840] That seems very cool. [49:32.800 --> 49:33.600] I think... [49:33.600 --> 49:35.760] It's probably a bit hard to determine intent. [49:36.280 --> 49:36.680] Although... [49:36.680 --> 49:37.000] Here... [49:37.000 --> 49:37.920] In the Chrome Web Store... [49:37.920 --> 49:39.540] You can fetch kind of like what the description is. [49:39.660 --> 49:40.680] So you can kind of tell if... [49:40.680 --> 49:41.900] If someone's being sketchy. [49:42.000 --> 49:44.000] At least match might be another data point. [49:44.220 --> 49:44.520] But... [49:44.520 --> 49:45.540] I don't know how it starts to... [49:45.540 --> 49:46.860] You know... [49:46.860 --> 49:47.500] Yeah... [49:47.500 --> 49:48.740] So that's the thing. [49:48.860 --> 49:49.940] With all AI tools... [49:49.940 --> 49:51.680] The cost is really the main problem right now. [49:52.420 --> 49:52.860] Um... [49:52.860 --> 49:54.020] Like the... [49:54.760 --> 49:57.020] Has anyone played with the cloud security review? [49:59.160 --> 49:59.600] Um... [49:59.600 --> 50:00.680] Thing like that came out. [50:00.820 --> 50:01.680] Like you can basically... [50:01.680 --> 50:02.720] Give cloud security review. [50:02.880 --> 50:03.240] And... [50:03.240 --> 50:04.120] I ran it on... [50:04.120 --> 50:05.280] On one pull request... [50:05.280 --> 50:07.160] That only had a hundred lines of code. [50:07.500 --> 50:08.840] And it was a dollar fifteen. [50:09.720 --> 50:11.880] So like imagine spending a dollar... [50:11.880 --> 50:12.800] Ten thousand... [50:12.800 --> 50:14.900] I would have spent eleven thousand dollars for... [50:14.900 --> 50:15.400] For... [50:15.400 --> 50:16.480] On this website. [50:16.600 --> 50:17.600] Like I don't wanna do that. [50:17.880 --> 50:18.220] So... [50:18.220 --> 50:19.740] So the main problem is the cost. [50:20.000 --> 50:20.320] But... [50:20.840 --> 50:21.120] But... [50:21.760 --> 50:22.040] Yeah... [50:22.040 --> 50:22.620] I... [50:22.620 --> 50:22.780] I... [50:22.780 --> 50:24.640] I wanna try it locally to see if... [50:24.640 --> 50:25.280] If... [50:25.280 --> 50:26.020] Running it locally... [50:26.020 --> 50:27.560] Would give meaningful results. [50:27.720 --> 50:29.780] For premium service. [50:38.840 --> 50:39.000] There... [50:39.000 --> 50:39.960] At some point it's gonna die. [50:40.080 --> 50:40.280] I'm not... [50:40.280 --> 50:41.580] I'm not even gonna bother to fix it. [50:43.380 --> 50:43.660] Okay. [50:43.780 --> 50:44.340] Thank you so much. [50:44.620 --> 50:44.940] Thank you so much. [50:44.940 --> 50:45.240] Thank you. [50:45.240 --> 50:45.320] Thank you so much.