[01:11.040 --> 01:12.080]  All right, everyone.
[01:12.280 --> 01:15.080]  Welcome to the next talk in our track three.
[01:15.460 --> 01:18.820]  Just a couple lines of business before we go into the talk itself.
[01:19.080 --> 01:19.800]  A couple reminders.
[01:20.560 --> 01:21.480]  It's really important.
[01:21.480 --> 01:24.200]  We ask you to please continue to wear your mask when you're indoors.
[01:24.240 --> 01:26.400]  If you need to take them off, step outside.
[01:26.640 --> 01:28.640]  Masks are fine outside.
[01:29.040 --> 01:30.800]  But as long as you're indoors, we really appreciate it.
[01:30.860 --> 01:33.660]  It helps us keep consistent with our policy and our code of conduct.
[01:34.500 --> 01:35.840]  Please try to stay hydrated.
[01:36.140 --> 01:40.340]  We don't want anybody to not be able to enjoy the rest of the conference or any other talks or the same.
[01:40.360 --> 01:41.200]  sessions or the tracks.
[01:41.920 --> 01:42.840]  So take care of yourself.
[01:44.500 --> 01:46.320]  Please mute your phone while you're in the session.
[01:46.680 --> 01:53.200]  The audio equipment is very sensitive and so it picks up any type of beeps or bings or whatever noise your phone makes.
[01:53.540 --> 01:56.060]  We have a fourth unscheduled track for talks.
[01:56.380 --> 02:00.240]  It is open to anybody who would like to give some sort of presentation or talk.
[02:00.840 --> 02:03.300]  It's in the coffee shop or in the coffee room.
[02:03.680 --> 02:05.340]  You go to the info desk to sign up.
[02:05.520 --> 02:06.620]  Just tell them what you want to talk about.
[02:06.780 --> 02:09.400]  They'll help you find a slot and you can go give your talk.
[02:10.360 --> 02:13.480]  It's entirely self-service, so you need to bring your own laptop and plug it in.
[02:13.620 --> 02:15.080]  We don't have any equipment to help you with that.
[02:15.620 --> 02:17.800]  Hacker Karaoke is tonight at 10 p.m.
[02:18.040 --> 02:21.900]  on track one, so it will be upstairs in DAC 416 on the fourth floor.
[02:22.960 --> 02:24.200]  Please come if you can.
[02:24.300 --> 02:25.200]  It should be a lot of fun.
[02:25.340 --> 02:29.200]  I won't be singing, thankfully, but lots of people who can sing well will be.
[02:29.860 --> 02:31.420]  There will be some judges there who judge.
[02:31.560 --> 02:32.360]  It is a competition.
[02:33.040 --> 02:35.100]  And lastly, volunteers are welcome.
[02:35.220 --> 02:37.620]  There's lots of volunteer opportunities for the rest of the conference.
[02:37.620 --> 02:44.080]  So please, if you're interested, stop in room 301 or go to the information desk and let them know what you would like to do to volunteer.
[02:44.760 --> 02:51.560]  So the talk right now is botnets are the best way to measure user hostile behavior on the Internet with David Seedy.
[02:51.780 --> 02:53.440]  So with that, I'll pass it off to David.
[02:53.700 --> 02:54.020]  Thank you.
[02:55.220 --> 02:56.260]  Thank you very much.
[02:57.540 --> 03:04.220]  Someone working at NSA once gently informed Bruce Schneier that worrying about automated surveillance was silly.
[03:04.720 --> 03:07.460]  It's like worrying about your dog seeing you naked.
[03:11.370 --> 03:13.150]  In many ways, that comparison is wrong.
[03:13.390 --> 03:21.370]  You can't communicate with... your dog can't communicate with others and it can't make decisions based on what it sees that will have a long-term impact on your life, for example.
[03:22.610 --> 03:31.790]  But one way it's not wrong is that, as we all know, on the Internet, nobody knows you're a dog or part of an automated surveillance botnet of user hostile behavior on the web.
[03:33.090 --> 03:35.990]  That means you can catch a service with its pants down.
[03:36.850 --> 03:38.990]  And this can be a good thing for everyone.
[03:41.650 --> 03:48.310]  So Sheldon Whitehouse is a senator and he was wrong when he said there's no good use for a botnet.
[03:49.650 --> 03:55.490]  I'm going to talk about botnets for measurement of bad behavior on the web as one example of beneficial botnets.
[03:55.690 --> 03:58.290]  But first, I want to talk about some other good uses.
[03:59.630 --> 04:05.770]  The general idea of coordinating use of volunteers' home systems to do beneficial things isn't new.
[04:06.190 --> 04:11.070]  The big example is that botnets have been used for distributed computing tasks.
[04:13.070 --> 04:27.530]  The first project of this sort that I personally became familiar with was Folding at Home, which today has more than 73,000 active nodes and focuses on discovering treatments for a broad range of diseases by simulating protein folding.
[04:27.530 --> 04:32.710]  This was way back when I was on my Packard Bell and my 14.4 modem.
[04:33.870 --> 04:39.110]  But the real first volunteer distributed computing project actually preceded Folding at Home.
[04:39.250 --> 04:41.610]  It was distributed.net in 1997.
[04:42.850 --> 04:44.370]  And I have links there.
[04:44.990 --> 04:48.630]  If you're interested in the history, it's actually very interesting to read.
[04:50.410 --> 04:55.790]  So one volunteer computing project on distributed.net aimed to break reduced round RC5 encryption.
[04:56.390 --> 04:59.730]  And there's a team of hackers from L0pht that participated.
[05:00.970 --> 05:03.830]  Similarly, COVID at Home was a Folding at Home project.
[05:04.670 --> 05:10.730]  And the RootFolds team, which was started by people from RootOwns, gave a talk at HOPE 2020.
[05:11.290 --> 05:16.610]  If you happen to see it, if you didn't, go back and watch that talk.
[05:17.730 --> 05:22.070]  So hackers have been involved in this stuff from the beginning and continue to be.
[05:22.910 --> 05:25.070]  And I think it's not hard to see why.
[05:26.190 --> 05:36.590]  David P. Anderson is the guy behind Boink, which is an infrastructure used by several distributed volunteer computing projects of this kind, including the very early SETI at Home.
[05:37.150 --> 05:42.870]  So he's not a hacker as far as I know, but he voiced a very kind of hacker-y idea.
[05:44.090 --> 05:47.110]  He said, my worldview is deeply anti-establishment.
[05:47.390 --> 05:48.790]  I root for the underdog.
[05:49.030 --> 05:58.130]  So the idea of volunteer computing, which shifts power away from corporations and the academic establishment and puts it in the hands of the people resonated strongly with me.
[05:59.110 --> 06:02.810]  He also went on to say, I have problems with authority figures.
[06:03.830 --> 06:07.090]  So to me, privacy and hacking is tied to this mentality.
[06:07.470 --> 06:13.250]  Groups of people coordinating to use technology to empower themselves directly without dependence on others.
[06:14.710 --> 06:26.450]  Boink is also interesting because it's not just a particular project, a distributed computing project, but it provides a platform and an infrastructure for developing new projects effectively or efficiently.
[06:27.390 --> 06:30.410]  And that's what I think we need for measurement botnets too.
[06:31.930 --> 06:48.330]  So these distributed computing projects are definitely examples of beneficial botnets, but from the beginnings and through to today, most of them have been focused on basic science projects rather than on sort of an applied measurement of the web.
[06:48.930 --> 06:50.810]  There are exceptions to that.
[06:51.270 --> 06:52.910]  Surveil at home is one.
[06:53.090 --> 06:58.190]  It used Boink for web measurement, but it hasn't been active since 2011, I think.
[06:58.850 --> 07:01.390]  But I'm interested to hear if you know something else about that.
[07:02.350 --> 07:07.110]  And there may be a few other examples that I've missed, but for the most part, they're not used for measurement.
[07:07.690 --> 07:11.510]  But there are projects doing distributed volunteer web measurements.
[07:13.250 --> 07:22.570]  Some sites attack the privacy of their visitors, and I'll call that user hostile behavior on the web for short.
[07:24.630 --> 07:28.150]  One use of web measurement is to record user hostile behavior.
[07:28.450 --> 07:31.990]  And so I want to talk about a particular case of that, dark patterns.
[07:33.150 --> 07:37.970]  So dark patterns are a kind of user interface design pattern.
[07:37.970 --> 07:45.350]  In general, user interface design patterns arise from common problems or use cases that designers have to handle.
[07:45.690 --> 07:50.750]  For example, a customer needs to keep track of their purchases and then provide payment and shipping information.
[07:51.010 --> 07:54.630]  Or a reader needs to find contact information on a website.
[07:57.530 --> 08:02.630]  So what are the problems or the common problems that give rise to dark design patterns?
[08:03.030 --> 08:05.230]  And what are the approaches to solving them?
[08:06.290 --> 08:14.230]  So the problems solved by dark patterns, first of all, belong to the service provider and not to you, the customer, visitor, or other user of the service.
[08:14.230 --> 08:21.810]  In fact, your behavior is what is treated as a user interface design problem addressed by dark patterns.
[08:23.550 --> 08:35.510]  The problems designers are trying to fix are things like stopping a visitor from declining to receive promotional emails while conveying to those who never tried to decline that they have some choice in the matter.
[08:36.710 --> 08:39.810]  Or stopping a customer from canceling a service.
[08:40.270 --> 08:49.270]  So this example is from Project Iliad, a project by Amazon to make cancellation difficult with dark patterns that was revealed in a leak this year.
[08:49.790 --> 08:55.710]  So here's what it looked like to try to cancel with Amazon at the time.
[08:58.010 --> 09:07.030]  Go to the website, click the hamburger or the menu button, scroll down to find the prime membership option, click manage membership.
[09:07.830 --> 09:09.950]  You're then given some options in gray.
[09:10.250 --> 09:12.350]  You click the lower one, which says end membership.
[09:12.830 --> 09:14.270]  Seems legit so far.
[09:14.730 --> 09:23.250]  But then you're presented with a screen you have to scroll down through and read about the exclusive benefits you will lose and look at a warning symbol.
[09:24.690 --> 09:27.910]  Those include prime free delivery, prime video.
[09:28.330 --> 09:30.330]  There's several things you have to scroll through.
[09:30.430 --> 09:31.470]  It's a pretty lengthy scroll.
[09:31.750 --> 09:42.810]  And then you get to a sort of stack of buttons and you have to pick not the blue one on top, which says use your benefits, but the central orange one, which says cancel my benefits.
[09:44.410 --> 09:50.230]  After you click on that, you have to scroll again and read about how you'll save money by switching to annual payments.
[09:51.270 --> 09:55.330]  Keep scrolling to learn about how canceling you won't be eligible for unclaimed offers.
[09:56.070 --> 09:58.570]  Then you'll get to another stack of buttons.
[09:58.990 --> 10:03.970]  And once again, you might think you could click one of those to cancel.
[10:04.130 --> 10:07.310]  But in fact, nope, none of those lets you cancel your subscription.
[10:07.430 --> 10:08.590]  You have to continue to scroll.
[10:08.590 --> 10:17.250]  And then at the very bottom of that resulting stack, there is an option to try to cancel.
[10:18.250 --> 10:19.630]  It says end now.
[10:20.210 --> 10:27.610]  And if you click that, it will tell you that your subscription will be canceled in a month or so.
[10:29.370 --> 10:33.730]  So this came from the Norwegian data protection authority.
[10:34.590 --> 10:35.450]  It's on YouTube.
[10:35.670 --> 10:36.850]  I gave the Nvidia's link there.
[10:37.290 --> 10:42.390]  If you're interested to watch it, it's kind of good to watch moving.
[10:47.030 --> 10:49.030]  So firstly, it would be absurd.
[10:49.130 --> 10:49.590]  Let me go back.
[10:49.710 --> 10:57.330]  It would be absurd to contact Amazon, even if there were some means to do that without just flapping your wings hard enough on Twitter to shift the zeitgeist.
[10:58.150 --> 11:02.390]  It'd be absurd to reach out and complain about the inability to cancel using their site.
[11:03.030 --> 11:08.930]  The problem is not that Amazon's designers messed up and they're unaware of the effects of this cancellation process for visitors.
[11:08.930 --> 11:11.830]  The design of that process is not an accident.
[11:12.130 --> 11:19.290]  On the contrary, it's often the result of very careful A-B testing against huge samples drawn from the service's users.
[11:21.950 --> 11:29.490]  So to understand dark patterns like these, you have to understand that people hate them, but that it doesn't matter.
[11:29.670 --> 11:31.730]  That doesn't matter to the people who use them.
[11:32.350 --> 11:34.170]  So people do hate dark patterns.
[11:34.170 --> 11:40.450]  You can see there are many name and shame sites where users take time out of their day to complain in great numbers.
[11:41.670 --> 11:43.890]  Here's our asshole design.
[11:44.810 --> 11:50.570]  There's also a tip line called Futurity Tip Line where people can just complain.
[11:51.430 --> 11:57.210]  There's a darkpatterns.games, which focuses more on addictive sorts of patterns.
[11:57.450 --> 11:59.150]  There's a darkpatterns hashtag.
[11:59.150 --> 12:00.450]  You can get on Mastodon.
[12:01.090 --> 12:07.640]  They're all vigorously stocked on an ongoing basis with comments from exasperated users.
[12:08.850 --> 12:18.330]  So that's how the term darkpatterns caught on from this sort of scattered reports from frustrated users wanting to name and shame services with a particular kind of user interface design.
[12:18.330 --> 12:23.030]  And that happened for long enough that abstractions started to become invisible.
[12:23.390 --> 12:32.550]  A similar kind of UI solution to these recurring problems that services were having with users and were fixing with UI.
[12:34.310 --> 12:37.500]  So darkpatterns emerged in plain sight from user dissatisfaction.
[12:37.910 --> 12:44.570]  It's not something that was revealed by research work about recondite practices that had to be uncovered.
[12:44.570 --> 12:46.450]  These are flagrant.
[12:46.750 --> 12:47.530]  They're insulting.
[12:48.650 --> 12:53.470]  They're contemptuous of the user in that they don't even really bother to hide.
[12:55.250 --> 12:57.690]  At the same time, the web is a big place.
[12:57.950 --> 13:02.730]  Visitors are busy and mostly won't report even openly hostile user interfaces.
[13:02.730 --> 13:08.190]  And the variety and implementations of darkpatterns is large and they're evolving.
[13:09.170 --> 13:15.110]  So something more systematic is needed to maintain an updated empirical understanding of darkpattern use.
[13:15.470 --> 13:18.770]  And that, friends, is where web measurement comes in.
[13:19.730 --> 13:29.750]  So darkpatterns is a kind of nice example for discussing measurement because they're hard to model and so are hard to automatically measure with a kind of a trained classifier approach.
[13:29.750 --> 13:36.510]  But they also exist on the web in a zillion different places, which makes them hard to measure just manually.
[13:38.250 --> 13:39.950]  You need sort of components of both.
[13:40.730 --> 13:40.970]  Okay.
[13:41.350 --> 13:49.050]  So let's talk about the two common approaches to web measurement and why I think neither of them is as good as botnets.
[13:49.910 --> 13:52.090]  One approach is bot farming.
[13:52.410 --> 14:01.770]  And that's to kind of create and manage a fleet of automated browsers that you host yourself, like a bunch of VPSs or servers in a university lab.
[14:02.430 --> 14:06.650]  And that have some sort of instrumentation on the browser that supports measurement.
[14:08.010 --> 14:14.270]  The other approach is called piggybacking, which is to convince a large number of users to allow their own browsers to be instrumented.
[14:14.270 --> 14:20.170]  So that's like using a special browser extension so that the sites that they visit can be measured.
[14:22.130 --> 14:22.730]  Okay.
[14:22.850 --> 14:24.750]  So let's start with farming, bot farming.
[14:25.050 --> 14:30.370]  So creating a fleet of bespoke automated browser bots reproduces a lot of effort.
[14:30.370 --> 14:44.130]  Lots of studies of the web share requirements, including things like invoking a specifiable number of bots or recording the requests and responses as a bot interacts, supporting keyword searches.
[14:45.670 --> 14:52.150]  It makes little sense to reimplement these features every time a new web measurement study is conducted.
[14:53.330 --> 15:07.310]  Unfortunately, in practice, adopting some existing web measurement project doesn't help much because a large amount of the existing code in this area is unmaintained or it's difficult to understand or modify or doesn't build without special knowledge.
[15:07.590 --> 15:18.810]  Particularly for people who are trained not to produce such things, but to kind of design experiments and analyze data or to understand privacy, you know, legal enforcement to do with privacy.
[15:20.490 --> 15:33.790]  Bot farming also requires a farm, the infrastructure for running the bots, and that can be expensive in terms of coin and time, putting pressure on long term projects, long running ones, like you might use for monitoring.
[15:34.170 --> 15:45.530]  So, for example, for an academic, there's little incentive to maintain the necessary infrastructure for a measurement study once the article that it's sort of directed towards has been published.
[15:45.530 --> 16:03.190]  So, the benefits of a one-off kind of bot farm project are usually pretty limited for a regulator like the FTC, Federal Trade Commission in the United States, who might want... who enforces privacy laws at the federal level, and who might want to monitor the web over time,
[16:03.830 --> 16:06.470]  rerunning the study repeatedly to sweep for new developments.
[16:08.850 --> 16:16.730]  Another reason bot services aren't ideal for web measurements is that a bot's behavior at a website has to be automated, in contrast to piggybacking, which we'll see in a moment.
[16:17.030 --> 16:21.270]  And that kind of artificial behavior can be detected, leading sites to alter their behavior.
[16:21.490 --> 16:24.210]  So, that's a kind of demand characteristic, if you want.
[16:24.270 --> 16:30.770]  In the study, the thing being measured is changing based on its understanding of being measured.
[16:31.570 --> 16:35.190]  And that's a challenge to external validity in web measurement studies.
[16:37.770 --> 16:48.510]  Finally, whereas economically and administratively, sharing resources among the bots in a farm kind of makes sense, viewpoint diversity improves measurement quality.
[16:48.730 --> 16:53.150]  But achieving good diversity is hard with centrally controlled infrastructure.
[16:53.150 --> 17:02.670]  You have to sort of try to get your hands on resources that are spread around diverse...
[17:04.270 --> 17:08.930]  in diverse places on the network, for whatever that means in your context.
[17:10.510 --> 17:15.090]  So, this is something, for example, that studies of censorship are very interested in, right?
[17:15.170 --> 17:18.430]  Diverse vantage points, very crucial to understanding how censorship really works.
[17:20.450 --> 17:21.090]  Okay.
[17:21.510 --> 17:23.570]  So, how about piggybacking?
[17:23.750 --> 17:25.230]  That's the other approach I mentioned.
[17:25.230 --> 17:33.390]  It's convincing users to run some instrumentation in an extension for a measurement study on their own browsers.
[17:33.990 --> 17:36.070]  So, that comes with its own problems.
[17:37.310 --> 17:47.210]  A nice example of it is Ad Observer, which is an NYU project that uses an extension to measure targeted political ads from Facebook and Google.
[17:47.210 --> 17:49.510]  It grew out of Ad Observatory.
[17:49.850 --> 17:53.870]  And at one point, a few years ago, Facebook sent a...
[17:53.870 --> 17:57.550]  I think it was actually before the 2020 election, I want to say.
[17:58.150 --> 18:07.210]  Facebook sent a threatening letter to Ad Observatory, which was a predecessor project, and suspended accounts that they detected to be using the tool.
[18:08.830 --> 18:18.810]  And I wondered if that might have something to do with Ghost Owl, which is a Facebook team discussed in an article by John Payne in 2600 a while back.
[18:19.770 --> 18:24.070]  And Ghost Owl apparently works to circumvent privacy tools for Facebook.
[18:27.590 --> 18:39.450]  And to me, you know, it's interesting to reflect on how these companies have formalized a practice for circumventing users' attempt to protect their privacy with these teams.
[18:39.630 --> 18:40.850]  Teams like Ghost Owl.
[18:41.050 --> 18:42.290]  Maybe it's not called that anymore.
[18:42.430 --> 18:43.750]  This is a few years ago.
[18:43.870 --> 18:46.450]  I think the article is a little older, but not that old.
[18:47.010 --> 18:50.670]  And also with projects like Project Iliad, which we saw before from Amazon.
[18:51.230 --> 18:54.070]  So if you know more about those, please do reach out.
[18:54.150 --> 18:56.030]  I'm interested to learn more.
[18:58.910 --> 18:59.430]  Okay.
[19:00.030 --> 19:05.430]  But maybe the most obvious problem with piggybacking is that instrumentation is a burden on the user's browsing.
[19:05.670 --> 19:09.790]  It slows it down, and it adds to personal privacy risk for them, right?
[19:09.790 --> 19:17.190]  So they're participating in a study which is maybe measuring privacy practices or dark patterns or something else that's user hostile on the web.
[19:17.190 --> 19:27.890]  So they're contributing to our collective privacy, but they're also enduring some personal privacy risk for running this thing on their own browser.
[19:28.870 --> 19:41.270]  The example I always think of is there was something called the Security Behavior Observatory, run by Carnegie Mellon, CMU, that collected data related to privacy and security behaviors from participant systems.
[19:41.270 --> 19:43.690]  So they were interested in usability questions in particular.
[19:45.110 --> 19:57.790]  And sometimes this meant, you know, collecting in particular for sensitive content, you know, something that they've gathered in incognito mode to try and understand how people use private browsing or incognito mode.
[19:58.790 --> 20:00.990]  So, obviously, that work is very important.
[20:01.230 --> 20:04.330]  The studies were valuable additions to our understanding of privacy and usability.
[20:04.630 --> 20:10.290]  But it's also clear that the privacy threat, there's a privacy threat for participants, and that's in those kinds of studies.
[20:12.590 --> 20:17.470]  Another problem is that piggybacking measurements are taken on sites that the user happens to visit.
[20:17.750 --> 20:23.530]  So as a crawl strategy, whatever happens to be visited is sometimes desirable, but sometimes not.
[20:23.530 --> 20:28.310]  So, particularly for studies with only a small number of participants, it doesn't work very well.
[20:29.450 --> 20:32.290]  And with piggybacking, there's no real choice in the matter.
[20:35.260 --> 20:47.140]  So, despite our experience of dark patterns as overt when we encounter them, I just want to point out that they rely on these kind of subtle features of human psychology.
[20:47.720 --> 20:52.700]  How to word a warning that's presented to you when you try to take an action that the server doesn't want you to take.
[20:53.320 --> 20:59.280]  At what length a path of links is likely to wear people down enough to give up cancelling their Amazon subscription.
[20:59.980 --> 21:02.500]  How many or how much scrolling they have to do.
[21:03.440 --> 21:07.900]  How to make clear that a decision will result in a future bother forever.
[21:08.220 --> 21:14.140]  You have a maybe later button instead of a yes and a maybe later, which is left open.
[21:14.140 --> 21:21.740]  How to color the accept and decline buttons in a dialogue asking for consent to share your information so that declining is less likely.
[21:23.500 --> 21:39.900]  The prevalence of dark patterns on the web and their use of psychology like this that's difficult to model means that to measure them, or especially to monitor them over time, you need to combine automation with manual human judgment.
[21:39.900 --> 21:43.380]  You need a measurement botnet.
[21:46.040 --> 21:52.860]  So in a measurement botnet, the bots performing measurements are hosted on a volunteers, let's say home machine.
[21:53.700 --> 21:56.800]  Volunteers install a client that manages their bot.
[21:57.200 --> 22:02.740]  The client is responsible for pulling down new studies, running them and returning results.
[22:03.440 --> 22:14.780]  In our implementation, which is called Mbotnet, we built the client UI with Argos, which parses the output of shell scripts and generates a kind of simple panel drop-down menu in GNOME.
[22:16.100 --> 22:28.060]  There's a related project called XBar that uses the same syntax to generate drop-downs in the menu bar of Darwin for Mac OS.
[22:28.060 --> 22:31.720]  So you get that kind of for free by using either one of the two.
[22:32.380 --> 22:33.820]  They're both great projects.
[22:33.920 --> 22:37.320]  They're sort of like a Zenity on steroids, if you've ever used Zenity.
[22:39.160 --> 22:46.520]  We try to isolate in our implementation the client from the host system and to wait until system resources become available to run.
[22:46.860 --> 22:50.220]  These are lessons learned a long time ago from distributed computing projects.
[22:51.240 --> 22:58.140]  We also try to claim only a fraction of available resources on the system while it's when the client is running.
[22:59.080 --> 23:11.420]  So for the right mix of broad availability and kind of, you know, possibility of installation and security, we implemented the client side isolation on Linux with systemd sandboxing.
[23:13.920 --> 23:18.640]  We're building for Linux and Mac OS first, but I just want to say I also love BSD.
[23:18.840 --> 23:21.340]  I would love to expand if I can find any help.
[23:23.440 --> 23:27.440]  So I'm happy to get into the details of what we did for isolation if you're interested.
[23:27.440 --> 23:32.140]  But to summarize, we used kind of straightforward set comp filtering.
[23:32.620 --> 23:39.840]  We restricted capabilities, use namespaces, and then to manage client resources, we use cgroups.
[23:40.540 --> 23:44.940]  With Mac OS, we basically don't know how it's going to go yet.
[23:46.080 --> 23:52.820]  I initially thought since it was BSD based, I could use jails, but I think Mac OS doesn't doesn't do jails.
[23:54.140 --> 23:56.360]  I'm interested to know more if you have a view on that, though.
[23:57.620 --> 23:57.820]  Okay.
[23:59.740 --> 24:04.740]  So when the client is running, the user volunteer can schedule ride-alongs to see what the bot is doing.
[24:04.740 --> 24:07.680]  This is sort of a not that interesting as a picture.
[24:07.720 --> 24:20.400]  It's just a Selenium instance running, but it was run by clicking in that drop-down menu, schedule a ride-along, and the next site that was visited by the bot was shown to the user.
[24:21.280 --> 24:26.740]  And this is also the functionality that allows us to ask the user for manual judgments.
[24:27.440 --> 24:28.840]  So I'll get back to that in a second.
[24:30.820 --> 24:45.440]  The automated browser code we use is a fork of OpenWPM, which has been used for some great bot farm style projects, like the web census, and actually for a study, kind of a one-off study, but a really interesting one on dark patterns.
[24:45.440 --> 24:49.940]  It's developed by the people at CITP and still actively maintained.
[24:50.380 --> 24:53.380]  And it has some great instrumentation that's implemented.
[24:53.980 --> 24:57.180]  And we're hoping to expand on it and kind of share back upstream.
[25:00.020 --> 25:07.560]  The client connects to the server to get more studies whenever it has extra capacity, rather than allowing connections to come in from the server.
[25:07.560 --> 25:15.340]  And we did that because it's easy, it doesn't require opening up ports, and it reuses functionality we need anyway to administer the server.
[25:16.120 --> 25:21.900]  A lot of this is beating a direct path to what we wanted.
[25:22.280 --> 25:25.880]  And it's sort of up in the air still, we're looking at design decisions.
[25:26.140 --> 25:31.260]  But we wanted to have something we could work with and use to understand the issues as well.
[25:32.020 --> 25:33.380]  You know, sort of running.
[25:35.040 --> 25:43.680]  So once the study is complete, it's compressed, the results are compressed and signed and returned to the server, subject to disk quota.
[25:43.980 --> 25:46.680]  And it's made accessible through a researcher portal.
[25:47.220 --> 25:52.680]  Researcher portal just allows researchers to apply to register and to specify studies.
[25:52.680 --> 25:54.880]  So I said apply to register.
[25:55.120 --> 25:55.420]  I don't...
[25:55.420 --> 25:59.780]  We're not going to allow anyone to register to run studies on this.
[26:00.100 --> 26:02.180]  At least, we don't think we will.
[26:03.140 --> 26:05.980]  How we're going to do vetting, though, is still open.
[26:05.980 --> 26:07.600]  So we're interested to think more.
[26:07.760 --> 26:10.560]  We're sort of thinking about that right now.
[26:12.820 --> 26:17.620]  Study designs include a variety of options, including things like whether to anonymize.
[26:17.620 --> 26:21.480]  You know, you can have the bot work over Tor.
[26:22.580 --> 26:25.400]  Whether to try to resist bot detection.
[26:26.260 --> 26:32.060]  And then some more detailed configuration parameters for the browser done through UserJS.
[26:33.560 --> 26:36.120]  And again, we hope to expand on these.
[26:36.300 --> 26:41.120]  If you have ideas about things that would be interesting, we'd like to know.
[26:42.500 --> 26:46.680]  One important option is whether to include in the study manual judgments to measure something.
[26:47.000 --> 26:53.220]  Because that helps us to sidestep the problem of modeling phenomena that are difficult to measure automatically.
[26:54.620 --> 27:01.500]  So for example, the researcher can schedule a manual judgment about the wording used on a button to sign up for a subscription.
[27:01.840 --> 27:05.500]  And it might be a question of whether this is high-pressure shaming of the user.
[27:07.580 --> 27:11.460]  It says, yes, send me the meal plan in a big green button.
[27:12.160 --> 27:15.040]  Or, I can't actually...
[27:15.040 --> 27:17.600]  No thanks, I don't like delicious food in grey.
[27:18.440 --> 27:20.560]  Underneath it, very faintly with low contrast.
[27:21.540 --> 27:23.960]  So is that shaming the user?
[27:24.140 --> 27:27.520]  Or maybe is it too high pressure for a particular group of users?
[27:27.520 --> 27:32.280]  Maybe the elderly or someone who's buying something to get help with a problem they're ashamed of?
[27:32.460 --> 27:33.460]  Or something else?
[27:33.460 --> 27:35.240]  Or is it just being funny?
[27:36.040 --> 27:41.280]  Is the contrast difference between these two elements too great for people with visual impairments of different kinds?
[27:41.460 --> 27:41.820]  And so on.
[27:43.720 --> 27:51.560]  So right now, we've implemented just a placeholder version of this option, which pops up a window to ask a question for each page visited that matches a regex.
[27:52.420 --> 28:00.220]  So more complex filters are possible to trigger manual judgment, but we wanted them to be driven by actual things that the researchers want.
[28:02.000 --> 28:02.560]  Okay.
[28:03.080 --> 28:11.060]  So getting manual judgment like this is building on the idea from the history of distributed computing called distributed thinking.
[28:11.700 --> 28:17.780]  And that involves the use of volunteers on the Internet to perform tasks that use human cognition, knowledge or intelligence.
[28:19.920 --> 28:24.740]  Over the years, there have been, so this is actually from Bossa, which is a Boink project.
[28:25.420 --> 28:32.300]  Not much has come out of it, but the idea is certainly present and articulated and has been implemented in this really cool way.
[28:34.860 --> 28:42.380]  So over the years, there have been many projects that have done sort of manual judgment with bots for like one-off studies.
[28:42.680 --> 28:47.220]  And again, what we're doing is not a standalone study, but a platform for running new ones.
[28:48.040 --> 28:50.420]  So an example of one of these one-offs is DIMES.
[28:50.420 --> 28:59.020]  That was a project taking measurements to try to understand the topology of the Internet, who's connected to whom, where, how things are functioning.
[28:59.680 --> 29:01.380]  It ran around the turn of the century.
[29:01.600 --> 29:06.820]  I believe it was an Israeli project, but they had participants from all around the world.
[29:06.920 --> 29:09.020]  That's what this pie chart is showing.
[29:10.120 --> 29:15.300]  Yeah, the large segment of Israeli participants makes me wonder, since it's a small country.
[29:16.880 --> 29:22.720]  And so you had these kind of lightweight services running on clients.
[29:22.900 --> 29:25.620]  And then after they ran, you could get...
[29:25.620 --> 29:30.180]  One of the kind of cool things you got to see was your perspective on the network.
[29:30.400 --> 29:31.160]  So that's what's shown.
[29:31.480 --> 29:33.220]  I guess it's figure two here.
[29:35.840 --> 29:41.200]  Yeah, in 2004, there were 800 agents in 50 countries for DIMES.
[29:41.860 --> 29:47.680]  And here's a few more projects using botnets that might be of some interest.
[29:48.760 --> 29:54.000]  I just list them due to time constraints, but they're all definitely worth a look.
[29:55.800 --> 30:04.540]  Citizen Browser, in particular, is work by the folks at The Markup, who also made this shirt, incidentally, beginning in 2010.
[30:05.880 --> 30:09.320]  It's a platform for studies on Facebook in particular.
[30:09.820 --> 30:18.820]  It uses the user's own credentials to log in and then to capture information about how Facebook interacts with their account.
[30:19.560 --> 30:20.780]  So that...
[30:21.260 --> 30:30.700]  One of the things I think is interesting about that project in particular is the approach to redaction, because a lot of what they record is specific to the account.
[30:31.380 --> 30:36.260]  And, you know, in order to collect it, they want to minimize the privacy risk.
[30:37.040 --> 30:41.000]  And so they have this kind of automated redaction process.
[30:43.220 --> 30:44.280]  It's worth a look.
[30:45.300 --> 30:49.440]  They did also have a security audit back when it was first introduced.
[30:51.000 --> 30:57.600]  And it uses ngFetch, which is, I guess, a proprietary automation tool from NetoGraph.
[30:57.740 --> 30:58.980]  But I don't know much about that.
[30:59.140 --> 31:03.220]  So if you do reach out, we can try to learn from it.
[31:03.340 --> 31:03.760]  I think it...
[31:04.280 --> 31:10.460]  It seems that Citizen Browser has produced several really interesting measurement studies.
[31:11.000 --> 31:15.180]  And so it's probably a decent amount to learn from what they've gone through.
[31:16.840 --> 31:18.700]  Also, Brave Redirection Tracking.
[31:19.180 --> 31:20.720]  So that's from 2018.
[31:21.160 --> 31:26.120]  It was a study, again, one-off, measuring prevalence of redirection tracking.
[31:26.460 --> 31:31.980]  So a website or maybe a third party automatically redirects the browser to another domain.
[31:32.240 --> 31:38.600]  So if it's a third party, what they're doing is embedding frames, and then they kick off a kind of redirection chain to persist data.
[31:38.600 --> 31:43.340]  In the first party case, it's just the top-level frame that does the redirection.
[31:46.160 --> 31:52.880]  Censored Planet is another relatively early censorship project.
[31:53.520 --> 31:56.580]  There are actually many of these.
[31:57.820 --> 31:58.540]  Not...
[31:58.540 --> 32:05.740]  It's not that common for them to be botnet sort of design.
[32:07.040 --> 32:22.950]  So an earlier iteration of this talk involved much more discussion of censorship circumvention projects and things that had developed that were peer-to-peer in one way or another, like Peek-a-Booty or a 6-4 and some of their successors.
[32:24.680 --> 32:32.220]  But really, there's a connection there between the architecture and what I'm talking about, but it's not super direct.
[32:32.840 --> 32:33.800]  So it's...
[32:33.800 --> 32:36.040]  You know, it's interesting, I think.
[32:36.120 --> 32:38.560]  I think especially questions like isolation of the client.
[32:38.820 --> 32:41.240]  It raises some interesting questions, but...
[32:43.280 --> 32:44.480]  You know, it's not the same.
[32:45.240 --> 32:55.120]  And then Surveil at Home, which I've already mentioned, was a BOINC project, which was botnet design.
[32:56.640 --> 33:07.580]  So that's kind of the simple, broad picture of measurement botnets, which support a broad range of empirical research questions on the web, not available to bot farms, piggybacking approaches.
[33:07.580 --> 33:25.780]  It includes questions about phenomena that are resistant to automatic detection and questions that require ongoing monitoring over a large number of sites, something like an enforcement agency like the FTC might be interested to do.
[33:27.040 --> 33:31.500]  So I'm happy to take questions about the details of our implementation if there's interest.
[33:31.500 --> 33:33.540]  And of course, also after the talk, especially.
[33:34.160 --> 33:41.180]  We've got a mailing list to talk about some of the stuff, whether it's our project in particular or more generally, if you're doing related work.
[33:42.060 --> 33:45.700]  But, you know, of course, we like to talk about the design of our project as well.
[33:46.320 --> 33:49.900]  So that's the email address.
[33:51.400 --> 34:03.460]  And if you're interested in contributing, we've got projects going, looking at client isolation, client scheduling, bot detection, circumvention, also just packaging, you know, we're...
[34:04.620 --> 34:12.600]  It's important that the installation be relatively painless and that the system be... the client be usable.
[34:12.940 --> 34:14.740]  So also interface design.
[34:15.680 --> 34:20.680]  And then, you know, sort of on a less technical note, the design of incentives.
[34:20.680 --> 34:23.780]  We have fake Internet points involved in this...
[34:24.400 --> 34:25.600]  in this design.
[34:25.860 --> 34:30.820]  But, you know, how to... how to do them, how to incorporate teams effectively, especially.
[34:31.040 --> 34:34.080]  I mean, I think that's a big driving force.
[34:34.460 --> 34:43.140]  I think there have been interesting insights from other projects about the failures of external motivating factors for participation.
[34:43.340 --> 34:45.540]  So paying people tends not to work well.
[34:45.640 --> 34:48.300]  It's kind of an interesting history with Boink in that.
[34:50.320 --> 34:53.560]  So if you're kind of more economics-minded, that's something we'd like to talk about.
[34:54.600 --> 34:57.540]  And then also from people with ideas for specific measurement studies.
[34:57.820 --> 35:04.480]  So we'll work with you to develop features you want, so long as they're likely to benefit others as well.
[35:05.080 --> 35:06.300]  Okay, so just to close...
[35:06.960 --> 35:07.140]  Whoops.
[35:08.360 --> 35:10.320]  That's actually not something I'm gonna...
[35:10.320 --> 35:12.560]  Well, we can talk about it if you want to, but not officially.
[35:14.840 --> 35:16.000]  Sheldon Whitehouse is wrong.
[35:16.740 --> 35:22.500]  Botnets can be beneficial and shouldn't be blocked indiscriminately, using bad botnets as a pretext.
[35:23.700 --> 35:38.580]  From the early 90s out-of-home projects and related distributed computing projects to today, we've got... we've had volunteers banding together to coordinate the use of private resources in the public interest, including teams of hackers who were present from the very beginning.
[35:39.080 --> 35:52.100]  So there's a... like I said, there's still a rich ecosystem of measurement studies that are more or less one-off, particularly for social media, but also for censorship, and for a variety of different kinds of practices on the web.
[35:53.420 --> 35:55.020]  So, hope I've convinced you.
[35:55.600 --> 36:00.400]  My view is botnets are the best way to study user hostile web practices empirically on the web.
[36:01.540 --> 36:02.140]  All right.
[36:02.480 --> 36:02.760]  Thanks.
[36:02.760 --> 36:04.640]  All right.
[36:04.940 --> 36:09.480]  So, we have the opportunity, if anyone in the live audience wants to ask questions, there's an open mic right there.
[36:09.500 --> 36:13.360]  Or if you feel uncomfortable with the open mic, just shout it out and I can repeat it for David.
[36:13.360 --> 36:15.120]  He can't actually hear you directly.
[36:15.240 --> 36:16.740]  He can only hear us through the mic.
[36:18.800 --> 36:20.940]  There are a couple of questions in the Matrix chat.
[36:21.340 --> 36:28.120]  So, the first one is, for Mac, would running the Linux version of the bot in a Docker container be both functional and sufficiently isolated?
[36:29.580 --> 36:30.220]  Yes.
[36:30.540 --> 36:31.480]  Thank you for this question.
[36:32.240 --> 36:35.200]  Sorry if there's a little delay while I get Matrix up.
[36:36.180 --> 36:37.720]  It's a little bit heavy.
[36:37.920 --> 36:38.040]  Wait.
[36:38.520 --> 36:39.380]  One second.
[36:40.120 --> 36:40.640]  Okay.
[36:42.840 --> 36:43.480]  Yes.
[36:43.700 --> 36:44.540]  So, Docker.
[36:44.740 --> 36:44.840]  Right.
[36:45.280 --> 36:45.620]  Yeah.
[36:45.680 --> 36:49.240]  We're doing a lot of things that Docker does to achieve isolation.
[36:50.440 --> 36:58.040]  Our feeling is that Docker is introducing a kind of general purpose thing that our users might not need.
[36:58.500 --> 37:00.960]  Remember, this is kind of a volunteer-based thing.
[37:01.080 --> 37:03.680]  It's really, you know, adoption is really essential.
[37:05.080 --> 37:09.900]  Just like anonymity, you know, measurement loves company.
[37:11.440 --> 37:18.860]  And so, we sort of felt like there wasn't much to be gained by working with Docker.
[37:21.360 --> 37:22.620]  That's the short answer.
[37:22.800 --> 37:25.540]  But, yeah, we could talk more.
[37:27.460 --> 37:31.780]  It's also a nice approach to, you know, interoperability, making it work on different platforms.
[37:31.940 --> 37:33.360]  So, there's a trade-off there, for sure.
[37:35.520 --> 37:39.480]  But, for now, we thought better to do it this way.
[37:41.140 --> 37:41.760]  All right.
[37:41.880 --> 37:44.360]  Next question from the chat room.
[37:44.600 --> 37:52.260]  Are there safeguards to keep, say, Amazon employees running the bot and saying, nah, that's a perfectly reasonable design pattern that would skew the results?
[37:54.420 --> 37:54.860]  Right.
[37:54.860 --> 37:55.300]  Yes.
[37:55.540 --> 37:56.700]  So, this is a great question.
[37:56.880 --> 37:58.660]  So, this always reminds me.
[37:59.140 --> 38:10.680]  David Clark, who was one of the early architects of the Internet, complains, I forget, I think in his book, Designing an Internet, that people always come to him and say that the early architects didn't think about security.
[38:11.120 --> 38:12.600]  And he says, it's not really right.
[38:12.720 --> 38:15.620]  Actually, we just had this really limited conception of what security meant.
[38:15.780 --> 38:27.020]  And we didn't really consider the idea that, you know, we were really concerned with two trusting parties communicating and then there's some Eve or Mallory in between who's, you know, messing things up.
[38:27.300 --> 38:33.940]  But we didn't consider the possibility enough of kind of one of the two communicants, you know, being untrustworthy.
[38:34.680 --> 38:39.360]  You know, you've got a man-at-the-end attack, you've got a denial-of-service attack, you've got Sybils, you've got...
[38:41.280 --> 38:45.440]  So, that's something we are thinking about.
[38:45.600 --> 39:02.360]  We do have a... and this actually comes up in the earlier history of distributing computing as well, where you want to have some policy for re-measuring the same site so that you have some notion of inter-annotator agreement.
[39:02.700 --> 39:10.260]  And, you know, it's possible to overwhelm those things, but difficult and maybe detectable.
[39:11.000 --> 39:21.780]  So, our current idea basically is to incorporate re-measurement of the same thing to prevent kind of poisoning like that.
[39:24.730 --> 39:26.050]  Yeah, there's the...
[39:26.050 --> 39:28.250]  Yeah, there was a funny...
[39:28.910 --> 39:39.910]  In the history of hacking, there was kind of an interesting case of this where the Time Top 100 Most Influential People poll was attacked by Anonymous to make it say...
[39:39.910 --> 39:41.990]  Well, this was the slide I almost went to.
[39:43.050 --> 39:45.970]  Marble Cake, also the game, I think.
[39:46.150 --> 39:47.810]  It was sort of an acrostic down the name...
[39:47.810 --> 39:49.490]  the first letter of all the names in the Top 100.
[39:50.170 --> 39:58.810]  And at one point, Time introduced a CAPTCHA because it was, you know, they were just stuffing the poll automatically.
[40:00.630 --> 40:02.530]  Anonymous, you know, participants in the off-work.
[40:03.230 --> 40:13.730]  And so, the first response made to that was to try to poison, to make it more effective, more efficient by poisoning the CAPTCHA results.
[40:13.730 --> 40:27.130]  So, the idea is that, at that time, CAPTCHA was showing two different sort of unclear words, one of which the answer was known, what the word said, and the other one was an unknown thing that OCR failed on.
[40:27.550 --> 40:43.110]  And so, the approach of this group was to say penis for every unknown word, just penis, penis, penis, you know, and then that would become one of the known words and it would become more automatable to defeat the system.
[40:44.410 --> 40:45.550]  Unfortunately, it didn't work.
[40:45.690 --> 40:50.630]  CAPTCHA is too big and there wasn't enough, you know, capacity to do it, but it's kind of a fun...
[40:50.630 --> 40:52.730]  Since we're talking about poisoning, it's a fun...
[40:52.730 --> 41:17.830]  And actually, even further, the subsequent approach by Anonymous was to do a sort of botnet style thing where they automated the manual solution of CAPTCHAs and thought carefully about the incentive structure, including things like, like a link to view porn while you're solving CAPTCHAs one after the other.
[41:18.870 --> 41:19.410]  So...
[41:20.050 --> 41:26.030]  I think I have a, let's see, I have a slide actually that might be germane to that.
[41:27.130 --> 41:29.770]  So, yeah, here's Marble Cake or the game.
[41:29.970 --> 41:32.030]  It's funny that this came up actually, I didn't think it was going to...
[41:33.070 --> 41:37.150]  That was the top 100 and Moot was the number one person.
[41:38.870 --> 41:41.550]  And, yeah, these are the different front ends they made.
[41:45.630 --> 41:48.050]  This is the one, I think, that shows the porn link.
[41:48.250 --> 41:49.250]  There it is at the top.
[41:49.970 --> 41:50.450]  So, anyway.
[41:50.930 --> 41:53.830]  Sorry to go off a little bit.
[41:55.450 --> 41:55.990]  All right.
[41:55.990 --> 41:58.810]  Are there any questions from here in the audience, the live audience?
[42:04.010 --> 42:04.910]  No, none.
[42:05.630 --> 42:06.790]  Well, I have one.
[42:07.310 --> 42:14.690]  Do you face any legal or regulatory issues or concerns that you run into in trying to promote or propagate this?
[42:15.450 --> 42:16.250]  Oh, yeah.
[42:16.390 --> 42:17.030]  That's so great.
[42:17.150 --> 42:17.230]  Yeah.
[42:17.370 --> 42:17.490]  Wow.
[42:17.630 --> 42:18.290]  This is...
[42:18.290 --> 42:18.390]  Yeah.
[42:19.310 --> 42:19.850]  Right.
[42:20.050 --> 42:21.590]  So, yes.
[42:21.950 --> 42:25.410]  One of the concerns involves the Computer Fraud and Abuse Act.
[42:26.790 --> 42:38.330]  So, is it exceeding authorized access to have an automated bot overcoming barriers that have been put in place by the service provider to keep out bots?
[42:40.090 --> 42:41.670]  Well, it's not clear.
[42:41.670 --> 42:52.210]  CFAA obviously is a dumpster fire, but at least Van Buren was a recent decision that went the right way in terms of protecting the ability to use bots.
[42:54.970 --> 42:59.450]  I interacted with a legal clinic at what was my home institution.
[42:59.450 --> 43:00.110]  I've since left.
[43:00.330 --> 43:09.330]  But, and asked them this question and they were very sort of gun shy, I guess would be the word, because the CFAA has been interpreted so broadly.
[43:11.090 --> 43:17.730]  And there were questions about, you know, tortious interference with another company by doing this kind of measurement.
[43:19.130 --> 43:24.310]  I think, there's big picture questions like that about how the law will be applied.
[43:24.510 --> 43:41.910]  There's also, like, narrow ones, like, if your rate limiting isn't very good, if you design this thing poorly and then, like, lots of people suddenly start banging on some sites and they go down and you effectively DDoS them, you're definitely going to get into some legal problems.
[43:43.430 --> 43:58.950]  But I think, you know, one of the lessons from the kind of private world of scraping is that thinking carefully about how to do that scheduling to prevent, you know, rate limiting, to prevent, you know, resource exhaustion is important and is doable.
[43:59.170 --> 44:01.770]  I mean, it's being done, I think, pretty well.
[44:02.050 --> 44:03.690]  So, yeah.
[44:03.810 --> 44:04.070]  All right.
[44:04.130 --> 44:05.310]  We have another question from the chat.
[44:05.590 --> 44:11.610]  Do websites generally list a user's participation in a monitoring botnet as a violation of their terms of service?
[44:13.730 --> 44:14.150]  Right.
[44:14.150 --> 44:14.570]  Yeah.
[44:16.830 --> 44:18.190]  So, I don't know.
[44:18.670 --> 44:20.590]  You know, generally, I don't read terms of service.
[44:20.790 --> 44:23.390]  I'm not a person with nothing to do in my life.
[44:23.510 --> 44:25.020]  I enjoy agony.
[44:27.370 --> 44:29.010]  But, sorry, no, that's not fair.
[44:29.090 --> 44:29.730]  I'm not a lawyer.
[44:29.930 --> 44:30.830]  I have lawyers read it, maybe.
[44:34.950 --> 44:51.050]  But, I guess there's a question about whether, first of all, whether something like CFAA can be applied based on violating a contract like a terms of service, I think is more or less settled that it can't, you can't just put in your terms of service, you're not allowed to come here,
[44:51.210 --> 44:54.390]  and then start, you know, criminal prosecution.
[44:58.650 --> 45:04.750]  But, yeah, I'm not sure how much, how that'll develop.
[45:04.990 --> 45:16.590]  I know, for example, in the HiQ case, which is another legal case of interest here, I believe LinkedIn sent a letter explicitly saying, we do not want you doing this.
[45:17.970 --> 45:25.510]  And, there was a further legal question about once you have that letter, sort of, then is it more, is there different issues that arise?
[45:26.850 --> 45:29.270]  So, I think it's a, there's a kind of thicket of things.
[45:29.630 --> 45:45.730]  And it's going to be, I mean, one of the things, one of the reasons I think measurement botnets are a nice approach, is that at least for the technical measures that are deployed to try to prevent bots, botnets are great, right?
[45:45.850 --> 45:58.630]  Because you can, you have a human volunteer there who's willing to explicitly say, I endorse this bot, acting on my behalf, going to the site and doing its thing, right?
[45:58.710 --> 46:11.310]  To sort of ask someone who comes to a commercial site to have a real intention of purchasing and not to do anything that's not directly related to that is, seems like it's starting to infringe on some other important issues.
[46:11.310 --> 46:27.010]  So, right, so there'll probably be a combination, I think, of things like terms of service, legal action, maybe trying to make an example of people, and then technical approaches as well.
[46:27.470 --> 46:39.450]  But both of them, I think, are well handled by a kind of community-driven platform for producing botnet measurements, measurement studies.
[46:42.650 --> 46:43.370]  Yeah.
[46:43.390 --> 46:47.490]  And are there any other implications for doing this internationally?
[46:47.530 --> 46:51.390]  Or is this something that is more protected domestically?
[46:52.630 --> 46:53.350]  Yeah.
[46:53.790 --> 47:00.390]  I, well, I, one application that we've been exploring is tied to disinformation.
[47:00.390 --> 47:08.030]  So there's a project called IDMO, the Italian Digital Media Observatory, I think is the acronym.
[47:08.550 --> 47:20.250]  And we are interacting with them and talking about whether we could use something like a measurement botnet to gain some insight into disinformation campaigns.
[47:20.250 --> 47:23.630]  That's another area that's another area that's difficult to automate.
[47:24.030 --> 47:25.630]  It's rapidly changing.
[47:25.970 --> 47:27.550]  Different people see different things.
[47:27.930 --> 47:29.510]  Viewpoint diversity is really important.
[47:30.750 --> 47:38.170]  So there's, there's maybe some potential, but at this point, yeah, we're kind of more focusing on making this thing.
[47:38.870 --> 47:40.630]  Well, I guess I don't want to say that.
[47:40.710 --> 47:53.850]  I mean, in some sense, we're, we're trying to be responsive to the people who come to us with ideas about how to do measurement studies and then produce the features that'll, you know, be beneficial across other studies as well.
[47:55.410 --> 48:00.030]  So that, that implies some, yeah, so there's been some connection to international studies with IDMO.
[48:01.190 --> 48:02.790]  But other than that, nothing specific.
[48:06.980 --> 48:08.360]  All right, any more questions?
[48:11.620 --> 48:13.160]  Looks like we don't have any more questions.
[48:13.360 --> 48:16.300]  So thank you so much for your presentation, David.
[48:16.440 --> 48:18.220]  It was very interesting and enlightening.
[48:19.020 --> 48:26.500]  And if you have any other information or links or references, everyone can view them inside the matrix chat for this, this talk.
[48:26.500 --> 48:33.040]  So you can share them there and everyone can continue to interact with David after the talk if you want to learn more or converse with them more.
[48:33.200 --> 48:34.320]  So again, thank you so much, David.
[48:34.420 --> 48:36.220]  And thank you, audience, for participating.
[48:37.140 --> 48:37.840]  All right.
[48:37.980 --> 48:38.620]  Thank you very much.
[48:43.620 --> 48:51.960]  Stick around for the next talk in, well, 15 minutes, 20 minutes, which is going to be on creating a general purpose wireless through network mesh.