[00:33.660 --> 00:34.820]  Welcome, ladies and gentlemen.
[00:36.020 --> 00:39.940]  Our next session, how to run a top 10 website publicly and transparently.
[00:40.480 --> 00:47.020]  Kunal is going to be our presenter, and we've got to make sure we stay on target for timeframe, because after this is keynote in here.
[00:47.380 --> 00:48.720]  So, take it away.
[00:49.640 --> 00:50.260]  Thank you.
[00:51.320 --> 01:00.200]  My name is Kunal, or user LEGO KTM, and today we're going to talk about Wikipedia, if you couldn't guess what the top 10 website was.
[01:01.600 --> 01:03.560]  I hope all of you know what Wikipedia is.
[01:03.780 --> 01:05.400]  It's a free encyclopedia.
[01:06.960 --> 01:10.100]  The main goal of this talk is to discuss transparency.
[01:10.640 --> 01:18.440]  And so, each slide has a QR code where you can find the resources that are on that slide, if there's a dashboard, a graph, or a chart, or something like that.
[01:18.660 --> 01:23.700]  I'm not sure you'll be able to scan them here, but at the end, there's a very big QR code with links to the slides.
[01:25.700 --> 01:31.800]  So, Wikipedia is an encyclopedia, you know, the things that they used to, like, print on dead trees.
[01:33.060 --> 01:40.040]  And it's primarily edited by volunteers, and it's available under a free Creative Commons license.
[01:40.040 --> 01:46.560]  You can take the content and reuse it and remix it as long as you attribute it that it came from Wikipedia.
[01:47.420 --> 01:52.100]  And the goal of Wikipedia is really to compile the sum of all human knowledge.
[01:52.100 --> 01:55.540]  And that's a pretty daunting task, you know.
[01:55.940 --> 02:05.440]  On the left, that's a picture of what remains of the Library of Alexandria, which at one point was, you know, like, the biggest collection of human knowledge.
[02:06.740 --> 02:11.160]  And it's a pretty daunting task, but a lot of people are working hard and are up to it.
[02:11.720 --> 02:16.580]  And as a side effect, you know, we really make the Internet not suck, in my opinion.
[02:19.320 --> 02:31.220]  But just to give us, you know, to give an idea of the scale of the problem is that estimates by looking at topic areas suggest that we're about 5% done with collecting, you know, all of human knowledge.
[02:32.240 --> 02:36.700]  And I think that, you know, whenever I go on Wikipedia, no matter what I look up, I'll find something.
[02:36.960 --> 02:42.780]  And yet to think that we're only 5% there really, you know, gives an estimate of the magnitude of the tasks that's in front of us.
[02:45.260 --> 02:51.840]  So first, I want to talk about how Wikipedia operates, you know, just building the encyclopedia.
[02:51.980 --> 02:53.420]  And we'll get to the technical part afterwards.
[02:54.180 --> 02:58.080]  And transparency is really a core principle of Wikipedia itself.
[02:58.340 --> 03:02.700]  So this is the article for Webb's first deep field as it looked about a week ago.
[03:02.880 --> 03:04.340]  It looks actually much better now.
[03:04.560 --> 03:08.500]  And it was the first image taken by the James Webb Space Telescope.
[03:09.580 --> 03:11.020]  And you can read the article.
[03:11.280 --> 03:12.100]  It's a nice article.
[03:12.220 --> 03:13.480]  It explains a picture of the background.
[03:13.980 --> 03:18.100]  But you can also, in the top right, there's a little tab that says View History.
[03:18.560 --> 03:23.400]  And if you click on the tab, it'll show you the history of the page.
[03:23.400 --> 03:25.400]  And this is, again, the history from a week ago.
[03:26.160 --> 03:31.260]  But then you can go through and look at each individual revision of why people made the change.
[03:31.260 --> 03:33.960]  And then you can actually look at a diff of the changes themselves.
[03:34.300 --> 03:40.760]  And so this diff is someone just adhering to the manual of style and changing the numeral six to be spelled out S-I-X.
[03:43.620 --> 03:49.560]  And that's all great, but it doesn't really explain, like, how things happen or why things happen.
[03:49.720 --> 03:51.680]  And so there's another tab that says Talk.
[03:51.960 --> 03:55.340]  And that's really where all the discussions happen on Wikipedia.
[03:55.340 --> 03:59.680]  And each page has an associated talk page that you'll find some of these discussions on.
[03:59.880 --> 04:01.400]  And people will ask different questions.
[04:01.560 --> 04:03.320]  They'll discuss the validity of different sources.
[04:03.320 --> 04:06.700]  They'll see if something is phrased properly or could be phrased better.
[04:06.940 --> 04:10.380]  Or, you know, whether something is unclear or jargony or could be better.
[04:10.560 --> 04:19.280]  And this is really, you know, the heart of the collaborative spirit of Wikipedia is that if you're not sure of something, instead of making the edit, you can just discuss it with other people.
[04:20.840 --> 04:25.060]  And really, the technical infrastructure works the same way.
[04:25.060 --> 04:29.340]  We adhere to these same principles of collaboration, transparency, and openness.
[04:31.640 --> 04:35.960]  So just at a glance, Wikipedia is the seventh most visited website.
[04:35.960 --> 04:38.940]  And I got that stat from Wikipedia, so I don't know if it's true.
[04:40.060 --> 04:45.780]  It's maintained by a collaboration of volunteers and staff members.
[04:46.200 --> 04:49.660]  And all of the source code is available under free licenses.
[04:49.940 --> 04:53.240]  The majority of it is under the GPL copyleft license.
[04:53.240 --> 05:00.300]  And like I said, it's developed in a collaborative manner that you can observe and what I'll kind of walk through today.
[05:01.840 --> 05:03.340]  But first, a quick segue.
[05:03.800 --> 05:06.320]  Naming things is a hard computer science problem.
[05:06.600 --> 05:09.240]  And Wikipedians kind of suck at this.
[05:09.480 --> 05:12.300]  So, the globe logo is Wikipedia.
[05:12.680 --> 05:14.860]  The encyclopedia, what you're really familiar with.
[05:15.060 --> 05:17.100]  In the middle is Wikimedia.
[05:17.820 --> 05:22.380]  The Wikimedia movement is a social movement designed to spread free knowledge.
[05:22.560 --> 05:28.640]  But it's also the name of the nonprofit organization that maintains trademarks, legal status, and runs the servers.
[05:28.840 --> 05:30.920]  That's the Wikimedia Foundation, or WMF.
[05:31.720 --> 05:37.720]  And just as an analogy, Wikimedia is to Mozilla as Wikipedia is to Firefox.
[05:37.720 --> 05:39.020]  So that's kind of the relationship.
[05:39.620 --> 05:51.720]  And then, if you flip Wikimedia around, you'll get MediaWiki, which someone thought was a brilliant idea of how to name the wiki software that powers Wikipedia and hundreds of other wikis around the Internet.
[05:52.500 --> 05:53.840]  I may use them interchangeably.
[05:54.640 --> 05:56.140]  This is kind of what it is.
[05:56.660 --> 05:58.740]  But hopefully, it'll make sense.
[06:00.640 --> 06:04.320]  So, a brief technical history of where Wikipedia came from.
[06:04.520 --> 06:10.480]  And that's what the Wikipedia homepage looked like roughly in November, December 2001.
[06:11.580 --> 06:14.480]  So there was a .com company called Bomus.
[06:14.960 --> 06:20.880]  And one of their side projects was Newpedia, which was a public encyclopedia.
[06:21.240 --> 06:24.060]  But everything had to be reviewed before it could be published.
[06:25.220 --> 06:26.660]  And it was very slow.
[06:26.660 --> 06:35.820]  And so they decided, why don't we create this project called Wikipedia, where anyone can edit and eventually, once the articles are good enough quality, we'll move them over to Newpedia.
[06:36.800 --> 06:42.160]  And about a few months into the project, Wikipedia had hundreds, if not a thousand articles.
[06:42.340 --> 06:44.300]  And Newpedia had 60 or 70.
[06:44.500 --> 06:46.320]  It was very clear which project won.
[06:47.780 --> 06:50.340]  And at this time, the servers were in San Diego, California.
[06:50.340 --> 06:53.500]  And they were owned by this for-profit company called Bomus.
[06:53.800 --> 06:55.400]  But volunteers got access.
[06:55.400 --> 07:08.760]  And within a few years, the leaders of Bomus, including Jimmy Wales, realized how important Wikipedia was, not just as a commercial thing, but also as just a resource for public good.
[07:08.860 --> 07:15.860]  And they split it off into the Wikimedia Foundation as a non-profit that would safeguard the legacy of it going forwards.
[07:17.920 --> 07:21.340]  And then in 2004, the servers would move to Tampa, Florida.
[07:21.680 --> 07:25.080]  And that was primarily because the co-founder of Wikipedia, Jimmy Wales, lived there.
[07:25.180 --> 07:27.800]  And he was the one who installed the first Tampa servers.
[07:28.960 --> 07:35.740]  Coincidentally, this was also the time that the first off-site backup of Wikipedia was taken because they were afraid of hurricanes.
[07:36.280 --> 07:49.840]  And it is kind of wild to think today that no one bothered to take a backup of Wikipedia for three years, knowing how important it is today that we were like one failed hard drive away from not having Wikipedia or having to start all over.
[07:51.660 --> 07:54.180]  Today, the story is very different.
[07:54.780 --> 07:59.460]  There are six different data centers around the globe that, you know, serve Wikipedia to users.
[07:59.680 --> 08:06.840]  There are two core data centers in Virginia and in Texas that run the MediaWiki software and serve the Wiki platform.
[08:07.160 --> 08:19.080]  And then there are four caching point of presence data centers that are in San Francisco, the Netherlands, Singapore, and the new one just opened up in France earlier this year and is starting to serve traffic.
[08:22.900 --> 08:34.900]  And this is an overview of the technical architecture of how, like, your web request flows starting, like, coming in from the caching layer to the application servers and then how it hits the storage layer.
[08:35.080 --> 08:36.520]  I'm not going to explain this.
[08:36.600 --> 08:38.400]  I'm mainly showing this for two purposes.
[08:38.840 --> 08:41.980]  One, to give you an idea of the complexity of the technical stack.
[08:42.360 --> 08:44.320]  You can see that there's a lot of different components.
[08:44.320 --> 08:50.860]  There's a lot of, you know, different caching layers and, you know, ways that data moves through the flow.
[08:51.080 --> 08:55.360]  And also the second thing is to show you that this diagram is publicly available on the Internet.
[08:55.380 --> 08:56.320]  You can read it.
[08:56.400 --> 08:57.460]  You can, like, look at it.
[08:57.540 --> 09:01.000]  It gets updated every few years or so as technology changes.
[09:01.680 --> 09:15.100]  And if you go to our technical wiki, which I'll link later, you can just type in any of these different components, you know, whether it's, like, the back-end cache or whether it's, you know, how we use, you know, things like McRouter and Envoy.
[09:15.520 --> 09:23.940]  And you can just type them into the wiki and you will find, like, all of our technical documentation on how we use it, what we use it for, where it's deployed, where you can find the configuration.
[09:24.400 --> 09:26.060]  All of that is publicly available.
[09:28.920 --> 09:35.120]  So these are, you know, like, the main, like, technical, like, points you get in that I'll go over today.
[09:35.120 --> 09:39.580]  So the first is, like, where we host our code, which is on this code review system called Garrett.
[09:39.760 --> 09:43.580]  And it's also mirrored to GitHub just for convenience and to make it easier for people to find.
[09:44.400 --> 09:48.520]  We publish our metrics and statistics at grafana.wikimedia.org.
[09:48.860 --> 09:51.560]  Our bugs are tracked in a system called Fabricator.
[09:51.700 --> 09:52.620]  And you can see the URL.
[09:52.860 --> 09:54.840]  And then our documentation is kind of spread out.
[09:54.900 --> 10:02.300]  We have two different wikis, wikitech.wikimedia.org, which is mainly focused on the wikimedia-specific documentation of how we deploy things.
[10:02.300 --> 10:09.560]  And mediawiki.org, which is really, you know, about the media wiki software, which can be used by anyone, but is also...
[10:09.560 --> 10:13.820]  has stuff that's applicable to how, you know, Wikipedia uses media wiki.
[10:14.300 --> 10:17.100]  And doc.wikimedia.org is another place for documentation.
[10:17.100 --> 10:20.760]  And that's mostly, like, you know, auto-generated documentation from code.
[10:24.100 --> 10:31.320]  So in the last 90 days, we saw around 13,000 patches submitted.
[10:31.960 --> 10:38.760]  And from, you know, over 350 different authors, some of whom are staff, and some of whom are volunteers.
[10:40.060 --> 10:45.880]  All patches to the media wiki code base have to be approved by someone with, like, we call them plus two rights.
[10:46.020 --> 10:50.680]  And basically, you have to... you vote plus two on the change, and then the change will get merged if it passes CI.
[10:51.340 --> 10:53.420]  And code gets deployed once a week.
[10:53.420 --> 10:59.400]  We create a branch, and then that branch is progressively rolled out from the smaller wikis to the biggest ones, including the English Wikipedia.
[10:59.400 --> 11:01.540]  And we call that the deployment train.
[11:02.240 --> 11:07.700]  And all the servers are, like, maintained using a system called Puppet, and that code is deployed immediately.
[11:07.900 --> 11:09.540]  And I'll talk about Puppet a little bit later.
[11:11.620 --> 11:14.040]  So, who is submitting media wiki patches?
[11:14.220 --> 11:17.700]  And remember how I said earlier it's a collaboration between staff and volunteers.
[11:18.200 --> 11:31.400]  And aside from the fact that a dude named Sam is incredibly awesome, you know, it's actually a pretty clear breakdown of, you know, about half the patches, about, you know, half of the people on the list are staff and half of them are volunteers.
[11:31.980 --> 11:39.720]  And when I looked at the, like, the actual breakdown of everyone, it was around, like, 53% of patches are submitted by staff, and the rest come from volunteers.
[11:39.960 --> 11:44.920]  So it actually is, like, a pretty close equilibrium between staff and volunteers.
[11:46.780 --> 11:49.420]  And then on the flip side, who is approving these patches?
[11:49.760 --> 11:55.340]  And it's actually pretty similar that, like, in the top six people, you know, half are staff and half are volunteers.
[11:55.960 --> 12:00.640]  And this is, like, you know, like, once a patch is merged, it'll automatically get deployed.
[12:00.680 --> 12:08.460]  And so, you know, volunteers actually do have significant power in getting, you know, patches from, you know, reviewing them and getting them deployed into production.
[12:11.920 --> 12:18.260]  So Puppet is, it's like a way to declaratively state what should be installed or running on a server.
[12:18.560 --> 12:21.780]  You know, it's similar to Ansible and other tools in that category.
[12:23.060 --> 12:24.980]  It basically is root.
[12:25.100 --> 12:26.300]  Puppet runs as root.
[12:26.300 --> 12:32.040]  So it's limited to our site reliability engineers and a few volunteers who also have root.
[12:33.240 --> 12:35.140]  And the Puppet code is public.
[12:35.460 --> 12:36.600]  You can find the gear repository.
[12:36.600 --> 12:43.660]  But all the passwords and secret keys are in a private repository that only exists in the production servers.
[12:45.520 --> 12:50.280]  And other people who don't have access to servers can actually test their Puppet patches in virtual machines.
[12:51.860 --> 12:54.780]  If you've never seen Puppet code before, this is kind of what it looks like.
[12:54.900 --> 12:59.640]  It's this, like, mishmash of a Ruby syntax with this, like, custom Puppet syntax.
[12:59.980 --> 13:00.180]  But...
[13:01.000 --> 13:03.980]  And this is, like, how we deploy MediaWiki web servers.
[13:05.520 --> 13:11.200]  But, like I mentioned, the really cool thing is that anyone can grab our Puppet code and test it in a virtual machine.
[13:11.580 --> 13:20.820]  So we have another team called Cloud Services that basically provides, you know, computing resources, which is an OpenStack deployment for volunteers and staff.
[13:20.980 --> 13:35.020]  And you can take the same Puppet code that is running in production, and you can just spin up a VM, apply the Puppet role, and you can be running these roughly, as close as you can, the same exact code that's running in production on your VM.
[13:35.320 --> 13:42.680]  The only difference is, is you don't have any private user data, so we can give out access to this VM that is running, you know, again, nearly identical code to anyone.
[13:42.980 --> 13:53.760]  And this means that it's really easy for volunteers to test their code, have a simulated working environment using the same system that is running in production.
[13:54.820 --> 13:57.460]  And it really makes it a lot more accessible to contribute.
[13:58.660 --> 14:14.340]  And so then, like, one of the key features is that we have this beta cluster, which is a replica of some of the production wikis that we have, and it's just running the latest version of MediaWiki, like as it's been merged into master, to catch different integration issues.
[14:14.500 --> 14:29.340]  And again, we can give out access to this beta cluster where people can test their MediaWiki code or debug problems in MediaWiki code without having... we can give it out access much more liberally than we do on production servers, just because there's really no private user data here.
[14:32.740 --> 14:36.620]  And in a similar vein, we have database replicas.
[14:36.920 --> 14:40.720]  And so we... everything is stored in...
[14:40.720 --> 14:43.300]  or most of the database stuff is stored in MariaDB.
[14:43.640 --> 14:49.200]  And so we have redacted replicas of these databases are available to cloud service VMs.
[14:49.420 --> 14:52.280]  And, you know, like, if you want to do stuff locally, you can set up an SSH tunnel.
[14:53.100 --> 14:57.940]  And so basically, we have all these rules that strip out all of the private data out of...
[14:57.940 --> 15:03.060]  or, like, you know, like, null the fields or zero them, and then return the rows that are public.
[15:03.660 --> 15:06.080]  And so then, you know, now what's the next step after that?
[15:06.220 --> 15:10.420]  We have... we have a web tool that allows you to just run SQL queries straight from the web.
[15:10.420 --> 15:13.300]  It's, like, PHP my admin, but, like, actually secure.
[15:15.820 --> 15:23.320]  And one of the really cool things is that once, like, a technical user who understands SQL, like, writes a query, we see non-technical users, or...
[15:23.320 --> 15:31.660]  I don't even consider them non-technical, but, like, traditionally non-technical users, will just start learning SQL via copy, paste, and modify.
[15:31.960 --> 15:33.160]  They'll start tweaking the queries.
[15:33.360 --> 15:36.100]  They'll, you know, like, just first it'll be strings that are really easy to modify.
[15:36.100 --> 15:40.780]  Then they'll start adding new conditions, or copying and pasting from other queries, combining them together.
[15:41.320 --> 15:42.440]  And it's really powerful.
[15:42.580 --> 15:57.420]  And it's really cool to see that, you know, like, just because we made this resource available, people are... who are, you know, would consider themselves traditionally non-technical users, or don't have a computer science or programming background, are actually writing SQL queries to get data based on,
[15:57.600 --> 16:00.520]  you know, like, the maintenance tasks that they want to do on wikis.
[16:03.700 --> 16:09.080]  So, shifting gears, like I said earlier, we publish all...we publish most of our statistics and metrics.
[16:09.280 --> 16:15.440]  And so, this is the...this is how many requests per second we're getting globally.
[16:15.820 --> 16:22.000]  And so, as the seventh most visited website, that amounts to, at peak, around 130,000 requests per second.
[16:22.800 --> 16:27.200]  And the shape of the graph is actually people being awake and going to sleep.
[16:27.380 --> 16:29.240]  That's how our traffic patterns are determined.
[16:30.580 --> 16:32.180]  And these are just some more statistics.
[16:32.420 --> 16:34.400]  You know, we have the total...total request volume.
[16:34.800 --> 16:36.580]  You know, how many errors we were serving.
[16:36.720 --> 16:42.160]  And you can see that in the error chart, clearly, we were having some problems at the beginning of it before they all, like, zeroed out.
[16:42.840 --> 16:45.420]  And another metric we measure is successful wiki edits.
[16:45.560 --> 16:48.460]  Like, if everyone stops editing, that means there's a problem on the wikis.
[16:51.780 --> 16:55.520]  And you can, like, get this breakdown for the health of, like, individual servers.
[16:55.520 --> 17:00.420]  So this is MW1312, which is a media wiki app server living in the Virginia cluster.
[17:01.000 --> 17:04.240]  And you can see, like, the CPU, the memory usage, networking.
[17:04.600 --> 17:09.320]  You know, if I...if you, like, scroll down farther, you can see, like, temperature, load averages.
[17:09.620 --> 17:11.580]  You know, you can, like, break this down for...
[17:11.580 --> 17:16.320]  And every server and virtual machine that we run in our production cluster is on this page.
[17:18.000 --> 17:19.840]  And you can look at it at the database layer.
[17:19.980 --> 17:27.260]  This is an aggregated shot of all of the database rows that are being written and read to every single one of our database clusters.
[17:29.000 --> 17:31.560]  And you can also look at it at the data center level.
[17:31.920 --> 17:34.140]  Like, this is our Singapore data center.
[17:34.300 --> 17:42.280]  And you can see that, like, Asia's is clearly waking up and starting to access more data on Wikipedia, so that way the networking charts are, like, going up.
[17:43.260 --> 17:46.860]  And you can look at, basically, like, how many servers there are in our data center.
[17:47.060 --> 17:52.200]  All of this information is publicly accessible, and anyone can just start browsing through it and learning things.
[17:56.760 --> 17:59.680]  And so, like on Wikipedia, we have talk pages.
[17:59.940 --> 18:01.060]  A lot of the...
[18:01.060 --> 18:04.980]  Most of the communication around this development happens in public as well.
[18:05.220 --> 18:08.840]  So we have a traditional mail-in list called Wikitech-L.
[18:09.180 --> 18:12.080]  Then we have the, like, fabricator and Garrett comments.
[18:12.080 --> 18:16.160]  And then we have a whole host of IRC channels on the Libera chat network.
[18:16.440 --> 18:24.080]  And the biggest channels of these are our bridge to Matrix to, you know, provide a little more nicer user interface for people who are not used to IRC.
[18:26.720 --> 18:33.540]  So now I kind of, like, want to walk through an example of how transparency was key in solving an outage.
[18:33.540 --> 18:39.380]  And so this happened towards the end of 2021.
[18:40.920 --> 18:44.380]  And users can upload files to Wikipedia.
[18:44.640 --> 18:47.840]  And we have a max file size of four gigabytes.
[18:48.240 --> 18:50.900]  But like I mentioned earlier, we have two core data centers.
[18:51.040 --> 18:55.980]  And so when you upload a file in the back end, it's actually being copied over to both data centers.
[18:55.980 --> 18:58.160]  So you have one upload...
[18:58.160 --> 19:02.680]  And it ends up in a system called Swift, which is, like, the OpenStack version of S3.
[19:03.720 --> 19:07.440]  And so, you know, your file is uploaded very quickly to the local data center.
[19:08.120 --> 19:10.140]  And then it also has to take the round...
[19:10.140 --> 19:13.180]  It has to take the extra cost of going from one data center to the other.
[19:13.280 --> 19:15.500]  So the, you know, speed of light from Virginia to Texas.
[19:16.440 --> 19:24.500]  And so what we found was that for whatever reason, files were being... were going from to the other data center very slowly.
[19:25.760 --> 19:28.440]  And it was about two megabytes per second.
[19:28.500 --> 19:32.860]  And we have a 300-second timeout, which basically gives you a 600 megabyte file.
[19:33.200 --> 19:36.120]  And people are uploading files up to four gigabytes in size.
[19:36.260 --> 19:38.900]  And so this is a real problem for any large file upload.
[19:39.360 --> 19:45.000]  And this coincided with the upgrade of our servers from Debian Stretch to Debian Buster.
[19:45.040 --> 19:49.960]  So something in that large upgrade caused these file uploads to slow down.
[19:50.720 --> 19:54.800]  And, you know, we didn't really have a good clue of what it could be.
[19:56.260 --> 20:00.840]  And so as, like, I was the lead person investigating this.
[20:01.400 --> 20:09.580]  And so my first step was basically to, like, try and reconstruct, you know, like, can I minimize the code?
[20:09.700 --> 20:12.980]  Can I just use plain a curl command and send the code off?
[20:13.180 --> 20:15.000]  Can I get the upload to finish in time?
[20:15.000 --> 20:16.680]  Or can I... or will it be really slow?
[20:17.040 --> 20:28.360]  And then also, can I just write, like, now using, like, a very minimized PHP reproduction of, like, stripping the code out of the two main classes that are responsible for handling file uploads?
[20:28.520 --> 20:36.100]  Can I, like, minimize it down to just, like, the very basics so that still reproduce the issues so we can, you know, like, run it under strace and a few other things?
[20:37.240 --> 20:41.740]  And what I found was that when I ran it with, you know, the command line curl, it finished instantly.
[20:41.740 --> 20:45.520]  But when I ran it with the PHP code, it was still slow, which is great.
[20:45.620 --> 20:47.400]  Now I have, like, two different things that we can compare.
[20:47.920 --> 20:54.120]  And so in the fabricator ticket, I pasted, like, everything that I was doing, all the commands I was running and all of the output that I got.
[20:55.560 --> 20:57.360]  And, you know, I have a little more debugging.
[20:57.540 --> 21:03.520]  Like, okay, if I, like, switch to using, instead of using curl in PHP, if I use stream wrappers, then it still works.
[21:03.920 --> 21:09.080]  But, you know, stream wrappers have a much smaller max file size.
[21:10.160 --> 21:19.180]  And I noted that, like, we had, as part of the upgrade from stretch to buster, we had upgraded libcurl, you know, from, you know, 7.52 to 7.64.
[21:19.380 --> 21:21.740]  So there were going to be changes, and maybe it was one of those changes.
[21:23.700 --> 21:28.740]  And my, like, final strace showed that it was, like, calling fread a lot, and I thought that was suspicious.
[21:28.740 --> 21:31.060]  And then I went to sleep, and my colleagues took over.
[21:32.540 --> 21:37.420]  And they said that, you know, okay, my hypothesis is correct, but it's not, like, the fread syscalls that are causing it.
[21:37.460 --> 21:43.180]  It's the fact that we're making 820 round trips between the data centers because we're making really small chunks.
[21:44.080 --> 21:51.160]  And the round trips will add up a lot just because we're in a different data center, and that's what was causing it to take so long.
[21:51.540 --> 21:55.860]  And, you know, they did some more, like, statistical analysis in Wireshark that showed, like, this is clearly the problem.
[21:56.000 --> 21:57.920]  We're sending too many round trips, and it's going too slowly.
[21:58.860 --> 22:09.540]  And then a volunteer who was following the ticket, and this volunteer has no server access, no access to private data, and, you know, probably doesn't have a full understanding of, like, the technical architecture.
[22:09.780 --> 22:17.080]  But they were reading the ticket, they were following along, and all of us who had been commenting so far had been doing so in, like, a very verbose way where everything we were doing was written out.
[22:17.560 --> 22:25.440]  And the volunteer says, you know, but the command line curl output is using HTTP 1, and the lib curl output from PHP is using HTTP 2.
[22:25.920 --> 22:30.080]  Can we, you know, either force one of them to use HTTP 2 or force HTTP 1?
[22:30.400 --> 22:44.180]  And that was actually the issue, was the fact that the use of HTTP 2 was causing it to do way more round trips than necessary, and as soon as we forced it to use HTTP 1, as the volunteer noticed was wrong, you know, that immediately fixed the issue.
[22:44.300 --> 22:46.080]  The uploads finished incredibly quickly.
[22:46.920 --> 22:52.680]  And at that point it was, okay, just write a patch to force it to use HTTP 1 and reset.
[22:53.020 --> 23:02.620]  And then the volunteer in a later comment actually found, like, in the lib curl changelog where it says, okay, we're going to default to HTTP 2 now if the method is not explicitly specified during...
[23:02.620 --> 23:05.180]  in the changelog period from stretch to buster.
[23:05.440 --> 23:07.620]  And so that, like, totally validated our theory.
[23:07.840 --> 23:10.760]  And at the same time, you know, this came from a volunteer.
[23:12.940 --> 23:19.820]  And in the end, it was the verbosely pasting publicly is what allowed this volunteer to come up and solve the issue.
[23:20.440 --> 23:26.820]  And it's likely that, you know, the staff SREs, you know, we would have figured out the real cause eventually.
[23:27.240 --> 23:30.300]  But I think it would have taken more time, a few days at least.
[23:31.400 --> 23:34.380]  And we have, like, a full incident report that you can read.
[23:34.400 --> 23:44.740]  And that incident report also links to a really good Cloudflare blog post that explains why, in certain conditions, HTTP 2 is actually slower than HTTP 1.
[23:44.860 --> 23:47.880]  So if you're interested in the technical reasoning behind it, I would recommend reading that.
[23:50.560 --> 23:52.540]  So jumping over to IRC.
[23:52.760 --> 23:54.100]  We have a ton of IRC channels.
[23:54.240 --> 23:59.460]  This is the list of technical IRC channels that go from M to T, because that's how much I could fit into this screenshot.
[24:00.040 --> 24:01.300]  There's a lot more.
[24:02.160 --> 24:04.900]  And you can, like, peruse the list of IRC channels.
[24:05.060 --> 24:05.720]  You can join them.
[24:06.160 --> 24:08.440]  Nearly all of them are publicly open.
[24:08.440 --> 24:13.800]  And a lot of them are actually publicly logged, so you can actually go back and read old conversations.
[24:15.540 --> 24:19.800]  These are the main three channels that, you know, if you're interested.
[24:20.440 --> 24:27.500]  There's the operations channel, which is the main coordination channel where deployments happen, where incident discussion happens.
[24:27.500 --> 24:31.660]  There's the SRE channel, which is the discussion amongst SREs.
[24:31.940 --> 24:35.780]  And there's the tech channel, which is for, like, end user support and help.
[24:35.780 --> 24:42.680]  And people will often just ask questions and just have general, like, anything that fits under technical discussion is, like, totally on topic there.
[24:43.500 --> 24:49.740]  And we also have a private security channel for dealing with DDoS attacks and active security issues.
[24:50.600 --> 25:02.340]  And that is used pretty sparingly just because everyone in that channel also recognizes that using that channel is excluding a significant amount of volunteers and even staff members who are not in that channel.
[25:02.520 --> 25:04.480]  And that's why discussing things publicly is preferred.
[25:06.260 --> 25:08.400]  So this is from Thursday.
[25:10.540 --> 25:14.500]  And this is just a snippet of an IRC log from the operations channel.
[25:14.500 --> 25:23.560]  And this is Tavi, a volunteer, pinging to the people who are responsible for the deployment, asking if the deployment can be rolled back because it's broken.
[25:23.860 --> 25:25.200]  Coincidentally, again, uploads.
[25:26.880 --> 25:35.660]  Can you imagine going on an IRC channel and pinging someone from Amazon, Google, or Facebook, saying, hey, your deployment is broken, can you please roll back?
[25:36.040 --> 25:41.980]  Like, this level of, like, access is really, like, you don't have that anywhere else.
[25:43.140 --> 25:48.660]  And within, you know, like, three or four minutes, you know, Gina says, okay, I'll roll back.
[25:48.800 --> 25:55.800]  And then, you know, the deployment tool says, you know, yes, Gina has rolled back the wikis to a previous version.
[25:57.040 --> 26:05.960]  And the deployment tool uses this thing called, like, an exclamation mark log, which means that all of the entries end up in what we call the server admin log.
[26:06.860 --> 26:13.380]  And this is a log for, like, any action that happens on a server that isn't going to be reflected anywhere else.
[26:13.560 --> 26:17.860]  Like, you know, like, when you, like, merge a patch, it gets, it's logged in Git, right?
[26:17.980 --> 26:29.820]  But for a lot of options, like, if you're rebooting a server or if you're adjusting some configuration in a database like etcd, then those things, or depooling a server, like, those things are not reflected anywhere else except on the server in your, like,
[26:29.880 --> 26:30.500]  bash history.
[26:30.500 --> 26:33.540]  And so the server admin log, like, collects all of that.
[26:33.700 --> 26:38.780]  And all of our deployment tools will automatically send logs to the server admin log.
[26:39.220 --> 26:44.400]  And you just type in, you know, exclamation mark log in IRC, and it'll go to a wiki page.
[26:44.540 --> 26:46.280]  It'll go to masked on, and it'll go to Twitter.
[26:46.740 --> 26:51.720]  And on the right, I have the Twitter search for the word oops in the server admin log.
[26:52.820 --> 26:55.300]  And the other fun one to search is WTF.
[26:55.640 --> 26:59.080]  And, you know, you can see what people were frustrated with, like, five or six years ago.
[27:00.840 --> 27:16.260]  And the Twitter and Mastodon integration is kind of new, but the wiki archives go back to June 2004, which is, you know, you can see what Wikimedia sys admins have been doing all the way starting in June 2004 up until, like, literally right now.
[27:16.740 --> 27:27.100]  There's, you know, just pages of archives, and you can search through them, you can read through them, you can see, like, people, like, reverting stuff, dealing with outages, not knowing what they're doing, where you can just, like, see, like, general maintenance,
[27:27.140 --> 27:29.120]  like, pooling and depooling servers.
[27:29.540 --> 27:31.640]  So there's a lot of, like, valuable history there.
[27:35.080 --> 27:42.280]  The other thing that's really cool on IRC and the operations channel is most of our alerts and monitoring will show up there.
[27:42.440 --> 27:47.120]  And so we use a combination of Asinga and Alert Manager to do notifications.
[27:47.120 --> 27:49.380]  And this is when there was, like, a blip in networking.
[27:49.600 --> 27:55.580]  And so Asinga thought a bunch of hosts were down just because it couldn't reach them.
[27:55.940 --> 28:01.380]  And the two lines that I bolded have the word, like, hash page in them.
[28:02.540 --> 28:06.360]  And that's special because it means that SREs were paged for that.
[28:06.520 --> 28:08.920]  So you can see that SREs have already been paged.
[28:09.000 --> 28:13.360]  And you don't need to, like, try and ping someone independently that all of these alerts have gone off.
[28:13.980 --> 28:15.580]  And both of those are database servers.
[28:15.720 --> 28:17.140]  That's why they triggered pages.
[28:18.260 --> 28:22.180]  And then at the very bottom, you can see Reuven going, uh-oh.
[28:22.360 --> 28:27.480]  Like, because he's been paged and within a minute he's on IRC being, like, why are all these servers disappearing?
[28:31.640 --> 28:34.860]  So far, everything I've talked about has been public, right?
[28:35.060 --> 28:41.520]  You know, there's, like, a lot of information that's public that's on, you know, on the wikis and in Fabricator and IRC that you can all access.
[28:41.520 --> 28:44.280]  But there is some private information that we do have.
[28:44.940 --> 28:47.540]  Overall, Wikipedia has, like, very little private information.
[28:47.720 --> 28:49.620]  Compared to most websites, we don't collect your address.
[28:49.860 --> 28:51.120]  We don't collect your credit card information.
[28:51.320 --> 28:53.120]  We don't, you know, collect your phone number.
[28:53.380 --> 28:56.020]  And even registering with your email is optional.
[28:56.220 --> 29:01.560]  Literally, the only private thing we have is your password and possibly your email address if you chose to give it.
[29:01.980 --> 29:04.100]  So there's very little private information.
[29:05.320 --> 29:13.260]  But still, volunteers need to sign an NDA to get access to debug logs, web request logs.
[29:13.500 --> 29:19.120]  We have slow SQL queries that can be used as DOS vectors or security tickets.
[29:19.300 --> 29:23.720]  And so the NDA process, the NDA is not even secret.
[29:23.820 --> 29:25.460]  You can read the terms on the wiki itself.
[29:25.840 --> 29:29.780]  And the bar for signing an NDA has been lowered over time.
[29:30.920 --> 29:36.100]  Previously, you needed C-level sign-off on everything to give a volunteer access.
[29:36.300 --> 29:39.400]  And now any Wikimedia Foundation employee can vouch for you.
[29:39.500 --> 29:43.420]  And then the legal department will prepare the NDA and get it signed.
[29:43.900 --> 29:50.880]  And when I checked earlier this week, there were exactly 100 users in our NDA LDAP group, which controls it.
[29:51.000 --> 29:55.520]  And so there's a decent amount of... And all those people are volunteers.
[29:56.280 --> 29:58.720]  None of them are Foundation staff, which has a separate group.
[29:58.720 --> 30:01.800]  So that really shows the scope of people.
[30:01.980 --> 30:06.220]  Even for this private information, we've been able to expand the access for it.
[30:08.200 --> 30:13.580]  And server access, not like the little kitten has gone, but real SSH access.
[30:14.080 --> 30:20.320]  Again, volunteers need to sign an NDA and go through a form of deployment training.
[30:20.560 --> 30:26.960]  And this access is controlled by the release engineering team, which is the one that coordinates and manages most of the deployments.
[30:28.760 --> 30:32.440]  There are about seven volunteers with MediaWiki deployment access.
[30:32.620 --> 30:33.460]  They can deploy patches.
[30:33.620 --> 30:34.680]  They can deploy security patches.
[30:35.020 --> 30:36.960]  You know, they can inspect the state of code.
[30:37.640 --> 30:38.880]  They can do interactive debugging.
[30:40.460 --> 30:42.420]  There are only two volunteers with Root.
[30:42.600 --> 30:43.400]  One of them is me.
[30:44.720 --> 30:49.000]  And me and the other person who is a volunteer with Root, we were both former staff.
[30:49.400 --> 31:00.920]  And in the past, there used to be volunteers with Root, which I think was a really good, like, equalizing principle and good governance that it's a mix of volunteers and staff who are making these, like, Root-level decisions.
[31:01.600 --> 31:16.800]  But over the years, you know, the SRE team has grown more professionalized and it's expanded a lot that it's really hard for volunteers to justify Root access because anytime you need something with Root access, it's not like, oh, I'm blocked on it because you just ping someone and,
[31:16.940 --> 31:20.000]  you know, someone on the SRE team will be awake and be able to help you.
[31:20.000 --> 31:22.940]  So, I would like to see that change.
[31:23.120 --> 31:30.280]  I think it's important that there are volunteers with Root access, but, you know, it's kind of a practical problem right now.
[31:33.760 --> 31:43.740]  All of what I have talked about so far about, like, transparency and access, I think it's important not to understate how hard this can be to keep up, to keep this culture up.
[31:44.560 --> 31:48.380]  It is a constant fight to keep things transparent.
[31:49.280 --> 31:53.620]  A lot of the systems that we have are designed to be open by default.
[31:54.580 --> 31:55.760]  You know, like, Garrett is public.
[31:56.060 --> 31:57.120]  Fabricator is mostly public.
[31:57.260 --> 31:58.980]  All of our statistics are public.
[31:59.120 --> 32:00.380]  Our IRC channels are public.
[32:00.700 --> 32:04.980]  You know, so as long as you, like, use those venues, you know, your things will be public by default.
[32:05.540 --> 32:09.080]  But regardless of this, people still, like, trend towards closed platforms.
[32:09.240 --> 32:11.520]  Like, people want to use Slack instead of IRC.
[32:11.620 --> 32:13.760]  People want to use Google Docs instead of Wikipages.
[32:14.380 --> 32:18.580]  And, you know, it can be, it can be difficult to, like, constantly be the source of friction.
[32:18.580 --> 32:21.160]  Being like, hey, I know you shared a Google Doc with me.
[32:21.300 --> 32:24.460]  Can you actually, like, just post it on the Wiki and I'll give you my feedback there?
[32:24.700 --> 32:28.800]  Or when someone pings you on, like, the internal, like, Wikimedia Foundation Slack.
[32:28.800 --> 32:33.340]  Like, actually, this is a discussion we should be having in public, you know, so that we volunteers can participate.
[32:33.340 --> 32:34.960]  Can we move it to IRC?
[32:35.900 --> 32:39.820]  And, you know, it's difficult, especially when, like, you're being pinged and the answer is just, like, yes.
[32:40.020 --> 32:44.360]  You know, like, do you really, are you really going to force the person to, like, context shift just to give them that answer?
[32:47.500 --> 32:49.820]  And then there's some things that have to be private.
[32:50.060 --> 32:51.100]  Like, legal advice.
[32:51.380 --> 32:55.440]  Lawyers are, you know, can only give legal advice to, like, the company.
[32:55.620 --> 33:00.140]  And so, you know, the only people who can actually read their, like, detailed advice are foundation staff.
[33:00.140 --> 33:05.160]  And then the foundation staff have to, like, summarize it to the volunteers and be like, this is what the lawyers said.
[33:05.160 --> 33:09.420]  And I'm sorry, I can't explain it to you, but this is what we have to do because the lawyers said so.
[33:09.760 --> 33:14.100]  And that can be frustrating, especially if you disagree with the lawyers.
[33:15.900 --> 33:21.380]  And then I think that people can really be intimidated by having to do everything publicly.
[33:22.580 --> 33:26.740]  You know, Wikimedia also has, like, a very strong component of value of privacy.
[33:26.940 --> 33:29.140]  And, you know, we're asking everyone to do everything in public.
[33:29.140 --> 33:32.880]  And, you know, these things are archived, you know, once they're public on the Internet, they're archived forever.
[33:33.020 --> 33:34.800]  And we keep our own records forever.
[33:35.300 --> 33:45.380]  And the idea that, you know, any mistake or silly comment or bad joke, you know, especially if it, you know, has aged poorly over time, you know, is going to be public and archived forever.
[33:46.040 --> 33:47.180]  Do you really want that?
[33:47.340 --> 33:50.980]  And I was in an interview for my new job.
[33:50.980 --> 33:56.740]  And the person interviewing me pulled up a random Wiki page that, you know, like, I had worked on, like a technical proposal.
[33:57.140 --> 34:00.780]  And being like, all of these users, you know, didn't like your proposal.
[34:00.880 --> 34:03.120]  What do you have to say about that, you know?
[34:03.440 --> 34:10.000]  And I was like, like, oh my goodness, like, you know, like, yeah, they did say that, you know, like, they didn't like my proposal.
[34:10.000 --> 34:11.900]  But, you know, I discussed it.
[34:11.980 --> 34:18.000]  And, like, the idea is that, like, any future employer or interviewer can just pull up what you've been working on and read all of your comments.
[34:18.340 --> 34:19.740]  I think it's kind of intimidating.
[34:20.780 --> 34:26.960]  I think the one thing that, you know, works in our favor is that we're very happy for people to be, you know, publicly anonymous.
[34:27.080 --> 34:34.940]  And even, like, there are people with, you know, merge rights to MediaWiki who we don't actually know their real name or their real life identity.
[34:34.940 --> 34:36.260]  It's just that they've done good work.
[34:36.260 --> 34:38.980]  They've become trusted in the community, and so we've given them the rights.
[34:39.760 --> 34:43.900]  To sign the NDA, you do have to use your legal name, but there's no...
[34:43.900 --> 34:48.360]  The legal name can just be restricted to the lawyers, and it can be made... it can be kept private.
[34:48.500 --> 34:53.080]  So the fact that we allow people to contribute anonymously does kind of mitigate that a bit.
[34:53.220 --> 34:56.940]  But I think it can turn people off and is something to keep in mind.
[35:00.040 --> 35:02.100]  So finally, what can you do?
[35:02.100 --> 35:10.200]  You know, I think that this model for transparency is not something that is, like, uniquely special to Wikimedia that only we can do it.
[35:10.300 --> 35:14.740]  I think that any other public interest website should try and do the same.
[35:14.740 --> 35:18.540]  And I think that even if you do it for your personal website, I think that's good for the Internet.
[35:18.680 --> 35:22.220]  I think this level of transparency, you know, helps make the Internet not suck.
[35:24.160 --> 35:26.240]  And really, the thing is to start gradually.
[35:26.300 --> 35:28.120]  Don't try and make everything public at once.
[35:28.300 --> 35:31.620]  That can be very difficult and real, like, culture clash.
[35:32.480 --> 35:35.360]  I find that documentation is the easiest place to start.
[35:35.520 --> 35:37.520]  And I will say that wikis are the best, of course.
[35:37.780 --> 35:43.980]  But, you know, like, even, like, you know, markdown files in a Git repository are decent as well, you know, as long as it's there.
[35:45.180 --> 35:46.700]  And just start writing things down.
[35:46.880 --> 35:49.700]  Even if it's just a list of, like, these are the server components we use.
[35:49.780 --> 35:50.560]  We use Nginx.
[35:50.600 --> 35:53.180]  We use Python with, like, G Unicorn.
[35:53.500 --> 35:55.980]  And, you know, it's managed by systemd units.
[35:56.100 --> 35:59.860]  And even if that's literally the list, you know, it's a place to start.
[35:59.860 --> 36:03.820]  And it's much easier to iteratively add things than it is to start from scratch.
[36:04.320 --> 36:10.080]  And, you know, once you start building a community, other people will start documenting things for you.
[36:11.620 --> 36:14.120]  And I would also say that, like, it's okay to lose control.
[36:14.360 --> 36:22.880]  You know, like, when it comes to our code, we're, like, very picky about, like, the syntax and the spacing and the formatting and how methods are named and how, you know, like, all of that.
[36:23.040 --> 36:25.500]  And, like, you kind of have to, like, lose control with wikis.
[36:25.620 --> 36:27.320]  Like, people will organize it the way they want.
[36:27.320 --> 36:32.920]  And as long as it's not, like, destructive, you know, then accept it and just go with it.
[36:32.980 --> 36:38.880]  The important thing is that people are contributing and eventually you'll move in the right direction and you'll grow to like it.
[36:41.560 --> 36:46.700]  And then, you know, after, once you've started getting some documentation done, you know, start publishing your server configuration.
[36:46.920 --> 36:49.380]  You know, what does your Apache configuration look like?
[36:49.420 --> 36:51.320]  What does your Nginx configuration look like?
[36:51.460 --> 36:53.480]  What are the systemd units that you're using?
[36:54.480 --> 36:58.360]  I like analogizing this to, like, how people publish their .files.
[36:58.560 --> 37:03.980]  You know, like, people, like, want to show off the cool tricks they have in their .dsh and their .bashrz.
[37:03.980 --> 37:10.020]  And there's, like, lists of, like, awesome .files where you can see, like, a list of, like, other people who have really cool .files.
[37:10.040 --> 37:12.080]  And, like, how they've customized their .bash prompt.
[37:12.240 --> 37:15.880]  Like, people have, like, really cool tricks hidden in their server configuration.
[37:16.200 --> 37:17.380]  Like, you should publish that, too.
[37:17.460 --> 37:22.180]  There's a lot of, like, material that could be learned or just that's interesting to look at.
[37:22.960 --> 37:26.760]  You know, the catch is that you have to figure out how to separate your passwords and your private data from that.
[37:26.760 --> 37:34.540]  But, really, that's a best practice you should be aiming for anyways to separate private, you know, private data from, you know, data that's not private.
[37:35.600 --> 37:38.060]  And then, finally, you know, track issues publicly.
[37:38.760 --> 37:43.300]  This can be difficult because oftentimes you don't want to, like, you know, show that you have bugs.
[37:43.460 --> 37:48.820]  But this is also the best way to get people started is to show that it's not perfect and that there's room for improvement.
[37:48.820 --> 37:58.680]  And, you know, even if you just literally, like, write, like, a line, like, this is buggy, you know, and file that as the issue, that's fine as long as it's there.
[37:58.840 --> 38:02.920]  And, you know, you might, you know, know that if there's a volunteer or, you know, we've had issues...
[38:02.920 --> 38:08.600]  We've had cases where, like, the upstream author will, like, see the ticket in our repository and they'll be like, hey, I'm the upstream for this.
[38:08.720 --> 38:11.140]  Can I help you, you know, solve what your problem is?
[38:11.140 --> 38:18.740]  And then, you know, even though you just had, like, one line, then you, like, are prompted to go into more detail about, you know, what the issue is.
[38:19.420 --> 38:24.440]  And having, like, an easy or good first task category is a good way for people to get started.
[38:25.860 --> 38:29.480]  And then, just in general, just keep up the fight for transparency.
[38:30.380 --> 38:40.820]  You know, like, be the little source of friction, you know, choose your battles of when you want to ask people to document something publicly or when you want to have a conversation in public as well.
[38:43.100 --> 38:46.980]  And if you're interested in contributing to Wikipedia, we have a lot of work to do.
[38:47.160 --> 38:49.060]  Like I mentioned earlier, we're 5% there.
[38:49.500 --> 38:54.920]  You know, whether you want to edit the projects or whether you want to, you know, contribute technically, there's a lot of work.
[38:55.380 --> 38:58.980]  The easiest way is to just lurk in our IRC and matrix channels.
[38:59.180 --> 39:00.420]  You know, something might catch your attention.
[39:00.580 --> 39:04.320]  Someone might be asking a question about the thing you have, like, subject matter expertise in.
[39:04.840 --> 39:07.100]  Or you might just find the conversations interesting.
[39:08.260 --> 39:10.460]  I have two QR codes on this slide.
[39:10.600 --> 39:15.520]  The first is the guide of how to become a MediaWiki hacker, which walks you through, like, setting up our Git setup.
[39:15.800 --> 39:19.660]  How to use, like, our Docker or VM-based, you know, MediaWiki setup.
[39:20.340 --> 39:30.820]  And the second is actually this brand-new Wikimedia developer portal, which gives you a lot of links of different other projects that are not explicitly MediaWiki that still need contributions.
[39:30.820 --> 39:32.440]  You know, we have, like, bot frameworks.
[39:32.720 --> 39:34.740]  We have, you know, like, different developer tooling.
[39:34.920 --> 39:36.300]  There's a lot of stuff out there.
[39:39.260 --> 39:40.220]  And thank you.
[39:40.900 --> 39:41.400]  That's it.
[39:41.520 --> 39:41.960]  We have...
[39:42.380 --> 39:44.860]  That's the QR code for the slides.
[39:45.040 --> 39:47.460]  That's different ways you can reach me if you have questions.
[39:47.760 --> 39:49.400]  And, yeah, if people have questions.
[39:59.800 --> 40:00.170]  Yeah.
[40:00.380 --> 40:02.340]  I think there's a mic back there if you want to use it.
[40:10.450 --> 40:10.970]  Thank you.
[40:11.050 --> 40:11.930]  This was really interesting.
[40:11.930 --> 40:24.390]  And I'm wondering, since it sounds like you're sort of the biggest website to scale doing so much in public, have you run into security issues by sort of having the biggest version of something so public?
[40:24.390 --> 40:26.990]  And how have you resolved that friction?
[40:27.150 --> 40:27.870]  Does the question make sense?
[40:28.130 --> 40:28.410]  Yeah.
[40:29.550 --> 40:30.030]  Yes.
[40:30.410 --> 40:35.650]  There's, like, a lot of people who are, like, afraid to, like, publish version numbers or, you know, details about that.
[40:36.870 --> 40:42.090]  But I think the main thing is that you have to stay on top of security practices just in general.
[40:42.270 --> 40:45.430]  Like, you have to be tracking, you know, the upstream security issues.
[40:45.470 --> 40:48.190]  You have to be tracking, you know, like, so we're all based on Debian.
[40:48.490 --> 40:52.590]  And, in fact, one of the members of the Debian security team works for the Wikimedia Foundation.
[40:52.670 --> 40:56.430]  And so we're, like, always up to date on what the existing security issues are.
[40:57.710 --> 40:59.410]  And we actually take it a step farther.
[41:00.410 --> 41:01.710]  And I have this prepared.
[41:02.110 --> 41:03.950]  We, like, publish all of the version numbers.
[41:03.970 --> 41:12.970]  If you go to Special Version on any MediaWiki installation, you can see, like, what the MediaWiki version is, what the PHP version, what the database is, you know, and all these different services.
[41:13.050 --> 41:17.510]  And you can see all the extensions installed, what their versions are, what the, you know, PHP libraries we use.
[41:18.430 --> 41:21.270]  In general, I think security through obscurity doesn't work.
[41:21.470 --> 41:29.230]  You know, like, at best, all we get is all these, like, automated scanner reports from people looking for bug bounty saying, like, oh, you're using an unsupported version of this.
[41:29.330 --> 41:29.950]  You have this bug.
[41:30.070 --> 41:31.850]  And then you're, like, did you actually test the bug?
[41:31.870 --> 41:34.570]  It doesn't work because we're using a patched version of the software.
[41:35.610 --> 41:46.150]  So, yeah, I, you know, like, there are always times, you know, like, when, that people do find security holes and we just have to patch them.
[41:46.250 --> 41:53.790]  And we have practices for deploying security patches privately that then get, you know, responsibly disclosed to, like, other people using MediaWiki.
[41:54.450 --> 41:58.390]  But the answer is, like, you just have to stay on top of security, whether you make it public or private.
[42:07.280 --> 42:07.880]  Hi there.
[42:08.760 --> 42:09.860]  First of all, thanks.
[42:10.180 --> 42:11.280]  That was a great talk.
[42:11.840 --> 42:19.120]  One thing that I'm wondering about is, are there any sort of, like, statistics on volunteer burnout if such a thing exists?
[42:20.920 --> 42:24.800]  I don't have any statistics offhand, but I know it's a real thing.
[42:25.300 --> 42:27.160]  Volunteer burnout is real.
[42:28.280 --> 42:31.980]  You know, I think that people, you know, like, oftentimes...
[42:31.980 --> 42:34.200]  So, it works both ways.
[42:34.320 --> 42:38.740]  There are volunteers who are still involved who are, like, around in, like, 2005, 2006.
[42:39.080 --> 42:53.480]  And then there are also people who, like, will show up for, you know, like, one year, provide these awesome contributions, you know, like, you get them, they're, like, merge rights, and then they just disappear and you never hear from them again and have no way to contact them.
[42:55.240 --> 42:56.160]  It's a mix of both.
[42:56.400 --> 43:05.180]  I think, you know, the main thing is that we have to, like, strengthen and grow the community so that no one person feels overburdened, that they have too many maintenance responsibilities, and we can share the load.
[43:05.260 --> 43:12.840]  And if people need time off, we should give them time off rather than letting them go all the way up, burn out, and then we lose them forever.
[43:14.400 --> 43:14.760]  Thanks.
[43:30.420 --> 43:31.640]  Thank you for the talk.
[43:31.780 --> 43:32.660]  I have a few questions for you.
[43:33.200 --> 43:34.880]  You can pick your favorite amongst.
[43:35.780 --> 43:36.700]  One technical question.
[43:36.880 --> 43:45.720]  I'm curious if you could say something about your usage of OpenStack, how you use it, your experience, where you chose to use it.
[43:46.140 --> 43:55.560]  Second question, you said fighting for transparency, which I think is a commitment to transparency at the foundation of Wikipedia that is wonderful and awesome and so strong.
[43:57.400 --> 44:02.600]  In your own word, what would you say is that strength coming from?
[44:03.480 --> 44:07.580]  I could say it in my own word, but I'm really curious to see how you would articulate it.
[44:09.000 --> 44:11.140]  Sorry, could you clarify the last part about transparency?
[44:11.380 --> 44:11.940]  What were you asking?
[44:12.340 --> 44:14.720]  Where does that strength of commitment come from?
[44:15.320 --> 44:17.420]  When the going gets hard, right?
[44:17.460 --> 44:20.020]  You mentioned that it takes an effort to maintain that transparency.
[44:20.020 --> 44:21.980]  Where does that strength come from?
[44:23.300 --> 44:25.400]  Yeah, so I'll answer this one first.
[44:26.200 --> 44:33.120]  You know, transparency is really about, like, making sure that in the future we have access to this information, you know?
[44:33.240 --> 44:41.520]  Like, if I discussed a technical problem in, like, PM with someone and lost the IRC logs, then, like, we have no way of knowing what the rationale was.
[44:41.960 --> 44:48.380]  And, you know, I think that doing things in public really builds up the ability for other developers.
[44:48.380 --> 44:56.060]  Like, I've learned a lot just from observing other IRC conversations with other people, seeing how they discuss things, and seeing how they were able to solve problems.
[44:56.340 --> 45:00.260]  And that really brings up the whole community, that sense of feeling involved.
[45:00.680 --> 45:04.400]  And really the idea that, you know, like, we have a long ways to go.
[45:04.740 --> 45:08.780]  You know, we've been here for 20 years, and we have probably, like, we want to be here forever.
[45:09.300 --> 45:13.700]  And so retaining our history and that kind of institutional knowledge is incredibly important.
[45:13.700 --> 45:16.300]  And really the easiest way to do that is through transparency.
[45:17.400 --> 45:18.880]  And then about OpenStack.
[45:19.460 --> 45:22.620]  So the OpenStack cluster, I don't have any size details.
[45:23.000 --> 45:24.800]  It's called Wikimedia Cloud Services.
[45:25.260 --> 45:30.700]  And basically people can create VMs, and we have, like, a few other OpenStack things.
[45:31.560 --> 45:35.140]  I don't know the exact reason why OpenStack was selected.
[45:35.140 --> 45:37.500]  It was picked in, like, 2011, 2012.
[45:37.540 --> 45:43.600]  I do know that the main person who was hired for it was, like, a member of the OpenStack Foundation and contributing to OpenStack already.
[45:43.720 --> 45:45.580]  But I don't know why.
[45:45.580 --> 45:52.760]  I'm happy to, like, put you in touch with the people who do know the answer, or at least point you to the wiki pages that might contain the information, if you want to find me afterwards.
[45:54.400 --> 45:54.900]  Thank you.
[45:55.180 --> 45:55.480]  Thank you.
[45:55.480 --> 45:56.680]  Thank you.